# Network Capacity Optimization for Cellular-Assisted Vehicular Systems by Online Learning-Based mmWave Beam Selection

Authors: Jinsong Gui; Yao Liu; Xiaoheng Deng; Bin Liu
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Directional communication is helpful to improve the performance of millimeter Wave (mmWave) links. However, the dynamic nature of vehicular scenarios raises the complexity of directional mmWave vehicular communications. Also, a mmWave link is susceptible to blockages. Therefore, a mmWave vehicular communication system requires high environmental adaptability and context-awareness. Due to inadequate context information and insufficient beam settings in the existing related algorithm, it is difficult to pick out the set of beams with more reasonable widths and directions, which hinders the further promotion of network capacity in vehicular networks. Therefore, we propose an improved fast machine learning (IFML) algorithm to overcome this shortcoming. In order to improve network capacity while suppressing the additional beam search overhead, a partitioned search method is designed in the IFML. Also, in order to be robust to occasional fluctuations and timely adapt to significant changes in communication environments, the IFML adopts a flexible beam performance update approach based on adjustable weight coefficient. The simulation results show that the IFML significantly outperforms the existing related algorithm in terms of aggregate received data after a certain number of online learning time periods.


---

## Body

## 1. Introduction

As the Internet of Things (IoT) is applied to the automobile industries, the vehicles equipped with communication modules are networked to their surroundings and further connected to the Internet, which can improve user experience and promote safe driving. Such vehicular communicating modes as Vehicle to Network (V2N), Vehicle to Vehicle (V2V), Vehicle to Infrastructure (V2I), and Vehicle to Pedestrian (V2P) are collectively referred to as Vehicle to Everything (V2X). In Intelligent Transport System (ITS), the V2X applications include autonomous driving services, traffic management services and road safety, and infotainment V2X services.To ensure Quality of Service (QoS) for these V2X applications, it is essential to provide adequate radio spectrum resources to increase network capacity, which can support the rapid exchange of information between infrastructure, vehicles, and pedestrians. Since radio spectrum resources are limited, the Long-Term Evolution V2X (LTE-V2X) systems and Dedicated Short-Range Communication (DSRC) systems mainly focus on the life-critical V2X applications (e.g., autonomous driving services and road safety management). Due to the controllable QoS and wide radio coverage, the LTE-V2X system outperforms the DSRC system in terms of providing connectivity for the V2X applications.With the rapid growth of interconnected vehicles and the continuous pursuit of experience quality by vehicle users, higher requirements are put forward for network capacity. However, sub-6 GHz frequency bands used by the LTE system cannot meet the increasing capacity demand, so the Fifth-Generation (5G) communication system adopts the underutilized millimeter Wave (mmWave) frequency bands to make up for the shortfall. This underutilization is mainly caused by the penetration loss and high path loss of mmWave frequency bands. Fortunately, the existing researches [1, 2] indicate that (1) the high path loss is compensated by directional transmission and beamforming and (2) the short communication range in mmWave bands can be formed by higher deployment density of base stations.The sub-6 GHz frequency bands have the wider coverage but the smaller capacity, while the mmWave frequency bands have the larger capacity but the narrower coverage. Therefore, the 5G systems and the LTE systems are widely considered to be the main driving forces for full support for all the V2X applications. However, the dynamic nature of vehicular scenarios raises the complexity of cellular-based directional mmWave vehicular communications. This is mainly because the directional communication needs an accurate beam alignment between a vehicle and a base station [3]. This alignment needs to be constantly adjusted due to vehicle mobility, which is not necessary for sub-6 GHz frequency bands since the omnidirectional transmission mode is adopted. Also, a mmWave link is susceptible to blockages (e.g., foliage, trucks, and buildings). Therefore, inaccurate beam selection raises high penetration loss and thus leads to the performance degradation of mmWave systems, since it does not guarantee effective avoidance of obstacles.The related means and mechanisms like beamforming training, beam tracking, and beam selection are usually used to ensure beam alignment. With the increase of the number of beams and users, the search space of choosing a reasonable beam pair for communication link is increasing, which greatly increases the delay of establishing mmWave communication link. Because beam selection involves many communication parameters, pure mathematical modeling is very complicated. Although beam selection methods based on machine learning are widely accepted, most require training before they can be used. However, online learning-based beam selection methods can be used without prior training, so it is well suited to dynamic vehicular communication systems. In this respect, an excellent example is the work in the literature [4].The authors in [4] argue that, through self-exploration, learning, and adaptation to communication environments, mmWave base stations can select beam accurately and maintain continuous scalability. Through modeling the beam selection as a contextual multi-armed bandit (MAB) problem, they propose a contextual online learning algorithm based on contextual information (i.e., a fast machine learning (FML) algorithm) to solve it. By this algorithm, any mmWave base station can autonomously learn from its previous decisions and the relationship to available contextual information, and thus, it knows the performance of every beam. Since this algorithm learns the expected beam performance in different contexts over time, it does not require a training phase. Here, the assumption is that a particular beam’s performance is basically similar under similar vehicle contexts.In [4], the beams are not overlapping, which limits the optional beam width and azimuth range. Although the different beams can serve the vehicles in their respective coverage at the same time, the same beam can only serve one of the vehicles in its coverage at any time since only unicast is considered. That is, only one vehicle is lucky to be served if there are more vehicles in the same fine place. For brevity, we call such a case the location conflict.Therefore, in order to reduce the possibility that the vehicles will be excluded from the scope of simultaneous services due to the location conflict, we must get rid of the limitations of beam width and azimuth range, which is helpful to increase the amount of data received by the vehicles in the system during each time period.Currently, only a limited number of vehicles can be selected to receive data from a base station at a time. This is because a base station only has a limited number of radio frequency (RF) chains (including power amplifiers, low noise amplifiers, up and down converters, and RF switches), which limits a base station’s ability to concurrently transmit data. Unlike a RF chain, in this paper, a beam is only viewed as a region that limits energy propagation and its shape is usually a cone (or sector), which we call a virtual beam for convenience.Therefore, the number of virtual beams can be unlimited at a base station. If the overlapped virtual beams are allowed at a base station, they can be used concurrently by assigning the different frequency bands and RF chains to them, where the number of the different frequency bands is not more than that of RF chains. Due to the abundant spectrum resources in mmWave frequency bands, it is not difficult to provide these frequency bands. Under such conditions, the selected beams for concurrent usage have the more reasonable beam width and beam direction and, thus, serve for the more appropriate set of vehicles at the same time.On the other hand, if we do not want to adopt the additional frequency bands, overlapping beams may be used simultaneously on the same frequency band, where the corresponding transmission powers have to be coordinated to control the interference between each other. However, it is not always the case that mutual interference is well controlled. If this case happens, overlapping beams cannot be used simultaneously in the same frequency band, and thus, its effect is reduced to that of any nonoverlapping beam scheme.As the number of vehicles that want to be connected increases, it is time-consuming to select the right beam for each vehicle from the large number of virtual beams at a base station, especially when the transmission power of the RF chain allocated to a virtual beam also needs to be selected. When a virtual beam is assigned an RF chain and the RF chain’s transmission power is specified, it is called a physical beam. In addition, the extension of the context dimension is beneficial to the construction of richer context decision space. The authors in [4] claim to consider only the small amount of vehicle arrival direction information, though they believe that richer contextual information will further improve system performance.Finally, in [4], with the increasing times that a beam is used, the latest observed beam performance value contributes less and less to the current updated beam performance than the previous updated values. This update method is robust to occasional fluctuations in beam performance. However, when the latest observed performance of a beam reflects a significant change in its communication environment, the real-time performance of this update method is very poor. Therefore, to address the above challenges, we propose an Improved FML (IFML) algorithm and the main contributions are as follows.
(1)
By using the IFML, we can select the set of beams with the more reasonable beam widths and beam directions. These selected beams can serve for the more appropriate set of vehicles at the same time during a time period, which is helpful to increase the amount of data received by vehicles(2)
In the IFML, the richer contexts are exploited besides allowing beam overlap, which is helpful to pick out the desired beams. Also, the coverage of a base station is divided into multiple subcoverages to search for the desired beams from them, respectively. The used beams in a time period will come from the beams selected from these subcoverages. This partitioned search method reduces the beam search overhead(3)
The beam performance update approach in the IFML is more flexible, which can be robust to occasional fluctuations and timely adapt to significant changes in communication environments by adjusting the weight coefficient in terms of the contribution of the latest observed value to the current updated beam performance value(4)
Compared with the existing related algorithm, we demonstrate by means of the simulation results that the IFML substantially improves the amount of data received by the vehicles in the system at a slightly increased overhead. However, there is no difference in terms of beam performance update cost after a certain number of time periodsThe rest of the paper is organized as follows. Section2 gives an overview of related works. In Section 3, we describe the system model, including network architecture and problem formulation. In Section 4, we propose an improved algorithm, including theoretical analysis of regret. The algorithm performance analysis is provided in Section 5, while the simulation results are presented in Section 6. Section 7 concludes the paper.


## 2. Related Work

### 2.1. V2X Communications and Architectures

The authors of [5] proposed a context-awareness heterogeneous vehicle network architecture, which is compatible with the current European Telecommunications Standards Institute (ETSI) and International Organization for Standardization (ISO) standardized ITS station reference architectures. The architecture is particularly suitable for this kind of applications that connected vehicles require the different radio communication technologies to satisfy the requirements in diverse vehicular scenarios, since it has the ability for dynamic selection and configuration of communication profiles based on the context conditions and the application requirements.The authors of [6] described the advantage of cellular infrastructure used for V2X applications alongside the architectures for cellular-assisted V2X systems and specifically discussed the security requirements of V2X in cellular networks. Besides the security challenges and requirements of V2X, the authors of [7] surveyed the key features of V2X and focused on the standardization techniques used for vehicular communication technologies. The V2X entities include vehicles equipped with various radio interfaces (e.g., DSRC, LTE, and mmWave), V2X application servers, LTE base stations (evolved node B, eNodeB), and mmWave small base stations. They are connected by various short radio communication technologies (e.g., IEEE 802.11p [8], IEEE 1609.x [9], ETSI ITS-G5 [10], and 5G New Radio [11]) and long-range radio communication technologies (e.g., LTE-V2X [6]).Due to only supporting short radio communication and adopting carrier sense multiple access with collision avoidance mechanism, the DSRC technology has small coverage and suffers from high collision probabilities under high vehicle density scenarios. Moreover, it has a limited bandwidth in the DSRC frequency spectrum and thus hardly meets the high traffic demand for V2V applications [9, 12]. The 3rd Generation Partnership Project (3GPP) release 14 [13] specifies that the LTE should both support V2I and V2N communications for latency tolerant V2X services and extend the current proximity service (ProSe) (e.g., Device-to-Device (D2D) communication) to implement direct V2V communications for delay-sensitive vehicular applications.The authors of [14] envisaged the potential benefits of using LTE infrastructure for V2X services and the superiority of LTE-V2X technologies over DSRC-based approaches, including wide coverage capacity, high reliability, low latency, high date, and controlled QoS. The 3GPP release 16 will specify the advanced solutions for V2X communications in the 5G New Radio access technology [15], which can support ultrahigh reliability, ultrahigh data rates, and ultralow latency. Also, the 3GPP radio access network working group agreed that the 5G-based V2X system supplements LTE-based V2X system to provide advanced V2X services instead of replacing the services offered by LTE-based V2X system [16].


### 2.2. mmWave Beam Management

Beam selection problems have been researched in traditional vehicular networks which operate at sub-6 GHz frequency bands to achieve maximum throughput. As mentioned above, the propagation characteristics of radio signals in sub-6 GHz frequency bands are essentially different from those of mmWave frequency bands.There are also studies on beam selection in mmWave networks. The authors of [17] addressed a hierarchical beam search problem, while the authors of [18] exploited the sparse multipath structure of mmWave channels to optimally select beam directions. The authors of [19] proposed a cell discovery method, where a base station periodically transmits synchronization signals to scan its entire angular space in time-varying random directions. Based on adaptive subspace sampling and hierarchical beam codebooks, the authors of [20] designed a beam alignment technique. The authors of [21] addressed the beam alignment issue in mmWave networks, where the transmitting and receiving antenna arrays require to frequently find the optimal beam pair. Therefore, when the number of transmitting and receiving beams is very large, the search space for beam matching is also large.The authors of [22] proposed a new beamforming scheme to minimize the transmission power and weighted symbol error rates, while the authors of [23] described a hybrid beamforming structure for small cell base station and formulated the joint allocation of the transmission power and beam into a mixed integer nonlinear programming problem. In [24], the authors proposed a codebook-based beam tracking scheme for mobile mmWave networks, which shows that the average tracking error probability can be lowered by improving the power allocation algorithm over the training beams.The authors of [25] analyzed the average search delay of mmWave systems, where the average search times for discovering user equipment is taken as the number of search sectors in the beamspace. The authors of [26] proposed a deep learning-based beam selection, which exploits sub-6 GHz channel information to select a mmWave beam. The authors of [27] provided the overview on beam-forming architecture and deployment scenarios in the 5G New Radio (NR) standard. To address power-efficient beam design, the authors of [28] proposed a feasible point search method and then developed a hybrid analog-digital mapping algorithm. To lay the foundation for beam tracking, the authors of [29] used machine learning to track user mobility, in which a deep neural network is learned and then used to predict the direction of user movement.To accelerate the beam training process, the authors of [30] proposed a low-cost hybrid architecture assisted by a digital beamforming module and designed a fast beam training method by utilizing the proposed architecture and using the sparsity of mmWave channels. To improve the latency performance of mmWave networks for industrial automation, the authors of [31] proposed an adaptive beam selection strategy to select the best set of beams among multiple users to reduce the overall latency for all users. The authors of [32] studied the beam alignment problem of multiuser mmWave massive Multi-input Multi-output (MIMO) system and proposed an alignment algorithm with partial beams by using machine learning. The authors of [33] proposed an efficient hierarchical beamforming training mechanism to establish directional links in dense mmWave cellular networks.Given the severe interbeam interferences caused by mobility of users, the authors of [34] proposed a beam resource allocation and mobility-aware subband scheme for the mmWave subband-beam massive MIMO systems. The authors of [35] focused on beam training problem of wireless local area networks in the 60-GHz band and designed an advanced beamforming training protocol for multiple access points and multiple users. The authors of [36] applied per-beam synchronization for a massive MIMO system to alleviate the delay dispersion effects and channel Doppler due to mobility and blockage.


### 2.3. Solutions of mmWave Vehicular Networks

The above works are not specific to mmWave vehicular networks, and thus, the corresponding solutions cannot be directly applied to the field. For the characteristics of mmWave vehicular networks, the readers are encouraged to refer to the overview in [37]. The authors of [38] analyzed the mmWave propagation characteristics in V2V communication scenarios and derived the relationship between beam width and channel coherence time in vehicular networks. The authors of [39–42] focused on the adaptability of mmWave beams in vehicular scenarios, and they believed that mmWave base stations can track vehicles by exploiting DSRC and adjusting the beam accordingly.The authors of [43] utilized Kernel-based machine learning algorithms and channel state information of sub-6 GHz bands to predict vehicles’ positions and then used them to preactivate target mmWave remote radio units. Also, they used K-nearest neighbor machine learning algorithms and historical handover data to predict handover decisions.Since achieving perfect beam-alignment is challenging in vehicular scenarios, the authors of [44] proposed the method of fingerprint-based beam-alignment, where the fingerprint of a given location is viewed as a set of beam pairs. Based on the proposed multifingerprint database for a given location, the base station intelligently adapts the fingerprints with the aid of learning, which is capable of maintaining the target performance in mmWave vehicular environments.The authors of [45] aimed to maximize the overall network throughput for highly-dynamic vehicular communications. They proposed a reinforcement learning approach to handle the beam selection problem in a high-mobility vehicular environment. The authors of [46] addressed optimal beam selection in mmWave-based V2X communications by designing the multitype-2 fuzzy inference system.The authors of [4] addressed the beam selection issue at mmWave base stations in vehicular scenarios and proposed the FML algorithm. The FML is essentially an online learning algorithm, and it does not require either accurate localization information or prior statistical knowledge of dynamic vehicular scenarios. If the work in [21] is used to extend the FML to achieve longer connectivity, the additional learning cost is required for beam alignment, which increases the complexity of the FML and, thus, is unsuitable for vehicular scenarios.In [47], the authors used a multiarmed bandit framework to develop online learning algorithms for beam pair selection and refinement. They used a type of sensor data (e.g., position) to identify the good mmWave beam directions, which is one approach to reduce the overhead since such information is widely available in vehicular applications. However, the work in [47] only deals with how to choose the more promising beam directions and refine them but does not focus on the optimal selection of beam width, where beams are spaced by the 3 dB beam widths in a fixed manner.Although the work in [21] investigated an online stochastic optimization problem and proposed an equivalent structured multiarmed bandit model to exploit contextual information, it is difficult to adapt to vehicular scenarios since it requires accurate localization information. The work in [4] is most related to our work. However, for the reasons mentioned in the introduction, the FML still has room for improvement, which is the motivation for the work in this paper.


## 2.1. V2X Communications and Architectures

The authors of [5] proposed a context-awareness heterogeneous vehicle network architecture, which is compatible with the current European Telecommunications Standards Institute (ETSI) and International Organization for Standardization (ISO) standardized ITS station reference architectures. The architecture is particularly suitable for this kind of applications that connected vehicles require the different radio communication technologies to satisfy the requirements in diverse vehicular scenarios, since it has the ability for dynamic selection and configuration of communication profiles based on the context conditions and the application requirements.The authors of [6] described the advantage of cellular infrastructure used for V2X applications alongside the architectures for cellular-assisted V2X systems and specifically discussed the security requirements of V2X in cellular networks. Besides the security challenges and requirements of V2X, the authors of [7] surveyed the key features of V2X and focused on the standardization techniques used for vehicular communication technologies. The V2X entities include vehicles equipped with various radio interfaces (e.g., DSRC, LTE, and mmWave), V2X application servers, LTE base stations (evolved node B, eNodeB), and mmWave small base stations. They are connected by various short radio communication technologies (e.g., IEEE 802.11p [8], IEEE 1609.x [9], ETSI ITS-G5 [10], and 5G New Radio [11]) and long-range radio communication technologies (e.g., LTE-V2X [6]).Due to only supporting short radio communication and adopting carrier sense multiple access with collision avoidance mechanism, the DSRC technology has small coverage and suffers from high collision probabilities under high vehicle density scenarios. Moreover, it has a limited bandwidth in the DSRC frequency spectrum and thus hardly meets the high traffic demand for V2V applications [9, 12]. The 3rd Generation Partnership Project (3GPP) release 14 [13] specifies that the LTE should both support V2I and V2N communications for latency tolerant V2X services and extend the current proximity service (ProSe) (e.g., Device-to-Device (D2D) communication) to implement direct V2V communications for delay-sensitive vehicular applications.The authors of [14] envisaged the potential benefits of using LTE infrastructure for V2X services and the superiority of LTE-V2X technologies over DSRC-based approaches, including wide coverage capacity, high reliability, low latency, high date, and controlled QoS. The 3GPP release 16 will specify the advanced solutions for V2X communications in the 5G New Radio access technology [15], which can support ultrahigh reliability, ultrahigh data rates, and ultralow latency. Also, the 3GPP radio access network working group agreed that the 5G-based V2X system supplements LTE-based V2X system to provide advanced V2X services instead of replacing the services offered by LTE-based V2X system [16].


## 2.2. mmWave Beam Management

Beam selection problems have been researched in traditional vehicular networks which operate at sub-6 GHz frequency bands to achieve maximum throughput. As mentioned above, the propagation characteristics of radio signals in sub-6 GHz frequency bands are essentially different from those of mmWave frequency bands.There are also studies on beam selection in mmWave networks. The authors of [17] addressed a hierarchical beam search problem, while the authors of [18] exploited the sparse multipath structure of mmWave channels to optimally select beam directions. The authors of [19] proposed a cell discovery method, where a base station periodically transmits synchronization signals to scan its entire angular space in time-varying random directions. Based on adaptive subspace sampling and hierarchical beam codebooks, the authors of [20] designed a beam alignment technique. The authors of [21] addressed the beam alignment issue in mmWave networks, where the transmitting and receiving antenna arrays require to frequently find the optimal beam pair. Therefore, when the number of transmitting and receiving beams is very large, the search space for beam matching is also large.The authors of [22] proposed a new beamforming scheme to minimize the transmission power and weighted symbol error rates, while the authors of [23] described a hybrid beamforming structure for small cell base station and formulated the joint allocation of the transmission power and beam into a mixed integer nonlinear programming problem. In [24], the authors proposed a codebook-based beam tracking scheme for mobile mmWave networks, which shows that the average tracking error probability can be lowered by improving the power allocation algorithm over the training beams.The authors of [25] analyzed the average search delay of mmWave systems, where the average search times for discovering user equipment is taken as the number of search sectors in the beamspace. The authors of [26] proposed a deep learning-based beam selection, which exploits sub-6 GHz channel information to select a mmWave beam. The authors of [27] provided the overview on beam-forming architecture and deployment scenarios in the 5G New Radio (NR) standard. To address power-efficient beam design, the authors of [28] proposed a feasible point search method and then developed a hybrid analog-digital mapping algorithm. To lay the foundation for beam tracking, the authors of [29] used machine learning to track user mobility, in which a deep neural network is learned and then used to predict the direction of user movement.To accelerate the beam training process, the authors of [30] proposed a low-cost hybrid architecture assisted by a digital beamforming module and designed a fast beam training method by utilizing the proposed architecture and using the sparsity of mmWave channels. To improve the latency performance of mmWave networks for industrial automation, the authors of [31] proposed an adaptive beam selection strategy to select the best set of beams among multiple users to reduce the overall latency for all users. The authors of [32] studied the beam alignment problem of multiuser mmWave massive Multi-input Multi-output (MIMO) system and proposed an alignment algorithm with partial beams by using machine learning. The authors of [33] proposed an efficient hierarchical beamforming training mechanism to establish directional links in dense mmWave cellular networks.Given the severe interbeam interferences caused by mobility of users, the authors of [34] proposed a beam resource allocation and mobility-aware subband scheme for the mmWave subband-beam massive MIMO systems. The authors of [35] focused on beam training problem of wireless local area networks in the 60-GHz band and designed an advanced beamforming training protocol for multiple access points and multiple users. The authors of [36] applied per-beam synchronization for a massive MIMO system to alleviate the delay dispersion effects and channel Doppler due to mobility and blockage.


## 2.3. Solutions of mmWave Vehicular Networks

The above works are not specific to mmWave vehicular networks, and thus, the corresponding solutions cannot be directly applied to the field. For the characteristics of mmWave vehicular networks, the readers are encouraged to refer to the overview in [37]. The authors of [38] analyzed the mmWave propagation characteristics in V2V communication scenarios and derived the relationship between beam width and channel coherence time in vehicular networks. The authors of [39–42] focused on the adaptability of mmWave beams in vehicular scenarios, and they believed that mmWave base stations can track vehicles by exploiting DSRC and adjusting the beam accordingly.The authors of [43] utilized Kernel-based machine learning algorithms and channel state information of sub-6 GHz bands to predict vehicles’ positions and then used them to preactivate target mmWave remote radio units. Also, they used K-nearest neighbor machine learning algorithms and historical handover data to predict handover decisions.Since achieving perfect beam-alignment is challenging in vehicular scenarios, the authors of [44] proposed the method of fingerprint-based beam-alignment, where the fingerprint of a given location is viewed as a set of beam pairs. Based on the proposed multifingerprint database for a given location, the base station intelligently adapts the fingerprints with the aid of learning, which is capable of maintaining the target performance in mmWave vehicular environments.The authors of [45] aimed to maximize the overall network throughput for highly-dynamic vehicular communications. They proposed a reinforcement learning approach to handle the beam selection problem in a high-mobility vehicular environment. The authors of [46] addressed optimal beam selection in mmWave-based V2X communications by designing the multitype-2 fuzzy inference system.The authors of [4] addressed the beam selection issue at mmWave base stations in vehicular scenarios and proposed the FML algorithm. The FML is essentially an online learning algorithm, and it does not require either accurate localization information or prior statistical knowledge of dynamic vehicular scenarios. If the work in [21] is used to extend the FML to achieve longer connectivity, the additional learning cost is required for beam alignment, which increases the complexity of the FML and, thus, is unsuitable for vehicular scenarios.In [47], the authors used a multiarmed bandit framework to develop online learning algorithms for beam pair selection and refinement. They used a type of sensor data (e.g., position) to identify the good mmWave beam directions, which is one approach to reduce the overhead since such information is widely available in vehicular applications. However, the work in [47] only deals with how to choose the more promising beam directions and refine them but does not focus on the optimal selection of beam width, where beams are spaced by the 3 dB beam widths in a fixed manner.Although the work in [21] investigated an online stochastic optimization problem and proposed an equivalent structured multiarmed bandit model to exploit contextual information, it is difficult to adapt to vehicular scenarios since it requires accurate localization information. The work in [4] is most related to our work. However, for the reasons mentioned in the introduction, the FML still has room for improvement, which is the motivation for the work in this paper.


## 3. Network Model

### 3.1. Network Architecture

As shown in Figure1, we consider an integrated mmWave/sub-6 GHz cellular system, where mmWave small base stations (mmSBSs) are deployed in the coverage area of an LTE eNode B (eNB). Each mmSBS is connected to the eNB via a backhaul link in a wired mode (e.g., an optical fiber) or a wireless mode (e.g., a mmWave link). The vehicles are equipped with two types of radio frequency interfaces (i.e., an LTE interface, which is used to connect to the eNB, and a mmWave interface, which is used for ultrahigh-speed data communication).Figure 1
A mmWave vehicular communication scenario.We assume that the eNB has the function that can assist each vehicle in its coverage to obtain the information of the mmSBS with which the vehicle should be associated. Since the LTE interface has the wider coverage, each vehicle can easily request help from the eNB via the LTE interface. If the eNB judges that there is not any appropriate mmSBS for the vehicle to associate with, it will reject the request, which will occur when the eNB judges that the vehicle is about to move out of its coverage. Even if the eNB agrees to handle a request that it should not have accepted, it can also transfer the request to an infrastructure component (e.g., a V2X server). With the help of the Home Subscriber Server (HSS) and Mobility Management Entity (MME) in the LTE infrastructure, this component will specify another suitable eNB to assist the vehicle.In theory, each mmSBS can plan an infinite number of virtual beams, and the beam width of each beam can be set to the range from 0 to 360°. Therefore, the overlap between virtual beams should be allowed. In fact, if all the virtual beams of a mmSBS are used simultaneously, its potential transmission capacity can be multiplied. To this end, each virtual beam must actually have an exclusive RF chain. Moreover, if the virtual beams overlap each other, they must be assigned to different frequency bands. Since there are abundant bandwidth resources in mmWave frequency bands, it is not difficult to allocate different channels for overlapping virtual beams. If each mmSBS coverage area only shares the same frequency band in an underlay mode, the RF chains’ transmission powers on overlapping beams must be reasonably selected to control mutual interference.Due to the limitation of form-factor and manufacturing cost, it is undesirable that the number of RF chains is proportional to the number of virtual beams when the number of virtual beams is set to a very large value. The literatures [2, 48] discuss the technical details on this limitation. Therefore, the number of vehicles that a mmSBS can simultaneously serve is limited by the maximum number of RF chains which a mmSBS can be equipped with. When the number of the virtual beams of a mmSBS and the number of the vehicles in its coverage exceed the number of its RF chains, respectively, it should select the best subset of virtual beams to provide the best downlink sum-data.In order to achieve this goal, we model a virtual beam selection problem in a mmSBS as a MAB problem, which allows this mmSBS to identify the best virtual beams by carefully selecting subsets of virtual beams over time. Also, each selected virtual beam must have its reasonable RF transmission power to control mutual interference if the same frequency band is shared in an underlay mode.As described in [49], a decision-maker must select a subset of actions with unknown expected earnings for the purpose of maximizing the earnings over time in MAB problems. The challenge in MAB problems is to address the exploration and exploitation dilemma, since all the actions should be fully explored to learn their earnings, but those which have already generated high earnings should also be exploited. Based on the contextual online learning idea, the authors in [4] address the abovementioned challenge. However, in our concerns, the online learning algorithm will have a larger search space since the number of possible virtual beams is larger, especially when the corresponding powers also need to be reasonably selected. Moreover, as the number of vehicles increases, the contextual space search overhead also increases accordingly. Therefore, we propose the IFML algorithm that tackles this problem.


### 3.2. Problem Formulation

We do not limit the number of virtual beams which a mmSBS can possess, since a larger virtual beamspace is more likely to find a more suitable beam for a vehicle. However, in order to keep this advantage while reducing the search time of online learning algorithm, we evenly divide the coverage area of a mmSBS intoL sectors (e.g., L=4 in Figure 2, that is, Sector 1, Sector 2, Sector 3, and Sector 4), where at most Ml virtual beams are set in the lth sector and l∈1,⋯,L. The virtual beams among different sectors do not overlap, while the virtual beams within the same sector are allowed to overlap.Figure 2
Integration feasibility of online learning method.The mmSBS can employ a finite setMl of Ml=∣Ml∣ virtual beams in the lth sector. For any sector, the mmSBS may simultaneously select a subset of at most n virtual beams to serve at most n (n<Ml,∀l∈1,⋯,L) vehicles, where n corresponds to the maximum number of RF chains at the mmSBS. However, considering all the sectors, the n virtual beams in the same sector are not necessarily the best n virtual beams. Therefore, the mmSBS should firstly select at most n best virtual beams from each sector separately to serve at most n vehicles in each sector, respectively, and then it reselects at most n best virtual beams from all the selected virtual beams to serve at most n vehicles in the entire coverage area of the mmSBS.When a virtual beam is selected to limit the energy propagation area of the transmitted signal, the RF chain and frequency band are assigned to it. At this point, the selected virtual beam becomes the physical beam. For the sake of simplicity, we will not distinguish between a virtual beam and a physical beam in the following text.We assume that the eNB can provide the necessary information to the mmSBS, which will be regarded as the vehicle context below. With the aid of the eNB, a vehicle will know the location of the mmSBS and the selected beam for it. Figure2 shows how the information is communicated within the network. Since a vehicle keeps a continuous connectivity with the eNB via its LTE interface, it can send a registration request message (see “1: registration request” in Figure 2) to this eNB when it requires a mmWave connectivity between itself and a mmSBS, which contains the vehicle’s location and velocity.Upon receipt of the registration request, the eNB sends a mmWave service request message (see “2: service request” in Figure2) to a potential mmSBS after it makes a certain decision and judgment based on the vehicle’s location and velocity. This message contains the vehicle’s identifier in the cellular system, the identifier of the road on which the vehicle is travelling, and its expected direction of arrival at the mmSBS. By running the IFML algorithm, the mmSBS responds to the mmWave service request (see “3: service response” in Figure 2) with the information with respect to the selected beam. Next, the eNB forwards the related information about the mmSBS (i.e., the mmSBS’s location and its selecting beam) to the vehicle (see “4: registration response” in Figure 2).Once reaching the coverage area, the vehicle starts a mmSBS associating process by sending an associating request, and then, it is replied by an associating response from the mmSBS, in which the vehicle measures the channel state information (CSI) from the associating response message and sends the CSI feedback for modulation and coding assignment. Next, the mmSBS enters the data transmission phase (see “5: association & communication” in Figure2). When the data transmission phase is successful, the mmSBS will get acknowledgments of the transmitted data frames and thus any other feedback is not required. If a vehicle cannot detect the mmSBS within a selected beam, it will send the feedback to the eNB (see “6: service feedback” in Figure 2). This feedback will be forwarded to the mmSBS as a reference for future decisions (see “7: service feedback” in Figure 2).Because the beam selection results need to be adjusted in time to serve the most appropriate set of vehicle users, we assume that the mmSBS adopts a discrete time setting. In other words, system time is discretized into equal time periods, which is denoted astt∈1,⋯,T. After each time period t, all the beam selection results will be updated. If a time period is shorter, the beam selection results are updated more timely, but it leads to more system overhead. Therefore, a reasonable tradeoff is a wise choice, and the specific value needs to be selected empirically. The detailed process of beam selection and update is described below.
(1)
At the first time slot of each time periodt, a set ℧tl=vt,il∣i∈1,⋯,Vtl of vehicles will be registered to the lth (l∈1,⋯,L) sector of the mmSBS via the eNB, where Vtl is the number of vehicles and satisfies Vtl=∣℧tl∣>n. n is the maximum number of supported RF chains in the mmSBS, and it corresponds to the maximum number of vehicles that can simultaneously get downlink transmission service within the coverage area of the mmSBS. As the foregoing, during the registration, the mmSBS gets the information about the context ct,il of each incoming vehicle vt,il, which is formally regarded as a D-dimensional vector. Since the information about a vehicle can be described by using D context dimensions, it is encoded as a bit string with the fixed length in each dimension. After the first time slot of each time period t, the new context space is acquired by the mmSBS, which is represented as ℂVtl=ct,il=<ct,il,1,⋯,ct,il,D>∣i∈1,⋯,Vtl. In this paper, the context vector is two-dimensional (i.e., D=2), since we only consider the identifier of road and direction of arrival as the context for a vehicle(2)
The mmSBS selects a subset of at mostn best beams from the lth sector, where the set of selected beams in each time period t is denoted as Btl=bt,jl∣j∈1,⋯,n,n<Ml. Then, it reselects at most n beams from ⋃l=1LBtl to serve at most n vehicles in the entire coverage area of the mmSBS. Finally, at most n vehicles in ⋃l=1L℧tl are informed about the selected beams by the associated eNB through using their LTE interfaces(3)
When each selected vehicle (e.g.,vt,il) enters a mmSBS’s coverage area, it receives data from this mmSBS and feeds back this situation to it. Therefore, the mmSBS will observe the amount of data rbt,jlct,il that vehicle vt,il successfully receives via the selected beams bt,jl, until the time period t is over or the vehicle is out of the reach of its beamIn general, the amount of datarbt,jlct,il that a vehicle with context ct,il∈ℂVtl can successfully receive from the mmSBS by using beam bt,jl∈Btl during the time period t is a random variable that relies on the communication environment of the mmSBS (e.g., road conditions, vehicle routes, and blockages). The random variable rbt,jlct,il is also regarded as the beam performance (i.e, the aggregate data received by the vehicle) of the beam bt,jl under the context ct,il, which is usually bounded in 0,rmax, where rmax is the maximum amount of data that can be received by a vehicle. The Shannon theorem and the contact time can be used to estimate rmax. The Shannon theorem determines the maximum achievable data rate of the channel, which also relies on the selected modulation and coding scheme. The contact time is defined as the time during which mmSBS can transmit data to the vehicle, which is bounded by the coverage area of the beam and depends on beam width, beam direction, and vehicle speed.The expected value of the random variablerbt,jlct,il can be denoted as μbt,jlct,il, which is also seen as the expected beam performance of the beam bt,jl under the context ct,il. The purpose of the mmSBS’s selecting a subset of beams is to maximize the expected received data at a subset of vehicles. That is, it aims at maximizing the sum of the expected beam performances. The optimal subset of beams in the time period t of the lth sector is denoted as Btl∗ℂVtl=bt,jl∗ℂVtl∣j∈1,⋯,n,n<Ml, which relies on ℂVtl=ct,il=<ct,il.1,⋯,ct,il,D>i∈1,⋯,Vtl and its n beams formally meet the following relational expression.
(1)bt,jl∗∈=argmaxbt,jlϵM1/⋃k−1j−1b^t,jlct,jlϵℂvtl/⋃k′−1i−1c^t,k′l∗μbt,jlct,il.If the mmSBS knows the expected beam performanceμbt,jlct,il for each vehicle context ct,il∈ℂVtl and each beam bt,jl∈Ml in advance, it can be easy to select the optimal subset of beams for the set of incoming vehicles in the lth sector according to (1). For all the time periods, this will generate the amount of data expected to be received in total.
(2)∑t=1T∑i=1Vtl∑j=1nErbt,jl∗ct,il∗=∑t=1T∑i=1Vtl∑j=1nμbt,jl∗ct,il∗.Unfortunately, the mmSBS does not understand the communication environment, so it must learn the expected beam performanceμbt,jlct,il over time for each vehicle vt,il with the context ct,il. The mmSBS must attempt different beams for different vehicle contexts over time to learn these values. Meanwhile, it should also ensure that those beams that have been proved to have good performance are fully utilized. Therefore, the mmSBS must seek out a trade-off between exploring the beams with the unknown performance and exploiting the beams with the proven high average beam performance.In the following section, we will describe the IFML algorithm, where for each time period with incoming vehicles of contexts⋃l=1LℂVtl, the best n beams are selected from ⋃l=1LMl. The IFML’s selection relies on the historical record of selected beams in previous time periods and the corresponding observed beam performance values. Given any set of the vehicles arriving in any contexts in the lth sector, the expected amount of received data is estimated as follows:
(3)∑t=1T∑i=1Vtl∑j=1nErbt,jlct,il=∑t=1T∑i=1Vtl∑j=1nEμbt,jlct,il.The expected difference in the amount of data received by vehicles that are also observed by the mmSBS from the vehicles’ feedback and that learned by the learning algorithm is called the regret of learning. Based on (2) and (3), the regret of learning can be estimated by the following formula:
(4)RT=E∑t=1T∑i=1Vtl∑j=1nrbt,jl∗ct,il∗−rbt,jlct,il=∑t=1T∑i=1Vtl∑j=1nμbt,jl∗ct,il∗−Eμbt,jlct,il.


## 3.1. Network Architecture

As shown in Figure1, we consider an integrated mmWave/sub-6 GHz cellular system, where mmWave small base stations (mmSBSs) are deployed in the coverage area of an LTE eNode B (eNB). Each mmSBS is connected to the eNB via a backhaul link in a wired mode (e.g., an optical fiber) or a wireless mode (e.g., a mmWave link). The vehicles are equipped with two types of radio frequency interfaces (i.e., an LTE interface, which is used to connect to the eNB, and a mmWave interface, which is used for ultrahigh-speed data communication).Figure 1
A mmWave vehicular communication scenario.We assume that the eNB has the function that can assist each vehicle in its coverage to obtain the information of the mmSBS with which the vehicle should be associated. Since the LTE interface has the wider coverage, each vehicle can easily request help from the eNB via the LTE interface. If the eNB judges that there is not any appropriate mmSBS for the vehicle to associate with, it will reject the request, which will occur when the eNB judges that the vehicle is about to move out of its coverage. Even if the eNB agrees to handle a request that it should not have accepted, it can also transfer the request to an infrastructure component (e.g., a V2X server). With the help of the Home Subscriber Server (HSS) and Mobility Management Entity (MME) in the LTE infrastructure, this component will specify another suitable eNB to assist the vehicle.In theory, each mmSBS can plan an infinite number of virtual beams, and the beam width of each beam can be set to the range from 0 to 360°. Therefore, the overlap between virtual beams should be allowed. In fact, if all the virtual beams of a mmSBS are used simultaneously, its potential transmission capacity can be multiplied. To this end, each virtual beam must actually have an exclusive RF chain. Moreover, if the virtual beams overlap each other, they must be assigned to different frequency bands. Since there are abundant bandwidth resources in mmWave frequency bands, it is not difficult to allocate different channels for overlapping virtual beams. If each mmSBS coverage area only shares the same frequency band in an underlay mode, the RF chains’ transmission powers on overlapping beams must be reasonably selected to control mutual interference.Due to the limitation of form-factor and manufacturing cost, it is undesirable that the number of RF chains is proportional to the number of virtual beams when the number of virtual beams is set to a very large value. The literatures [2, 48] discuss the technical details on this limitation. Therefore, the number of vehicles that a mmSBS can simultaneously serve is limited by the maximum number of RF chains which a mmSBS can be equipped with. When the number of the virtual beams of a mmSBS and the number of the vehicles in its coverage exceed the number of its RF chains, respectively, it should select the best subset of virtual beams to provide the best downlink sum-data.In order to achieve this goal, we model a virtual beam selection problem in a mmSBS as a MAB problem, which allows this mmSBS to identify the best virtual beams by carefully selecting subsets of virtual beams over time. Also, each selected virtual beam must have its reasonable RF transmission power to control mutual interference if the same frequency band is shared in an underlay mode.As described in [49], a decision-maker must select a subset of actions with unknown expected earnings for the purpose of maximizing the earnings over time in MAB problems. The challenge in MAB problems is to address the exploration and exploitation dilemma, since all the actions should be fully explored to learn their earnings, but those which have already generated high earnings should also be exploited. Based on the contextual online learning idea, the authors in [4] address the abovementioned challenge. However, in our concerns, the online learning algorithm will have a larger search space since the number of possible virtual beams is larger, especially when the corresponding powers also need to be reasonably selected. Moreover, as the number of vehicles increases, the contextual space search overhead also increases accordingly. Therefore, we propose the IFML algorithm that tackles this problem.


## 3.2. Problem Formulation

We do not limit the number of virtual beams which a mmSBS can possess, since a larger virtual beamspace is more likely to find a more suitable beam for a vehicle. However, in order to keep this advantage while reducing the search time of online learning algorithm, we evenly divide the coverage area of a mmSBS intoL sectors (e.g., L=4 in Figure 2, that is, Sector 1, Sector 2, Sector 3, and Sector 4), where at most Ml virtual beams are set in the lth sector and l∈1,⋯,L. The virtual beams among different sectors do not overlap, while the virtual beams within the same sector are allowed to overlap.Figure 2
Integration feasibility of online learning method.The mmSBS can employ a finite setMl of Ml=∣Ml∣ virtual beams in the lth sector. For any sector, the mmSBS may simultaneously select a subset of at most n virtual beams to serve at most n (n<Ml,∀l∈1,⋯,L) vehicles, where n corresponds to the maximum number of RF chains at the mmSBS. However, considering all the sectors, the n virtual beams in the same sector are not necessarily the best n virtual beams. Therefore, the mmSBS should firstly select at most n best virtual beams from each sector separately to serve at most n vehicles in each sector, respectively, and then it reselects at most n best virtual beams from all the selected virtual beams to serve at most n vehicles in the entire coverage area of the mmSBS.When a virtual beam is selected to limit the energy propagation area of the transmitted signal, the RF chain and frequency band are assigned to it. At this point, the selected virtual beam becomes the physical beam. For the sake of simplicity, we will not distinguish between a virtual beam and a physical beam in the following text.We assume that the eNB can provide the necessary information to the mmSBS, which will be regarded as the vehicle context below. With the aid of the eNB, a vehicle will know the location of the mmSBS and the selected beam for it. Figure2 shows how the information is communicated within the network. Since a vehicle keeps a continuous connectivity with the eNB via its LTE interface, it can send a registration request message (see “1: registration request” in Figure 2) to this eNB when it requires a mmWave connectivity between itself and a mmSBS, which contains the vehicle’s location and velocity.Upon receipt of the registration request, the eNB sends a mmWave service request message (see “2: service request” in Figure2) to a potential mmSBS after it makes a certain decision and judgment based on the vehicle’s location and velocity. This message contains the vehicle’s identifier in the cellular system, the identifier of the road on which the vehicle is travelling, and its expected direction of arrival at the mmSBS. By running the IFML algorithm, the mmSBS responds to the mmWave service request (see “3: service response” in Figure 2) with the information with respect to the selected beam. Next, the eNB forwards the related information about the mmSBS (i.e., the mmSBS’s location and its selecting beam) to the vehicle (see “4: registration response” in Figure 2).Once reaching the coverage area, the vehicle starts a mmSBS associating process by sending an associating request, and then, it is replied by an associating response from the mmSBS, in which the vehicle measures the channel state information (CSI) from the associating response message and sends the CSI feedback for modulation and coding assignment. Next, the mmSBS enters the data transmission phase (see “5: association & communication” in Figure2). When the data transmission phase is successful, the mmSBS will get acknowledgments of the transmitted data frames and thus any other feedback is not required. If a vehicle cannot detect the mmSBS within a selected beam, it will send the feedback to the eNB (see “6: service feedback” in Figure 2). This feedback will be forwarded to the mmSBS as a reference for future decisions (see “7: service feedback” in Figure 2).Because the beam selection results need to be adjusted in time to serve the most appropriate set of vehicle users, we assume that the mmSBS adopts a discrete time setting. In other words, system time is discretized into equal time periods, which is denoted astt∈1,⋯,T. After each time period t, all the beam selection results will be updated. If a time period is shorter, the beam selection results are updated more timely, but it leads to more system overhead. Therefore, a reasonable tradeoff is a wise choice, and the specific value needs to be selected empirically. The detailed process of beam selection and update is described below.
(1)
At the first time slot of each time periodt, a set ℧tl=vt,il∣i∈1,⋯,Vtl of vehicles will be registered to the lth (l∈1,⋯,L) sector of the mmSBS via the eNB, where Vtl is the number of vehicles and satisfies Vtl=∣℧tl∣>n. n is the maximum number of supported RF chains in the mmSBS, and it corresponds to the maximum number of vehicles that can simultaneously get downlink transmission service within the coverage area of the mmSBS. As the foregoing, during the registration, the mmSBS gets the information about the context ct,il of each incoming vehicle vt,il, which is formally regarded as a D-dimensional vector. Since the information about a vehicle can be described by using D context dimensions, it is encoded as a bit string with the fixed length in each dimension. After the first time slot of each time period t, the new context space is acquired by the mmSBS, which is represented as ℂVtl=ct,il=<ct,il,1,⋯,ct,il,D>∣i∈1,⋯,Vtl. In this paper, the context vector is two-dimensional (i.e., D=2), since we only consider the identifier of road and direction of arrival as the context for a vehicle(2)
The mmSBS selects a subset of at mostn best beams from the lth sector, where the set of selected beams in each time period t is denoted as Btl=bt,jl∣j∈1,⋯,n,n<Ml. Then, it reselects at most n beams from ⋃l=1LBtl to serve at most n vehicles in the entire coverage area of the mmSBS. Finally, at most n vehicles in ⋃l=1L℧tl are informed about the selected beams by the associated eNB through using their LTE interfaces(3)
When each selected vehicle (e.g.,vt,il) enters a mmSBS’s coverage area, it receives data from this mmSBS and feeds back this situation to it. Therefore, the mmSBS will observe the amount of data rbt,jlct,il that vehicle vt,il successfully receives via the selected beams bt,jl, until the time period t is over or the vehicle is out of the reach of its beamIn general, the amount of datarbt,jlct,il that a vehicle with context ct,il∈ℂVtl can successfully receive from the mmSBS by using beam bt,jl∈Btl during the time period t is a random variable that relies on the communication environment of the mmSBS (e.g., road conditions, vehicle routes, and blockages). The random variable rbt,jlct,il is also regarded as the beam performance (i.e, the aggregate data received by the vehicle) of the beam bt,jl under the context ct,il, which is usually bounded in 0,rmax, where rmax is the maximum amount of data that can be received by a vehicle. The Shannon theorem and the contact time can be used to estimate rmax. The Shannon theorem determines the maximum achievable data rate of the channel, which also relies on the selected modulation and coding scheme. The contact time is defined as the time during which mmSBS can transmit data to the vehicle, which is bounded by the coverage area of the beam and depends on beam width, beam direction, and vehicle speed.The expected value of the random variablerbt,jlct,il can be denoted as μbt,jlct,il, which is also seen as the expected beam performance of the beam bt,jl under the context ct,il. The purpose of the mmSBS’s selecting a subset of beams is to maximize the expected received data at a subset of vehicles. That is, it aims at maximizing the sum of the expected beam performances. The optimal subset of beams in the time period t of the lth sector is denoted as Btl∗ℂVtl=bt,jl∗ℂVtl∣j∈1,⋯,n,n<Ml, which relies on ℂVtl=ct,il=<ct,il.1,⋯,ct,il,D>i∈1,⋯,Vtl and its n beams formally meet the following relational expression.
(1)bt,jl∗∈=argmaxbt,jlϵM1/⋃k−1j−1b^t,jlct,jlϵℂvtl/⋃k′−1i−1c^t,k′l∗μbt,jlct,il.If the mmSBS knows the expected beam performanceμbt,jlct,il for each vehicle context ct,il∈ℂVtl and each beam bt,jl∈Ml in advance, it can be easy to select the optimal subset of beams for the set of incoming vehicles in the lth sector according to (1). For all the time periods, this will generate the amount of data expected to be received in total.
(2)∑t=1T∑i=1Vtl∑j=1nErbt,jl∗ct,il∗=∑t=1T∑i=1Vtl∑j=1nμbt,jl∗ct,il∗.Unfortunately, the mmSBS does not understand the communication environment, so it must learn the expected beam performanceμbt,jlct,il over time for each vehicle vt,il with the context ct,il. The mmSBS must attempt different beams for different vehicle contexts over time to learn these values. Meanwhile, it should also ensure that those beams that have been proved to have good performance are fully utilized. Therefore, the mmSBS must seek out a trade-off between exploring the beams with the unknown performance and exploiting the beams with the proven high average beam performance.In the following section, we will describe the IFML algorithm, where for each time period with incoming vehicles of contexts⋃l=1LℂVtl, the best n beams are selected from ⋃l=1LMl. The IFML’s selection relies on the historical record of selected beams in previous time periods and the corresponding observed beam performance values. Given any set of the vehicles arriving in any contexts in the lth sector, the expected amount of received data is estimated as follows:
(3)∑t=1T∑i=1Vtl∑j=1nErbt,jlct,il=∑t=1T∑i=1Vtl∑j=1nEμbt,jlct,il.The expected difference in the amount of data received by vehicles that are also observed by the mmSBS from the vehicles’ feedback and that learned by the learning algorithm is called the regret of learning. Based on (2) and (3), the regret of learning can be estimated by the following formula:
(4)RT=E∑t=1T∑i=1Vtl∑j=1nrbt,jl∗ct,il∗−rbt,jlct,il=∑t=1T∑i=1Vtl∑j=1nμbt,jl∗ct,il∗−Eμbt,jlct,il.


## 4. The Improved Fml Algorithm

The proposed beam selection algorithm (i.e., the IFML algorithm) is run in each mmSBS. It first evenly partitions the context space into small sets of similar contexts in each of the sectors of a mmSBS and, then, learns about the individual performance of each beam in each of these small sets. Next, in each time period, the IFML goes through either an exploration process or an exploitation phase, which relies on the contexts of arrival vehicles and the preselected control function. For an exploration process, the IFML selects a random subset of beams. However, in an exploitation phase, it selects the beams that performed best during the previous time periods. Finally, by observing the amount of received data from the vehicles in the coverage area of the mmSBS, the IFML gets performance estimating values of selected beams. Therefore, it learns the performance of each beam under each vehicle context over time.The pseudocode of the IFML algorithm is described in Algorithms1–3. In the lines 1~5 of Algorithm 1, the IFML evenly partitions the context space ⋃l=1LℂVtl into L∙ℂTD-dimensional subspaces with the same size, where L and ℂT are the inputting values to the algorithm. Then, the IFML initializes each counter Nb,slt for each beam b∈⋃l=1LMl and each subspaces∈⋃l=1LℂVtl. The purpose of setting these counters is to describe how many vehicles of a certain context have reached the mmSBS in previous time periods, in which the mmSBS had selected a certain beam. Moreover, the counter Nb,slt formally denotes the total number of vehicles with the context in subspace s that reached the mmSBS whenever beam b had been selected in any of the time periods 1,⋯,t−1 of any of the sectors 1,⋯,L. The algorithm also initializes each estimatorμ^b,slt for each beam b∈⋃l=1LMl and each subspace s∈⋃l=1LℂVtl, which denotes the estimated performance of beam b for vehicles with the context in subspace s.In a time periodt, the IFML observes the contexts ⋃l=1LℂVtl of the ∑l=1LVtl incoming vehicles. For each context ct,il, the IFML decides to which subspace this context belongs (lines 3-4 in Algorithm 2). That is, it seeks st,il∈⋃l=1LℂVtl with ct,il∈st,il. According to the set ⋃l=1LHtl≔⋃l=1Lst,il∣i∈1,⋯,Vtl of subspaces, the IFML computes the set BHtluet of underexplored beams (line 5 in Algorithm 2) by using the following formula:
(5)BHtluet≔⋃i=1Vtlb∈Ml:Nb,st,illt≤Kt.In (5), K:1,⋯,T↦ℝ is a deterministic, monotonically increasing control function, which is used to determine whether to go through an exploration process or enter an exploitation phase. The control function Kt needs to be selected adequately to guarantee that the IFML achieves a desired fine performance in terms of its regret. The Theorem 1 in [4] provides a suitable selection for the control function, which is also repeated below for the convenience of readers.Theorem 1 (bound forRT): for the lth sector, let Kt=Kt=t2α/3α+Dlogt and ℂT=T1/3α+D. If the IFML is run by using these parameters and if Assumption 1 holds true, the leading order of the regret R(T) is OnVtlrmaxMlT2α+D/3α+DlogT.The detailed proof of Theorem 1 can be found in [4] (although the parameters are somewhat different, they are essentially the same), where Assumption 1 is repeated below for the convenience of readers.Assumption 1: there existα>0 and β>0 such that for all b∈Ml and for all x,y∈ℂVtl in the lth sector, it holds that μbx−μby≤βx−yα, where · denotes the Euclidean norm in ℝD.When there exist underexplored beams, the IFML goes through an exploration phase (lines 6-14 in Algorithm2). If the number ult≔BHtluet of underexplored beams is at least n, the IFML randomly selects n beams from them. If the number ult of underexplored beams is smaller than n, the IFML selects all ult underexplored beams. Moreover, it selects the n−ult beams b^1,Htllt, …, b^n−ul,Htllt from Ml\BHtluet, which meet the following formula:
(6)b^j,Htllt=argmaxblϵM1/Bj,Htluet∪⋃k−1j−1b^j,Htlltclϵℂvtl/⋃k′−1i−1c^k′,Htll∗tμbt,jlct,il.In (6), j=1,⋯,n−ult. If there are no underexplored beams, the IFML adopts an exploitation action (lines 15-17 in Algorithm 2), it selects the n beams b^1,Htllt, …, b^n,Htllt from Ml, which meet the following formula:
(7)b^j,Htllt=argmaxblM1/⋃k−1j−1b^j,Htlltclϵℂvtl/⋃k′−1i−1c^k′,Htll∗tμbt,jlct,il.In (7), j=1,⋯,n. After selecting the n beams from each sector, respectively, the IFML will reselect at most n beams from all the selected beams of all the sectors (lines 8-18 in Algorithm 1). After beam reselection, the IFML observes the beam performance of each selected beam for each vehicle within this time period (line 1 in Algorithm 3). According to these observations, the IFML updates its internal counters (lines 2-11 in Algorithm 3), where the weight coefficient ζ represents the contribution of the recently observed beam performance to the updated beam performance and is usually determined by the empirical values in terms of the communication performance of the system.Algorithm 1: Pseudocode of the IFML Algorithm.
Input:T, L, ℂT, KtOutput: the selected beams and the specified vehicles using the beams1:For each l=1, …, Ldo2:  Initialize context partition: Create partitionℂVtl into ℂT subspaces with same size.3:  Initialize counters: for allb∈Ml and s∈ℂVtl, set Nb,sl=04:  Initialize estimators: for allb∈Ml and s∈ℂVtl, set μ^b,sl=05:End for6:For each t=1, …, Tdo7:  Invoke Algorithm2 to get BHuet and BHvet8:IfBHuet≠∅then    ■ Exploration9:u=sizeBHuet10:Ifu≥nthen11:    Select then beams randomly from BHuet12:Else13:    Treat all the members ofBHuet as the u beams14:    Select then−u best beams from BHvet15:End if16:Else    ■ Exploitation17:   Select then beams with the best performance from BHvet18:End if19:  Invoke Algorithm3 to update estimators and counters20:End forAlgorithm 2: Algorithm for getting BHuet and BHvet..
Input:nullOutput:BHuet and BHvet1:BHuet=BHvet=∅;2:For each l=1, …, Ldo3:  Observe vehicle contextsℂVtl=ct,il∣i∈1,⋯,Vtl4:  FindHtl=st,il∣i∈1,⋯,Vtl such that ct,il∈st,il∈ℂVtl5:  Compute the set of under-explored beamsBHtluet in (5)6:IfBHtluet≠∅then    ■ Exploration7:ul=sizeBHtluet8:Iful≥nthen9:    Selectbt,il∣i∈1,⋯,n randomly from BHtluet10:    Addbt,il∣i∈1,⋯,n to BHuet11:Else12:    Selectbt,il∣i∈1,⋯,ul as theul beams from BHtluet and then add them to BHuet13:    Selectbt,ul+1l, …, bt,nl as then−ul beams b^1,Htllt, …, b^n−ul,Htllt from (6) and then add them to BHvet14:End if15:Else    ■ Exploitation16:   Selectbt,1l, …, bt,nl as then beams b^1,Htllt, …, b^n,Htllt from (7) and then add them to BHvet17:End if18:End forAlgorithm 3: Algorithm for updating estimators and counters.
Input: the selected beamsOutput: the updated estimators and counters1: The random variablerbt,jlct,il is is initialized to a negative number2: Observe the received data of each vehiclevt,ilwith the context ct,il∈st,il∈ℂVtl, in each beam bt,jl, and assign it to rbt,jlct,il3:For each l=1, …, Ldo4:For each i=1, …, Vtldo5:For each j=1, …, ndo6:Ifrbt,jlct,il≥0then7:μ^bt,jl,st,ill=1−ζ·μ^bt,jl,st,ill+ζ·rbt,jlct,il8:Nbt,jl,st,ill=Nbt,jl,st,ill+19:End if10:End for11:End for12:End for


## 5. Algorithm Performance Analysis

For a mmSBS with coverage area partition, its online learning overhead isO⋃l=1LℂT∙Ml. However, For a mmSBS without coverage area partition, the corresponding overhead is OL∙ℂT∙⋃l=1LMl. Obviously, the overhead of the former is much smaller than that of the latter, especially when the number of the divided coverage areas (i.e., the value of L) is larger.When the number of beams at a mmSBS is larger and there are more context subspaces in its coverage area, it will take a longer time to learn the beam performance under each context subspace. In the scheme proposed in this paper, the corresponding learning overhead isO⋃l=1LℂT∙Ml, since coverage area partition is employed. The overhead savings are due to the neglect of a large number of unreasonable beam-context matching operations. For example, a beam obviously cannot be assigned to any vehicle in any context subspace outside its coverage, since such beam-context matching operation makes no sense.When the number of beams at a mmSBS is not large and there are not many context subspaces in its coverage area, the coverage partition for a mmSBS is not necessary since it may not reduce overhead significantly. In fact, only if the number of beams is not limited, it is possible to set more types of beam widths and beam directions, and thus, it is more possible to select the beam with a more reasonable width and direction for each vehicle.


## 6. Performance Evaluation

### 6.1. Simulation Scenario and Parameter Settings

The IFML is evaluated via simulation experiments. The IFML is divided into two categories (i.e., the IFML-I and the IFML-II), depending on whether the overlapping beams use the different frequency bands or the same one. In the IFML-I, the overlapping beams use the different frequency bands, while they use the same one in the IFML-II.In the following text, we first describe the simulation scenario and, then, introduce the mmWave channel propagation model that is used in our simulation experiments. Next, we describe the benchmark algorithms and performance metrics. Finally, we give the relevant simulation parameter settings according to the 3GPP technical specification in [50].Our simulation scenario is shown in Figure3, where the mmSBS’s coverage is divided into the four sectors (i.e., L=4) with the same size. For each sector, the two roads pass through it, where the location and size of the blockages on each road are fixed and account for about 10 percent of the road length. These blockages block the mmWave channels between the mmSBS and the vehicles on the roads.Figure 3
Illustration of our simulation scenario.There are seven types of beams according to the various beam width values (i.e., from 30° to 90° with the step size of 10°) for each sector, where the number of beams per type is one. Therefore, the number of the mmSBS’s beams in thelth sector is set to Ml=7 in the IFML-I, while Ml is 7∣P∣ in the IFML-II. The setP is a set of discrete values with respect to transmission power of a RF chain, while P is the number of the members in the setP. For the IFML-II, the discrete power values are set from 0.1 to 1 Watts with the step size of 0.1, and thus, P is equal to 10.In our implementation, a time period is defined by the mmSBS as a fixed length of time. During this time period, the IFML-I and the IFML-II receive registration information and, then, learn from the context and received data of the other vehicles passing through the selected beams. We select the identifier of the road and the direction of arrival as context. Therefore, the context vector is a two-dimensional vector (i.e.,D=2). The arrival direction of a vehicle is defined as the angle between the line connecting this vehicle with a mmSBS and the positive x-axis in the plane coordinate system with this mmSBS as the origin.The length of a time periodt, the parameter α, and the number of two-dimensional subspaces in each sector of the mmSBS are set to t=3s, α=0.33, and ℂT=10, respectively. Therefore, based on Theorem 1, the time horizon T is about 1000, while the value of control functionKt is about 2.02When then beams with the best performance are selected, the contexts in which these beams match are also determined at the same time. If the number of vehicles in such contexts is greater than the number of the selected beams, a strategy for vehicle selection needs to be determined in advance. For example, depending on the order in which vehicles enter a context space, a First-Come-First-Served (FCFS) policy can be used, or a First-Come-Last-Served (FCLS) policy can be adopted. In addition, a Random Selection (RASE) policy can also be employed.When the vehicles under the specific beam coverage in thelth sector are counted as the members of Vtl in the first time slot of the tth time period, every counted vehicle has entered this coverage area before the beginning of this time slot but does not leave before the end of this time slot. So the earlier they enter, the earlier they may leave.The above simulation scenario is implemented on the OMNeT platform (i.e., omnetpp-5.4.1) with the application package (i.e., veins-4.7.1) in terms of vehicle communication simulation framework, where the traffic pattern is generated by the third-party software: simulation of urban mobility (i.e., sumo-0.30.0). The speed of vehicles is set to the range from 5 to 10 m/s.The following mmWave channel propagation model is adopted in our simulation experiments:(8)pir=pitGitGirGic.In (8), pit is the transmission power at a directional beam from the mmSBS to the vehicle i, while pir is the received power at the vehicle i side when the mmSBS transmits at the power pit;Git, and Gir are the directional transmitting gain and directional receiving gain, respectively; Gic is the channel gain between the mmSBS and the vehicle i. When the beam between the mmSBS and the vehicle i is aligned, Git and Gir can be estimated by the following formula [51]:
(9)Git=2π−2π−θit∂θit,9aGir=2π−2π−θir∂θir,9b.In (9), θit is the beam width of the transmitter, while θir is the beam width of the receiver; ∂ is the gain of the side lobe and 0<∂≪1, while θit and θir are the main lobe in radian. The channel gain Gic can be estimated by the following formula [52]:
(10)Gic=χicδτ−τi2.In (10), δ· is the Dirac delta function; τi and χic are the propagation delay and the amplitude of the path from the mmSBS to the vehicle i, respectively. τi is estimated by the following formula:
(11)τi=dic.In (11), di is the distance of the path from the mmSBS to the vehicle i, and c is the speed of light. When there is a line-of-sight (LOS) path between the mmSBS and the vehicle i, the amplitude is estimated by the following formula [52]:
(12)χic,los=λ4πdi.In (12), λ is the wavelength, and λ=c/fc,fc is the carrier frequency. When there is not any LOS path between the mmSBS and the vehicle i, the amplitude is related to both path loss and reflection coefficients. Due to very high reflection loss in mmWave band [53], only one reflection of a given path is considered, which is estimated by the following formula [52]:
(13)χic,nlos=λΓ4πdi.In (13), Γ is the reflection coefficient of the mmWave reflection path. Therefore, based on (12) and (13), χic is estimated by
(14)χic=χic,los,Inunblockedarea14aχic,nlos,Inblockedarea14b.In the IFML-II, when the overlapping beams use the same frequency band, there is the same frequency interference between them. Let the beam of the vehiclei and the beam of the vehicle j be overlapping in the same mmSBS coverage. When the mmSBS transmits data to vehicle j, the interference power received at vehicle i should be
(15)pj→ir=pjtGj→itGj→irGic.In (15), Gj→it and Gj→ir are the directional transmitting gain and directional receiving gain of the mmWave link between the mmSBS and vehicle i when the mmSBS transmits data to vehicle j. The directional transmit-receive gain of each mmWave link can be derived by the following formula:
(16)Gj→itGj→ir=2π−2π−θjt∂θjt∙2π−2π−θir∂θirConditionA2π−2π−θjt∂θjt∙∂ConditionB∂∙2π−2π−θir∂θirConditionC∂∙∂ConditionD.Letϕj→it and ϕj→ir be the beam offset angle from the mmSBS’s (the mmSBS transmits to vehicle j) transmitting beam direction to the position of vehicle i and that from vehicle i’s (vehicle i receives from the mmSBS) receiving beam direction to the position of the mmSBS, respectively; Condition A is both ϕj→it≤θjt/2 and ϕj→ir≤θjr/2; Condition B is both ϕj→it≤θjt/2 and ϕj→ir>θjr/2; Condition C is both ϕj→it>θjt/2 and ϕj→ir≤θjr/2; Condition D is both ϕj`→it>θjt/2 and ϕj→ir>θjr/2.The work in [4] is most similar to ours. However, in the implementation in [4], the context vector is one-dimensional since the only direction of arrival is considered as the context. Also, a time period is defined as the time during which the observed vehicle enters and leaves the cell coverage area. Since this type of time period is influenced by a variety of factors (e.g., speed of vehicle, size of cell coverage area, and road orientation), it has great uncertainty and thus is difficult to determine its value in practical applications.Furthermore, the direction of arrival in [4] only takes some simple and abstract values (i.e., north, south, east, and west) due to the consideration for wide adaptation. Therefore, in order to make a fairer comparison, we design a variant of the algorithm (i.e., FML) in [4] and call it VFML, which basically retains the core idea of the FML algorithm except for the fixed time period (i.e., t=3s) and the adoption of vehicle arrival direction defined in our scheme.Based on the same bound forRT as that in the IFML, the time horizon T in the VFML should be about 1000. In addition, based on our simulation scenario in Figure 3, the simulation area for the VFML is divided into 24 one-dimensional subspaces according to the arrival direction of a vehicle. Therefore, the other parameters in the VFML should be set to α=0.39 andKt=2.35. Since the maximum number of supported RF chains in a mmSBS is set to six and the beams do not overlap in the VFML, the six orthogonal beams with variable beam width from 30° to 60° cover the 360° azimuth.In addition, we also consider an Optimal Scheme (OPSC) as a comparison of our scheme, which has a priori knowledge about the expected beam performanceμbt,jlct,il of each beam bt,jl∈Btl in each context ct,il∈ℂVtl. In each period, the OPSC selects the optimal subset Btl∗ℂVtl of n beams as in (1). Therefore, the results achieved by the OPSC is the expected performance upper bound of the system.The performance metrics used in the evaluation are the aggregate received data, the cumulative received data, and the online learning cost. We define the aggregate received data as the received data (inGbits) from all the vehicles during a time period t, while we define the cumulative received data as the received data from all the vehicles during the time horizon T. The online learning cost is defined as the number of exploration rounds required for the IFML-I, the IFML-II, and the VFML to reach a certain percentage of the OPSC’s performance, where all the exploration operations during a time period are classified as one round. The values of the remaining simulation parameters are listed in Table 1, unless otherwise stated in the following text.Table 1
The simulation parameters.
SymbolDescriptionValuefcCarrier frequency60 GHzWSystem bandwidth1 GHzpmaxtThe maximum transmission power of a mmSBS30 dBmN0Noise power density at a vehicle-174 dBm/HzBWrEach vehicle’s receiving beam width90°BWtA mmSBS’s transmitting beam width30°~90°nThe maximum number of supported RF chains in a mmSBS6ΓThe reflection coefficient of mmWave reflection path0.3tThe length of a time period3 sTThe number of time periods1000ζThe weighted coefficient0.125RRadius of simulation area of a mmSBS100 m


### 6.2. Simulation Results and Analysis

Firstly, we evaluate the performance metrics of the IFML-I and the IFML-II compared to the benchmark algorithms such as the VFML and the OPSC. The corresponding results are shown in Figures4–11. When there are multiple vehicles in the coverage area of a selected beam at the same time, the VFML selects one of the vehicles at random for communication. Therefore, in this set of simulations, the IFML-I, the IFML-II, and the OPSC also adopt the RASE strategy. Next, we evaluate the performance metrics of the IFML-I in terms of the FCFS strategy, the FCLS strategy, and the RASE strategy. The corresponding results are shown in Figures 12–19. Finally, we compare the impact of learning information space size on the performance metrics of the IFML-I and the VFML, respectively. The corresponding results are shown in Figures 20–23.Figure 4
Impact of vehicle density on cumulative received data.Figure 5
Impact of vehicle density on online learning costs.Figure 6
Impact of the number of selected beams per period on cumulative received data.Figure 7
Impact of the number of selected beams per period on online learning costs.Figure 8
Impact of thermal noise power density on cumulative received data.Figure 9
Impact of thermal noise power density on online learning costs.Figure 10
Aggregate received data over the time horizon.Figure 11
The number of exploration operations per period over the time horizon.Figure 12
Impact of vehicle density on cumulative received data under the three vehicle scheduling strategies.Figure 13
Impact of vehicle density on online learning costs under the three vehicle scheduling strategies.Figure 14
Impact of the number of selected beams per period on cumulative received data under the three vehicle scheduling strategies.Figure 15
Impact of the number of selected beams per period on online learning costs under the three vehicle scheduling strategies.Figure 16
Impact of thermal noise power density on cumulative received data under the three vehicle scheduling strategies.Figure 17
Impact of thermal noise power density on online learning costs under the three vehicle scheduling strategies.Figure 18
Impact of weight coefficientζ on cumulative received data under the three vehicle scheduling strategies.Figure 19
Impact of weight coefficientζ on online learning costs under the three vehicle scheduling strategies.Figure 20
Impact of learning information space size on the IFML-I’s cumulative received data.Figure 21
Impact of learning information space size on the IFML-I’s online learning costs.Figure 22
Impact of learning information space size on the VFML’s cumulative received data.Figure 23
Impact of learning information space size on the VFML’s online learning costs.In Figure4, we investigate the impact of the number of vehicles in the simulation area on the cumulative received data achieved by the different algorithms, where at most 6 selected beams are used simultaneously in each time period and the number of vehicles varies from 35 to 95 with the step size of 15. We can see from Figure 4 that as the number of vehicles in the simulation area increases, the cumulative received data also increase. This is because the smaller number of vehicles means the more insufficient context information and thus results in a poorer learning effect. Therefore, the probability of accurately selecting the beams that can make vehicles maximize the cumulative received data is relatively small. With the increase of the number of vehicles, more context information is provided, which is conducive to improving the learning effect. Therefore, the probability of accurately selecting the beams that can make vehicles maximize the cumulative received data is relatively large.From Figure4, we also see that the cumulative received data hardly increase when the number of vehicles in the simulation area increases to a certain amount. This is because the number of selected beams per time period is fixed. At the same time, when the context information is large enough, the specified number of served vehicles can be selected reasonably through online learning. Figure 4 shows that the IFML-I is closer to the optimal one (i.e., the OPSC) than the IFML-II and the VFML, where the IFML-II outperforms the VFML. The main reason lies in two aspects. On the one hand, the IFML-I and the IFML-II can find the more reasonable beam width and beam direction for each served vehicle when compared with the VFML. On the other hand, the two IFMLs’ two-dimensional context can bring more accurate information than the VFML’s one-dimensional context, which can make a more reasonable decision for each vehicle. Compared with the IFML-II, the IFML-I pays more frequency band resources for better performance.Based on the same simulation configuration as that in Figure4, the online learning costs required for the IFML-I, the IFML-II, and the VFML under the different number of vehicles are shown in Figure 5. We can observe from Figure 5 that the changes of the online learning costs result from the number of vehicles in the system. As the number of vehicles increases, the costs increase as well. This is because the more vehicles that enter the system in a time period will give each beam the opportunity to probe the more context subspaces. Therefore, the performances of each beam on the more subspaces can be detected in a time period. If there is no performance history or the recorded history is insufficient, the corresponding beams are scheduled for exploration as soon as possible in a time period, which is helpful to speed up the detection process of the performance of each beam in each context subspace, and thus reduce the number of exploration rounds.Moreover, we can see from Figure5 that the two IFMLs’ online learning costs are larger than those of the VFML. This is because the two IFMLs have more beams and context subspaces than those of the VFML. By setting the more number of beams and using the more context information, each selected vehicle can be assigned the beam with a more reasonable width and direction. However, there are larger combinations of beams and contexts that need to be explored. As the number of vehicles increases, the difference in the number of exploration rounds between the two schemes is narrowed. This indicates that the operation mode (that the two IFMLs first perform a partitioned search for a mmSBS coverage area and then select the exploration beams from the partitioned search results) has higher exploration efficiency with the increase of the number of vehicles. In the IFML-II, in order to control cochannel interference, the beam transmission power must be adjusted, so it needs to be discretized into various values, which makes the beam exploration space be larger and the online learning cost be higher.In Figure6, we analyze the impact of the number of selected beams per time period on the cumulative received data, where the number of vehicles in the simulation area is set to 65 and the number of selected beams varies from 2 to 6 with the step size of 1. It can be seen from Figure 6 that as the number of simultaneously selected beams increases, the cumulative received data in the system increase as well. The reason for this increase is that the increasing number of beams used in a concurrent manner leads to the enhanced coverage area and thus the more vehicles can be served at the same time. However, as mentioned earlier, the larger the number of concurrent beams is, the higher the hardware complexity and energy consumption at a mmSBS are. It can also be seen that the two IFMLs achieve a larger cumulative received data than the VFML, where the explanation for the difference in terms of cumulative received data between the different schemes is similar to the explanation of the results in Figure 4.Based on the same simulation configuration as that in Figure6, the online learning costs required for the two IFMLs and the VFML under the different number of selected beams per period are shown in Figure 7. From Figure 7, as the number of selected concurrent beams increases, the number of exploration rounds decreases. The larger number of selected concurrent beams per time period means that the larger number of beams with unknown or uncertain performance can be explored in each time period. When the combined space size of beams and context subspaces remains unchanged, the number of required exploration rounds will decrease. In Figure 7, we see that the online learning costs of the two IFMLs are larger than that of the VFML, and the explanation of this difference is similar to the explanation of the results in Figure 5.In Figure8, we analyze the impact of thermal noise power density on the cumulative received data, where the number of vehicles in the simulation area is set to 65, the number of selected beams is set to 6, and the thermal noise power density varies from -170 dBm/Hz to -150 dBm/Hz with the step size of 5 dBm/Hz. Obviously, as the thermal noise power density in the system increases, the cumulative received data decrease. This is because the increase of thermal noise power density will reduce the signal-to-noise ratio. When the transmission power of a mmSBS is constant, the data rate will decrease according to the Shannon theorem. Also, we can see from Figure 8 that the cumulative received data of the two IFMLs are higher than that of the VFML, where the explanation for this difference is similar to the explanation of the results in Figure 4.Based on the same simulation configuration as that in Figure8, the online learning costs required for the two IFMLs and the VFML under the different thermal noise power density are shown in Figure 9. It is observed from Figure 9 that the impact of the thermal noise power density on the online learning costs of the algorithms is not obvious in general. The reason is that the variation of thermal noise power density does not change the size of the information space of online learning.However, at higher thermal noise density, the number of the VFML’s beam exploration rounds decreases slightly. As shown in Figure8, the thermal noise power density has a great impact on the cumulative received data. Therefore, the possible reason is that, due to the combination of the high thermal noise environment and the small number of optional beams in the VFML, the performance of each beam under each context subspace is not easily affected by the changes of other factors in the environment, and thus, a beam performance is considered stable and does not require additional exploration.In Figure10, we analyze the aggregate received data achieved by the algorithms over the time horizon with 1000 time periods, where the number of vehicles in the simulation area is set to 65. Figure 10 illustrates that the IFML-I can achieve performance closer to the OPSC than the IFML-II and the VFML. The fluctuations on the graph result from the number of vehicles in the system, the speed of each vehicle, and the contact time. Clearly, the OPSC gives an upper bound to the other algorithms due to a priori knowledge of the expected beam performance. So, the OPSC can select the optimal subset of beams according to the relational expression (1).The two IFMLs obviously outperform the VFML in terms of aggregate received data after 300 time periods, and the IFML-I is close to the OPSC’s aggregate received data after 600 time periods. The main reason is that after sufficient online learning, the two IFMLs can select a set of beams with more reasonable widths and directions than the VFML, which can provide greater transmission capacity for the vehicles in their coverage. However, before 300 time periods, due to insufficient online learning, most of the beams assigned to the vehicles are selected at random, and thus, there is no significant difference between the two IFMLs’ aggregate received data and the VFML’s.Based on the same simulation configuration as that in Figure10, the number of exploration operations per time period required for the algorithms over the time horizon with 1000 time periods are shown in Figure 11. We can see from Figure 11 that the two IFMLs needs more time periods for beam performance exploration than the VFML. This indicates that the two IFMLs takes longer time to sufficient online learning effect than the VFML. The main reason is that there is a difference of the learning information space resulting from the combination of beams and context subspaces between the two IFMLs and the VFML.In the two IFMLs, the larger learning information space leads to higher aggregate received data but increases the online learning costs. Fortunately, after a certain amount of time periods, there is no difference in the online learning costs between the three schemes, where they still have a similar beam performance update process after each beam is used.Based on the same simulation configuration as that in Figure4, under the three vehicle scheduling strategies adopted by the IFML-I, Figure 12 shows that the impact of the number of vehicles in the system on the cumulative received data, while Figure 13 shows that the impact of the number of vehicles in the system on online learning costs.As explained in Figure4, the cumulative received data under all the vehicle scheduling strategies in Figure 12 varies with the number of vehicles in the simulation area. However, there is a difference in terms of cumulative received data between the three vehicle scheduling strategies. The main for this difference is that the different strategies result in the different contact time, during which a vehicle stays in a beam coverage area. Among the three strategies, the FCLS leads to the longest average contact time, while the FCFS leads to the shortest one. So the cumulative data received under the FCLS is the largest, while that under the FCFS is the least.The variation trend of the online learning costs with the number of vehicles in the simulation area under all the vehicle scheduling strategies in Figure13 is similar to that in Figure 5. Therefore, the explanation in Figure 5 applies here as well. There is no difference in the online learning costs among the three strategies under the different number of vehicles. The main reason is that the different vehicle scheduling strategies do not affect the size of the learning information space.Based on the same simulation configuration as that in Figure6, under the three vehicle scheduling strategies adopted by the IFML-I, Figure 14 shows that the impact of the number of selected beams per period on cumulative received data, while Figure 15 shows that the impact of the number of selected beams per period on online learning costs.The variation trend of cumulative received data with the number of selected concurrent beams under all the vehicle scheduling strategies in Figure14 is similar to that in Figure 6. Therefore, the explanation in Figure 6 applies here as well. Also, in the case of different number of beams that can be used simultaneously, the main reason for the difference in cumulative received data under the three strategies is similar to that in Figure 12.As explained in Figure7, the online learning costs under all the vehicle scheduling strategies in Figure 15 vary with the number of beams that could be used simultaneously. However, there is no difference in the online learning costs under the three vehicle scheduling strategies. The main reason is that the different vehicle scheduling strategies do not affect the size of the information space for online learning.Based on the same simulation configuration as that in Figure8, under the three vehicle scheduling strategies adopted by the IFML-I, Figure 16 shows that the impact of thermal noise power density on cumulative received data, while Figure 17 shows that the impact of thermal noise power density on online learning costs.The variation trend of cumulative received data with the thermal noise power density under all the vehicle scheduling strategies in Figure16 is similar to that in Figure 8. Therefore, the explanation in Figure 8 applies here as well. Also, in the case of different thermal noise power density, the main reason for the difference in cumulative received data under the three vehicle scheduling strategies is similar to that in Figure 12.The variation trend of the online learning costs with the thermal noise power density under all the vehicle scheduling strategies in Figure17 is similar to that in Figure 9. However, there is no difference in the online learning costs under the three vehicle scheduling strategies. The main reason is that the different vehicle scheduling strategies do not affect the size of the information space for online learning.For the three vehicle scheduling strategies adopted by the IFML-I, we investigate the impact of the weight coefficientζ on the cumulative received data and the online learning costs, where the number of vehicles in the simulation area is set to 65 and ζ varies from 0.125 to 0.875 with the step size of 0.125. Figure 18 shows that the impact of ζ on cumulative received data, while Figure 19 shows that the impact of ζ on online learning costs.Combined with Figures18 and 19, we can see that when the IFML-I adopts the weight coefficient value of 0.125, the cumulative received data is relatively large and the corresponding online learning costs are relatively small, where the ratio of performance to cost is relatively high.For different communication environments, the weight coefficient value of achieving a relatively high ratio of performance to cost may be different. In the IFML-I, it is easy to adjust the weight coefficient value to adapt to different communication environments, while in the VFML, this flexibility is not available because of an incremental absolute average evaluation approach used in beam performance update.Based on the same simulation configuration as that in Figure4, we compare the impact of learning information space size on the performance metrics of the IFML-I, where “IFML 7×4×10”, “IFML 7×1×10”, “IFML 3×4×10”, and “IFML 3×1×10” represent the different learning information spaces used by the IFML-I, respectively. “IFML 7×4×10” means that there are 7 types of beam widths (i.e., from 30° to 90° with the step size of 10°) for each sector, the number of each type of beam widths is 4, and each sector is divided into 10 contextual subspaces. By analogy, the meanings of “IFML 7×1×10”, “IFML 3×4×10”, and “IFML 3×1×10” are also easy to understand, so the detailed description is omitted.The cumulative received data of the IFML-I with different learning information space sizes under the different number of vehicles are shown in Figure20, while the online learning costs required for the IFML-I with different learning information space sizes under the different number of vehicles are shown in Figure 21.Combined with Figures20 and 21, we can see that the IFML-I with a larger learning information space usually performs better than those with a smaller learning information space in terms of cumulative received data except for “IFML 7×1×10”. However, in terms of online learning costs, the IFML-I with a larger learning information space performs worse than those with a smaller learning information space.Although “IFML7×1×10” has less learning information space than “IFML 3×4×10”, it performs better than “IFML 3×4×10” in terms of both cumulative received data and online learning costs. This indicates that setting more types of beam widths is helpful to pick out more reasonable beam widths and thus improve performance. In other words, only increasing the number of beams with the same width has a limited improvement on performance. Also, we see that the cumulative received data of “IFML 7×1×10” is slightly worse than that of “IFML 7×4×10”, but the online learning costs of the former are much less than that of the latter, especially in the case of a large number of vehicles. This indicates that “IFML 7×1×10” has the relatively high ratio of performance to cost.Based on the same simulation configuration as that in Figure4, we compare the impact of learning information space size on the performance metrics of the VFML, where “VFML 6×24”, “VFML 12×24”, “VFML 24×24”, and “VFML 48×24” represent the different learning information spaces used by the VFML, respectively. “VFML 6×24” means that there are 6 types of beam widths (i.e., from 60° to 360° with the step size of 60°) and the entire base station coverage area is divided into 24 contextual subspaces. By analogy, the meanings of “VFML 12×24”, “VFML 24×24”, and “VFML 48×24” are also easy to understand, so the detailed description is omitted.The cumulative received data of the VFML with different learning information space sizes under the different number of vehicles are shown in Figure22, while the online learning costs required for the VFML with different learning information space sizes under the different number of vehicles are shown in Figure 23.Combined with Figures22 and 23, we can see that the VFML with a smaller larger learning information space performs worse than those with a larger learning information space in terms of cumulative received data, while the opposite is true in terms of online learning costs.Also, we see from Figures22 and 23 that the cumulative received data gains from expanding the learning information space are extremely limited for the VFML. This indicates that without changing the design structure or idea of the VFML algorithm, it is difficult to improve the cumulative received data by expanding the learning information space, which further illustrates the necessity of the work in this paper.


## 6.1. Simulation Scenario and Parameter Settings

The IFML is evaluated via simulation experiments. The IFML is divided into two categories (i.e., the IFML-I and the IFML-II), depending on whether the overlapping beams use the different frequency bands or the same one. In the IFML-I, the overlapping beams use the different frequency bands, while they use the same one in the IFML-II.In the following text, we first describe the simulation scenario and, then, introduce the mmWave channel propagation model that is used in our simulation experiments. Next, we describe the benchmark algorithms and performance metrics. Finally, we give the relevant simulation parameter settings according to the 3GPP technical specification in [50].Our simulation scenario is shown in Figure3, where the mmSBS’s coverage is divided into the four sectors (i.e., L=4) with the same size. For each sector, the two roads pass through it, where the location and size of the blockages on each road are fixed and account for about 10 percent of the road length. These blockages block the mmWave channels between the mmSBS and the vehicles on the roads.Figure 3
Illustration of our simulation scenario.There are seven types of beams according to the various beam width values (i.e., from 30° to 90° with the step size of 10°) for each sector, where the number of beams per type is one. Therefore, the number of the mmSBS’s beams in thelth sector is set to Ml=7 in the IFML-I, while Ml is 7∣P∣ in the IFML-II. The setP is a set of discrete values with respect to transmission power of a RF chain, while P is the number of the members in the setP. For the IFML-II, the discrete power values are set from 0.1 to 1 Watts with the step size of 0.1, and thus, P is equal to 10.In our implementation, a time period is defined by the mmSBS as a fixed length of time. During this time period, the IFML-I and the IFML-II receive registration information and, then, learn from the context and received data of the other vehicles passing through the selected beams. We select the identifier of the road and the direction of arrival as context. Therefore, the context vector is a two-dimensional vector (i.e.,D=2). The arrival direction of a vehicle is defined as the angle between the line connecting this vehicle with a mmSBS and the positive x-axis in the plane coordinate system with this mmSBS as the origin.The length of a time periodt, the parameter α, and the number of two-dimensional subspaces in each sector of the mmSBS are set to t=3s, α=0.33, and ℂT=10, respectively. Therefore, based on Theorem 1, the time horizon T is about 1000, while the value of control functionKt is about 2.02When then beams with the best performance are selected, the contexts in which these beams match are also determined at the same time. If the number of vehicles in such contexts is greater than the number of the selected beams, a strategy for vehicle selection needs to be determined in advance. For example, depending on the order in which vehicles enter a context space, a First-Come-First-Served (FCFS) policy can be used, or a First-Come-Last-Served (FCLS) policy can be adopted. In addition, a Random Selection (RASE) policy can also be employed.When the vehicles under the specific beam coverage in thelth sector are counted as the members of Vtl in the first time slot of the tth time period, every counted vehicle has entered this coverage area before the beginning of this time slot but does not leave before the end of this time slot. So the earlier they enter, the earlier they may leave.The above simulation scenario is implemented on the OMNeT platform (i.e., omnetpp-5.4.1) with the application package (i.e., veins-4.7.1) in terms of vehicle communication simulation framework, where the traffic pattern is generated by the third-party software: simulation of urban mobility (i.e., sumo-0.30.0). The speed of vehicles is set to the range from 5 to 10 m/s.The following mmWave channel propagation model is adopted in our simulation experiments:(8)pir=pitGitGirGic.In (8), pit is the transmission power at a directional beam from the mmSBS to the vehicle i, while pir is the received power at the vehicle i side when the mmSBS transmits at the power pit;Git, and Gir are the directional transmitting gain and directional receiving gain, respectively; Gic is the channel gain between the mmSBS and the vehicle i. When the beam between the mmSBS and the vehicle i is aligned, Git and Gir can be estimated by the following formula [51]:
(9)Git=2π−2π−θit∂θit,9aGir=2π−2π−θir∂θir,9b.In (9), θit is the beam width of the transmitter, while θir is the beam width of the receiver; ∂ is the gain of the side lobe and 0<∂≪1, while θit and θir are the main lobe in radian. The channel gain Gic can be estimated by the following formula [52]:
(10)Gic=χicδτ−τi2.In (10), δ· is the Dirac delta function; τi and χic are the propagation delay and the amplitude of the path from the mmSBS to the vehicle i, respectively. τi is estimated by the following formula:
(11)τi=dic.In (11), di is the distance of the path from the mmSBS to the vehicle i, and c is the speed of light. When there is a line-of-sight (LOS) path between the mmSBS and the vehicle i, the amplitude is estimated by the following formula [52]:
(12)χic,los=λ4πdi.In (12), λ is the wavelength, and λ=c/fc,fc is the carrier frequency. When there is not any LOS path between the mmSBS and the vehicle i, the amplitude is related to both path loss and reflection coefficients. Due to very high reflection loss in mmWave band [53], only one reflection of a given path is considered, which is estimated by the following formula [52]:
(13)χic,nlos=λΓ4πdi.In (13), Γ is the reflection coefficient of the mmWave reflection path. Therefore, based on (12) and (13), χic is estimated by
(14)χic=χic,los,Inunblockedarea14aχic,nlos,Inblockedarea14b.In the IFML-II, when the overlapping beams use the same frequency band, there is the same frequency interference between them. Let the beam of the vehiclei and the beam of the vehicle j be overlapping in the same mmSBS coverage. When the mmSBS transmits data to vehicle j, the interference power received at vehicle i should be
(15)pj→ir=pjtGj→itGj→irGic.In (15), Gj→it and Gj→ir are the directional transmitting gain and directional receiving gain of the mmWave link between the mmSBS and vehicle i when the mmSBS transmits data to vehicle j. The directional transmit-receive gain of each mmWave link can be derived by the following formula:
(16)Gj→itGj→ir=2π−2π−θjt∂θjt∙2π−2π−θir∂θirConditionA2π−2π−θjt∂θjt∙∂ConditionB∂∙2π−2π−θir∂θirConditionC∂∙∂ConditionD.Letϕj→it and ϕj→ir be the beam offset angle from the mmSBS’s (the mmSBS transmits to vehicle j) transmitting beam direction to the position of vehicle i and that from vehicle i’s (vehicle i receives from the mmSBS) receiving beam direction to the position of the mmSBS, respectively; Condition A is both ϕj→it≤θjt/2 and ϕj→ir≤θjr/2; Condition B is both ϕj→it≤θjt/2 and ϕj→ir>θjr/2; Condition C is both ϕj→it>θjt/2 and ϕj→ir≤θjr/2; Condition D is both ϕj`→it>θjt/2 and ϕj→ir>θjr/2.The work in [4] is most similar to ours. However, in the implementation in [4], the context vector is one-dimensional since the only direction of arrival is considered as the context. Also, a time period is defined as the time during which the observed vehicle enters and leaves the cell coverage area. Since this type of time period is influenced by a variety of factors (e.g., speed of vehicle, size of cell coverage area, and road orientation), it has great uncertainty and thus is difficult to determine its value in practical applications.Furthermore, the direction of arrival in [4] only takes some simple and abstract values (i.e., north, south, east, and west) due to the consideration for wide adaptation. Therefore, in order to make a fairer comparison, we design a variant of the algorithm (i.e., FML) in [4] and call it VFML, which basically retains the core idea of the FML algorithm except for the fixed time period (i.e., t=3s) and the adoption of vehicle arrival direction defined in our scheme.Based on the same bound forRT as that in the IFML, the time horizon T in the VFML should be about 1000. In addition, based on our simulation scenario in Figure 3, the simulation area for the VFML is divided into 24 one-dimensional subspaces according to the arrival direction of a vehicle. Therefore, the other parameters in the VFML should be set to α=0.39 andKt=2.35. Since the maximum number of supported RF chains in a mmSBS is set to six and the beams do not overlap in the VFML, the six orthogonal beams with variable beam width from 30° to 60° cover the 360° azimuth.In addition, we also consider an Optimal Scheme (OPSC) as a comparison of our scheme, which has a priori knowledge about the expected beam performanceμbt,jlct,il of each beam bt,jl∈Btl in each context ct,il∈ℂVtl. In each period, the OPSC selects the optimal subset Btl∗ℂVtl of n beams as in (1). Therefore, the results achieved by the OPSC is the expected performance upper bound of the system.The performance metrics used in the evaluation are the aggregate received data, the cumulative received data, and the online learning cost. We define the aggregate received data as the received data (inGbits) from all the vehicles during a time period t, while we define the cumulative received data as the received data from all the vehicles during the time horizon T. The online learning cost is defined as the number of exploration rounds required for the IFML-I, the IFML-II, and the VFML to reach a certain percentage of the OPSC’s performance, where all the exploration operations during a time period are classified as one round. The values of the remaining simulation parameters are listed in Table 1, unless otherwise stated in the following text.Table 1
The simulation parameters.
SymbolDescriptionValuefcCarrier frequency60 GHzWSystem bandwidth1 GHzpmaxtThe maximum transmission power of a mmSBS30 dBmN0Noise power density at a vehicle-174 dBm/HzBWrEach vehicle’s receiving beam width90°BWtA mmSBS’s transmitting beam width30°~90°nThe maximum number of supported RF chains in a mmSBS6ΓThe reflection coefficient of mmWave reflection path0.3tThe length of a time period3 sTThe number of time periods1000ζThe weighted coefficient0.125RRadius of simulation area of a mmSBS100 m


## 6.2. Simulation Results and Analysis

Firstly, we evaluate the performance metrics of the IFML-I and the IFML-II compared to the benchmark algorithms such as the VFML and the OPSC. The corresponding results are shown in Figures4–11. When there are multiple vehicles in the coverage area of a selected beam at the same time, the VFML selects one of the vehicles at random for communication. Therefore, in this set of simulations, the IFML-I, the IFML-II, and the OPSC also adopt the RASE strategy. Next, we evaluate the performance metrics of the IFML-I in terms of the FCFS strategy, the FCLS strategy, and the RASE strategy. The corresponding results are shown in Figures 12–19. Finally, we compare the impact of learning information space size on the performance metrics of the IFML-I and the VFML, respectively. The corresponding results are shown in Figures 20–23.Figure 4
Impact of vehicle density on cumulative received data.Figure 5
Impact of vehicle density on online learning costs.Figure 6
Impact of the number of selected beams per period on cumulative received data.Figure 7
Impact of the number of selected beams per period on online learning costs.Figure 8
Impact of thermal noise power density on cumulative received data.Figure 9
Impact of thermal noise power density on online learning costs.Figure 10
Aggregate received data over the time horizon.Figure 11
The number of exploration operations per period over the time horizon.Figure 12
Impact of vehicle density on cumulative received data under the three vehicle scheduling strategies.Figure 13
Impact of vehicle density on online learning costs under the three vehicle scheduling strategies.Figure 14
Impact of the number of selected beams per period on cumulative received data under the three vehicle scheduling strategies.Figure 15
Impact of the number of selected beams per period on online learning costs under the three vehicle scheduling strategies.Figure 16
Impact of thermal noise power density on cumulative received data under the three vehicle scheduling strategies.Figure 17
Impact of thermal noise power density on online learning costs under the three vehicle scheduling strategies.Figure 18
Impact of weight coefficientζ on cumulative received data under the three vehicle scheduling strategies.Figure 19
Impact of weight coefficientζ on online learning costs under the three vehicle scheduling strategies.Figure 20
Impact of learning information space size on the IFML-I’s cumulative received data.Figure 21
Impact of learning information space size on the IFML-I’s online learning costs.Figure 22
Impact of learning information space size on the VFML’s cumulative received data.Figure 23
Impact of learning information space size on the VFML’s online learning costs.In Figure4, we investigate the impact of the number of vehicles in the simulation area on the cumulative received data achieved by the different algorithms, where at most 6 selected beams are used simultaneously in each time period and the number of vehicles varies from 35 to 95 with the step size of 15. We can see from Figure 4 that as the number of vehicles in the simulation area increases, the cumulative received data also increase. This is because the smaller number of vehicles means the more insufficient context information and thus results in a poorer learning effect. Therefore, the probability of accurately selecting the beams that can make vehicles maximize the cumulative received data is relatively small. With the increase of the number of vehicles, more context information is provided, which is conducive to improving the learning effect. Therefore, the probability of accurately selecting the beams that can make vehicles maximize the cumulative received data is relatively large.From Figure4, we also see that the cumulative received data hardly increase when the number of vehicles in the simulation area increases to a certain amount. This is because the number of selected beams per time period is fixed. At the same time, when the context information is large enough, the specified number of served vehicles can be selected reasonably through online learning. Figure 4 shows that the IFML-I is closer to the optimal one (i.e., the OPSC) than the IFML-II and the VFML, where the IFML-II outperforms the VFML. The main reason lies in two aspects. On the one hand, the IFML-I and the IFML-II can find the more reasonable beam width and beam direction for each served vehicle when compared with the VFML. On the other hand, the two IFMLs’ two-dimensional context can bring more accurate information than the VFML’s one-dimensional context, which can make a more reasonable decision for each vehicle. Compared with the IFML-II, the IFML-I pays more frequency band resources for better performance.Based on the same simulation configuration as that in Figure4, the online learning costs required for the IFML-I, the IFML-II, and the VFML under the different number of vehicles are shown in Figure 5. We can observe from Figure 5 that the changes of the online learning costs result from the number of vehicles in the system. As the number of vehicles increases, the costs increase as well. This is because the more vehicles that enter the system in a time period will give each beam the opportunity to probe the more context subspaces. Therefore, the performances of each beam on the more subspaces can be detected in a time period. If there is no performance history or the recorded history is insufficient, the corresponding beams are scheduled for exploration as soon as possible in a time period, which is helpful to speed up the detection process of the performance of each beam in each context subspace, and thus reduce the number of exploration rounds.Moreover, we can see from Figure5 that the two IFMLs’ online learning costs are larger than those of the VFML. This is because the two IFMLs have more beams and context subspaces than those of the VFML. By setting the more number of beams and using the more context information, each selected vehicle can be assigned the beam with a more reasonable width and direction. However, there are larger combinations of beams and contexts that need to be explored. As the number of vehicles increases, the difference in the number of exploration rounds between the two schemes is narrowed. This indicates that the operation mode (that the two IFMLs first perform a partitioned search for a mmSBS coverage area and then select the exploration beams from the partitioned search results) has higher exploration efficiency with the increase of the number of vehicles. In the IFML-II, in order to control cochannel interference, the beam transmission power must be adjusted, so it needs to be discretized into various values, which makes the beam exploration space be larger and the online learning cost be higher.In Figure6, we analyze the impact of the number of selected beams per time period on the cumulative received data, where the number of vehicles in the simulation area is set to 65 and the number of selected beams varies from 2 to 6 with the step size of 1. It can be seen from Figure 6 that as the number of simultaneously selected beams increases, the cumulative received data in the system increase as well. The reason for this increase is that the increasing number of beams used in a concurrent manner leads to the enhanced coverage area and thus the more vehicles can be served at the same time. However, as mentioned earlier, the larger the number of concurrent beams is, the higher the hardware complexity and energy consumption at a mmSBS are. It can also be seen that the two IFMLs achieve a larger cumulative received data than the VFML, where the explanation for the difference in terms of cumulative received data between the different schemes is similar to the explanation of the results in Figure 4.Based on the same simulation configuration as that in Figure6, the online learning costs required for the two IFMLs and the VFML under the different number of selected beams per period are shown in Figure 7. From Figure 7, as the number of selected concurrent beams increases, the number of exploration rounds decreases. The larger number of selected concurrent beams per time period means that the larger number of beams with unknown or uncertain performance can be explored in each time period. When the combined space size of beams and context subspaces remains unchanged, the number of required exploration rounds will decrease. In Figure 7, we see that the online learning costs of the two IFMLs are larger than that of the VFML, and the explanation of this difference is similar to the explanation of the results in Figure 5.In Figure8, we analyze the impact of thermal noise power density on the cumulative received data, where the number of vehicles in the simulation area is set to 65, the number of selected beams is set to 6, and the thermal noise power density varies from -170 dBm/Hz to -150 dBm/Hz with the step size of 5 dBm/Hz. Obviously, as the thermal noise power density in the system increases, the cumulative received data decrease. This is because the increase of thermal noise power density will reduce the signal-to-noise ratio. When the transmission power of a mmSBS is constant, the data rate will decrease according to the Shannon theorem. Also, we can see from Figure 8 that the cumulative received data of the two IFMLs are higher than that of the VFML, where the explanation for this difference is similar to the explanation of the results in Figure 4.Based on the same simulation configuration as that in Figure8, the online learning costs required for the two IFMLs and the VFML under the different thermal noise power density are shown in Figure 9. It is observed from Figure 9 that the impact of the thermal noise power density on the online learning costs of the algorithms is not obvious in general. The reason is that the variation of thermal noise power density does not change the size of the information space of online learning.However, at higher thermal noise density, the number of the VFML’s beam exploration rounds decreases slightly. As shown in Figure8, the thermal noise power density has a great impact on the cumulative received data. Therefore, the possible reason is that, due to the combination of the high thermal noise environment and the small number of optional beams in the VFML, the performance of each beam under each context subspace is not easily affected by the changes of other factors in the environment, and thus, a beam performance is considered stable and does not require additional exploration.In Figure10, we analyze the aggregate received data achieved by the algorithms over the time horizon with 1000 time periods, where the number of vehicles in the simulation area is set to 65. Figure 10 illustrates that the IFML-I can achieve performance closer to the OPSC than the IFML-II and the VFML. The fluctuations on the graph result from the number of vehicles in the system, the speed of each vehicle, and the contact time. Clearly, the OPSC gives an upper bound to the other algorithms due to a priori knowledge of the expected beam performance. So, the OPSC can select the optimal subset of beams according to the relational expression (1).The two IFMLs obviously outperform the VFML in terms of aggregate received data after 300 time periods, and the IFML-I is close to the OPSC’s aggregate received data after 600 time periods. The main reason is that after sufficient online learning, the two IFMLs can select a set of beams with more reasonable widths and directions than the VFML, which can provide greater transmission capacity for the vehicles in their coverage. However, before 300 time periods, due to insufficient online learning, most of the beams assigned to the vehicles are selected at random, and thus, there is no significant difference between the two IFMLs’ aggregate received data and the VFML’s.Based on the same simulation configuration as that in Figure10, the number of exploration operations per time period required for the algorithms over the time horizon with 1000 time periods are shown in Figure 11. We can see from Figure 11 that the two IFMLs needs more time periods for beam performance exploration than the VFML. This indicates that the two IFMLs takes longer time to sufficient online learning effect than the VFML. The main reason is that there is a difference of the learning information space resulting from the combination of beams and context subspaces between the two IFMLs and the VFML.In the two IFMLs, the larger learning information space leads to higher aggregate received data but increases the online learning costs. Fortunately, after a certain amount of time periods, there is no difference in the online learning costs between the three schemes, where they still have a similar beam performance update process after each beam is used.Based on the same simulation configuration as that in Figure4, under the three vehicle scheduling strategies adopted by the IFML-I, Figure 12 shows that the impact of the number of vehicles in the system on the cumulative received data, while Figure 13 shows that the impact of the number of vehicles in the system on online learning costs.As explained in Figure4, the cumulative received data under all the vehicle scheduling strategies in Figure 12 varies with the number of vehicles in the simulation area. However, there is a difference in terms of cumulative received data between the three vehicle scheduling strategies. The main for this difference is that the different strategies result in the different contact time, during which a vehicle stays in a beam coverage area. Among the three strategies, the FCLS leads to the longest average contact time, while the FCFS leads to the shortest one. So the cumulative data received under the FCLS is the largest, while that under the FCFS is the least.The variation trend of the online learning costs with the number of vehicles in the simulation area under all the vehicle scheduling strategies in Figure13 is similar to that in Figure 5. Therefore, the explanation in Figure 5 applies here as well. There is no difference in the online learning costs among the three strategies under the different number of vehicles. The main reason is that the different vehicle scheduling strategies do not affect the size of the learning information space.Based on the same simulation configuration as that in Figure6, under the three vehicle scheduling strategies adopted by the IFML-I, Figure 14 shows that the impact of the number of selected beams per period on cumulative received data, while Figure 15 shows that the impact of the number of selected beams per period on online learning costs.The variation trend of cumulative received data with the number of selected concurrent beams under all the vehicle scheduling strategies in Figure14 is similar to that in Figure 6. Therefore, the explanation in Figure 6 applies here as well. Also, in the case of different number of beams that can be used simultaneously, the main reason for the difference in cumulative received data under the three strategies is similar to that in Figure 12.As explained in Figure7, the online learning costs under all the vehicle scheduling strategies in Figure 15 vary with the number of beams that could be used simultaneously. However, there is no difference in the online learning costs under the three vehicle scheduling strategies. The main reason is that the different vehicle scheduling strategies do not affect the size of the information space for online learning.Based on the same simulation configuration as that in Figure8, under the three vehicle scheduling strategies adopted by the IFML-I, Figure 16 shows that the impact of thermal noise power density on cumulative received data, while Figure 17 shows that the impact of thermal noise power density on online learning costs.The variation trend of cumulative received data with the thermal noise power density under all the vehicle scheduling strategies in Figure16 is similar to that in Figure 8. Therefore, the explanation in Figure 8 applies here as well. Also, in the case of different thermal noise power density, the main reason for the difference in cumulative received data under the three vehicle scheduling strategies is similar to that in Figure 12.The variation trend of the online learning costs with the thermal noise power density under all the vehicle scheduling strategies in Figure17 is similar to that in Figure 9. However, there is no difference in the online learning costs under the three vehicle scheduling strategies. The main reason is that the different vehicle scheduling strategies do not affect the size of the information space for online learning.For the three vehicle scheduling strategies adopted by the IFML-I, we investigate the impact of the weight coefficientζ on the cumulative received data and the online learning costs, where the number of vehicles in the simulation area is set to 65 and ζ varies from 0.125 to 0.875 with the step size of 0.125. Figure 18 shows that the impact of ζ on cumulative received data, while Figure 19 shows that the impact of ζ on online learning costs.Combined with Figures18 and 19, we can see that when the IFML-I adopts the weight coefficient value of 0.125, the cumulative received data is relatively large and the corresponding online learning costs are relatively small, where the ratio of performance to cost is relatively high.For different communication environments, the weight coefficient value of achieving a relatively high ratio of performance to cost may be different. In the IFML-I, it is easy to adjust the weight coefficient value to adapt to different communication environments, while in the VFML, this flexibility is not available because of an incremental absolute average evaluation approach used in beam performance update.Based on the same simulation configuration as that in Figure4, we compare the impact of learning information space size on the performance metrics of the IFML-I, where “IFML 7×4×10”, “IFML 7×1×10”, “IFML 3×4×10”, and “IFML 3×1×10” represent the different learning information spaces used by the IFML-I, respectively. “IFML 7×4×10” means that there are 7 types of beam widths (i.e., from 30° to 90° with the step size of 10°) for each sector, the number of each type of beam widths is 4, and each sector is divided into 10 contextual subspaces. By analogy, the meanings of “IFML 7×1×10”, “IFML 3×4×10”, and “IFML 3×1×10” are also easy to understand, so the detailed description is omitted.The cumulative received data of the IFML-I with different learning information space sizes under the different number of vehicles are shown in Figure20, while the online learning costs required for the IFML-I with different learning information space sizes under the different number of vehicles are shown in Figure 21.Combined with Figures20 and 21, we can see that the IFML-I with a larger learning information space usually performs better than those with a smaller learning information space in terms of cumulative received data except for “IFML 7×1×10”. However, in terms of online learning costs, the IFML-I with a larger learning information space performs worse than those with a smaller learning information space.Although “IFML7×1×10” has less learning information space than “IFML 3×4×10”, it performs better than “IFML 3×4×10” in terms of both cumulative received data and online learning costs. This indicates that setting more types of beam widths is helpful to pick out more reasonable beam widths and thus improve performance. In other words, only increasing the number of beams with the same width has a limited improvement on performance. Also, we see that the cumulative received data of “IFML 7×1×10” is slightly worse than that of “IFML 7×4×10”, but the online learning costs of the former are much less than that of the latter, especially in the case of a large number of vehicles. This indicates that “IFML 7×1×10” has the relatively high ratio of performance to cost.Based on the same simulation configuration as that in Figure4, we compare the impact of learning information space size on the performance metrics of the VFML, where “VFML 6×24”, “VFML 12×24”, “VFML 24×24”, and “VFML 48×24” represent the different learning information spaces used by the VFML, respectively. “VFML 6×24” means that there are 6 types of beam widths (i.e., from 60° to 360° with the step size of 60°) and the entire base station coverage area is divided into 24 contextual subspaces. By analogy, the meanings of “VFML 12×24”, “VFML 24×24”, and “VFML 48×24” are also easy to understand, so the detailed description is omitted.The cumulative received data of the VFML with different learning information space sizes under the different number of vehicles are shown in Figure22, while the online learning costs required for the VFML with different learning information space sizes under the different number of vehicles are shown in Figure 23.Combined with Figures22 and 23, we can see that the VFML with a smaller larger learning information space performs worse than those with a larger learning information space in terms of cumulative received data, while the opposite is true in terms of online learning costs.Also, we see from Figures22 and 23 that the cumulative received data gains from expanding the learning information space are extremely limited for the VFML. This indicates that without changing the design structure or idea of the VFML algorithm, it is difficult to improve the cumulative received data by expanding the learning information space, which further illustrates the necessity of the work in this paper.


## 7. Conclusion

In this paper, we proposed the two improved fast machine learning algorithms (i.e., the IFML-I and the IFML-II) to improve network capacity in mmWave vehicular networks based on contextual multi-armed bandits. To cope with unknown performance values of mmWave beams assigned to the dynamically arriving vehicles, the two IFMLs periodically observe the amount of data received by the connected vehicles in each time period and update performance values of the mmWave beams under the context subspaces corresponding to these vehicles by adopting the proposed beam performance update approach based on adjustable weight coefficient. In this way, the two IFMLs learn context-specific mmWave beam performances online. Moreover, the two IFMLs exploit the richer contexts and more beam types so that the set of beams with the more reasonable beam width and beam direction is picked out. Also, the additional beam search overhead can be suppressed by the partitioned search method. Compared with the benchmark algorithm, the simulation results showed that the two IFMLs substantially improve the amount of data received by the vehicles in the system during each time period at a slightly increased online learning overhead. However, there is no difference in terms of beam performance update cost after a certain number of time periods between the two IFMLs and the benchmark algorithm.



---

*Source: 8876186-2021-03-20.xml*