Two-Stage Multiarmed Bandit for Reconfigurable Intelligent Surface Aided Millimeter Wave Communications  
[PERSON]  
1 Electrical Engineering Department, College of Engineering at Wadi Addwasir, Prince Sattam Bin Abdulaziz University, Wadi Addwasir 11991, Saudi Arabia; [EMAIL_ADDRESS]  
complex estimation of the massive mmWave channel state information (CSI) at both RIS and mmWave users. Moreover, as RIS is entirely passive, it does not support any channel estimation operations. In this paper, for practical realization of the mmWave-RIS communication system and to avoid the estimation of the massive mmWave CSI, antenna codebooks are suggested for both mmWave TX and RIS. Without loss of generality, we follow the mmWave codebook design given in [12; 13], where the codebooks are generated with 90-degree phase resolution without amplitude adjustment. We choose this codebook design due to its simplicity and standardization by mmWave standards, i.e., Wireless Gigabit (WiGig) standard [12; 13]. Furthermore, to maximize the spectral efficiency at the intended mmWave user, PS vectors of both mmWave TX and RIS should be jointly optimized. However, this will consume a considerable beamforming training (BT) overhead, especially when using many PS vectors of sizeable antenna arrays. To efficiently address this problem, an online learning approach is proposed using a single-player multiarmed bandit (MAB) game. MAB is an online learning strategy where an agent tries to maximize its profit via playing over the bandit's available arms. The agent attempts to compromise between consistently exploiting the arm, giving the maximum profit so far, or exploring new ones, known as _exploitation-exploration_ trade-off. Based on the reward's distribution, MAB games can be categorized as _stochastic_ or _adversarial_ MAB, where in the former the rewards come from a pre-known distribution while in the latter the rewards come arbitrarily. A complete survey of the MAB approach, including its categories and algorithms, can be found in [14; 15]. In the formulated MAB game, the mmWave TX will be the player, the available space of the candidate PS vectors will be the arms of the bandit game, and the achievable spectral efficiency at the mmWave user will be the reward. Thus, one set of joint PS vectors can be tested at a time, which highly relaxes the BT overhead. Furthermore, a nested two-stage MAB methodology is proposed to reduce the complexity and increase the convergence rate of the proposed MAB game. Thus, the main contributions of this paper can be summarized as follows:  
* The RIS-assisted mmWave communication system is considered, where an optimization problem is formulated to jointly adjust the PS vectors of both mmWave TX and RIS. * Discrete PSs in the form of codebook design are suggested to relax the complicated CSI estimation problem at both mmWave TX and RIS. In this design, the PSs are assumed to be generated with 90-degree phase resolution with constant amplitude like the codebook design used by WiGig standards [12; 13]. * A stochastic single-player MAB game is constructed to jointly optimize the PS vectors of mmWave TX and RIS. This facilitates the adjustment of both PS vectors successively in a time-by-time fashion, which highly reduces the required BT overhead. Typically, the only available information for a MAB player is its reward observation, without any details about the environment. Thus, considering mmWave PSs optimization as a MAB game eliminates the need for CSI estimation as the observed achievable spectral efficiency, i.e., the reward of the game, is the only needed information. This information can be easily obtained via the feedback channel between the mmWave RX and TX. Moreover, the suggested codebook design facilitates the implementation of the MAB game, where the PS vectors are considered as its arms. To reduce the complexity of the arm optimization as we have two sets of arms, one belonging to the mmWave TX and the other to the RIS, a nested two-stage MAB game is proposed in this paper.