## 4 Proposed Antenna Codebook and MAB Approach
### Proposed Nested Two-Stage MAB Approach
#### 4.2.1 Proposed Nested Two-Stage TS Algorithm
Then, its corresponding reward \(\psi_{\mathbf{\Phi}_{i};f_{i}^{\prime}}\) is achieved, and its model parameters are updated for the posterior distribution of the next round of the MAB game as given in Algorithm 1. #### 4.2.2 Proposed Nested Two-Stage UCB Algorithm

UCB is based on increasing the confidence of the chosen arm by decreasing its uncertainty. This is done by exploiting the arm with the maximum achievable average reward so far while exploring the less selected ones. Algorithm 2 summarizes the proposed nested two-stage UCB algorithm. Like the TS algorithm, the inputs to the algorithm are the spaces of codebooks \(\mathcal{R}\) and \(\mathcal{F}\), and the outputs are the adjusted values of \(\mathbf{\Phi}_{i}\)- and \(f_{j^{*}}\). For initialization, at \(t=1\), the average spectral efficiencies \(\overline{\psi}_{\mathbf{\Phi}_{i};f_{i}^{\prime}}\) corresponding to all values of \(\mathbf{\Phi}_{i}\). and \(f_{j}\) are set to uniform random values in the range of \([0,1]\), where \(1\leq i\leq|\mathcal{R}|\), \(1\leq j\leq|\mathcal{F}|\), and their corresponding numbers of selections, i.e., \(Z_{\mathbf{\Phi}_{i};f_{i}^{\prime}}\), are set to 1. Moreover, a PS vector \(f\), i.e., \(f_{j_{t}^{*}}\), is picked uniformly from its corresponding PS codebook, i.e., \(\mathcal{F}\). For \(2\leq t\leq T_{H}\), nested two-stage UCB algorithms are conducted as follows:

In the first UCB stage, based on the value of \(f_{j_{t-1}^{*}}\) and index of the \(\mathbf{\Phi}\) matrix, \(i_{t}^{*}\) is selected based on the UCB policy as follows [17]:

\[i_{t}^{*}=\text{arg max}_{i}\Bigg{(}\overline{\psi}_{\mathbf{\[PERSON]}_{i-1};f_{i-1}^ {\prime}}+\sqrt{\frac{2 ln(t)}{Z_{\mathbf{\[PERSON]}_{i-1};f_{i-1}^{\prime}}}}\Bigg{)} \tag{12}\]

where \(\overline{\psi}_{\mathbf{\[PERSON]}_{i-1};f_{i-1}^{\prime}}\) represents the exploitation term, while the term \(\sqrt{\frac{2 ln(t)}{Z_{\mathbf{\[PERSON]}_{i-1};f_{i-1}^{\prime}}}}\) represents the exploration term of the UCB policy. After selecting \(\mathbf{\[PERSON]}_{i}^{*}\), its corresponding reward \(\psi_{\mathbf{\[PERSON]}_{i};f_{i-1}^{\prime}}\) is obtained and its related parameters \(Z_{\mathbf{\[PERSON]}_{i};f_{i-1}^{\prime}}\) and \(\overline{\psi}_{\mathbf{\[PERSON]}_{i};f_{i-1}^{\prime}}\) are updated as given in Algorithm 2. Based on the selected value of \(\mathbf{\[PERSON]}_{i}^{*}\), the value of \(f_{j_{t}^{*}}\) is adjusted in a nested manner via the second stage UCB as given in Algorithm 2. Then, its corresponding reward \(\psi_{\mathbf{\[PERSON]}_{i};f_{i}^{\prime}}\) is obtained and its related parameters \(Z_{\mathbf{\[PERSON]}_{i};f_{i}^{\prime}}\) and \(\overline{\psi}_{\mathbf{\Phi}_{i};f_{i}^{\prime}}\) are updated as given in Algorithm 2. Figure 2 shows the spectral efficiency performances of the compared schemes i.e., nested two-stage TS, UCB, and random at no blockage against the used number of PS vectors (\(K\)), where \(K=M=N\). Generally speaking, as the number of \(K\) increases, the spectral efficiencies of all schemes increase due to the increase in the received power affected by the increment in the beamforming gain. Although the proposed nested two-stage MAB algorithms do not need CSI estimation and only use the observed spectral efficiency, they have good performances against the optimal performance compared to random PS selection. Moreover, the proposed nested two-stage TS algorithm outperforms all other compared schemes due to its Bayesian policy, which constructs prior/posterior distributions to the achievable spectral efficiency. As the assumed normal distribution highly matches the actual distribution of the attainable spectral efficiency, the proposed nested two-stage TS outperforms UCB-based one. Random PS selection shows the worst performance because it selects the PS vectors arbitrarily without any optimization objective. At \(K=4\), about 98.5%, 97%, and 85.3% of the optimal performance are obtained by the proposed nested two-stage TS, UCB, and random selection, respectively.