## 4 Proposed Antenna Codebook and MAB Approach
### Proposed Nested Two-Stage MAB Approach
#### 4.2.1 Proposed Nested Two-Stage TS Algorithm
This means that \(\mathbf{\psi}_{\mathbf{\Phi}_{i}\mathbf{f}_{j_{t}}}\ \sim\ \mathcal{N}\left(\overline{\mathbf{\psi}}_{\mathbf{\[PERSON]}_{i}\mathbf{f}_{j_{t}}} \sigma^{2}_{\mathbf{\[PERSON]}_{i}\mathbf{f}_{j_{t}}}\right)\), where \(\overline{\mathbf{\psi}}_{\mathbf{\[PERSON]}_{i}\mathbf{f}_{j_{t}}}\) and \(\sigma^{2}_{\mathbf{\[PERSON]}_{i}\mathbf{f}_{j_{t}}}\) are the mean and variance of the assumed normal distribution, and \(\overline{\mathbf{\psi}}_{\mathbf{\[PERSON]}_{i}\mathbf{f}_{j_{t}}}\) is the average value of \(\mathbf{\psi}_{\mathbf{\[PERSON]}_{i}\mathbf{f}_{j_{t}}}\). This assumption comes from the AWGN term given in (1). Algorithm 1 gives the proposed nested two-stage TS algorithm, where the inputs to the algorithm are the spaces of codebooks \(\mathcal{R}\). and \(\mathcal{F}\) and the outputs are the adjusted values of \(\mathbf{\[PERSON]}_{i^{*}}\) and \(\mathbf{f}_{j^{*}}\). At the beginning of the algorithm, i.e., \(t=1\) the average spectral efficiencies \(\overline{\mathbf{\psi}}_{\mathbf{\[PERSON]}_{i}\mathbf{f}_{j_{t}}}\), the variances \(\sigma^{2}_{\mathbf{\[PERSON]}_{i}\mathbf{f}_{j_{t}}}\), and the number of selections \(Z_{\mathbf{\Phi}_{i}\mathbf{f}_{j_{t}}}\) corresponding to all values of \(\mathbf{\Phi}_{i}\) and \(\mathbf{f}_{j^{*}}\) are set to 0, 1, and 0, respectively. In addition, a PS vector \(\mathbf{f}\), i.e., \(\mathbf{f}_{j_{t}^{*}}\), is initialized by picking it uniformly from its corresponding PS codebook. For \(2\leq t\leq T_{H}\), where \(T_{H}\) is the total time horizon, nested two-stage TS algorithms are performed as follows:```
Output:\(\mathbf{\Phi}_{i^{*}}\) and \(\mathbf{f}_{j^{*}}\) Input:\(\mathcal{R},\mathcal{F}\) Initialization: At \(t=1\), \(1\). \(\overline{\mathbf{\Phi}}_{\mathbf{\phi}_{i}f_{i}}=0\), \(\sigma^{2}_{\mathbf{\Phi}_{i}f_{i}}=1\), \(Z_{\mathbf{\Phi}_{i}f_{i}}=0\), \(1\leq i\leq|\mathcal{R}|\), \(1\leq j\leq|\mathcal{F}|\) 2. Select a value of \(\mathbf{f}_{j^{*}_{i}}\) at random from its finite set \(\mathcal{F}\) For\(t=2\),..., \(T_{H}\)
```

**Algorithm 1** Nested Two-Stage TS Algorithm

```
Output:\(\mathbf{\Phi}_{i^{*}}\) and \(\mathbf{f}_{j^{*}_{i}}\) Input:\(\mathcal{R},\mathcal{F}\) Initialization: At \(t=1\), \(1\). \(\overline{\mathbf{\Phi}}_{\mathbf{\phi}_{i}f_{i}}=0\), \(\sigma^{2}_{\mathbf{\Phi}_{i}f_{i}}=1\), \(Z_{\mathbf{\Phi}_{i}f_{i}}=0\), \(1\leq i\leq|\mathcal{R}|\), \(1\leq j\leq|\mathcal{F}|\) 2. Select a value of \(\mathbf{f}_{j^{*}_{i}}\) at random from its finite set \(\mathcal{F}\) For\(t=2\),..., \(T_{H}\)
```

**Algorithm 2** Nested Two-Stage TS

In the first stage and based on the value of \(\mathbf{f}_{j^{*}_{i-1}}\), a value of \(\mathbf{\Phi}_{i^{*}_{t}}\) is selected and its corresponding reward \(\psi_{\mathbf{\Phi}_{i^{*}_{t}}f_{i-1}^{*}}\) is obtained. This is done by sampling the prior distributions of \(\psi_{\mathbf{\[PERSON]}_{i^{*}_{t}}f_{i-1}^{*}}\), i.e., \(\tau_{\mathbf{\[PERSON]}_{i_{t}}-f_{i}^{*}_{t-1}}\sim\mathcal{N}\left(\overline{\psi}_ {\mathbf{\[PERSON]}_{i_{t}}-f_{i-1}^{*}}\sigma^{2}_{\mathbf{\[PERSON]}_{i_{t}}-f_{i-1}^{*}}\right)\), \(1\leq i\leq|\mathcal{R}|\). Then, the index \(i^{*}_{t}\) corresponding to the maximum \(\tau_{\mathbf{\[PERSON]}_{i_{t}-1}f_{i-1}^{*}}\) is selected as follows:

\[i^{*}_{t}=\arg\max_{i}\left(\tau_{\mathbf{\[PERSON]}_{i_{t}-1}f_{i-1}^{*}}\right), \tag{10}\]

Next, the value of \(\mathbf{\[PERSON]}\) matrix corresponding to this index, i.e., \(\mathbf{\[PERSON]}_{i^{*}_{t}}\), is obtained. Afterward, its corresponding reward \(\psi_{\mathbf{\[PERSON]}_{i^{*}_{t}}f_{i}}\) is achieved, and its model parameters \(Z_{\mathbf{\[PERSON]}_{i}f_{i-1}^{*}}\),\(\overline{\psi}_{\mathbf{\[PERSON]}_{i};f_{i-1}^{\prime}}\), and \(\sigma^{2}_{\overline{\psi}_{i};f_{i-1}^{\prime}}\) are updated for its posterior distribution as given in Algorithm 1, where the methodology presented in [33] is used for updating \(\overline{\psi}_{\mathbf{\[PERSON]}_{i};f_{i-1}^{\prime}}\) and \(\sigma^{2}_{\mathbf{\[PERSON]}_{i};f_{i-1}^{\prime}}\).

In the second stage TS and based on \(\mathbf{\[PERSON]}_{i}\) coming from the first stage, a value of \(f_{i}^{\prime}\) is adjusted, and its corresponding reward \(\psi_{\mathbf{\[PERSON]}_{i};f_{i}^{\prime}}\) is obtained. In this procedure, random samples \(\tau_{\mathbf{\[PERSON]}_{i};f_{i-1}}\sim\mathcal{N}\bigg{(}\overline{\psi}_{\mathbf{\[PERSON]} _{i};f_{i-1}},\sigma^{2}_{\mathbf{\[PERSON]}_{i};f_{i-1}}\bigg{)}\), \(1\leq j\leq|\mathcal{F}|\), are taken, and the index \(j_{t}^{*}\) corresponding to the maximum sample value is chosen as follows:

\[j_{t}^{*}=\text{arg max}_{j}\Big{(}\tau_{\mathbf{\[PERSON]}_{i};f_{i-1}} \Big{)}, \tag{11}\]

Again, the value of \(f\) matrix corresponding to this index, i.e., \(f_{j_{t}^{*}}\), is obtained.