## 3 Proposed Beam Learning with MARL
### Numerical Simulation with MARL  
In this proposed work, a 120\({}^{\circ}\) sector of a cell for simulation purposes is modeled, restricting transmissions within this azimuth range. Although the 4 RF channels can concurrently serve 4 users within this angular space, real-world scenarios typically involve more than four active users. To address this, users with similar channels are served with a single beam. The assignment of each user to a specific beam, whether before or after the MARL-based codebook learning process, is determined through beam sweeping. Consequently, the number of beams in the learned codebook remains consistent with the initial access codebook, which is adjustable for the performance assessment. Figure 6 illustrates the radiation pattern for one such codebook with nine beams, showcasing variations for different quantization bits.  
In the proposed MARL algorithm, the number of agents corresponds to the number of beams utilized in the sector. This configuration effectively breaks down the task of selecting a beam from a large set into finding a single beam within a smaller subset, thereby enhancing the efficiency of the codebook learning process. An additional and significant benefit of employing one agent per beam is the ability to identify optimal non-interfering beams within the sector, even in nLOS scenarios. Each agent in the MARL algorithm strives to maximize the individual beamforming gain while minimizing interference with other agents, as reflected in the reward processing outlined in Algorithm 1.  
Upon completion of the learning phase, the acquired codebook becomes readily deployable within the initial access procedure. Users can now be efficiently served using the learned codebook, rendering the traditional matched filter-based beam codebook obsolete. This transition marks a significant advancement in the efficiency and adaptability of beamforming techniques, as the learned codebook optimally caters to the dynamic needs and complexities of the communication environment without relying on pre-defined beam patterns.  
This learned codebook is valid until there is no significant change in terms of macro structures within the sector. Although such time will only be there occasionally, in the case of such large changes in the structure or the replacement of the BS, learning has be initiated again for all the beams.  
Next, the analog beamforming codebook selection for the UE is carried out. In this work, a conventional beamforming codebook tailored for the UE is employed. The process of selecting beams from the codebook for the UE involves a standard beam search procedure, encompassing steps such as sounding, measurement, and feedback.  
In the final step, the baseband beamforming vector (\(F_{BB}\)) at the BS is calculated. This computation follows the procedure outlined in [4]. In this process, the BS formulates its zero-forcing digital precoder \(F_{BB}\) based on the quantized channel feedback received from the UE. Due to the utilization of RF beamforming and the presence of sparse mmWave channels, it is anticipated that the effective MIMO channel will be well-conditioned [16; 17]. This favorable channel condition enables the utilization of a straightforward multi-user digital beamforming strategy like zero-forcing, which can achieve a performance close to the optimal level [18]. The algorithm for obtaining the baseband beamforming vector \(F_{BB}\) is detailed in the second stage of the procedure presented in [4].