## 4 Results and Discussion  
This section highlights the performance of the proposed MARL technique. This is demonstrated through a series of experiments where the agents are trained using the  
Figure 6: Radiation pattern for 32-element ULA antenna with number of phase shifter quantization bits (**a**) 0 bits, (**b**) 2 bits, (**c**) 4 bits (**d**) 6 bits. Colored lines represent distinct radiation patterns corresponding to specific steering angles in the azimuth plane. parameters outlined in Table 2. The effectiveness of the proposed MARL-based approach to learning beam codebooks across various scenarios is assessed. Unlike the process of creating a codebook for single-user MIMO systems, as discussed in [10], acquiring a beam codebook with multiple beams for multiuser MIMO involves not only learning the codebook but also identifying optimal combinations from a wide range of potential beamforming vectors. Figure 7 illustrates the average beamforming gain relative to the number of beams contained within the codebook, specifically in the LOS scenario. In this scenario, the BS employs a uniform linear array with isotropic elements, oriented in a back-baffled configuration. The graph demonstrates a consistent upward trend in average beamforming gain with an increasing number of beams. In line with the observations made in [10], the result in Figure 7 demonstrates nearly equivalent performance to a classical 32-beam beamsteering codebook when employing only 6 beams. Notably, the proposed approach not only matches but also surpasses the performance of [10]. This trend persists as the solution, employing 8 beams, consistently outperforms the 32-beam classical beamsteering codebook while also exceeding the capabilities of [10]. This achievement is particularly significant because it addresses a multiuser scenario with four co-channel users and a nonzero interference probability, representing an improvement over the single-user MIMO configuration in [10]. It is important to note that traditionally, single-user and multiuser codebooks are identical, meaning multiple beams from the same codebook are used for multiuser MIMO. For this comparison, the single-user codebook learned by deep reinforcement learning (DRL) in [10] is utilized and extended it to the multiuser case. This demonstrates that, even with a simple clustering scheme, which eliminates the need for extensive random beam searches with a large set of matched beams, the proposed MARL approach, utilizing the designed reward function, can effectively understand the wireless environment. The methodology illustrates its capability to dynamically adjust beam configurations based on user distributions and environmental topography, significantly reducing the beam training overhead in a massive MIMO system by minimizing the number of beams required for effective communication in the learned codebook. This ad  
Figure 7: Comparison of average beamforming gain versus number of beams in the learned codebook in LOS with 32-element ULA. justment effectively mitigates interference within densely populated urban environments, leading to notable performance improvements. Furthermore, users were strategically placed in nLOS areas within the scenario. The simulation under nLOS conditions highlights the superiority of the proposed MARL system compared to traditional beam codebooks and those proposed in [10]. In this scenario, MARL outperforms the 32-beam classical beamsteering codebook and [10] with just 4 beams. Given that only reflected paths of the channel are available under nLOS conditions, this improvement underscores the adaptability of the MARL system to varying environments. The simulation results for MARL under nLOS conditions are depicted in Figure 8.  
Figure 8 clearly shows that the learned codebook with only 4 beams in nLOS outperforms the traditional 32-beam beamsteering codebook. Additionally, as the number of beams in the learned codebook increases, the performance of the proposed algorithm approaches that of equal gain combining (EGC).