## 1 Introduction
### Motivation  
Quantized phase shifter with fixed numbers of bits makes the search space for beams very large. For example, there will be \(8^{64}\) beams for a 64-element antenna array with 3 quantization bits. In a multiuser case with four RF chains, this number will equate to \(4\times 8^{64}\). Finding optimal beams in such a finite but huge space is impractical with exhaustive search or any other traditional technique. Hence, it is a convention to use a beam codebook with large numbers of beams pointing in different directions in an effort to maximize the gain to the users in that direction. This approach is not optimal as this single-lobe beam, which matches filters to the array responses in a particular angle, is not guaranteed to offer maximum possible gain for occluded, non-line of sight (nLOS) users. Also, the large number of beams required in such a codebook renders the beam training inefficient and time consuming, and hence, is inapplicable to mobile users. Additionally,an accurate array response is required to form such a beamsteering codebook which may not be available for cost-effective systems as calibrating antenna arrays is a sophisticated and costly process.  
Motivated by the fact that the environment changes infrequently, the computation power and time are traded for accuracy by utilizing uplink CSI estimated at the base station, with re-clustering required only when the environment changes. CSI provides good autocorrelation properties and is relatively immune to hardware imperfections in the RF stage, making it suitable for beam learning applications. Perfect CSI is not assumed, as the channel is seen through the RF lens in hybrid beamforming. However, it is shown, in this work, that beam learning can be efficiently achieved even with uplink CSI estimates at the BS, by using uplink channel envelope estimates at the BS as fingerprints of specific user locations within the MARL framework.  
It is important to note that a learned multibeam codebook differs from simply parallelizing multiple single beams. Unlike fixed codebook beams, learned beams are not matched filters to the antenna array response and can adopt any arbitrary shape suitable for the scattering environment of deployment. Consequently, this may result in beam overlap, leading to inter-user interference, if cooperation between the beams for multiple simultaneous users is not established. The proposed MARL-based approach addresses this issue by cooperatively learning the beamforming vector for each beam per user group.  
In summary, RL is employed for beam learning due to its online learning capability, eliminating the need for an offline data collection phase. The Wolpertinger architecture is utilized for its efficiency in exploring vast search spaces for beam configurations. MARL is proposed to facilitate the selection of multiple parallel and diverse beams. Through centralized training, MARL can identify environment-specific, arbitrarily shaped beams that do not interfere with each other. Most existing approaches use RSRP only, which simplifies the design but RSRP does not provide good auto-correlation and is not a reliable indicator of a UE's position. In this work, uplink CSI is utilized, which, although it may not always replace downlink CSI for channel estimation in non-reciprocal systems with hardware imperfections, it offers a better representation of the implicit location of the UE.