## 1 Introduction
For example, low-complexity hybrid precoding algorithms exploit the sparse nature of mmWave channels using basis pursuit algorithmic concepts, assuming channel knowledge [5]. Similarly, low-complexity hybrid beamforming algorithms for single-user single-stream MIMO-OFDM systems aim to maximize the received signal strength or sum-rate over different sub-carriers [7]. However, these algorithms were designed for single-user channels, limiting supported streams. In multi-user systems, digital precoding in hybrid setups can design precoders that reduce interference between users, making the development of near-optimal, low-complexity hybrid precoding algorithms for multi-user mmWave systems particularly important. Large antenna arrays with quantized phase shifters have challenges. These phase shifters, with constant modulus, control only the phase, limiting applications to equal gain transmission schemes to maximize SNR or diversity gain [8]. Both artificial intelligence (AI) and non-AI methods have been explored in the literature to find optimal beamforming codebooks. A benchmark work [9] used deep learning (DL) to find optimal codebooks for transmit beamforming and combining at user terminals,though it required channel matrix information at both the training and prediction phases. Reinforcement learning (RL) offers a promising solution by eliminating the need for the offline training phases of static deep networks and facilitating adaptive, situation-aware systems capable of learning from the environment. Significant works, such as [10], have implemented RL-based systems with Wolpertinger-variant architectures for beam codebook design, preceded by beam-clustering to reduce the codebook size. Beam clustering in this implementation is actuated through sensing beams. The author in [11] combined the radar-aided direction of arrival (DoA) and direction of departure (DoD) estimation and CS-based position estimation with hybrid beamforming for vehicular communication systems, emphasizing accurate direction estimation to improve the beamforming efficiency. In this work, author has eliminated CSI estimation feedback completely by using the radar-based subsystem. Hybrid beamforming solutions for multi-user millimeter-wave heterogeneous networks is developed in [12] where orthogonal matching pursuit-based analog beamforming and minimum mean square error (MMSE)-based digital beamforming is used. To support multiple simultaneous users, the proposed work extends [10] by introducing MARL. MARL is an AI research area involving the development of intelligent agents that cooperate or compete to achieve common or individual goals. In a fully cooperative stochastic game, all agents share the same goal and work together to achieve it. Stochastic games involve uncertain action outcomes, with outcome probabilities depending on the current game state. In fully cooperative settings, centralized MARL is generally preferred for effective action coordination among agents. In this work, agent coordination translates to reducing interference between beams, serving different user groups simultaneously. Interference between users within a group served by a single beam is minimized through baseband precoding. The authors in [10] utilized receiver signal strength (beamforming gain) for user clustering and codebook learning. Although this metric simplifies the design, it underperforms in nLOS environments and with user mobility. The analysis of average spatial autocorrelation functions for individual multipath components in the 28 GHz mmWave band shows that signals lose correlation after about two wavelengths in LOS environments and after about five wavelengths in nLOS environments [13]. This unreliability of signal strength alone challenges the learning agent's ability to gather useful environmental information. The agent architecture in [10] is adopted for each RL agent in the proposed design. The Wolpertinger-variant structure adapts the continuous action space of deep deterministic policy gradient (DDPG) to work with large discrete action spaces [14].