## 3 Proposed Beam Learning with MARL
### Components of the Proposed MARL-Based Codebook Learning System and Its Implementation  
To make MARL applicable, the environment must be modeled as a Markov process. In [10], this is achieved by incorporating the current beamforming vector as a function of the previous beamforming vector. A similar approach as in [10] is followed, extending this method by also considering the partial and imperfect CSI acquired by the BS during uplink SRS transmission by the UE. The operation of the entire system is illustrated in Figure 5, with each processing block and signal flow explained subsequently. Figure 5: Block diagram of the proposed MARL-based beamforming codebook design. RL is a type of machine learning where an agent learns to make decisions by performing actions and receiving feedback from the environment in the form of rewards or penalties. This learning process involves trial-and-error, where the agent seeks to maximize cumulative rewards over time. Unlike supervised learning, where the correct actions are provided by a teacher, RL agents must discover optimal actions through interaction with the environment. This approach is formalized through frameworks like Markov decision processes, which provide a mathematical foundation for modeling decision-making scenarios involving uncertainty and delayed rewards. RL has been successfully applied in various domains, including robotics, game playing, and autonomous control systems, due to its ability to handle complex, dynamic environments. Recent advancements in deep reinforcement learning, which combines RL with deep neural networks, have led to significant breakthroughs, such as achieving human-level performance in complex games like Go and Atari. This field continues to evolve, offering promising solutions for real-world applications where adaptive, intelligent behavior is required. The proposed MARL algorithm builds upon the Wolpertinger Architecture [14], following a similar approach to that described in [10]. The Wolpertinger architecture in reinforcement learning is a sophisticated framework designed to address the challenges associated with large discrete action spaces. This architecture integrates the strengths of deep reinforcement learning with efficient action exploration mechanisms. It operates by embedding the discrete action space into a continuous representation, allowing for more efficient searching and action selection. The Wolpertinger policy effectively reduces the computational complexity by narrowing down potential actions through a k-nearest neighbors approach, ensuring that the most promising actions are evaluated without exhaustive searching. This makes it particularly useful in applications such as content caching at the edge of wireless networks and optimizing beamforming in MIMO systems, where traditional methods struggle with the vast number of possible actions. By efficiently managing large action spaces, the Wolpertinger architecture enhances the scalability and applicability of reinforcement learning in complex, real-world scenarios, ensuring robust and effective decision making. The Wolpertinger architecture adapts the DDPG, originally crafted for continuous action spaces, to function within a discrete action space through the utilization of a K-nearest neighbor (KNN) classifier. To address non-stationary environment issues in a multi-agent RL system with continuous action spaces, the MADDPG offers a solution. MADDPG achieves this through centralized training and decentralized execution. To accommodate a discrete action space in MARL, an improvisation on MADDPG is made by implementing each agent in MARL using the Wolpertinger architecture. Thus, the proposed MARL essentially embodies MADDPG, with each agent designed to adhere to the Wolpertinger architecture. The proposed beam learning problem presents a significant challenge due to the large number of possible actions. For instance, considering a base station with 32 antennas, 3-bit phase shifters, and 4 RF channels, each agent faces around \(8^{32}\) potential actions.