## 3 Proposed Beam Learning with MARL
### Data Collection through Initial Access Procedure  
The codebook learning process is initiated by first obtaining a fixed conventional codebook through the initial access procedures and beam management procedures outlined in the 5G New Radio (5 GNR) technical report. The steps involved in this fixed codebook learning process are summarized as follows:  
Procedure 1: When a connection is established between a transmitter and a receiver, an initial beam alignment is required. This involves finding an optimal transmit-receive beam pair that maximizes the signal strength between the devices. Various methods like synchronization signal blocks (SSBs) and reference signals are used to aid in this process. This is shown in Figure 4.  
Procedure 2: Refining transmit-end beam via non-zero-power CSI-RS and SRS. After initial beam acquisition, this beam management aims to refine the beams to improve the communication link further. In this step, reference signals are sent in different directions using finer beams within the initial angular range. UE or BS assesses these beams with the fixed receive beam and selects the best transmit beam.  
This proposed system initially employs a standard beamforming procedure and gradually transitions into a more efficient MARL-based system over time. This method essentially substitutes standard codebook beams with learned beams on a one-to-one correspondence  
Figure 4: SSB beam search method in initial access procedure.  
Figure 3: Topographic map of ray tracing environment with one LOS and one nLOS user. Radiation pattern for the 32 antenna BS is also shown forming beams toward each users.  
basis. The angular spacing between nearby beams is determined by the number of beams, which corresponds to the number of agents in the MARL framework. This approach simplifies implementation without necessitating alterations to the existing infrastructure. Once the codebook is learned, it can be utilized until the link's performance deteriorates due to significant changes in the deployment site.  
In this research, the procedures for initial beam acquisition and subsequent beam learning are segmented into the following major steps:  
1. SSB beam sweeping;
2. Beam measurement and determination at UE;
3. Beam reporting to BS by UE;
4. Send SRS to BS from UE for uplink transmit end beam refinement and also for MARL-based downlink transmit end beam refinement. This procedure differs from method in 5 GNR by the fact that the standard used NZ-CSI-RS for downlink transmit end beam refinement. This requires CSI feedback from UE and can work only with traditional matched filter-based codebooks as full-channel estimate feedback from UE which is required for non-codebook based beamforming is unavailable or impractical to achieve and resource intensive;
5. Send NZ-CSI-RP to UE only to obtain RSRP feedback (RSRP consumes very little resource);
6. Decode received SRS and estimate uplink Channel at BS;
7. Send RSRP measurement in SRS to UE for beam refinement at UE;
8. At BS, use RSRP and channel estimate acquired in step 5 and step 6 to learn downlink transmit end beam codebook through the proposed MARL algorithm.