## 2 Adaptive Beam Alignment Using RL  
Figure 2 depicts the proposed phase adaptive tuning architecture for beamforming, which uses the universal software radio peripheral (USRP) to transmit and receive signals. The first step is to generate the baseband 16-QAM signal after setting the parameters of the modulated signal through a personal computer (PC) using the radio hardware driver software LabVIEW. Subsequently, the baseband signal will be converted to 6 GHz through the USRP by digital-analog conversion, and the bandwidth is 250 kHz. To transmit millimeter-wave signals, the 6 GHz signal is boosted to 28 GHz by a frequency upconverter (UPC) and then transmitted to the other end of the 4 \(\times\) 4 array antenna through a 4 \(\times\) 4 array antenna, and then the 28 GHz signal is downconverted to 6 GHz by a frequency downconverter (DC). Then, the signal is transmitted to the USRP receiver, where it is demodulated, converted to the digital signal reduced to the base frequency, and analyzed for signal quality using LabVIEW. The beamforming steering angle setting options are \(-45^{\circ}\), \(-30^{\circ}\), \(-15^{\circ}\), \(0^{\circ}\), \(15^{\circ}\), \(30^{\circ}\), and \(45^{\circ}\), and the distance between the beamforming 4 \(\times\) 4 array antenna Tx and Rx is fixed at 2 m. Figure 3 shows the measured radiation patterns of the 4 \(\times\) 4 array antenna at different steering angles: (a) \(0^{\circ}\), (b) \(15^{\circ}\) and \(-45^{\circ}\), (c) \(30^{\circ}\) and \(-30^{\circ}\), and (d) \(45^{\circ}\) and \(-15^{\circ}\). It can be noticed that the maximum power point of the radiation pattern distribution is consistent with the set steering angle. Hence, the highest power associated with the main lobe is the transmission direction for the communication application. The main lobe's power already exceeds the side lobe's power by more than 10 dB to guarantee that the main lobe transmission will not encounter interference. Figure 1: Conceptual vision of synergistic operation of mmW beamforming and O-RAN. Collecting EVM values at various phases is necessary for the application of RL-based beamforming with phase-adaptive beam alignment for the set steering angles of \(-45^{\circ}\), \(-30^{\circ}\), \(-15^{\circ}\), \(0^{\circ}\), \(15^{\circ}\), \(30^{\circ}\), and \(45^{\circ}\). The RL model, which seeks to predict and modify the beamforming antenna's phase, is trained using collected data. RL comprises three fundamental components: state, action, and reward. The state denotes the present circumstances or environment, furnishing all necessary information for the agent to make decisions. The action is the choice taken by the agent according to the current state, which determines  
Figure 3: Radiation pattern measurement at the different steering angles (**a**) \(0^{\circ}\), (**b**) \(15^{\circ}\) and \(-45^{\circ}\), (**c**) \(30^{\circ}\) and \(-30^{\circ}\), and (**d**) \(45^{\circ}\) and \(-15^{\circ}\). Figure 2: The arrangement of \(4\times 4\) array antenna beamforming with RL. how the agent interacts with the environment and may lead to different outcomes. The reward is the score that the agent gets from the environment's feedback signals after action. This score indicates how well the action accomplished the agent's objective and serves as a basis for making subsequent decisions. In real-world scenarios, the system relies on trained RL algorithms to anticipate and modify the phase of beamforming, as the precise angle of the user is not always known. Based on past data, these models can quickly determine the optimal phase configuration for situations when the user's position is uncertain or fluctuating, resulting in improved signal coverage. To run efficiently and provide improved communication quality even in situations when the user's angle is not instantly known, the system relies on broad patterns in the environment to draw inferences and optimizations instead of real-time user position information.