## 2 Adaptive Beam Alignment Using RL
RL is capable of dynamically adapting to changing circumstances in addition to determining the current ideal phase angle. For example, RL can swiftly adapt each beamforming phase to offer the best possible signal strength to each user even when many users' locations change. This is done by using the knowledge it has gained over time. This approach is more efficient and adaptable to dynamically changing environments than manually testing all possible phase combinations one by one. Thus, while the optimal phase angle can be found manually in static environments, RL can respond more quickly to changes and continuously optimize system performance in real-world applications. In this setting, instead of receiving feedback from the EVM in real time, the RL model is trained offline on previously collected data. The model learns to predict the EVM based on different antenna phase configurations by analyzing these data to optimize its strategy, effectively \"knowing\" about the EVM because it learns the relationship between the EVM and antenna phase from experimental data. The Q-learning algorithm is one type of off-policy RL technique that allows an agent to learn from behaviors based on an alternative policy. It continuously adjusts Q-values, or state-action pairs, in order to improve the agent's decision-making abilities over time. In this context, the \(\varepsilon\)-greedy algorithm is used to choose the next course of action, striking a balance between exploitation and exploration [33]. The impact of future rewards is taken into account using a discount factor of 0.88, which reduces their ability to affect present actions. In essence, Q-learning chooses the course of action that maximizes the Q-table by updating the value function. The agent is provided with input on the EVM and the associated variable phases of the array antenna to optimize the EVM. Until the optimal EVM performance is reached, the agent iteratively modifies the array antenna's variable phase over a predetermined number of steps. Thus, the performance of the system may be optimized by the adaptive beam alignment approach that has been suggested, which can independently adjust the phase in response to the EVM input from the RL agent. The beamforming Q-tables for five actions in the phase state that correspond to steering angles of \(45^{\circ}\), \(30^{\circ}\), \(15^{\circ}\), \(0^{\circ}\), \(-15^{\circ}\), \(-30^{\circ}\), and \(-45^{\circ}\) are displayed in Figure 4a-g. The regions in the figure that are indicated in dark red have higher Q-values, meaning that the RL agent favors choosing the movements in these areas. Conversely, the dark blue regions denote lower Q-values, suggesting that the RL agent would rather not function in these regions. On the other hand, the system takes one measurement of the EVM each time the beamforming antenna's phase is changed, and it reports whether the EVM grows or shrinks. If the EVM becomes larger, it means the signal quality has deteriorated, and the system will give a negative reward score and vice versa, as depicted in Figure 2.  
Figure 5a-g illustrate the actions that the agent can take to achieve optimal convergence within a specific time period, which represents the time of a single interaction between the agent and the environment and is used to adjust the phase of the beamforming over consecutive time periods to optimize its performance. Therefore, the time period helps the  
RL algorithm to adjust the phase change strategy at different time intervals. At different beamforming angles of 45\({}^{\circ}\), 30\({}^{\circ}\), 15\({}^{\circ}\), 0\({}^{\circ}\), \(-\)15\({}^{\circ}\), \(-\)30\({}^{\circ}\), and \(-\)45\({}^{\circ}\), there were 50 random phase settings employed in the initial stage, and the optimal phase convergence and EVM values obtained after RL operation were 300\({}^{\circ}\) and 8.54%, 105\({}^{\circ}\) and 7.61%, 240\({}^{\circ}\) and 7.55%, 0\({}^{\circ}\) and 7.38%, 120\({}^{\circ}\) and 7.3%, 240\({}^{\circ}\) and 7.53%, and 60\({}^{\circ}\) and 8.86%, respectively.