## 4 Proposed RL-Based Antenna Beamwidth and Beam Alignment Interval Optimization
### RL-Based Joint Antenna Beamwidth and Beam Alignment Interval Optimization  
In the proposed RL-based joint optimization scheme, the state \(s_{t}\) is defined as \(s_{t}=(v_{t},d_{t})\), where \(v_{t}\) is the vehicle speed and \(d_{t}\) is the distance traveled by the vehicle at time \(t\). The action \(a_{t}\) is defined as \(a_{t}=(\varphi_{t},L_{t})\), where \(\varphi_{t}\) is the beamwidth and \(L_{t}\) is the beam alignment interval at time \(t\). The \(\epsilon\)-greedy policy is used to select the action in order to balance exploration and exploitation. However, the action, the antenna beamwidth, and the beam alignment interval affect not only the amount of received data during the current time slot, but also the amount of received data while the vehicle moves along the rest of the road in the future. Consequently, the reward \(r_{t}\) is defined as the weighted sum of the amount of data received in the past, \(U^{(t-)}=\sum_{j=1}^{t-1}S_{j}\); the amount of data received during the current beam alignment interval, \(U^{(t)}=\sum_{j=t}^{t+L_{t}-1}S_{j}\); and the amount of data received in the future, \(U^{(t+)}=\sum_{j=t+L_{t}}^{N}S_{j}\). \[r_{t}=(1-\beta)\cdot\Big{(}U^{(t-)}+U^{(t)}\Big{)}+\beta\cdot U^{(t+)}, \tag{11}\]  
where \(\beta\) is the weight for the future. However, because (11) is noncausal, we approximate the amount of data received in the future based on the received data in the current slot as \(U^{(t+)}=\sum_{j=t+L_{t}}^{N}S_{t}\). Figure 4 shows the structure of the Q-learning-based joint antenna beamwidth and beam alignment interval optimization. The procedure of the algorithm is summarized in Algorithm 1. The Q-value and the amount of data received in the past are initialized with zero (line 1). The parameters, learning rate, discount rate, and weight for the future are configured (line 2). The system model is initialized as the environment and vehicles are generated on a multi-lane road (lines 4-5). The RSU agent obtains the state from the environment (line 7). The agent selects the beamwidth and the beam alignment interval as an action according to the \(\epsilon\)-greedy policy (line 9). According to the action, the amount of received data during the current beam alignment interval is calculated (line 10). The number of remaining time slots and the amount of data to be received in the future is estimated  
Figure 3: A general RL architecture. based on the current beam alignment interval (lines 11-12). The reward is calculated and the amount of data received in the past is updated (lines 13-14). The Q-value and policy is updated via the above process, and the optimal policy is established at the end of each episode (lines 15-20). ```
1:Initialize the Q-value function Q(\(s\), \(a\)) and \(U^{(t-)}\) with zeros;
2:Set the learning rate \(\alpha\), the discount rate \(\gamma\), and the weight for the future \(\beta\);
3:for each episode do
4: Initialize environment;
5: Generate \(K\) vehicles;
6:for each vehicle do
7:for each time slot do
8: The RSU get state \(s_{t}\) from the environment;
9: The RSU selects an action \(a_{t}\) based on the \(e\)-greedy policy;
10: Calculate \(U^{(t)}\);
11: Calculate the number of remaining time slot;
12: Calculate \(U^{(t+)}\);
13: Calculate the reward \(r_{t}\);
14: Update \(U^{(t-)}\);
15: Update \(Q_{\pi}(s_{t},a_{t})\) by using (9);
16: Update policy \(\pi\)
17:endfor
18:endfor
19: Update \(Q_{\pi^{*}}(s_{t},a_{t})\) by using (10);
20: Establish the optimal policy \(\pi^{*}\);
21:endfor
```  
**Algorithm 1** Q-learning-based antenna beamwidth and beam alignment interval optimization  
The computation time of Algorithm 1 can be ignored because of the simplicity of the Q-learning model. In Algorithm 1, the RSU obtains the state, i.e., the speed and location of vehicles. After the RSU chooses an action, i.e., the beamwidth and beam alignment interval, it calculates the estimated data rate of the vehicle according to (5) and updates the Q-table. Hence, the computational time for each step can be ignored.