# Reinforcement Learning Approach for Adaptive C-V2X Resource Management
## 3 Proposed Reinforcement Learning Approach for Adaptive Resource Management
### Impact of MCS Index
For the epsilon value (\(\epsilon\)), we used \(\epsilon=0.1\) and \(\epsilon=0.9\) to represent the \"exploitation\" and \"exploration\" actions, respectively, within the epsilon-greedy mechanism [37]. When the epsilon value is close to zero, the system will never explore but tend to exploit previously known knowledge. On the other hand, when the epsilon value is close to one, the system will prefer to use random action rather than to use the information from past knowledge. Similar to the epsilon, we investigated two sets of learning rate (\(\alpha\)) value scenarios. The learning rate scenarios were set to \(\alpha=0.1\) and \(\alpha=0.9\). The smaller alpha value meant the Q-value would change slower, and vice versa. For the discount factor (\(\gamma\)) value, since we aimed to achieve the optimal PRR performance, we only \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Values** \\ \hline Simulation time (\(T\)) & 60 s \\ Vehicle position update time (\(t_{r}\)) & 0.1 s \\ Vehicle’s speed & 120 km/h \\ Vehicle’s speed standard deviation & 40 km/h \\ Road length & 1000 m \\ Number of lanes & 1 per directions \\ Total number of vehicles (\(n_{V}\)) & 100 vehicles, 200 vehicles \\ Channel bandwidth (\(B_{c}\)) & 10 Mhz \\ Transmission power (\(P_{Tx}\)) & 23 dBm \\ Beacon size & 190 bytes \\ Sensing interval & 0.1 s \\ Modulation Coding Scheme (MCS) & 0–28 \\ Probability resource keep (\(P_{rk}\)) & 0.8 \\ Q-learning predefined epsilon (\(\epsilon\)) & 0.1, 0.9 \\ Q-learning discount factor (\(\gamma\)) & 0.1 \\ Q-learning learning rate (\(\alpha\)) & 0.1, 0.9 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameter settings. investigated the discount factor value at the low value at \(\gamma=0.1\). Under the simulation parameters, as explained in Table 1 and Figures 2 and 3, are the initial simulation results of the corresponding MCS index from the related works for 100 vehicles and 200 vehicles. Figure 3: PRR performance of static MCS index value against vehicle’s distance with \(n_{V}=200\) vehicles. Figure 2: PRR performance of static MCS index value against vehicle’s distance with \(n_{V}=100\) vehicles. Figure 2 shows the PRR performance with corresponding MCS index values based on Table 1 against the vehicle's distance. We conducted our initial simulation as a benchmark foundation to investigate our proposed RL algorithm approach for the adaptive MCS index value later. As shown in Figure 2, the total number of vehicles was set to 100 vehicles. Since the vehicle's density was notably at a fair amount, most of the MCS index values PRR performance were relatively high. For instance, the most significant performance difference was developed on the higher MCS index, 20 and 27. As explained in the previous section, a higher MCS index will result in a larger number of physical resource blocks. The larger number of physical resource blocks required more transmission power. Hence, the higher MCS index suffered poor PRR performance. A similar result was also obtained in Figure 3, where the number of vehicles was higher, at 200 vehicles in total. Identically with Figure 2, the higher MCS index also suffered poor PRR performance. The condition worsened with the higher vehicle number. More vehicles deployed in the system meant more RBs needed to be occupied for data transmission. Thus, when the number of physical RBs is large and the number of vehicles is also high, this may cause a lack of radio resources. From Figures 2 and 3, we can see that using a high MCS index value will produce low PRR performance. Instead, the overall PRR performance may drop significantly along with the increasing vehicle distance. This situation may be caused by the lack of the transmission power to satisfy the SINR threshold on each TB. respectively, where \(R_{t}\) is the reward function in current time step and \(\gamma\in[0\,1]\) is the discount factor.