## 5 Q-Learning-Based CCM-SPS Resource Allocation Scheme Proposed for JCS Sidelink
### Q-Learning-Based CCM-SPS Scheme for JCS Sidelink
#### 5.2.1 Vehicular Agent Based on the Reinforce Learning Model  
In a typical reinforcement learning framework [47], as illustrated in Figure 4, the agent achieves its learning objectives through an iterative process of receiving rewards from the environment. Intelligent agents improve their strategies by exchanging rewards with the environment.  
In the proposed sidelink JCS system, each vehicle updates its current state after transmitting data packets and detecting collisions. The current state includes the collision status and the current \(RC\) value. For vehicle \(i\), the state \(s_{i}\in S\) can be described as follows:  
\[s_{i}=(COL_{i},RC_{i}) \tag{17}\]  
where \(\text{COL}_{i}\) indicates whether the vehicle \(i\) experiences a resource collision when transmitting a data packet. \(\text{RC}_{i}\) represents the current value of the reselection counter, with a range of \([0,15]\). Therefore, the state set consists of a discrete state space.  
The traditional SB-SPS scheme reduces the \(RC\) by 1 after each packet transmission. To regulate the decreased efficiency of \(RC\), we propose a set of a discrete action space that represents the selection of actions based on acquired state information. This enables resource collision vehicles to adjust the \(RC\) decreasing step size (\(\Delta RC\)).  
\[A=\{a_{i}\in\Delta RC\} \tag{18}\]  
where \(\Delta RC\) represents the value of the \(RC\) that the vehicle \(i\) has to reduce after each transmission. The vehicle will dynamically select an action in the action space of \([0,15]\) based on the observed state. When the \(RC-\Delta RC<0\), the vehicle enters the reselection process and reselects a new resource.  
After a vehicle executes an action, its state will transition, and it will learn the instantaneous reward \(R\) from the environment.  
\[R=\begin{cases}(\exp(-N_{col})-1)/log_{10}(RC+1),&\text{COL}=1\\ 1,&\text{COL}=0\end{cases} \tag{19}\]  
Figure 4: Reinforcement learning framework.  
where \(N_{col}\) represents the times of a consecutive resource collision for the vehicle. When the collision does not occur, the reward is 1. When a collision occurs, the greater the \(N_{col}\) is, the greater the penalty is, ensuring that the collision state can be removed as quickly as possible. In addition, the current \(RC\) value is introduced as a constraint. The smaller the \(RC\), the greater the penalty. When different \(RC\) vehicles collide in the same resources, the probability of the vehicle with a small \(RC\) entering the reselection stage increases, while the vehicle with a large \(RC\) can continue to transmit using the current resources. This mechanism can effectively reduce the number of vehicles in the reselection stage simultaneously, thus helping to maintain the stability of the system.  
In Q-learning, Q value \(Q(s,a)\) was calculated and updated using a reward model to evaluate the state-action mapping policy under the state action pair \((s,a)\). The Q value is updated by the Bellman equation, as shown below:  
\[\begin{split} Q(s,a)&\gets Q(s,a)+\alpha[R\\ &+\eta\max_{s\in S}Q(s^{\prime},a)-Q(s,a)]\end{split} \tag{20}\]  
When vehicles optimize the search for the optimal action, they need to balance the exploitation and exploration of learned knowledge to ensure that each action has a possibility of being selected.  
\[a=\left\{\begin{array}{ll}\arg\max_{a\in A}Q(s,a),&\text{with prob. }1- \varepsilon\\ \text{random},&\text{with prob. }\varepsilon\end{array}\right. \tag{21}\]  
This paper adopts an \(\varepsilon\)-greedy strategy to balance the exploitation-exploration process. During the training process, the initial \(\varepsilon\) is set to 0.9 and gradually decreased to 0.1. The training lasted for 100 s, consisting of 1000 iterations. The initial learning rate was set to 0.01, and the discount factor was set to 0.9.