## 3 RL-Based Resource Allocation Algorithm
### Link Configuration Decision
#### 3.2.3 Reward Design  
The ultimate goal of this paper is to maximize the long-term sum rate of V2V links, ensure the QoS of V2I links in V2X scenarios and ensure the transmission performance of priority links. V2V links are generally used to transfer key information during vehicle running, such as vehicle speed, location, driving direction and braking. V2I communication  
Figure 2: The interaction between an agent and the environment in vehicle network.  
is mainly used for real-time information services, vehicle monitoring and management, and charging without parking. Therefore, the primary goal should be to ensure V2V link transmission, while ensuring that V2V transmission should not cause too much influence on V2I links, and priority links in the network of vehicles should be interfered with less. To achieve this goal, this paper assumes that the first V2V link is a priority link, and the reward of RL can be designed as follows:  
\[R=\lambda_{c}\sum_{i=1}^{I}C_{j}^{c}[i]+\lambda_{d}\sum_{j=1}^{J}C_{j}^{d}- \lambda_{f}I_{1}, \tag{10}\]  
where \(\lambda_{c}\), \(\lambda_{d}\) and \(\lambda_{f}\), respectively, correspond to the non-negative weight of the total rate of V2I links, total rate of V2V links and total interference of priority links.  
The optimization goal of the reinforcement learning algorithm is to find an optimal strategy \(\pi(a,s)\) to maximize the cumulative reward value returned by the training set. Strategy \(\pi(a,s)\) refers to the probability distribution of action \(a\) given state \(s\). In reinforcement learning, there is always an optimal strategy \(\pi^{*}(a,s)\) that maximizes the expected reward. The expected reward can be expressed as \(R_{t}=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}\), where \(\gamma\) is the attenuation of future rewards.  
In this paper, \(Q\) learning is used to solve RL problems because it has model-independent properties where for \(P(s^{\prime},r|s,a)\) no prior is required. Concerning \(Q\) learning based on a given strategy \(\pi\)  
\[Q^{\pi}(s,a)=E_{\pi}[R_{t}|S_{t}=s,A_{t}=a]\]  
So the agent takes action \(a\) in state \(S\) and gets a reward based on probability \(\pi\). The optimal action value function \(Q^{*}(s,a)\) under the optimal strategy \(\pi^{*}\) satisfies the Bellman equation, which can be approximated by the iterative updating method:  
\[Q(S_{t},A_{t})\gets Q(S_{t},A_{t})+\alpha[R_{t+1}+\gamma\text{max}_{a}Q(S_ {t+1},a)-Q(S_{t},A_{t})], \tag{11}\]  
where \(0<\alpha<1\) represents the learning rate, which determines how much of the error is to be learned.  
In [28], the \(Q\)-table was turned into a network model to obtain a \(Q\) value. In the past, a \(Q\) value needed to be queried in a \(Q\)-table, but now we only need to input the state and action to obtain the corresponding \(Q\) value.  
DQN uses the \(\varepsilon-greedy\) strategy to store the transfer samples \((S_{t},A_{t},R_{t},S_{t+1})\) obtained by the interaction between each time-stepping agent and the environment in the empirical memory unit. At each slot, according to the behavioral strategy and feedback from the environment, a set of sample data \(D\) of the agent is stored in the memory bank every training session, and network parameters \(\theta\) are updated with random gradient descent variables to minimize the squared error:  
\[L=\sum_{t\in D}{[R_{t+1}+\gamma\text{max}_{a}Q(S_{t+1},a;\theta^{-})-Q(S_{t},A _{t};\theta)]^{2}}, \tag{12}\]  
where \(\theta^{-}\) represents the parameter of the target Q-network, which is updated synchronously with Q-network parameter \(\theta\) every time step. Meanwhile, in order to further improve the stability of DQN, DQN is updated several times. During training, a certain number of data are randomly extracted from the empirical memory unit for training. So you can constantly optimize the network model.