## 4 Video Application Based on Deep Reinforcement Learning Algorithm
### Deep Reinforcement Learning Algorithm Based on Value Function
The problem is described as an MDP whose state space is defined as the QOS satisfaction, resource utilization and the number of resource blocks occupied by each slice; the action is defined as the proportion of resources allocated to the slice and the reward is defined as the composition of resource utilization and QOS satisfaction. The optimal strategy is obtained by using the DQN model, and the performance results of the resource utilization, QOS satisfaction and throughput are given. [PERSON] et al. [69] studied joint communication, caching and computing design issues to achieve excellent operation and cost efficiency for in-vehicle networks. Because of the high complexity of CVIS, the author used the multi-time scale framework [70] to develop deep reinforcement learning to reduce the complexity. The multi-time scale framework consists of two models. The model for each cycle is called the large time scale DQN model, and the model for selecting the action decision for each time slot is called the small time scale DQN model. Numerical results are given to prove its performance gain. To maximize the long-term sum ratio of all vehicles, [PERSON] et al. [71] proposed a hybrid architecture which consists of centralized decision making and distributed resource sharing (C-Decision scheme). Each vehicle uses a deep neural network to compress its observed information, and then feeds it back to the central decision-making unit. The centralized decision-making unit uses DQN to allocate resources, and then sends the decision results to all vehicles. In order to promote distributed resource sharing, the author proposed a distributed decision and spectrum sharing architecture (D-Decision scheme) for each V2V link. Simulation results show that the proposed C-Decision and D-Decision schemes can achieve near-optimal performance and are robust to feedback interval variation, input noise and feedback noise. [PERSON] et al. [72] studied the federated optimization of transmission mode selection and resource allocation for cellular V2X communication. The problem is expressed as an MDP whose state space is defined as the receiving interference power, the resource block neighbor information of the previous subframe, the channel gain, the current load and the remaining time to meet the delay threshold; the actions are defined as the resource block allocation, communication mode selection and transmission power level; the rewards are defined as the sum of the total capacity benefits, the unmet capacity penalties and the effects of reliability and delay requirements. A decentralized algorithm based on DQN is proposed to maximize the total capacity of vehicles to infrastructure users and to meet the delay and reliability requirements of V2V communication. [PERSON] et al. [73] proposed a deep double Q-network on the premise of an in-depth study of the double Q-learning algorithm. The traditional DQN may overestimate the Q-value, but DDQN reduced the overestimation by improving the training algorithm, so as to obtain a more accurate Q-value and optimize the performance of the model. [PERSON] et al. [74] proposed a competitive network structure as a network model of DQN; its structure is shown in Figure 7. The competitive network structure divided the abstract features extracted by DNN into two places, in which the state value function represents the value of the environment itself, and its value does not change according to the action choice, while the action dominance function represents the additional reward value after the execution of the selected action. When the two are combined, the Q-value is generatedfor each action. The duel DQN improves the prediction accuracy of the value function by improving the structure of the model. In order to improve the performance of vehicle network, [PERSON] et al. [75] proposed an integrated framework which can realize the dynamic allocation of network, cache and computing resources. The resource allocation strategy of the framework is described as a joint optimization problem.