## 4 Video Application Based on Deep Reinforcement Learning Algorithm
### Deep Reinforcement Learning Algorithm Based on Value Function  
After the Deep Q-Network (DQN) was first proposed by [PERSON] et al. [62] in 2013, deep reinforcement learning has attracted increased attention. With the deepening of video transmission research over CVIS, DQN is widely used to solve video business-related problems. Because of the large dimension of state space, the Q-value of traditional reinforcement learning is difficult to predict. The deep neural network can extract eigenvalues through multiple network layers to reduce the dimension, so DQN approximately represents the Q-value function through the deep convolutional network (DNN). The evaluated Q-value function may be unstable and deviated. The target network and the experience playback mechanism proposed by [PERSON] et al. [63] solve this issue. In order to improve the revenue of RSU operators, [PERSON] et al. [64] proposed a scenario in which vehicles covered by an RSU cache off-the-shelf content (video streams, etc.) to the RSU, and the RSU then provides services for newly arrived vehicles, which is defined as an MDP. The author defined the environmental state as the vehicle location information, the request content and the available content information, defined the action as whether or not to distribute the content and defined the reward as the available income. The DQN algorithm is used to solve the MDP problem; the structure of the DQN algorithm is shown in Figure 6. First, observe the state space: DNN extracts the abstract representation through the state space, RSU selects the action to execute and the environment gives feedback (the vehicle delivery cost). The DQN model proposed by the author is superior to other methods in terms of total income, cost and service rate. [PERSON] et al. [65] solve the security and QOS problems of green, balanced, linked and efficient vehicle networks. Using the DQN model, the completion request rate is increased by 10-25%, the average request delay is reduced by 10-15% and the total network life is increased by 5-65%. [PERSON] et al. [66] developed a network slicing model based on the fog node (FN) cluster coordinated with the edge controller (EC) to solve the resource allocation problem. The author described the problem as an MDP, and the state space is defined as the resource block information, the action is defined as whether the edge controller sends the resource block to the corresponding FN cluster for processing and the reward is defined as the key performance indicator (KPI). Through DQN, self-adaptive learning is the best slicing strategy to maximize KPI, so as to achieve efficient resource allocation. [PERSON] et al. [67] proposed an asynchronous federated DQN-based and URLLC-aware computation offloading algorithm (ASTEROID) to maximize throughput under long-term  
Figure 6: The structure of the DQN algorithm. URLLC constraints. In this paper, a URLLC constraint model, based on extreme value theory, is established. Secondly, Lyapunov is used to optimize the decomposition of task unloading and computing resource allocation. Finally, an asynchronous federated DQN (AF-DQN) algorithm is proposed to solve the problem of vehicle-side task unloading. Its state space is defined as the backlog of information in actual and virtual queues, the action is defined as whether the task is unloaded and the reward is defined as the throughput. Simulation results show that ASTEROID achieves excellent throughput and URLLC performance. [PERSON] et al. [68] proposed a novel slicing framework and optimization solution based on dynamic reinforcement learning for efficient resource allocation in virtual networks with D2D-based V2V communication. The goal is to balance resource utilization and QOS satisfaction across multiple slices.