## 4 Video Application Based on Deep Reinforcement Learning Algorithm
### Policy Gradient Algorithm  
The complexity of the DQN seeks out the optimal policy by estimating that the Q-value indirectly increases exponentially when the dimension of continuous or discrete action space is large, so some researchers have focused on solving the optimal strategy directly. The stochastic policy gradient algorithm was proposed in 1999, which parameterizes the policy and updates the policy parameters by calculating the policy gradient. The deterministic policy gradient (DPG) algorithm, on the basis of a random policy gradient, was proposed in 2014. However, because all kinds of policy gradient algorithms need a large number of datasets to provide training--otherwise they will fall into the local optimal situation--the problem can be well solved by the combination of an AC algorithm and a policy gradient. The AC algorithm framework is shown in Figure 8. The AC algorithm is based on the idea of time difference (TD). The policy network is an actor who selects the action according to the environment state. The value function network is a critic who evaluates the action through the value function and generates the time difference error signal to update the improved policy network and the value function network. Figure 7: The DQN algorithm structure based on a competitive architecture. [PERSON] et al. [78] combined a deep neural network with DPG for the first time in 2016, and proposed a deep deterministic policy gradient (DDPG) algorithm based on an AC framework, which makes up for the deficiency that DRL is unable to find the optimal strategy when it comes to continuous action spaces. To minimize the content delivery latency, [PERSON] et al. [79] designed an optimized vehicle edge caching and content delivery strategy using DRL. Edge caching and content delivery problems are described as MDPs; the state space is defined as an adjacent vehicle set, link transmission rate, cached content information and vehicle driving direction; the actions are defined as whether content is cached, the request content ratio and the bandwidth allocation; and the reward is defined as the content delivery delay. The DDPG algorithm is proposed to solve the above problems. Compared with the two benchmark solutions, the numerical results show the effectiveness of the proposed DDPG-based algorithm. [PERSON] et al. [80] aimed to maximize video quality while reducing delay and rate switching in traffic fog computing (VFC) systems by jointly optimizing vehicle scheduling, rate selection, computing and spectrum resource allocation. Considering the dynamic characteristics of the vehicle network and the available computing/spectrum resources, the problem is modeled as an MDP. Due to the action space of an MDP being a mixture of multi-dimensional continuous variables and discrete variables, this study uses a soft-ACDRL algorithm to solve the MDP problem, and its algorithm structure is shown in Figure 9. [PERSON] et al. [81] proposed a dual-scale DRL framework based on a soft-ACDRL algorithm to solve the problem of joint communication, computing and cache resource allocation, reducing costs and improving user satisfaction as a result. The asynchronous advantage ACDRL algorithm (A3C) was proposed by [PERSON] [82]. Because of its multi-core CPU and multi-threading technology, the A3C algorithm makes multiple agents collect experience samples at the same time, and the correlation between samples is very low. It adopts the training mode of on-policy, and replaces the experience playback mechanism in the ACDRL algorithm by multi-line parallel and asynchronous updating parameters. The researchers have proved that the iterative speed of the algorithm is obviously better than that of the ACDRL algorithm. [PERSON] et al. [83] proposed a video analysis framework that integrates multi-access edge computing and blockchain technology into IoAV to maximize the throughput of blockchain systems and reduce the latency of MEC systems.