# Vehicle-Following Control Based on Deep Reinforcement Learning
## 2 Related Work  
Vehicle-following research has been an important research direction in traffic flow analysis and autonomous driving research. The related research on vehicle-following model can be traced back to the middle of the last century, and there have been many research advances since then. In terms of vehicle-following model research, the first vehicle-following model was proposed by [PERSON] [9] and was widely used in the description of vehicle flow. Later, different types of vehicle-following models based on different directions and fields were also proposed. [PERSON] et al. [10] jointly proposed the GM model, which is based on the driver's stimulus response while taking into account the safe distance. The subsequent stimulus-response vehicle-following models are mostly based on this model. The Gipps [11] model as the earliest safe distance model was proposed by [PERSON] and [PERSON]. Based on the Gipps model, [PERSON] et al. [12] proposed a safety distance model based on the headway. [PERSON] et al. [13] proposed a driver state vehicle distance model based on kinematics analysis, and a safe distance model based on the difference of driver response delay in emergency state. [PERSON], [PERSON] et al. proposed the classic IDM model [14]. The IDM model can fully describe the change of vehicle-following behavior from free flow to congested state with a unified structure. With the development of neural networks, the use of fuzzy logic to establish a neural network vehicle-following model has been realized. The results of the neural network vehicle-following model proposed by [PERSON] et al. [15] after comparing with Gipps show that its prediction accuracy is higher than the latter. The vehicle-following model based on sequence-to-sequence proposed by [PERSON] et al. [16] has memory effect and response delay capabilities, which further expands the spatial expectation and improves the accuracy of platoon simulation and the stability of traffic flow. [PERSON] et al. [17] proposed a novel platoon formation and optimization model combining graph theory and safety potential field (G-SPF) theory which can form a collision-free platoon in a short time. [PERSON] and [PERSON] [18] proposed an improved forward-considered vehicle-following model that uses an average expected velocity field to describe the flow of autonomous vehicles. The new model has three key parameters: adjustable sensitivity, intensity factor, and average expected velocity field size, which in general bear a large impact on the stability and congestion state of autonomous vehicle flow. The purpose of vehicle-following decision-making is to form a vehicle-following decision that can ensure the safety and rationality of vehicle-following according to the state information of the vehicle-following vehicle or the vehicle-following queue and certain decision-making methods. In terms of vehicle-following decision research, [PERSON] et al. [19] proposed an adaptive hierarchical control structure, in which the upper control layer is used to obtain the sliding mode control law of the required acceleration according to the inter-vehicle state information. In the lower control layer, switching logic with hysteresis boundaries is developed to ensure ride comfort, and the desired torque is calculated in real-time based on an inverse dynamics model to track the desired acceleration planned by the upper control layer. [PERSON] et al. [20] proposed a behavior estimation method based on contextual traffic information to identify and predict lane change intentions, and optimize the acceleration sequence by combining the lane change intentions of other vehicles. The above methods are based on traditional control methods and do not have the robustness to adapt to most scenarios, so many teams have turned their attention to deep reinforcement learning. The intelligent vehicle-following process can be abstracted into a state transition process that conforms to the Markovian property [21], so it is possible to use the deep reinforcement learning realize the vehicle-following.