# Vehicle Lane Change Models--A Historical Review
## 2 Vehicle Lane Change Models
### Empirical Models of Lane Change
#### 2.1.2 Reinforcement Learning Models  
Reinforcement learning models are a type of machine learning that can derive optimized strategies from the vehicle's interactions with the environment. In the lane change model, reinforcement learning trains intelligent vehicles to change lanes. Vehicles in lane change models mainly refer to autonomous vehicles and vehicles in traffic simulation systems. Reinforcement learning enables vehicles to realize lane change behavior by learning the optimal lane change strategy. Reinforcement learning includes environment, state, action, reward, strategy, etc. The environment can be a real road or a traffic simulation environment. The state mainly refers to the location, speed, and variable information of adjacent vehicles. The movement mainly consists of lane keeping and lane changing to the left or right. The reward is mainly to teach the lane change model what constitutes safe lane change behavior, direct it away from dangerous lane change behavior, and optimize the lane change strategy through continuous trial and error and learning. The strategy is the lane change mode selected by the intelligent vehicle according to the current environmental state. The performance of the reinforcement learning lane change model heavily depends on the quality and quantity of the training dataset. In the process of practical application, the model needs to be optimized. [PERSON] et al. (2018) proposed a deep Q-network inverse reinforcement learning (IRL) method to extract rewards with large state spaces. The explosion state space problem is solved, and a highway driving simulator is realized. [PERSON] et al. (2018) addressed the problem of automatic lane changing of autonomous vehicles in a multi-lane environment with adjacent vehicles. Traditional rule-based approaches are difficult to apply to formulate high-level policies, but this framework demonstrates a more structured and data-efficient end-to-end learning scheme for complete policies on a well-designed low-level controller. Q-masking incorporates prior knowledge, constraints, and information from low-level controllers directly into the learning process, simplifying the reward function and making learning faster and more efficient. Automatic lane changing is an important function of intelligent vehicles. In a complex lane change environment, the uncertainty of adjacent drivers needs to be taken into account. The agents in the simulation environment newly developed by [PERSON], [PERSON] et al., include vehicle dynamics, lateral controller, adaptive cruise controller ([PERSON] et al., 2018), and a safe lane change algorithm ([PERSON] et al., 2018) to optimize performance. Q-learning performs well in planning ([PERSON] et al., 2018) and autonomous system control ([PERSON] et al., 2018; [PERSON] et al., 2018), and it further optimizes the deep reinforcement learning algorithm ([PERSON] et al., 2018). This led prior research to train a deep reinforcement learning agent ([PERSON] et al., 2018). This agent shows excellent performance for uncertain and random environments, and it can perform well in complex lane-changing environments. The lane change behavior of the driver is based on adapting to the driving situation of the road ahead or based on their driving experience. Unreasonable lane change behavior is the main cause of traffic congestion and accidents. According to the uncertainty and complexity of the driving environment, [PERSON] et al. (2018) proposed a lane change strategy based on near-end strategy ([PERSON] et al., 2018) optimization, which, in turn, is based on deep reinforcement learning. Through training, a smooth, safe, and efficient lane change behavior can be achieved. Due to the variability and complexity of the