## 1 Introduction
They evaluated the simulation performance with a time-space diagram and reward values. [PERSON] et al. [12] used a SUMO and Middleware for vehicle navigation in an urban network. Furthermore, the reduction in delay time, fuel consumption, and emissions could happen with a higher average speed. Hence, the average speed has become a promising metric to verify a training policy in the real environment. Considering longitudinal vehicle motion modeling, the car-following models were applied to a microscopic traffic simulation in order to explain the vehicle following behavior. Adaptive cruise control (ACC), an advanced car-following model, was applied to set the relative distance between vehicles. Previous studies have used an automotive system to enhance safety and smooth traffic. [PERSON] and [PERSON] [13] used the ACC system in a semi-autonomous vehicle. However, ACC relies upon constant spacing. Recently, the car-following model was conducted for AVs according to discrete following interval in order to improve the traffic flow stability [14]. The intelligent driver model (IDM), which was proposed by [PERSON] and [PERSON] [15], showed a principal superiority against other ACC models. This means that the parameters of the IDM are available and intuitive to improve real capacity. The IDM was also conducted by the BMW vehicle manufacturer. Additionally, the advantage of the IDM was embedded in the SUMO simulator [15]. For example, the IDM was applied to instability in a traffic congestion flow [16]. Therefore, the IDM can take effective control of HVs in a simulation environment. Furthermore, recent breakthroughs in artificial intelligence (AI) have been designed to enhance autonomous driving domains. For example, the policy search guidance was conducted by deep convolutional neural networks (CNNs) [17]. The convolutional long short-term memory (Conv-LSTM) was designed for AV's motion planning [18]. Additionally, the NVIDIA drive is one of the leading AI platforms for automated driving applications [19]. The MobilEye EyeQ5 could be advantageous in complex tasks thanks to four parallel optimized machine learning paradigms [20]. However, these approachesneed a large number of datasets or a commercial platform. Recently, numerous studies have focused on RL-based driving tasks in dynamic conditions. RL, which is a subset of machine learning, is significantly different from unsupervised learning and supervised learning. It tries to maximize a reward from state and observation instead of finding a hidden structure in the input data. The traditional RL is the Markovian decision process (MDP) initiated by [PERSON] [21]. MDP relies on discrete stochastic algorithms to optimize the policy. Nevertheless, AVs operate in an uncertain condition due to the intentions of human drivers and the noise of the sensor. To overcome this issue, a partially observable MDP (POMDP) was applied to keep a probability distribution through a set of observations [22]. Recent studies have used RL for transportation issues--namely, adaptive traffic signal control [23; 24] and autonomous vehicle agents in roundabouts [25]. Furthermore, the development of a deep neural network (DNN) can enhance feature extraction representations for complex tasks based on multi-hidden layers. By integrating RL and DNN, which is named deep reinforcement learning (DRL), the training policy has achieved a more reliable performance. For example, [PERSON] et al. [26] used the DRL for large-scale adaptive traffic signal control (ATSC). [PERSON] et al. [27] applied DRL to left turn CAVs at a signalized intersection. [PERSON] and [PERSON] [28] applied DRL to control multiple signalized intersections. Additionally, [PERSON] et al. [29] used DRL for an intelligent roundabout. More importantly, policy optimization can help to enhance the DRL performance. Numerous studies have used neural network function approximators--namely, asynchronous advantage actor-critic (A3C) [30], deep Q-learning [31], trust region policy optimization (TRPO) [32], and proximal policy optimization (PPO) [33].