# A Deep Reinforcement Learning Approach for Ramp Metering Based on Traffic Video Data

Authors: Bing Liu; Yu Tang; Yuxiong Ji; Yu Shen; Yuchuan Du
Journal: Journal of Advanced Transportation
 (2021)

---

## Abstract

Ramp metering that uses traffic signals to regulate vehicle flows from the on-ramps has been widely implemented to improve vehicle mobility of the freeway. Previous studies generally update signal timings in real-time based on predefined traffic measurements collected by point detectors, such as traffic volumes and occupancies. Comparing with point detectors, traffic cameras—which have been increasingly deployed on road networks—could cover larger areas and provide more detailed traffic information. In this work, we propose a deep reinforcement learning (DRL) method to explore the potential of traffic video data in improving the efficiency of ramp metering. Vehicle locations are extracted from the traffic video frames and are reformed as position matrices. The proposed method takes the preprocessed video data as inputs and learns the optimal control strategies directly from the high-dimensional inputs. A series of simulation experiments based on real-world traffic data are conducted to evaluate the proposed approach. The results demonstrate that, in comparison with a state-of-the-practice method, the proposed DRL method results in (1) lower travel times in the mainline, (2) shorter vehicle queues at the on-ramp, and (3) higher traffic flows downstream of the merging area. The results suggest that the proposed method is able to extract useful information from the video data for better ramp metering controls.


---

## Body

## 1. Introduction

Ramp metering uses traffic signals to regulate vehicle flows from on-ramps to the mainline of the freeway. It alleviates the negative impacts of “capacity drop” resulting from massive merging behaviors and reduces the total time spent in the traffic system [1, 2]. Several field tests have demonstrated the effectiveness of ramp metering in terms of throughput, vehicle-miles-traveled, vehicle-hours-traveled, and travel time reliability [3–6]. For instance, a large-scale experiment conducted in Minneapolis-Saint Paul found that when the ramp meters were turned off, freeway capacity decreased by 9%, travel time increased by 22%, and crashes increased by 26% [7].Most of the ramp metering methods are driven by traffic measurements, such as volumes, queue lengths, and occupancies, obtained from point detectors (e.g., inductive loop detectors) [8, 9]. These measurements are predefined and only reflect partial information regarding traffic operations. Recently, traffic cameras have been increasingly deployed on road networks for monitoring traffic operations and detecting illegal driving behaviors. Compared with point detectors, traffic cameras cover larger areas and provide more detailed traffic information, such as vehicle locations, vehicle speeds, and headways between vehicles.We propose a deep reinforcement learning (DRL) method to explore the potential of traffic video data in improving the efficiency of ramp metering. The proposed method does not rely on traditional traffic measurements from loop detectors. Instead, it uses traffic video frames as inputs and learns the optimal control strategies directly from high-dimensional visual inputs by taking advantage of the special neutral structures in deep learning, which are capable of automatically extracting high-level features from raw data. The effectiveness of the proposed method is demonstrated in comparison with a state-of-the-practice method in a real-world case study.This paper is organized as follows. A brief review of the related literature is presented first, and then the methodology is described, followed by the demonstration in a real-world case study. The final section summarizes the paper and discusses possible directions for future research.


## 2. Related Works

The signal timing plans for ramp metering could be fixed or traffic responsive. Wattleworth [10] firstly formulated a linear programming model to optimize a fixed signal timing plan for ramp metering based on historical traffic volume data. Responsive approaches adapt to traffic flow fluctuations by updating signal timings in response to real-time traffic measurements. The responsive approaches can be classified as rule-based, optimization-based, and RL-based.The rule-based approaches update signal timings in real-time according to specific rules. Masher et al. [11] proposed a feedforward ramp metering algorithm, aiming at keeping the downstream traffic flow below the capacity. Papageorgiou et al. [8] proposed a feedback ramp metering algorithm, which is named ALINEA, based on the occupancy obtained by inductive loop detectors downstream of the merging area. The ALINEA regulates the inflows from the on-ramp in the scheme of closed-loop control so that the detected occupancy would approach a predefined target value. The ALINEA was further extended by considering more traffic measurements, such as queue lengths at the on-ramp and traffic volumes in the mainline [12]. Wang et al. [9] proposed the proportional-integral-based ALINEA (i.e., PI-ALINEA), considering the cases where a bottleneck with a capacity lower than that of the merging area is present further downstream of the merging area. They demonstrated numerically that the PI-ALINEA performs better than the original ALINEA. The ALINEA has been incorporated in several coordinated ramp metering algorithms, such as METALINE and HERO [13, 14].The optimization-based approaches optimize signal timings based on real-time traffic data. The model predictive control framework, which considers the interactions between ramp metering and future traffic states, is often employed to predict traffic evolution for proactive traffic controls. Nevertheless, the models used to describe traffic dynamics may lead to nonlinear control problems, which brings difficulty in finding the optimal signal timings [15–17]. In addition, the effectiveness of the ramp metering strategy depends on the degree of the fitness of the models to the actual traffic dynamics.The RL-based approaches search for a policy that determines the signal timings based on the current traffic state. Existing RL-based ramp metering approaches were mainly developed based on value-based RL methods, such as the Q-learning algorithm. The Q-learning algorithm evaluates the action value of each state-action pair and improves the policy with a heuristic search. The actions usually refer to the metering rate or signal phase selection. The states are represented by traffic measurements, such as traffic density and volume. The downstream and upstream traffic flow, ramp queue length, and traffic density are often selected to define the control reward [18–21]. The RL has also been introduced for coordinated ramp metering [22, 23]. Note that existing RL methods for ramp metering were driven by traditional traffic measurements. To the best of our knowledge, no studies have attempted to automatically extract information from traffic videos and learn the optimal control strategies directly from the visual inputs for ramp metering. Some studies have considered traffic video data as inputs in the RL framework for intersection signal controls [24–26]. Nevertheless, the elements in the RL framework, such as the action, reward, and state, for ramp metering and intersection signal controls are different.Compared with the methods based on traditional measurement, the proposed strategy shows great potential in improving ramp metering performance due to the spatiotemporal information contained in the video data. However, the high-dimension data increase the computational complexity in the training phase. Therefore, a well-designed RL framework and algorithm are necessary to guarantee the effectiveness of the proposed method.


## 3. Methodology

We propose a DRL method for local ramp metering based on traffic video data. The proposed method adopts a flexible two-phase control scheme, which is illustrated in Figure1. A policy trained from the DRL determines whether the phase in the next time step is green or red at a fixed time step L based on the current traffic state. The adopted control scheme is conceivably more flexible than the conventional “stop-and-go” scheme with fixed red and green phases flashing alternately. To reduce the computation burden, traffic video frames are used as control inputs. Vehicle locations from raw images are extracted to reconstruct visual representations in the DRL method [24].Figure 1
Two-phase control scheme for ramp metering.Figure2 illustrates the ramp metering problem in the general scheme of RL. The ramp meter acts as the agent to interact with the environment, namely, the traffic system including the mainline and the on-ramp. The arrows represent the direction of information interaction in the system. The ramp meter takes certain control action at based on current traffic state st. Then, the traffic system responds to the control action with the state transition from st to st+1. The ramp meter obtains reward rt that quantifies the effect of the control action at given state st. The interaction process is assumed to be a Markov decision process [27]. That is, given the current state st and action at, the following state st+1 and reward rt are independent of previous states and actions.Figure 2
Ramp metering in the general scheme of RL.The RL method aims at training the agent (Ramp meter) to find an optimal policyπ∗ that maximizes the discounted cumulative reward Gt after time step t. To achieve that goal, one approach is to learn the optimal action-value function Q∗s,a, which is defined by the following:(1)Q∗s,a=maxπEπGt|st=s,at=a,where π is a policy mapping traffic states to control actions. Once Q∗s,a is found, the action to be taken can be obtained by the following:(2)π∗s=argmaxa′Q∗s,a′.The process of learning policy from the action-value functionQ is termed Q-learning. The key step in Q-learning is to estimate Q∗s,a. To better capture the traffic features in video data. We adopt a deep neural network termed Q-network to approximate Q∗s,a. The neural network contains several convolutional layers and fully connected layers. The convolutional layers extract traffic features from the video frames, and the fully connected layers estimate the Q value based on the output of convolutional layers.


### 3.1. State, Action, and Reward Representation

#### 3.1.1. Action

The ramp meter takes two possible actions at fixed decision time stepL to determine whether the phase in the next time step is green (G) or red (R), namely, action set A=G,R.


#### 3.1.2. State

Figure3 illustrates the process of obtaining visual representations from the raw images to represent traffic states. For simplification purpose, the vehicle is represented by a location point xi,yi extracted from raw images. Considering a control area of size X×Y as shown in Figure 3(a), traffic state at time step t is represented by a position matrix mt∈ℝX×Y:(3)mtxi,yi=UV,i∈It,where UV is a constant and It refers to the set of vehicles in the control area at time step t.Figure 3
State representation: (a) control area; (b) position matrixmt–1; (c) traffic state st.The position matrix additionally contains the information of the signal states:(4)mtxR,yR=UG,ifat−1=G,UR,ifat−1=R,where xR,yR is the position of signal light in images and UG and UR are constants to represent actions execute in time step t−1.The other elements in matrixmt are set to zero:(5)mtx,y=0,x,y∉xi,yi|i∈It∪xR,yR.Since one matrix is insufficient to describe vehicle dynamics, we stack the matrices of consecutiveN time steps to represent the state st:(6)st=mt−N−1,…,mt.For computation efficiency, down-sampling is introduced before the matrix stack to reduce the input dimensions and maintain sufficient information about vehicle positions.


#### 3.1.3. Reward

The reward is critical since it motivates the agent to approach the objective. Vehicle travel times through the system are good measurements of vehicle mobility. However, they are not suitable rewards since they are greatly influenced by previous actions and cannot well quantify the value of the current action. Alternatively, the speed in the merging area and the queue length at the on-ramp are adopted to define the reward. The rewardrt after action at is defined by the following:(7)rt=μv¯t+ωq¯t,μ>0,ω<0,where v¯t and q¯t represent the average speed in the merging area and the average queue length at the on-ramp during time step t and t+1, respectively. Positive μ and negative ω denote the reward weights for the speed and the queue length, respectively. The defined rewards balance the priority needs between improving vehicle mobility in the mainline and reducing vehicle delays at the on-ramp.


### 3.2. Algorithm

The training problem can be viewed as the regression problem with the loss function defined by the following:(8)ℒwi=Es,aQ∗s,a−Qs,a,wi2,where wi denotes the weights in the deep neural network after the ith update and Q∗s,a is approximated by the following:(9)Q∗s,a≈rt+1+γmaxa′Qst+1,a′,wi,st=s,at=a.The gradient-based algorithm Adam [28] is used to update wi to minimize the loss function.Although Q-network could handle large state space, it raises the problem of instability in training [29]. Two approaches have been proposed to resolve the problem, illustrated in Figure 4. The experience replay [30] stores the data prepared for training into the replay buffer and samples these data uniformly for training the agent. It contributes to making the training data independently and identically distributed. The target network [29] is introduced for a stable estimation of target value Q∗s,a. When Q-network is being trained, the target network is freezing. Its parameters w− are updated by copying wi in Q-network with a certain frequency. The loss function is redefined as follows:(10)ℒ1wi=Es,art+1+γmaxa′Q0st+1,a′,w−−Q0s,a,wi2,where Q0s,a,wi represents the prediction values of the action-value function.Figure 4
Approaches for stabilizing the training process in deep Q-learning: (a) relay buffer; (b) target network.
(a)(b)A multitask learning strategy is implemented in the training process to improve learning efficiency. While training the ramp metering Q-network, the mean speed in the merging area and the queue length at the on-ramp are predicted simultaneously. The fully connected layer in the Q-network in multitask learning is extended to two layers to calculate the Q-value and predict the speed and queue length, respectively. Another loss function to measure the prediction accuracy is defined as follows:(11)ℒ2wi=Es,aQ1s,a,wi−v2+Q2s,a,wi−u2,where Q1s,a,wiandQ2s,a,wi represent the prediction values of the action-value function, mean speed, and queue length, respectively. u and v refer to the ground truth of the speed and queue length, respectively. The total loss function is as follows:(12)ℒwi=ℒ1wi+λℒ2wi.The algorithm for training the ramp metering policy is presented in Algorithm1. The ramp meter takes new action with the ε-greedy strategy, meaning that the agent does not always adopt the best action derived from current Qs,a,w. Doing so is helpful to explore unknown policies that may be better.Algorithm 1: Deep Q-learning for ramp metering.
Input: Batch size k, learning rate η, decision step size L, exploration rate ε,Freezing intervalF, number of training episode N,Number of decision time stepsT in one episodeFill the replay buffer randomlyInitialize the action-value functionQ with random weights w.Initialize the target networkQ–with weights w−=wf←0for episode =  1,…,Nfort =  1,…,TObserve statest and choose action at by ε-greedy strategyExecute actionat, obtain reward rt, state st+1Store transition <st, at, rt, vt, ut, st+1> into replay bufferUniformly samplek transitions from replay bufferCompute the gradient∇wℒ1+λℒ2Update weightsw in Q-network by Adam with learning rate ηf←f+1end ifif mod (f, F) = = 0Update target networkQ−:w−←wend ifend forend for


## 3.1. State, Action, and Reward Representation

### 3.1.1. Action

The ramp meter takes two possible actions at fixed decision time stepL to determine whether the phase in the next time step is green (G) or red (R), namely, action set A=G,R.


### 3.1.2. State

Figure3 illustrates the process of obtaining visual representations from the raw images to represent traffic states. For simplification purpose, the vehicle is represented by a location point xi,yi extracted from raw images. Considering a control area of size X×Y as shown in Figure 3(a), traffic state at time step t is represented by a position matrix mt∈ℝX×Y:(3)mtxi,yi=UV,i∈It,where UV is a constant and It refers to the set of vehicles in the control area at time step t.Figure 3
State representation: (a) control area; (b) position matrixmt–1; (c) traffic state st.The position matrix additionally contains the information of the signal states:(4)mtxR,yR=UG,ifat−1=G,UR,ifat−1=R,where xR,yR is the position of signal light in images and UG and UR are constants to represent actions execute in time step t−1.The other elements in matrixmt are set to zero:(5)mtx,y=0,x,y∉xi,yi|i∈It∪xR,yR.Since one matrix is insufficient to describe vehicle dynamics, we stack the matrices of consecutiveN time steps to represent the state st:(6)st=mt−N−1,…,mt.For computation efficiency, down-sampling is introduced before the matrix stack to reduce the input dimensions and maintain sufficient information about vehicle positions.


### 3.1.3. Reward

The reward is critical since it motivates the agent to approach the objective. Vehicle travel times through the system are good measurements of vehicle mobility. However, they are not suitable rewards since they are greatly influenced by previous actions and cannot well quantify the value of the current action. Alternatively, the speed in the merging area and the queue length at the on-ramp are adopted to define the reward. The rewardrt after action at is defined by the following:(7)rt=μv¯t+ωq¯t,μ>0,ω<0,where v¯t and q¯t represent the average speed in the merging area and the average queue length at the on-ramp during time step t and t+1, respectively. Positive μ and negative ω denote the reward weights for the speed and the queue length, respectively. The defined rewards balance the priority needs between improving vehicle mobility in the mainline and reducing vehicle delays at the on-ramp.


## 3.1.1. Action

The ramp meter takes two possible actions at fixed decision time stepL to determine whether the phase in the next time step is green (G) or red (R), namely, action set A=G,R.


## 3.1.2. State

Figure3 illustrates the process of obtaining visual representations from the raw images to represent traffic states. For simplification purpose, the vehicle is represented by a location point xi,yi extracted from raw images. Considering a control area of size X×Y as shown in Figure 3(a), traffic state at time step t is represented by a position matrix mt∈ℝX×Y:(3)mtxi,yi=UV,i∈It,where UV is a constant and It refers to the set of vehicles in the control area at time step t.Figure 3
State representation: (a) control area; (b) position matrixmt–1; (c) traffic state st.The position matrix additionally contains the information of the signal states:(4)mtxR,yR=UG,ifat−1=G,UR,ifat−1=R,where xR,yR is the position of signal light in images and UG and UR are constants to represent actions execute in time step t−1.The other elements in matrixmt are set to zero:(5)mtx,y=0,x,y∉xi,yi|i∈It∪xR,yR.Since one matrix is insufficient to describe vehicle dynamics, we stack the matrices of consecutiveN time steps to represent the state st:(6)st=mt−N−1,…,mt.For computation efficiency, down-sampling is introduced before the matrix stack to reduce the input dimensions and maintain sufficient information about vehicle positions.


## 3.1.3. Reward

The reward is critical since it motivates the agent to approach the objective. Vehicle travel times through the system are good measurements of vehicle mobility. However, they are not suitable rewards since they are greatly influenced by previous actions and cannot well quantify the value of the current action. Alternatively, the speed in the merging area and the queue length at the on-ramp are adopted to define the reward. The rewardrt after action at is defined by the following:(7)rt=μv¯t+ωq¯t,μ>0,ω<0,where v¯t and q¯t represent the average speed in the merging area and the average queue length at the on-ramp during time step t and t+1, respectively. Positive μ and negative ω denote the reward weights for the speed and the queue length, respectively. The defined rewards balance the priority needs between improving vehicle mobility in the mainline and reducing vehicle delays at the on-ramp.


## 3.2. Algorithm

The training problem can be viewed as the regression problem with the loss function defined by the following:(8)ℒwi=Es,aQ∗s,a−Qs,a,wi2,where wi denotes the weights in the deep neural network after the ith update and Q∗s,a is approximated by the following:(9)Q∗s,a≈rt+1+γmaxa′Qst+1,a′,wi,st=s,at=a.The gradient-based algorithm Adam [28] is used to update wi to minimize the loss function.Although Q-network could handle large state space, it raises the problem of instability in training [29]. Two approaches have been proposed to resolve the problem, illustrated in Figure 4. The experience replay [30] stores the data prepared for training into the replay buffer and samples these data uniformly for training the agent. It contributes to making the training data independently and identically distributed. The target network [29] is introduced for a stable estimation of target value Q∗s,a. When Q-network is being trained, the target network is freezing. Its parameters w− are updated by copying wi in Q-network with a certain frequency. The loss function is redefined as follows:(10)ℒ1wi=Es,art+1+γmaxa′Q0st+1,a′,w−−Q0s,a,wi2,where Q0s,a,wi represents the prediction values of the action-value function.Figure 4
Approaches for stabilizing the training process in deep Q-learning: (a) relay buffer; (b) target network.
(a)(b)A multitask learning strategy is implemented in the training process to improve learning efficiency. While training the ramp metering Q-network, the mean speed in the merging area and the queue length at the on-ramp are predicted simultaneously. The fully connected layer in the Q-network in multitask learning is extended to two layers to calculate the Q-value and predict the speed and queue length, respectively. Another loss function to measure the prediction accuracy is defined as follows:(11)ℒ2wi=Es,aQ1s,a,wi−v2+Q2s,a,wi−u2,where Q1s,a,wiandQ2s,a,wi represent the prediction values of the action-value function, mean speed, and queue length, respectively. u and v refer to the ground truth of the speed and queue length, respectively. The total loss function is as follows:(12)ℒwi=ℒ1wi+λℒ2wi.The algorithm for training the ramp metering policy is presented in Algorithm1. The ramp meter takes new action with the ε-greedy strategy, meaning that the agent does not always adopt the best action derived from current Qs,a,w. Doing so is helpful to explore unknown policies that may be better.Algorithm 1: Deep Q-learning for ramp metering.
Input: Batch size k, learning rate η, decision step size L, exploration rate ε,Freezing intervalF, number of training episode N,Number of decision time stepsT in one episodeFill the replay buffer randomlyInitialize the action-value functionQ with random weights w.Initialize the target networkQ–with weights w−=wf←0for episode =  1,…,Nfort =  1,…,TObserve statest and choose action at by ε-greedy strategyExecute actionat, obtain reward rt, state st+1Store transition <st, at, rt, vt, ut, st+1> into replay bufferUniformly samplek transitions from replay bufferCompute the gradient∇wℒ1+λℒ2Update weightsw in Q-network by Adam with learning rate ηf←f+1end ifif mod (f, F) = = 0Update target networkQ−:w−←wend ifend forend for


## 4. Case Study

### 4.1. Case Set-Up

The proposed method is evaluated in SUMO, an open-source microscopic traffic simulator [31]. The simulation is performed in the freeway connecting Qingdao and Huangdao through a tunnel in Shandong, China, as shown in Figure 5. The one-lane on-ramp and the upstream three-lane mainline merge. And the freeway gradually reduces to three lanes in the downstream segments.Figure 5
Road network considered in the case study.The simulation considers the road network covering the on-ramp, the upstream and downstream mainlines, and the merging area. The speed limits in the mainline and at the on-ramp are 80 and 40 km/h, respectively. The simulation parameters regarding driving behaviors are calibrated based on empirical data. The simulation time is 7:50 a.m. to 9:00 a.m. Figure6 presents the 10-min traffic volumes from 7:50 a.m. to 9:00 a.m. in the mainline and at the on-ramp.Figure 6
Traffic volumes in the mainline and the on-ramp.The agent’s observation in DRL covers the simulated road network. Three consecutive frames of the traffic video in the controlled area are needed as the observation input of each decision time step. After zooming and down-sampling, the size of one observation is4×512 in pixels. Three consecutive observations are stacked to represent the traffic state. To accommodate the input size for ramp metering, the Q-network in Mnih et al. [30] is revised as Figure 7, which consists of three convolutional neural network (CNN) layers and one full connection (FC) layer for each output layer. Thus, there are about 2 million weights in the deep neural network that are stored permanently after training. The hyperparameters in the deep Q-learning for ramp metering are listed in Table 1.Figure 7
Structure of the deep neural network.Table 1
Key parameters in deep Q-learning for ramp metering.
ParametersValueState matrix dimension3,4,512Replay buffer sizeB2×105Training stepsT4200Learning rateη2.5×10−4Batch sizek32Exploration rateε0.1Freeze intervalF104Decision step sizeL4sLoss weightλ0.001Reward weightμ0.5Reward weightω−0.1For comparative purposes, two scenarios are also considered in the evaluation. The first scenario does not apply ramp metering, and the second scenario adopts the PI-ALINEA method. The PI-ALINEA method fixes the length of the green phase and determines the length of the red phase in the current signal cycle based on the length of the red phase in the previous cycle and the downstream occupancies in the previous and current cycles. The ramp metering raterk for k-th cycle is given by the following:(13)rk=rk−1+KRok−o^−KPok−1−ok,where ok and ok−1 represent the mean mainline occupancy during the k-th and (k + 1)-th cycle, respectively. o^ refers to the desired value for mainline occupancy, which is set to 16% in this study. KR=90 and KP=50 are two regulator parameters in PI-ALINEA, which are fine-tuned to minimize mean travel times in the traffic system. To accommodate the “stop-and-go” scheme, the green time for each cycle is fixed with time step size L and the red time for each cycle is given by rkL.


### 4.2. Evaluation and Comparison

The Q-network is trained in106 training frames, and each scenario is evaluated in 20 simulation experiments. The training process, presented in Figure 8, can be finished in less than 24 hours on a computing server with NVIDIA Quadro K620 GPU and Intel Xeon CPU E5-1650 (3.6 GHz, 6 cores). The simulations for the training phase and each experiment are initialized with different random seeds. The episode reward and mean travel time are both converged in 400,000 training frames and can maintain a stable value until the end of the training process.Figure 8
Convergence curves for (a) episode reward and (b) mean travel time.
(a)(b)Figure9 presents the Q1 (25%), Q2 (50%), and Q3 (75%) of the travel times in the mainline for three scenarios. The results demonstrate that ramp metering not only reduces vehicle travel times but also improves the stability of traffic flows in the mainline. The resulting travel times are close when the demand is low at the beginning (8:00–8:10). With the increase in the demand, the median travel time increases faster in the no-control scenario and reaches the maximum value of 4.25 min at 8 : 28. The maximum values of the median travel times resulting from the PI-ALINEA method and the DRL method are 3.75 min and 3.4 min, respectively, which are 11.7% and 20% lower than those in the no-control scenario. As the demand decreases, the travel times of the three scenarios decrease to similar levels. In addition, all the ramp metering methods narrow the interquartile ranges (Q3-Q1) of the travel times. Overall, the proposed DRL method results in shorter travel times with narrower travel time ranges than that of the fine-tuned PI-ALINEA method.Figure 9
Vehicle travel times in the mainline.Table2 presents mean vehicle travel times in 20 simulation experiments. It is found that the performance of the PI-ALINEA method is not always better than that of the no-control scenario. For example, compared with the no-control scenario, the PI-ALINEA method increases vehicle travel time by 10% in the 6th experiment. In contrast, the proposed DRL method consistently results in a shorter travel time than that of the no-control scenario, demonstrating the stable performance of the proposed method. On average, the DRL method reduces vehicle travel time by 13% when compared with the no-control scenario.Table 2
Mean travel times in 20 simulation experiments (s).
No.No controlsPI-ALINEADRL1186179 [−4%]171 [−8%]2203173 [−15%]171 [−16%]3217175 [−19%]177 [−18%]4227181 [−20%]181 [−20%]5202173 [−14%]177 [−12%]6178195 [+10%]173 [−3%]7185178 [−4%]174 [−6%]8226180 [−20%]173 [−23%]9175214 [+22%]169 [−3%]10206183 [−11%]175 [−15%]11188173 [−8%]167 [−11%]12211182 [−14%]170 [−19%]13219175 [−20%]172 [−21%]14170172 [+1%]170 [−0%]15187182 [−3%]175 [−6%]16216170 [−21%]179 [−17%]17177192 [+8%]174 [−2%]18228169 [−26%]172 [−25%]19190179 [−6%]175 [−8%]20194180 [−7%]170 [−12%]Figure10 presents the temporal-spatial distribution of the mean speeds along the mainline. The results further demonstrate the efficiency of the ramp metering and the better performance of the proposed method. In the no-control scenario, the minimum speed in the mainline is 12 m/s (43.2 km/h). The minimum speeds resulting from the PI-ALINEA method and the DRL method are 13.3% and 20% higher than those in the no-control scenario, respectively. The area with a speed lower than 15 m/s (54 km/h) is reduced with the PI-ALINEA method, which is further reduced when the DRL method is implemented.Figure 10
Temporal-spatial distribution of speed in (a) no-control; (b) PI-ALINEA; (c) DRL.
(a)(b)(c)Figure11 presents the Q1, Q2, and Q3 of the queue lengths at the on-ramp in three scenarios. The results reveal that ramp metering improves vehicle mobility in the mainline at the cost of delays of vehicles at the on-ramp. Without ramp metering, no vehicle queues are formed at the on-ramp. Both ramp metering methods result in vehicle queues at the on-ramp. Nevertheless, the queues produced by the proposed method are shorter than those produced by the PI-ALINEA method. When the PI-ALINEA method and the proposed DRL method are adopted, the maximum values of the Q3 of the queue lengths are 126 m and 67 m, respectively.Figure 11
Queue length at the on-ramp.Figure12 presents the mean queue lengths at the on-ramp and the red ratios (i.e., the proportion of the red phase in the simulation time) in each simulation for the two ramp metering methods. The queue length generally increases with the red ratio. However, compared with the PI-ALINEA method, the queues produced by the DRL method are shorter, suggesting that the proposed method could better utilize the green time.Figure 12
Relationship of queue length and red ratio for (a) PI-ALINEA and (b) DRL.Figure13 presents the queue lengths at the on-ramp and the occupancy downstream of the merging area for the two ramp metering methods. Figure 13(a) demonstrates that the queue length and the occupancy are highly correlated when the PI-ALINEA method is adopted. The results are understandable. According to Figure 12, a higher red ratio leads to long queue length. Given high occupancy, the PI-ALINEA method tends to adopt a long red phase, which would lead to long queues at the on-ramp. In contrast, Figure 13(b) shows that the correlation between the queue length and the occupancy is relatively low when the DRL method is implemented. The queue length over time is relatively stable, ranging from 25–50 m. Although the occupancy is low after 8 : 45, the ramp meter is still active. The results indicate that the proposed DRL method not only relies on downstream occupancy but other information extracted from raw data.Figure 13
Queue length and occupancy for (a) PI-ALINEA and (b) DRL.
(a)(b)The better performance of the proposed DRL method is further revealed when considering traffic flows downstream of the merge area, which is demonstrated in Figure14. In the beginning, traffic flows in three scenarios are close. During the peak period between 8 : 10–8 : 35, the flows resulting from the DRL method are higher and more stable, suggesting that the proposed DRL method could better alleviate the negative impacts of “capacity drop” resulting from massive merging behaviors. When traffic demand decreases during 8 : 40–9 : 00, traffic flows in the two ramp metering scenarios also decline. Nevertheless, traffic flows in the no-control scenario are still high during 8 : 45–8 : 55 since more vehicles are held in the system in the previous periods.Figure 14
Downstream volume in three scenarios.A perturbation analysis is conducted to evaluate the impact of information contained in video data. As shown in Figure15, the control area is divided into eight subareas. The sensors applied in PI-ALINEA are located in subarea 2. The information in the subarea is removed from original data to generate perturbed data for each subarea. The impact of missing information for each subarea is evaluated in 20 simulation experiments with perturbed data.Figure 15
Division subareas.Figure16 presents the box plot of travel time when the information of different subareas is removed, and the baseline is the result of experiments that take original data as input. Apart from subarea 2, the information from other subareas such as subarea 3 and 4 also influences the control performance obviously, which proves that the DRL-based controller takes more information from video data into consideration besides the measurements applied in PI-ALINEA.Figure 16
Travel time with perturbed input data.


## 4.1. Case Set-Up

The proposed method is evaluated in SUMO, an open-source microscopic traffic simulator [31]. The simulation is performed in the freeway connecting Qingdao and Huangdao through a tunnel in Shandong, China, as shown in Figure 5. The one-lane on-ramp and the upstream three-lane mainline merge. And the freeway gradually reduces to three lanes in the downstream segments.Figure 5
Road network considered in the case study.The simulation considers the road network covering the on-ramp, the upstream and downstream mainlines, and the merging area. The speed limits in the mainline and at the on-ramp are 80 and 40 km/h, respectively. The simulation parameters regarding driving behaviors are calibrated based on empirical data. The simulation time is 7:50 a.m. to 9:00 a.m. Figure6 presents the 10-min traffic volumes from 7:50 a.m. to 9:00 a.m. in the mainline and at the on-ramp.Figure 6
Traffic volumes in the mainline and the on-ramp.The agent’s observation in DRL covers the simulated road network. Three consecutive frames of the traffic video in the controlled area are needed as the observation input of each decision time step. After zooming and down-sampling, the size of one observation is4×512 in pixels. Three consecutive observations are stacked to represent the traffic state. To accommodate the input size for ramp metering, the Q-network in Mnih et al. [30] is revised as Figure 7, which consists of three convolutional neural network (CNN) layers and one full connection (FC) layer for each output layer. Thus, there are about 2 million weights in the deep neural network that are stored permanently after training. The hyperparameters in the deep Q-learning for ramp metering are listed in Table 1.Figure 7
Structure of the deep neural network.Table 1
Key parameters in deep Q-learning for ramp metering.
ParametersValueState matrix dimension3,4,512Replay buffer sizeB2×105Training stepsT4200Learning rateη2.5×10−4Batch sizek32Exploration rateε0.1Freeze intervalF104Decision step sizeL4sLoss weightλ0.001Reward weightμ0.5Reward weightω−0.1For comparative purposes, two scenarios are also considered in the evaluation. The first scenario does not apply ramp metering, and the second scenario adopts the PI-ALINEA method. The PI-ALINEA method fixes the length of the green phase and determines the length of the red phase in the current signal cycle based on the length of the red phase in the previous cycle and the downstream occupancies in the previous and current cycles. The ramp metering raterk for k-th cycle is given by the following:(13)rk=rk−1+KRok−o^−KPok−1−ok,where ok and ok−1 represent the mean mainline occupancy during the k-th and (k + 1)-th cycle, respectively. o^ refers to the desired value for mainline occupancy, which is set to 16% in this study. KR=90 and KP=50 are two regulator parameters in PI-ALINEA, which are fine-tuned to minimize mean travel times in the traffic system. To accommodate the “stop-and-go” scheme, the green time for each cycle is fixed with time step size L and the red time for each cycle is given by rkL.


## 4.2. Evaluation and Comparison

The Q-network is trained in106 training frames, and each scenario is evaluated in 20 simulation experiments. The training process, presented in Figure 8, can be finished in less than 24 hours on a computing server with NVIDIA Quadro K620 GPU and Intel Xeon CPU E5-1650 (3.6 GHz, 6 cores). The simulations for the training phase and each experiment are initialized with different random seeds. The episode reward and mean travel time are both converged in 400,000 training frames and can maintain a stable value until the end of the training process.Figure 8
Convergence curves for (a) episode reward and (b) mean travel time.
(a)(b)Figure9 presents the Q1 (25%), Q2 (50%), and Q3 (75%) of the travel times in the mainline for three scenarios. The results demonstrate that ramp metering not only reduces vehicle travel times but also improves the stability of traffic flows in the mainline. The resulting travel times are close when the demand is low at the beginning (8:00–8:10). With the increase in the demand, the median travel time increases faster in the no-control scenario and reaches the maximum value of 4.25 min at 8 : 28. The maximum values of the median travel times resulting from the PI-ALINEA method and the DRL method are 3.75 min and 3.4 min, respectively, which are 11.7% and 20% lower than those in the no-control scenario. As the demand decreases, the travel times of the three scenarios decrease to similar levels. In addition, all the ramp metering methods narrow the interquartile ranges (Q3-Q1) of the travel times. Overall, the proposed DRL method results in shorter travel times with narrower travel time ranges than that of the fine-tuned PI-ALINEA method.Figure 9
Vehicle travel times in the mainline.Table2 presents mean vehicle travel times in 20 simulation experiments. It is found that the performance of the PI-ALINEA method is not always better than that of the no-control scenario. For example, compared with the no-control scenario, the PI-ALINEA method increases vehicle travel time by 10% in the 6th experiment. In contrast, the proposed DRL method consistently results in a shorter travel time than that of the no-control scenario, demonstrating the stable performance of the proposed method. On average, the DRL method reduces vehicle travel time by 13% when compared with the no-control scenario.Table 2
Mean travel times in 20 simulation experiments (s).
No.No controlsPI-ALINEADRL1186179 [−4%]171 [−8%]2203173 [−15%]171 [−16%]3217175 [−19%]177 [−18%]4227181 [−20%]181 [−20%]5202173 [−14%]177 [−12%]6178195 [+10%]173 [−3%]7185178 [−4%]174 [−6%]8226180 [−20%]173 [−23%]9175214 [+22%]169 [−3%]10206183 [−11%]175 [−15%]11188173 [−8%]167 [−11%]12211182 [−14%]170 [−19%]13219175 [−20%]172 [−21%]14170172 [+1%]170 [−0%]15187182 [−3%]175 [−6%]16216170 [−21%]179 [−17%]17177192 [+8%]174 [−2%]18228169 [−26%]172 [−25%]19190179 [−6%]175 [−8%]20194180 [−7%]170 [−12%]Figure10 presents the temporal-spatial distribution of the mean speeds along the mainline. The results further demonstrate the efficiency of the ramp metering and the better performance of the proposed method. In the no-control scenario, the minimum speed in the mainline is 12 m/s (43.2 km/h). The minimum speeds resulting from the PI-ALINEA method and the DRL method are 13.3% and 20% higher than those in the no-control scenario, respectively. The area with a speed lower than 15 m/s (54 km/h) is reduced with the PI-ALINEA method, which is further reduced when the DRL method is implemented.Figure 10
Temporal-spatial distribution of speed in (a) no-control; (b) PI-ALINEA; (c) DRL.
(a)(b)(c)Figure11 presents the Q1, Q2, and Q3 of the queue lengths at the on-ramp in three scenarios. The results reveal that ramp metering improves vehicle mobility in the mainline at the cost of delays of vehicles at the on-ramp. Without ramp metering, no vehicle queues are formed at the on-ramp. Both ramp metering methods result in vehicle queues at the on-ramp. Nevertheless, the queues produced by the proposed method are shorter than those produced by the PI-ALINEA method. When the PI-ALINEA method and the proposed DRL method are adopted, the maximum values of the Q3 of the queue lengths are 126 m and 67 m, respectively.Figure 11
Queue length at the on-ramp.Figure12 presents the mean queue lengths at the on-ramp and the red ratios (i.e., the proportion of the red phase in the simulation time) in each simulation for the two ramp metering methods. The queue length generally increases with the red ratio. However, compared with the PI-ALINEA method, the queues produced by the DRL method are shorter, suggesting that the proposed method could better utilize the green time.Figure 12
Relationship of queue length and red ratio for (a) PI-ALINEA and (b) DRL.Figure13 presents the queue lengths at the on-ramp and the occupancy downstream of the merging area for the two ramp metering methods. Figure 13(a) demonstrates that the queue length and the occupancy are highly correlated when the PI-ALINEA method is adopted. The results are understandable. According to Figure 12, a higher red ratio leads to long queue length. Given high occupancy, the PI-ALINEA method tends to adopt a long red phase, which would lead to long queues at the on-ramp. In contrast, Figure 13(b) shows that the correlation between the queue length and the occupancy is relatively low when the DRL method is implemented. The queue length over time is relatively stable, ranging from 25–50 m. Although the occupancy is low after 8 : 45, the ramp meter is still active. The results indicate that the proposed DRL method not only relies on downstream occupancy but other information extracted from raw data.Figure 13
Queue length and occupancy for (a) PI-ALINEA and (b) DRL.
(a)(b)The better performance of the proposed DRL method is further revealed when considering traffic flows downstream of the merge area, which is demonstrated in Figure14. In the beginning, traffic flows in three scenarios are close. During the peak period between 8 : 10–8 : 35, the flows resulting from the DRL method are higher and more stable, suggesting that the proposed DRL method could better alleviate the negative impacts of “capacity drop” resulting from massive merging behaviors. When traffic demand decreases during 8 : 40–9 : 00, traffic flows in the two ramp metering scenarios also decline. Nevertheless, traffic flows in the no-control scenario are still high during 8 : 45–8 : 55 since more vehicles are held in the system in the previous periods.Figure 14
Downstream volume in three scenarios.A perturbation analysis is conducted to evaluate the impact of information contained in video data. As shown in Figure15, the control area is divided into eight subareas. The sensors applied in PI-ALINEA are located in subarea 2. The information in the subarea is removed from original data to generate perturbed data for each subarea. The impact of missing information for each subarea is evaluated in 20 simulation experiments with perturbed data.Figure 15
Division subareas.Figure16 presents the box plot of travel time when the information of different subareas is removed, and the baseline is the result of experiments that take original data as input. Apart from subarea 2, the information from other subareas such as subarea 3 and 4 also influences the control performance obviously, which proves that the DRL-based controller takes more information from video data into consideration besides the measurements applied in PI-ALINEA.Figure 16
Travel time with perturbed input data.


## 5. Conclusion and Future Research

This study proposes a DRL method for local ramp metering based on traffic video data. The proposed method learns optimal strategies directly from high-dimensional visual inputs, overcoming the reliance on the traditional traffic measurements. The better performance of the proposed method is demonstrated in a real-world case study. Compared with the traditional PI-ALINEA method, the proposed method results in lower travel times in the mainline and shorter vehicle queue lengths at the on-ramp and could better alleviate the negative impacts of “capacity drop” resulting from massive merging behaviors. The results suggest that the proposed DRL method is able to extract useful information from the video data for better ramp metering controls.It is convenient to deploy the proposed model in practice since the well-trained model is lightweight, and the input image data are accessible with the latest communication technologies. For future research, additional studies are suggested to conduct to investigate the stability of the proposed method under various traffic and weather circumstances. We assume that vehicle location information from the videos is accurate. Actually, the data quality may be disturbed by factors such as weather, vehicle sizes, and colors. It is valuable to design a ramp metering algorithm that is robust to those disturbances. In addition, abnormal events, such as accidents, are not considered in this study. Efficiently training the ramp metering to adapt to abnormal events remains to be investigated. Finally, extending the proposed method for coordinated ramp metering controls is worth pursuing.



---

*Source: 6669028-2021-10-31.xml*