# Adaptive Coordinated Variable Speed Limit between Highway Mainline and On-Ramp with Deep Reinforcement Learning

Authors: Ming Cheng; Chenghao Zhang; Hui Jin; Ziming Wang; Xiaoguang Yang
Journal: Journal of Advanced Transportation
 (2022)

---

## Abstract

Highway merging bottleneck is challenged with serious traffic conflicts between on-ramp and mainline vehicles, causing significant capacity drop and drastic speed changes. The paper proposes an adaptive coordinated variable speed limit model to manage highway speed of on-ramp and mainline continuous sections without priority to mainline. That helps to remove the speed difference between the vehicles from on-ramp and mainline flooding into the merging zone, and to sustain actual traffic density close to critical density to counteract capacity drop as indicated with macroscopic fundamental diagram. The method of deep reinforcement learning based on deep deterministic policy gradient is employed to solve the proposed model with a row of continuous control variables. Simulation platform with VISSIM 5.3 is established, and the proposed method can enhance traffic flow through the merging zone by around 10% and 19% under static and dynamic demand, respectively, in addition to reduced density and speed variation by around 30%. This research provides insights into the management of highway capacity so as to secure traffic efficiency and reliability for the merging zone.


---

## Body

## 1. Introduction

Highway congestion has been increasingly emphasized to prevent or alleviate its detrimental effect on traffic mobility, safety, and environment [1]. Highway congestion is mostly attributed to the merging or diverging around the on-ramps and off-ramps, traffic speed of which can be much lower than that of the mainline. Thus drastic speed changes can be caused, in addition to aggressive lane changing and weaving. That hinders the traffic flow on highway mainline, and causes recurrent highway bottlenecks. Considering that off-ramp are always closely related to the connected surface road, off-ramp bottleneck can be more complex [2]. Therefore, this research targets at on-ramp bottleneck, where vehicles frequently decelerate or stop to wait before squeezing onto the mainline.Traffic of mainline and on-ramp can be coordinated with the combined strategies of variable speed limit (VSL) and ramp-metering so as to adjust highway flow into the merging zone. VSL issues dynamic speed limit with variable signs, depending on traffic conditions [3, 4]. Ramp-metering adjusts vehicle flow rate such that mainline density remains below the critical value to prevent traffic breakdown [5]. For example, VSL and ramp-metering are combined into model predictive control by incorporating dynamic speed limits on the mainline into the original on-ramp model of METANET [6], so as to address ramp-metering disadvantage of failing to prevent congestion upstream of the merging zone [7]. Papamichai et al. [8] combines VSL with ramp-metering in a discrete optimal control programming, where excessive on-ramp queue is treated with a penalty added to the objective. Assuming predetermined ramp-metering rate for on-ramp, a model predictive control is developed for VSL over a finite time horizon to address the recurrent highway on-ramp bottleneck, which is found to enhance traffic flow by 12.8% and reduce total travel time by 31.8% [9]. Then driver acceptance is taken into the coordination between VSL and ramp-metering upstream of and at the merging zone, which enhances bottleneck throughput significantly [10]. Recent research also attempts to introduce the connected and automated vehicles into the traffic control and management of highway merging zone, where microscopic vehicular trajectory is coordinated to enhance traffic efficiency [11, 12].Despite rich research on the coordination of mainline VSL and ramp-metering, the existing study generally assigns priority to mainline at the cost of delaying or stopping on-ramp vehicles. Such management strategy has been found to account for nearly half of highway accidents at the congested merging zone, where rear-end crashes are most frequent due to drastic deceleration [13]. In addition, sideswipe and angle highway collisions are also found to be significantly related to the speed difference between highway mainline and on-ramp [14]. Statistical analysis has validated that increasing ramp vehicle volume assists in reducing crashes, while increasing mainline vehicle volume tends to increase crashes. Thus, it is important to enhance ramp traffic efficiency [15]. That is consistent to the early empirical research that suggests to remove the criteria of setting ramp design speed to be 50th percentile of freeway design speed, so as to allow more freedom to ramp vehicles [16].Therefore, the research is encouraged to apply VSL to both on-ramp and mainline for their homogenization, instead of suppressing on-ramp and prioritizing mainline traffic. This highway management strategy is expected to alleviate the speed difference between mainline and on-ramp traffic, reducing on-ramp vehicle stops and shockwaves at the merging zone. That helps to maintain highway capacity and to reduce traffic fluctuations. To tune finely the variable speed limit, a set of continuous decision variables should be solved along the time horizon. Thus an efficient solution method is called for that is capable of finding the optimal control scheme under traffic dynamics.Recently, artificial intelligence has attracted increasing attention, where intelligent agents are defined to perceive environment, predict system evolution, and take proactive actions for maximal reward [17]. Advanced application of artificial intelligence includes the improvement of object tracking against occlusion [18], image inpainting against structure disconnecting with improved total variation minimization [19] or structural disorder with multilevel attention progression mechanism [20], and multiscale superresolution image with feature map attention mechanism [21]. When it comes to highway management, reinforcement learning is widely adopted for complex modelling solution. For example, a reinforcement learning based VSL control is proposed to reduce total travel time near a freeway recurrent bottleneck [22] and to smooth vehicle conflicts for reduced crashes [23]. Single agent Q learning was implemented on ramp-metering, which is found to reduce total network travel time by 17% compared to ALINEA algorithm [24]. Q-learning was also applied to VSL optimization to reduce total travel time of both mainline and on-ramp by 49.3% and 21.8% under stable and dynamic traffic demand, respectively [22]. Further, deep Q-learning is applied to VSL of highway mainline with on-ramp bottlenecks, reducing total travel time by 26% to 67% with stable demand, and 21% to 70% with dynamic demand [25]. To select continuous actions instead of discrete ones, deep deterministic policy gradient (DDPG) is proposed to learn competitive policies [26]. For example, updated research has applied DDPG learning model to VSL control against spatially dynamic speed limit zones based on vehicle position and speed [27].Therefore, research gap is identified on highway merging zone management, where previous research assigns priority to mainline over on-ramps, causing traffic shocks and conflicts upstream of the bottleneck area. To this end, the research proposes an adaptive coordinated VSL (ACVSL) to explore the potential of applying VSL to highway mainline and on-ramp, guiding vehicles of both to adjust to the same speed before the merging zone [28]. The proposed speed limit is deduced from fundamental diagram with traffic density close to the critical density that corresponds to road capacity. Thus capacity drop at the merging zone can be reduced under the objective of maximal vehicle throughput, to which the emerging solution method of deep reinforcement learning is employed for efficient control scheme.For this paper, the main contributions are as follows. (1) a novel ramp management strategy is proposed to remove mainline priority over on-ramp, where both are managed with VSL in an adaptive coordinated way. Thus equal speed can be achieved between mainline and on-ramp immediately upstream of the merging zone. That helps to alleviate traffic shockwaves at the merging bottleneck and to anticipate capacity drop. (2) Error state is structured to reflect the real-time difference between actual and critical traffic density, based on which speed control parameters are developed to update VSL of both mainline and on-ramp. Thus traffic density at the merging zone can be stabilized around the critical density to counteract road capacity drop. (3) DDPG-based deep reinforcement learning is established to solve the proposed model with continuous control parameters, which is calibrated and validated with simulation platform for enhanced vehicle throughput and alleviated speed fluctuations.The remaining of the paper is organized as follows. Section2 establishes ACVSL model for the continuous highway sections of mainline and on-ramp till the merging bottleneck. It develops nonlinear feedback to narrow the difference between actual and critical traffic density to sustain bottleneck capacity. Section 3 develops the deep reinforcement learning method with DDPG to solve the proposed ACVSL under critic-actor framework. Simulation and tests follow in Section 4, validating that the proposed model and solution method can significantly enhance the efficiency of highway merging. Section 5 briefly concludes the paper, and points out future research directions.


## 2. Mathematical Modelling

Figure1 shows a typical highway section with on-ramp. The mainline and on-ramp are divided into Nm and Nr sections, respectively. Considering the limited ramp length, we set Nr=2. VSL on the upstream sections controls the vehicles getting into the next section, and finally into the merging zone.Figure 1
Configuration of highway mainline and on-ramp.Objective of the control model is given by:(1)max∑tqdt=∑t∑xqNxxt,so as to maximize vehicle throughput from the merging zone, and to retain highway capacity. Parameter qdt represents the total flow through the merging bottleneck at time t, and qNxxt means flow on the last section of x at time t, where x=r and x=m refer to the mainline and on-ramp, respectively.As indicated by the studies of [2, 29], lower speed limit can increase the critical density to a higher value. Figure 2 shows that the speed lower than the free-flow speed (i.e. 0<vd<vd0) assists in shifting the critical density to a higher value (i.e. 0<ρd0<ρd) in the fundamental diagram without decreasing bottleneck capacity. Moreover, the lower speed causes the slope of the under-critical part to be decreased, making the fundamental diagram get closer to a straight line, i.e. the red dash line vs. green dash line on the left-side, to alleviate the speed deviation around the critical density and the resultant shockwaves. Therefore, we may design an adaptive coordinated VSL control on the continuous sections of highway mainline and on-ramp, letting the traffic density at the merging section converge to the critical density derived from fundamental diagram. Thus the region of fluent traffic for the merging zone can be extended in comparison to the no-control scenarios, helping to attenuate shockwaves, and to sustain bottleneck capacity.Figure 2
Effects of lower speed on fundamental diagram.To this end, we define an error state to describe the deviation of the actual traffic density from its critical value, given by:(2)eixt=ρixt−ρd,c∀x,i,(3)edt=ρdt−ρd,c.For the sections upstream of and at the merging zone, respectively, at time stept on highway x. Parameter ρixt means the actual traffic density on section i of road x at time step t. Parameter ρdt represents the actual density of the merging zone at time step t, while ρd,c describes the critical density between fluent and congested traffic flow, and is assumed to be constant.Now we have developed an open-loop control system joining both mainline and on-ramp. In the following, the nonlinear feed-back mechanism for adaptive coordinated variable speed limit (ACVSL) is established.Control parameters aim to adjust the guide speed on the VSL signs upstream of the merging zone, given by:(4)λixt+1=qNxxt−v¯ixt+1ρixtLi+1xei+1xt∀x,i≤Nx−1,so as to maintain the density of the merging zone around the critical value, and to counteract capacity drop. Notation λixt+1 represents the control parameter on section i of highway x at time step (t+1), qNxxt describes the traffic flow of section Nx, v¯ixt+1 means the intended guide speed that will be demonstrated on the VSL sign at time step (t+1), and Li+1x represents the length of section i+1 on x. Therefore, on the right-side of Eq. (4), the numerator is the flow difference between section Nx and section i, and the denominator is the vehicle count in section i+1 required to fill its density gap from the critical density at time t of highway x [30]. Eq. (4) can be reformulated to represent the intended guide speed v¯ixt+1, given by:
(5)v¯ixt+1=−λixt+1Li+1xei+1xt+vNxxtρNxxtρixt∀x,i≤Nx−1Thus the gap between actual density and critical density can be alleviated to promote efficient and stable traffic flow. Therefore, once control parameterλixt+1 is determined, the intended guide speed v¯ixt+1 can be calculated with Eq. (5).When it comes to the last sections either on the mainline or on-ramp, VSL there is set the same as that of the merging zone to secure smooth convergence, given by:(6)v¯Nxxt=Vd∀x,where Vd means the theoretical speed of the merging zone from macroscopic fundamental diagram at critical density [30]. Thus speed conflicts can be alleviated between highway mainline and on-ramp immediately upstream of the merging zone.Additionally, to avoid dynamic oscillation in speed limit, the speed design by Zhang and Ioannou [31] is borrowed, given by:
(7)v~ixt=minv¯ixt5,vixt−1+Cv,vi−1xt+Cv,vmax∀x,i≤Nx−1,(8)Vixt=maxv~ixt,vmin,vixt−1−Cv,vi−1xt−Cv∀x,i≤Nx−1,where v~ixt is the maximum speed limit that can be demonstrated on VSL sign for section i of highway x at step t. Parameter v¯ixt5 rounds v¯ixt to its closest multiple of 5, Cv represents the maximum speed difference in VSL between successive control steps and highway sections; vmax is the upper bound of VSL; and Vixt returns the final speed delivered to VSL sign for section i of highway x after considering the lower bound of speed limit, with vmin being the lower bound of VSL.Thus the optimization model ACVSL is summarized with the object of Equation (1) and constraints of Equations (2)–(8), by optimizing a set of control parameters λixt. Table 1 summarizes the notation of the proposed model.Table 1
ACVSL notation list.
No.NotationExplanation1NxSection count on highway partx, where x=m and x=r refers to mainline and on-ramp, respectively.2qdtTotal flow through the merging bottleneck at timet3qNxxtTraffic flow of sectionNx at time t4eixtError state of sectioni on highway x upstream of the merging zone at time t5ρixtActual traffic density on sectioni of highway x at time t6ρd,cCritical density between fluent and congested traffic flow7edtError state of merging zone at time stept8ρdtActual density of merging zone at time stept9λixt+1Control parameter on sectioni of highway x at time step t+110v¯ixt+1Intended speed on VSL sign for sectioni on highway x at time step t+111Li+1xLength of sectioni+1 on highway x12vNxxtActual speed on sectionNx at time step t13VdTheoretical speed of the merging zone from macroscopic fundamental diagram at critical density14v~ixtMaximum speed limit demonstrated on VSL sign for sectioni of x at time step t15vixt−1Actual speed on sectioni of highway x at time step t−116CvMaximum speed difference in VSL between successive time steps and sections17VixtFinal speed delivered to VSL sign after considering the lower bound of speed limit for sectioni on highway x at time step t18vmaxUpper bound of VSL19vminLower bound of VSL


## 3. Solution Method

In this section, we target at optimizing the vector of control parametersλixt from Eq. (4), a set of multi-dimensional continuous variables. The control parameters λixt indicate the ratio of the flow difference to the gap between current density and critical density for section i of highway x at time t, determining the VSL values in turn as in Eq. (5), so as to adjust the count of vehicles moving downward. Thus we may coordinate VSL of both mainline and on-ramp so as to adapt to real-time traffic state with optimized control parameters λixt, sustaining critical traffic density that is beneficial to bottleneck flow.DDPG-based deep reinforcement learning method first collects traffic statest, given by:
(9)st=vt,ρt,where vt and ρt refer to the vectors of traffic speed and density over highway sections at time t, which are of different physical units and normalized for effective learning, given by:
(10)vt=v1xt,v2xt,⋯,vNx−1xt,vdtvfx,(11)ρt=ρ1xt,ρ2xt,⋯,ρNx−1xt,ρdtρjamx,with vfx and ρjamx being free flow speed and jam density on highway section x, respectively.The proposed method entails the training of a software-controlled agent to take actionat in response to the current system state st for reward rt. Action sets the value of the control parameter λixt at the beginning of each control time step, given by:
(12)at=λ1xt,λ2xt,⋯,λNxxtλmax,where λmax means the upper bound of control variables for normalization to secure the action variables in the interval 0,1, a necessity for the input of neural network. Reward function aims to maximize highway bottleneck outflow, given by:
(13)rt=∫cTc+1TqdtQTdt,which is divided by the multiplication of Q and T for normalization, with Q and T being bottleneck capacity and step interval length, respectively. Reward at each step is achieved via actor network, which actually refers to control policy π, a function that maps system state to probability distribution with control actions. Critic network then develops a corresponding Q-function to represent the accumulated discounted reward if action at is taken for state st and strategy π is followed at step t+1 onwards, given by:
(14)Qπst,at=Ert+γQπst+1,πst+1st,at,where Qπ represents the Q value obtained under strategy π, and γ is a discount factor of interval 0,1. Thus the accumulated reward is kept a finite quantity where the temporally-proximate rewards are more heavily weighted than are the distant ones. To obtain the optimum policy π∗ for maximum EQπst,πst, the proposed network is customized for parameterized actors and critics, i.e. πsθπ and Qs,aθQ.Specifically, the training epochs tune the actors and critics alternately, first by updating the function approximatorsθQ to satisfy Equation (14), and then by updating θπ with a policy gradient defined by:
(15)E∇aQs,aθQa=πs∙∇θππsθπ.Transition pairst,at,st+1,rt is stored in the replay memory at the end of each control step and is sampled uniformly at random for state update. Figure 3 illustrates the actor-critic framework employed to establish the relationships between traffic states, action agents, and reward function, so as to address the problem with continuous action space of parameter λix. Referring to the limited dimension of input vt,ρt, i.e. 2 Nm+Nr−2, one hidden layer is constructed for both critic and actor network.Figure 3
Schematic illustration of the critic and actor network.Note: DDPG-based deep reinforcement learning method adopts exploration epochs before training. That is to overcome local minimum, a major challenge of learning in continuous action spaces. Thus the procedure of exploration is treated independently from the learning algorithm, so as to globally explore for the optimal strategy. Algorithm1 summarizes the proposed solution method.Algorithm 1: DDPG Algorithm.
Initialize critic networkQs,aθQ and actor network πsθπInitialize target critic networkQs,aθQ′ and target network πsθπ′Initialize prioritized experience replay memoryRWhile not converge doObserve system statestSelect actionat=μsθμ+N for the current state st following ε-greedy exploration policyObserve transition pairst,at,st+1,rt, and store in RSample a mini-batchst,at,st+1,rt of size N from RSetyi=ri+γQ′si+1,πsi+1θπ′θQ′Update critic network by minimizing:∑iyi−Qsi+1,siθQ2Update actor with the sampled policy gradient:∇θπJ∼1/N∑i∇aQs,aθQs=si,a=πsi∇θππsθπsiUpdate target networkθπ′⟵τθπ+1−τ·θπ′θQ′⟵τθQ+1−τ·θQ′


## 4. Simulation and Tests

In this section, we design and evaluate the performance of the proposed ACVSL model and solution method for a highway bottleneck with on-ramp. After validation, the proposed method is applied to extensive scenarios with varying traffic demand.


### 4.1. Set-up

The proposed ACVSL model and solution algorithm are developed and tested with MATLAB 2020, which is connected to simulation platform VISSIM 5.3 for traffic states collection. Specifically, the files of Layout.ini and Net.inp input road network information and other traffic parameters into VISSIM 5.3, respectively. Figure4 shows the workflow, where the controller observes the traffic state and reward from the environment of VISSIM simulator at each control step and returns the control parameters to the simulator with COM interface. Note the first 20-min simulation is ignored to allow vehicles to fill the convergence zone.Figure 4
The framework of ACVSL control platform.Highway mainline has 3 lanes, while on-ramp has a single lane. Assuming default traffic parameters of driving behaviour and traffic composition in VISSIM platform, capacity of mainline and on-ramp is tested and set 2000 and 1500 veh/h/ln, respectively. Other parameters of the proposed ACVSL model are summarized in Table2. Control step is set 2 min (i.e. T=2 min) for timely VSL update. That is, VSL signs are updated every 2 min so as to delicately control traffic state dynamics. Control parameter λix is specified in the range of [0,100], i.e. the action space.Table 2
Scenario parameters.
No.NotationValueNo.NotationValueNo.NotationValue1L1m500 m6L1r100 m11ρjamr144 veh/km/ln2L2m300 m7L2r100 m12λmax1003L3m200 m8vfm120 km/h13ρd80 veh/km/ln4L4m200 m9vfr=vmax80 km/h14vmin20 km/h5Ld100 m10ρjamm144 veh/km/ln15CvKm/hMeta-parameters of DDPG-based deep reinforcement learning are selected from repeated experiment, and the set with the best performance is adopted. Reward discount factorγ is set 0.85. Update probability during the training period is 20%. The size of replay memory and mini-batch is set 4000 and 400, respectively.


### 4.2. Training

We consider the following two scenarios to train the proposed method of DDPG-based deep reinforcement learning for the developed ACVSL model.(1)
The scenario with static demand, which is set 5400 and 1400 veh/h for mainline and on-ramp, respectively(2)
The scenario with dynamic demand, which follows the curve in Figure5 for mainline and on-ramp, respectively. That is, demand increases and decreases for the first and last 60 min, respectively. Sample data employed in the training process is summarized in Table 3Figure 5
Dynamic flow demand for DDPG exploration and training.Table 3
Sample data of traffic dynamics.
Training dataTesting dataNo.Interval (min)Mainline demand (veh/h/ln)On-ramp demand (veh/h/ln)Mainline demand (veh/h/ln)On-ramp demand (veh/h/ln)10-213408091507100922-413638351529103534-613878641553106446-814138951579109558-10144192916071129610-12147096516371165712-141502100316691203814-161536104317021243916-1815711085173712851018-2016071129177413291120-2216451175181213751222-2416851222185114221324-2617251270189114701426-2817651318193215181528-3018061368197315681630-3218471417201416171732-3418881465205416651834-3619281513209417131936-3819661560213317322038-4020041604217017742140-4220391647220617972242-4420721686223918062344-4621031723226918452446-4821301756229718722548-5021541785232119072650-5221751810234219262752-5421921830235819512854-5622041845237119702956-5822131856238019813058-602217186123841993Each epoch of both exploration and training persists 2-hour simulation windows. With 2-minute control interval, each epoch of the exploration and training procedure is thus divided into 60 steps. A total of 30 epochs are adopted in the exploration, while 30 and 70 epochs in the training for the scenarios with static demand and dynamic demand, respectively, where learning rateτ is set 0.01 and 0.02, due to the difficulty of capturing demand changes in the latter scenario.Figure6 shows the outcomes of exploration and training epochs, as measured by the resulting bottleneck flow vs. epoch number under static and dynamic demand, respectively. In the exploration epoch, the series of control parameters is stochastically selected without optimization. Thus the bottleneck volume varies greatly. In comparison, in the training epoch, control parameters are iteratively updated to tune actors and critics alternatively, where bottleneck volume exceeds that with the best control scheme from exploration epochs (i.e. EX-ACVSL) at the 7th and 8th epoch under the static and dynamic scenario, respectively. The optimal control schemes from the proposed deep reinforcement learning (i.e. DRL-ACVSL) are obtained at the 24th and 65th epoch, beating the best control scheme from the exploration epoch EX-ACVSL by 13% and 19%, respectively.Figure 6
Bottleneck flow acquired from exploration and training procedure in different scenarios. (a) Static scenario and (b) dynamic scenario.
(a)(b)


### 4.3. Test and Calibration

Test and calibration of the trained controller entails a 2 h simulation under static and dynamic scenarios, respectively. The static case is characterized with the demand of 5500 veh/h on the mainline and 1500 veh/h on the ramp. The dynamic case deals with the demand given in Figure7, data trend of which input to the proposed ACVSL method is similar to that of Table 3.Figure 7
Dynamic flow demand for DDPG test.The proposed scheme from the developed method DRL-ACVSL is compared with the strategies that adopts the optimal control parameters from the exploration epochs (i.e. EX-ACVSL), and that of without control (i.e. Do-nothing). Table4 summarizes the performance of DRL-ACVSL scheme from training together with EX-ACVSL selected from the exploration epoch as well as the Do-nothing strategy. It is observed that DRL-ACVSL scheme competes with EX-ACVSL and Do-nothing with higher and more stable traffic flow at the merging zone, where the variation of traffic density and speed is also reduced to promote stable traffic flow. Specifically, average bottleneck flow is enhanced with DRL-ACVSL strategy by 7% and 10% under static demand, while by 14% and 19% under dynamic demand, compared to EX-ACVSL and Do-nothing strategies, respectively. Such enhancement competes the existing research that reports flow increase by 4.7% [32]. DRL-ACVSL competence is also reflected with the fact that standard deviation of traffic density is reduced by 17% and 27% under static vs. 23% and 32% under dynamic demand and standard deviation of traffic speed is reduced by 25% and 33% under static vs. 9% and 9% under dynamic demand against EX-ACVSL and Do-nothing strategies, respectively.Table 4
Performance comparison of DRL-ACVSL, EX-ACVSL, and Do-nothing strategies.
ScenarioStatic demandDynamic demandStrategyDRL-ACVSLEX-ACVSLDo-nothingDRL-ACVSLEX-ACVSLDo-nothingTraffic flow (veh/h)Avg.4,2913,9983,9094,8184,2084,041Med.4,4824,0253,7564,8564,4003,945Std. DV517468660610656688Traffic density (veh/km)Avg.10813016811594156Med.10913417711990162Std. DV222730233034Traffic speed (km/h)Avg.413225444728Med.423022424925Std. DV689101111Figure8 demonstrates the dynamics of traffic flow for highway merging bottleneck under three strategies, it is observed that DRL-ACVSL scheme manages to smooth vehicle outflow and keep it at a higher level than that with EX-ACVSL scheme, though the latter also promotes stable vehicle throughput compared to the Do-nothing scheme. Moreover, DRL-ACVSL performs better under static demand than dynamic demand, bringing higher vehicle throughput, and better smoothing flow fluctuations. Another interesting finding is that, under dynamic demand, DRL-ACVSL manages to enhance traffic flow compared to the static scenario, because the changing demand allows better exploitation of highway capacity with gradually increasing or decreasing demand, instead of crowding into the bottleneck constantly.Figure 8
Bottleneck flow under different control strategies. (a) Static scenario and (b) dynamic scenario.
(a)(b)Traffic density in Figure9 shows that DRL-ACVSL is efficient in reducing and stabilizing traffic density at the merging zone under static scenario. In this case, DRL-ACVSL competes with EX-ACVSL and Do-nothing strategies, where significant ups and downs in traffic density are frequently observed. When it comes to the dynamic case, the advantage of DRL-ACVSL reducing density dynamics is kept, though density comes to a higher level compared to EX-ACVSL strategy. Therefore, DRL-ACVSL is validated to keep stable traffic density and reduce traffic shocks, promoting traffic efficiency and safety. Note traffic density is also related to the error state (see Figure 10) between actual and critical traffic density, where the latter is set constant based on fundamental diagram with speed limit. Thus Figure 10 shows similar trend to that of Figure 9.Figure 9
Bottleneck density under different control strategies (a) Static scenario and (b) dynamic scenario.
(a)(b)Figure 10
Bottleneck error state under different control strategies. (a) Static scenario and (b) dynamic scenario.
(a)(b)Figure11 comes to traffic speed of the merging zone. It is observed that DRL-ACVSL can enhance traffic speed and reduce speed oscillation to a higher degree than EX-ACVSL, while the Do-nothing strategy performs least satisfying. Moreover, the gap between DRL-ACVSL and EX-ACVSL is enlarged when it comes to the subsequent steps, showing that the proposed solution can well trace traffic dynamics under static demand. When it comes to the dynamic scenario, EX-ACVSL outperforms DRL-ACVSL with higher speed, though DRL-ACVSL manages to further reduce speed deviation. The Do-nothing strategy is still the worst with respect to speed average and variation.Figure 11
Bottleneck speed under different control strategies. (a) Static scenario and (b) dynamic scenario.
(a)(b)


### 4.4. Numerical Analyses

Extensive numerical analyses are conducted for the proposed method under variable traffic demands. Figure12 summarizes the results from the numerical analysis, where dynamic demand increases from half static demand to one and half at the same difference for the first hour, and decreases in the same style for the second hour. It is observed that the proposed DRL-ACVSL is efficient in enhancing bottleneck throughput against varying traffic demand. Specifically, with static demand, DRL-ACVSL always competes with EX-ACVSL and Do-nothing strategies, with DRL-ACVSL advantages strengthened when on-ramp demand increases, though the latter to a lesser degree. In contrast, the Do-nothing strategy brings decreased throughput when on-ramp demand increases, implicating the intensified conflicts between mainstream and on-ramp vehicles. Similar trend is observed under dynamic demand, except that vehicle throughput is generally at a higher level compared to static scenarios. Thus the proposed model and solution method is validated to be capable of adapting to various traffic conditions for enhanced bottleneck efficiency.Figure 12
Traffic throughput with different control strategies under varying traffic demand.Thus the proposed method has the potential to better enhance the capacity of the merging zone with increasing on-ramp demand. That can be explained with on-ramp vehicle density getting closer to the intended critical density to well sustain capacity of the merging zone for maximum throughput. In comparison, the increased demand of mainline decreases the vehicle flow of the merging zone. The findings are consistent to the previous research that increasing on-ramp flow and reducing mainline flow contributes to enhance bottleneck capacity [15] with a new attempt to coordinate the variable speed limits of both mainline and on-ramp.


## 4.1. Set-up

The proposed ACVSL model and solution algorithm are developed and tested with MATLAB 2020, which is connected to simulation platform VISSIM 5.3 for traffic states collection. Specifically, the files of Layout.ini and Net.inp input road network information and other traffic parameters into VISSIM 5.3, respectively. Figure4 shows the workflow, where the controller observes the traffic state and reward from the environment of VISSIM simulator at each control step and returns the control parameters to the simulator with COM interface. Note the first 20-min simulation is ignored to allow vehicles to fill the convergence zone.Figure 4
The framework of ACVSL control platform.Highway mainline has 3 lanes, while on-ramp has a single lane. Assuming default traffic parameters of driving behaviour and traffic composition in VISSIM platform, capacity of mainline and on-ramp is tested and set 2000 and 1500 veh/h/ln, respectively. Other parameters of the proposed ACVSL model are summarized in Table2. Control step is set 2 min (i.e. T=2 min) for timely VSL update. That is, VSL signs are updated every 2 min so as to delicately control traffic state dynamics. Control parameter λix is specified in the range of [0,100], i.e. the action space.Table 2
Scenario parameters.
No.NotationValueNo.NotationValueNo.NotationValue1L1m500 m6L1r100 m11ρjamr144 veh/km/ln2L2m300 m7L2r100 m12λmax1003L3m200 m8vfm120 km/h13ρd80 veh/km/ln4L4m200 m9vfr=vmax80 km/h14vmin20 km/h5Ld100 m10ρjamm144 veh/km/ln15CvKm/hMeta-parameters of DDPG-based deep reinforcement learning are selected from repeated experiment, and the set with the best performance is adopted. Reward discount factorγ is set 0.85. Update probability during the training period is 20%. The size of replay memory and mini-batch is set 4000 and 400, respectively.


## 4.2. Training

We consider the following two scenarios to train the proposed method of DDPG-based deep reinforcement learning for the developed ACVSL model.(1)
The scenario with static demand, which is set 5400 and 1400 veh/h for mainline and on-ramp, respectively(2)
The scenario with dynamic demand, which follows the curve in Figure5 for mainline and on-ramp, respectively. That is, demand increases and decreases for the first and last 60 min, respectively. Sample data employed in the training process is summarized in Table 3Figure 5
Dynamic flow demand for DDPG exploration and training.Table 3
Sample data of traffic dynamics.
Training dataTesting dataNo.Interval (min)Mainline demand (veh/h/ln)On-ramp demand (veh/h/ln)Mainline demand (veh/h/ln)On-ramp demand (veh/h/ln)10-213408091507100922-413638351529103534-613878641553106446-814138951579109558-10144192916071129610-12147096516371165712-141502100316691203814-161536104317021243916-1815711085173712851018-2016071129177413291120-2216451175181213751222-2416851222185114221324-2617251270189114701426-2817651318193215181528-3018061368197315681630-3218471417201416171732-3418881465205416651834-3619281513209417131936-3819661560213317322038-4020041604217017742140-4220391647220617972242-4420721686223918062344-4621031723226918452446-4821301756229718722548-5021541785232119072650-5221751810234219262752-5421921830235819512854-5622041845237119702956-5822131856238019813058-602217186123841993Each epoch of both exploration and training persists 2-hour simulation windows. With 2-minute control interval, each epoch of the exploration and training procedure is thus divided into 60 steps. A total of 30 epochs are adopted in the exploration, while 30 and 70 epochs in the training for the scenarios with static demand and dynamic demand, respectively, where learning rateτ is set 0.01 and 0.02, due to the difficulty of capturing demand changes in the latter scenario.Figure6 shows the outcomes of exploration and training epochs, as measured by the resulting bottleneck flow vs. epoch number under static and dynamic demand, respectively. In the exploration epoch, the series of control parameters is stochastically selected without optimization. Thus the bottleneck volume varies greatly. In comparison, in the training epoch, control parameters are iteratively updated to tune actors and critics alternatively, where bottleneck volume exceeds that with the best control scheme from exploration epochs (i.e. EX-ACVSL) at the 7th and 8th epoch under the static and dynamic scenario, respectively. The optimal control schemes from the proposed deep reinforcement learning (i.e. DRL-ACVSL) are obtained at the 24th and 65th epoch, beating the best control scheme from the exploration epoch EX-ACVSL by 13% and 19%, respectively.Figure 6
Bottleneck flow acquired from exploration and training procedure in different scenarios. (a) Static scenario and (b) dynamic scenario.
(a)(b)


## 4.3. Test and Calibration

Test and calibration of the trained controller entails a 2 h simulation under static and dynamic scenarios, respectively. The static case is characterized with the demand of 5500 veh/h on the mainline and 1500 veh/h on the ramp. The dynamic case deals with the demand given in Figure7, data trend of which input to the proposed ACVSL method is similar to that of Table 3.Figure 7
Dynamic flow demand for DDPG test.The proposed scheme from the developed method DRL-ACVSL is compared with the strategies that adopts the optimal control parameters from the exploration epochs (i.e. EX-ACVSL), and that of without control (i.e. Do-nothing). Table4 summarizes the performance of DRL-ACVSL scheme from training together with EX-ACVSL selected from the exploration epoch as well as the Do-nothing strategy. It is observed that DRL-ACVSL scheme competes with EX-ACVSL and Do-nothing with higher and more stable traffic flow at the merging zone, where the variation of traffic density and speed is also reduced to promote stable traffic flow. Specifically, average bottleneck flow is enhanced with DRL-ACVSL strategy by 7% and 10% under static demand, while by 14% and 19% under dynamic demand, compared to EX-ACVSL and Do-nothing strategies, respectively. Such enhancement competes the existing research that reports flow increase by 4.7% [32]. DRL-ACVSL competence is also reflected with the fact that standard deviation of traffic density is reduced by 17% and 27% under static vs. 23% and 32% under dynamic demand and standard deviation of traffic speed is reduced by 25% and 33% under static vs. 9% and 9% under dynamic demand against EX-ACVSL and Do-nothing strategies, respectively.Table 4
Performance comparison of DRL-ACVSL, EX-ACVSL, and Do-nothing strategies.
ScenarioStatic demandDynamic demandStrategyDRL-ACVSLEX-ACVSLDo-nothingDRL-ACVSLEX-ACVSLDo-nothingTraffic flow (veh/h)Avg.4,2913,9983,9094,8184,2084,041Med.4,4824,0253,7564,8564,4003,945Std. DV517468660610656688Traffic density (veh/km)Avg.10813016811594156Med.10913417711990162Std. DV222730233034Traffic speed (km/h)Avg.413225444728Med.423022424925Std. DV689101111Figure8 demonstrates the dynamics of traffic flow for highway merging bottleneck under three strategies, it is observed that DRL-ACVSL scheme manages to smooth vehicle outflow and keep it at a higher level than that with EX-ACVSL scheme, though the latter also promotes stable vehicle throughput compared to the Do-nothing scheme. Moreover, DRL-ACVSL performs better under static demand than dynamic demand, bringing higher vehicle throughput, and better smoothing flow fluctuations. Another interesting finding is that, under dynamic demand, DRL-ACVSL manages to enhance traffic flow compared to the static scenario, because the changing demand allows better exploitation of highway capacity with gradually increasing or decreasing demand, instead of crowding into the bottleneck constantly.Figure 8
Bottleneck flow under different control strategies. (a) Static scenario and (b) dynamic scenario.
(a)(b)Traffic density in Figure9 shows that DRL-ACVSL is efficient in reducing and stabilizing traffic density at the merging zone under static scenario. In this case, DRL-ACVSL competes with EX-ACVSL and Do-nothing strategies, where significant ups and downs in traffic density are frequently observed. When it comes to the dynamic case, the advantage of DRL-ACVSL reducing density dynamics is kept, though density comes to a higher level compared to EX-ACVSL strategy. Therefore, DRL-ACVSL is validated to keep stable traffic density and reduce traffic shocks, promoting traffic efficiency and safety. Note traffic density is also related to the error state (see Figure 10) between actual and critical traffic density, where the latter is set constant based on fundamental diagram with speed limit. Thus Figure 10 shows similar trend to that of Figure 9.Figure 9
Bottleneck density under different control strategies (a) Static scenario and (b) dynamic scenario.
(a)(b)Figure 10
Bottleneck error state under different control strategies. (a) Static scenario and (b) dynamic scenario.
(a)(b)Figure11 comes to traffic speed of the merging zone. It is observed that DRL-ACVSL can enhance traffic speed and reduce speed oscillation to a higher degree than EX-ACVSL, while the Do-nothing strategy performs least satisfying. Moreover, the gap between DRL-ACVSL and EX-ACVSL is enlarged when it comes to the subsequent steps, showing that the proposed solution can well trace traffic dynamics under static demand. When it comes to the dynamic scenario, EX-ACVSL outperforms DRL-ACVSL with higher speed, though DRL-ACVSL manages to further reduce speed deviation. The Do-nothing strategy is still the worst with respect to speed average and variation.Figure 11
Bottleneck speed under different control strategies. (a) Static scenario and (b) dynamic scenario.
(a)(b)


## 4.4. Numerical Analyses

Extensive numerical analyses are conducted for the proposed method under variable traffic demands. Figure12 summarizes the results from the numerical analysis, where dynamic demand increases from half static demand to one and half at the same difference for the first hour, and decreases in the same style for the second hour. It is observed that the proposed DRL-ACVSL is efficient in enhancing bottleneck throughput against varying traffic demand. Specifically, with static demand, DRL-ACVSL always competes with EX-ACVSL and Do-nothing strategies, with DRL-ACVSL advantages strengthened when on-ramp demand increases, though the latter to a lesser degree. In contrast, the Do-nothing strategy brings decreased throughput when on-ramp demand increases, implicating the intensified conflicts between mainstream and on-ramp vehicles. Similar trend is observed under dynamic demand, except that vehicle throughput is generally at a higher level compared to static scenarios. Thus the proposed model and solution method is validated to be capable of adapting to various traffic conditions for enhanced bottleneck efficiency.Figure 12
Traffic throughput with different control strategies under varying traffic demand.Thus the proposed method has the potential to better enhance the capacity of the merging zone with increasing on-ramp demand. That can be explained with on-ramp vehicle density getting closer to the intended critical density to well sustain capacity of the merging zone for maximum throughput. In comparison, the increased demand of mainline decreases the vehicle flow of the merging zone. The findings are consistent to the previous research that increasing on-ramp flow and reducing mainline flow contributes to enhance bottleneck capacity [15] with a new attempt to coordinate the variable speed limits of both mainline and on-ramp.


## 5. Conclusions

This paper proposes an adaptive coordinated variable speed limit (ACVSL) control strategy for highway bottleneck with on-ramp, where the priority to mainline over on-ramp is removed to alleviate the speed difference upstream of the merging zone and to improve highway merging efficiency. In this endeavour, error state is developed to indicate the gap between the actual and critical traffic density, with the latter derived from fundamental diagram that corresponds to road capacity. Thus the smaller the error state, the higher traffic flow, and the shifting can be reduced between fluent and congested traffic states. In the following, nonlinear feedback mechanism is established to adjust speed limits along the control horizon.To solve the proposed model, deep reinforcement learning algorithm is developed, which is trained with DDPG using a critic-actor network to finely tune the continuous control variables. With traffic state collected and control reward evaluated, exploration and training epochs search the feasible domain and establish the control scheme. Experiment platform with VISSIM is then employed to test the proposed method, followed with numerical analyses. After calibration and training, extensive scenarios are established for method validation. It is found that the proposed scheme is capable of enhancing traffic flow through the bottleneck by 10% and 19% under static and dynamic demand, respectively, compared to the Do-nothing strategy. That competes the existing literature with respect to the improvement in vehicle throughput, validating the efficiency of the proposed model. Moreover, the proposed scheme manages to reduce the variation of traffic density and speed by around 30%, helping to stabilize traffic states and reduce traffic shockwaves. Extensive numerical analyses further confirm the advantage of the proposed method to improve bottleneck capacity especially when on-ramp demand increases.This research adds to the existing literature with coordinated speed guide between mainline and on-ramp, providing insights for responsive agencies into highway on-ramp bottleneck management that prioritize mainline over ramps can be unnecessary. Thus highway merging bottleneck can be better addressed to relieve congestion and to reduce accident. Limitation of the research is threefold. First, the performance in pollution emissions and fuel consumption should be further explored. Second, detailed analysis has not been scratched on how mainline and on-ramp vehicles cooperate for acceptable gap and smooth merging. Third, the effect of acceleration lane at the merging zone on traffic states has not been considered. The on-going research is to extend the proposed method to include more ramps so as to explore the potential of adaptive coordinated variable speed limit for consecutive highway ramps.



---

*Source: 2435643-2022-09-29.xml*