# A Way to Automatically Generate Lane Level Traffic Data from Video in the Intersections

Authors: Zhongguo Yang; Sikandar Ali; Weilong Ding; Irshad Ahmed Abbasi; Muhammad Faizan Khan
Journal: Journal of Advanced Transportation
 (2021)

---

## Abstract

Lane level traffic data such as average waiting time and flow data at each turn direction not only enable navigation systems to provide users with more detailed and finer-grained information; it can also pave the way for future traffic congestion prediction. Although few studies considered extracting traffic flow data from a video at the lane level, it is not clear how many vehicles required turn left in fine-grained lanes during a fixed time. Many previous works focus on applying sensor data instead to videos to extract traffic flow. However, the reversible lanes and various shooting angles obstruct the progress of constructing a traffic data collection system. A framework is proposed to get these data in the intersection directly from a video and solve the problem of vehicle occlusion based on the delayed matching model. First, the different direction lanes are detected automatically by clustering trajectory data which are generated by tracking each vehicle. Experiments are conducted on urban intersections to show that our method can generate these traffic data effectively.


---

## Body

## 1. Introduction

In the era of big data, the data such as the waiting time, the turning type (going straight, turning left, and turning right) of vehicles, and traffic flow of each lane are important parts of traffic data. Although many intelligent traffic systems are applied, it is still difficult to answer the questions like how many vehicles will pass a certain intersection following a certain direction in the rush hour from the video directly. In general, getting these data using sensors or loops might be very expensive.These fine-grained traffic data on the road intersection and segments are very helpful for transport infrastructure, road design, traffic police resource allocation, and historical trend analysis. For instance, there are two equal distance ways fromA to B; an average waiting time in the intersection is useful when recommending travel ways for drivers. Besides, authorities in high-traffic public places such as stadiums, airports, exhibition halls, entertainment parks, shopping malls, museums, libraries, and bus/railway stations greatly benefit from a traffic big data collection tool to implement better traffic management to achieve low congestion and optimization of traffic control devices to achieve optimum travel times [1–3].As shown in Figure1, there are three fine-grained lanes in this intersection; the data of average waiting time and volume of each lane are beneficial for studying the traffic congestion and the saturation flow rate during the rush hour [4].Figure 1
The description of fine-grained traffic data.These data are especially valuable when constructing the intelligent traffic system. A typical application is that when the traffic flow in different lanes is known, managers could change the number of lanes in a certain direction in a fixed time automatically. Another application scenario is to adjust the red-blue light time which will increase the efficiency of road. No additional waiting time is needed if there are no vehicles in one direction. The red-blue light system could switch quickly and smartly.However, the lanes may change their direction, which is known as invertible lanes or tide lanes. Moreover, due to different shooting angles of intersection video, it is impossible to set all the lanes’ metadata manually in the software when we build an automatic data collection system.If we want the fine-grained traffic data from video, we should detect lanes automatically. Moreover, obtaining these traffic data through GPS and electronic police devices is time consuming and expensive. Also, not all the cameras in the street own the function of electronic police which recognizes and records the number plate of vehicles.On the contrary, with the quick development of video processing techniques, it is very economical and convenient to get these traffic data from the video directly. Although the object detection [5–7] or tracking problem [8, 9] had been well solved, there are still some problems needed for future study of complex traffic scene. For instance, as a machine could not understand traffic signs for each lane in the intersection, it is imperative to detect these lanes adaptively based on trajectory data.To solve the above problems, we propose a three-stage framework, including vehicle detection and tracking, lane detection, and traffic big data generating stages.In the stage of generating traffic big data, we propose a clustering method to estimate lanes in each intersection based on vehicle driving trajectory. To make the clustering task effective, the cycle of traffic phase switching is extracted from traffic flow and used for the initial partition trajectory into several small groups. Besides, an algorithm was proposed to calculate the waiting time for each turning type by computing the comparative speed of vehicles. Some experiments are conducted on intersection videos and the result shows that our method can generate these traffic big data through video automatically and effectively.The main contributions of the study are listed below:(1)
A framework is proposed to generate traffic big data from video in the intersection based on vehicle tracking and lane detection(2)
A clustering method is applied to estimate lanes in each intersection based on vehicle driving trajectory(3)
An algorithm is proposed to calculate the waiting time for each turning type by computing the comparative speed of vehiclesThe organization of this article is as follows. In Section2, some related works are reviewed. Section 3 demonstrates our main method in detail. The experiment and analysis are shown in Section 4. The conclusion is given in Section 5.


## 2. Related Works

The usage of video-based traffic data has been an area of interest in intelligent transportation systems for the past few decades. Especially, lane detection, vehicle trajectory, and traffic flow in the intersection are widely studied.Most of the researchers [10–12] who studied the traffic congestion are based on data collected from GPS or sensors. While some researchers try to collect data from video based on various methods, some of them are summarized below.


### 2.1. Video-Based Vehicle Trajectory Extraction Methods

To overcome the inadequacy of trajectory data, many researchers have attempted to develop computer vision algorithms to collect traffic data in the last two decades. Vehicle detection and tracking are the two main steps of traffic video analysis. The detection step separates the object of interest from the background and then recognizes the location and scale of the targeted object.The tracking step identifies the object of interest in consecutive frames to trace object movements. The vehicle detection and tracking methods are categorized in Table1.Table 1
Video-based vehicle trajectory extraction methods.
WorkKey methodCharacteristicVehicle detectionMotion-based [13]They are restricted to the constant brightness assumption, resulting in the incomplete detection of moving objectsModel-based (R-CNN [14], Faster-CNN [5], SSD [6], YOLO [7]They need large training datasets, complicated designs of deep convolutional network structures, and long training times to achieve optimal performance for targeted scenesVehicle trackingKalman filter (KF) [15]Tracked objects can still drift over time due to the error accumulation from the first to the last frameCNN-based trackers [16]It needs a large amount of training data, the significant manual-processing efforts in preparing the training datasetsAs shown in Table1, many well-known and good methods are developed to detect and track vehicles from video. In our work, we just utilize these methods as basic technology. The main work in this study is to combine these algorithms and develop a framework to generate traffic big data. The key algorithm in our work is to estimate the traffic lane automatically and generate traffic parameters such as traffic flow and average waiting time.


### 2.2. Application Scenarios for Trajectory Extraction Methods

Although there are many works focus on video-based trajectory data extraction, only very few of them considered the intersections in city roads. More details are shown in Table2.Table 2
Application scenarios for trajectory extraction methods.
WorkData extractedApplication scenariosMontanino [17]Vehicle trajectoryHighways and fixed lanes.Krajewski [18]Vehicles’ positions and movementsGerman highwaysZhang [19]Reconstruct high-resolution vehicle trajectoriesHigh-angle traffic video comes from NGSIMDey and Kundu [1]Volume dataIntersections in the city.On the highway, the lanes are separated from each other and no cross connection between them. As a result, it is easy to get the number of lanes and compute the volume data from them. However, due to the complex traffic scenes and cross connection of these lanes in the intersections, the aforementioned works failed to cope with them. It is necessary to recognize different lanes first and then compute volumes’ data in the intersection. Besides, occlusions from adjacent lanes are insignificant in the highway traffic video, while it commonly happens on the urban intersections.The proposed work is most similar to the work of Dey and Kundu [1] which proposes a method to turn video into traffic data on the urban intersections. However, they marked labels A, B, C, and D as the entry and exit point of trajectory data and ignored all the directions in the intersection.Different from their work, we try to automatically detect the lanes including invertible lanes in the intersection and generate flow data for each fine-grained lane. Considering there may be multiple lanes for vehicles to turn left or turn right, so a fine-grained lane is defined to represent each lane in the intersection.


## 2.1. Video-Based Vehicle Trajectory Extraction Methods

To overcome the inadequacy of trajectory data, many researchers have attempted to develop computer vision algorithms to collect traffic data in the last two decades. Vehicle detection and tracking are the two main steps of traffic video analysis. The detection step separates the object of interest from the background and then recognizes the location and scale of the targeted object.The tracking step identifies the object of interest in consecutive frames to trace object movements. The vehicle detection and tracking methods are categorized in Table1.Table 1
Video-based vehicle trajectory extraction methods.
WorkKey methodCharacteristicVehicle detectionMotion-based [13]They are restricted to the constant brightness assumption, resulting in the incomplete detection of moving objectsModel-based (R-CNN [14], Faster-CNN [5], SSD [6], YOLO [7]They need large training datasets, complicated designs of deep convolutional network structures, and long training times to achieve optimal performance for targeted scenesVehicle trackingKalman filter (KF) [15]Tracked objects can still drift over time due to the error accumulation from the first to the last frameCNN-based trackers [16]It needs a large amount of training data, the significant manual-processing efforts in preparing the training datasetsAs shown in Table1, many well-known and good methods are developed to detect and track vehicles from video. In our work, we just utilize these methods as basic technology. The main work in this study is to combine these algorithms and develop a framework to generate traffic big data. The key algorithm in our work is to estimate the traffic lane automatically and generate traffic parameters such as traffic flow and average waiting time.


## 2.2. Application Scenarios for Trajectory Extraction Methods

Although there are many works focus on video-based trajectory data extraction, only very few of them considered the intersections in city roads. More details are shown in Table2.Table 2
Application scenarios for trajectory extraction methods.
WorkData extractedApplication scenariosMontanino [17]Vehicle trajectoryHighways and fixed lanes.Krajewski [18]Vehicles’ positions and movementsGerman highwaysZhang [19]Reconstruct high-resolution vehicle trajectoriesHigh-angle traffic video comes from NGSIMDey and Kundu [1]Volume dataIntersections in the city.On the highway, the lanes are separated from each other and no cross connection between them. As a result, it is easy to get the number of lanes and compute the volume data from them. However, due to the complex traffic scenes and cross connection of these lanes in the intersections, the aforementioned works failed to cope with them. It is necessary to recognize different lanes first and then compute volumes’ data in the intersection. Besides, occlusions from adjacent lanes are insignificant in the highway traffic video, while it commonly happens on the urban intersections.The proposed work is most similar to the work of Dey and Kundu [1] which proposes a method to turn video into traffic data on the urban intersections. However, they marked labels A, B, C, and D as the entry and exit point of trajectory data and ignored all the directions in the intersection.Different from their work, we try to automatically detect the lanes including invertible lanes in the intersection and generate flow data for each fine-grained lane. Considering there may be multiple lanes for vehicles to turn left or turn right, so a fine-grained lane is defined to represent each lane in the intersection.


## 3. Proposed Method

The process of our method is shown in Figure2, and it contains three stages: getting trajectory data, lane detection, and vehicle data generating.Figure 2
The process of big traffic data generating.


### 3.1. Vehicle Tracking and Trajectory Data Collection

Vehicle detection in every frame of the video and the detection method are based on CNN (convolutional neural network) algorithm; specifically, Yolo [7] is used. For each vehicle, the position is detected in the picture. With this vehicle position information, these vehicle are tracked by feature matching; the process is illustrated in Figure 3. As shown in Figure 3, different features from a different modal are fused to enable more accurate vehicle matching and tracking based on original tracking algorithm.Figure 3
The matching features in the tracking process.In the tracking process, three different features are utilized to be a representation of one vehicle. We extend a single trajectory by matching the end of the trajectory and the newly detected vehicle by comparing their feature distance. Here, the feature applied includes the appearance features, histogram features, and motion features. The appearing features are image similarity and histogram features are large granularity of similarity from the view of colour distribution. Nevertheless, the motion feature means the degree of position overlapping.


### 3.2. Occlusions at Urban Intersection

In urban intersections, occlusions commonly happen for the low shooting angle of cameras; an example is demonstrated in Figure4. Hence, a delayed matching model (DMM) is designed in the process of getting trajectories from videos. The diagram of DMM is shown in Figure 5.Figure 4
The occlusion in the urban intersection.Figure 5
The delayed matching model.As shown in Figure4, two vehicles are turning left in the intersection. In the middle of the red lane, due to the camera angle, the minibus prevents the car from being seen in this view. Fortunately, with the two vehicles going by, they appear in the video again. Therefore, it is possible to connect the broken trajectory again based on the delayed matching technology.As demonstrated in Figure5, two collections will be maintained: one is matched trajectory and the other is an unmatched trajectory. One unmatched trajectory could be changed to a matched trajectory if it is matched with any vehicle in the matching process. In the delayed matching model, we fully take advantage of feature matching, which recognizing the same vehicle even when it they reappear after some time.


### 3.3. Lane Detection at Intersections

As lane direction may change in the scene of tide traffic, it is necessary to detect fine-grained lanes at different times of the day. Therefore, an adaptive algorithm is designed to detect lanes in the intersection. The proposed method consists of two steps to estimate lane, coarse-grained partition of trajectory data, and trajectory clustering method. An algorithm is proposed to estimate the lane in the intersection, as shown in Figure6.Figure 6
The process of lane detection.As shown in Figure6, the flow data of the intersection could be computed by simply counting the number of vehicles at a certain time. The extreme values of the flow data indicate the switching time points of the traffic phase, and due to the change in traffic direction, it will change following the switching of the traffic phase. Hence, the trajectory of vehicles could be divided into several groups, as shown in Figure 6.


#### 3.3.1. Coarse-Grained Partition of Trajectories

Usually, there are several phases at this intersection, as shown in Figure7.Figure 7
Recognized traffic phases.In different phases, vehicles of different directions will start to move orderly. It is possible to estimate the phase changes through traffic flow analysis. Furthermore, by tracking all the vehicles on the road, it is possible to get traffic flow data for all these lanes and directions. This peak and valley of flow data chart indicate the traffic phase changing time point, as shown in Figure8.Figure 8
Traffic flow distribution and phase switching diagram.As shown in Figure8, the horizontal axis is time; the vertical axis is the flow data of vehicles for all lanes. The phase switching time is the time when the flow changes drastically, so the simple extremum point of the flow can be used to judge the phase switching time. According to these switching times, trajectory data will be divided into several groups.


#### 3.3.2. Lane Detection Based on Clustering

The problem of lane recognition changes to clustering different groups of trajectories. DTW (dynamic time warping) is a sequence alignment method to find an optimal matching between two trajectories and measure the similarity without considering lengths and time ordering [20]:(1)WA,B=fmin1n∑i=1nai−bfi2,where A has n points and B has m points; all mappings fi,n⟶1,m should satisfy the requirements that f1=1,fn=m and fi≤fj, for all 1≤i≤j≤n.The positions of the estimated lane are average positions of one group of trajectories, and they are shown with green colour in Figure9.Figure 9
Trajectory data in different phases.
(a)(b)(c)It can be seen that the phase switching point based on the traffic flow extreme value computing can accurately separate the trajectories of adjacent phases.All the lanes estimated in the intersection are shown in Figure10. Although these lanes are easily separated from each other, it is still an important task considering the dynamic traffic state and invertible lanes existing. These lanes in different intersections could be recognized adaptively.Figure 10
Recognized lanes in an intersection.


## 3.1. Vehicle Tracking and Trajectory Data Collection

Vehicle detection in every frame of the video and the detection method are based on CNN (convolutional neural network) algorithm; specifically, Yolo [7] is used. For each vehicle, the position is detected in the picture. With this vehicle position information, these vehicle are tracked by feature matching; the process is illustrated in Figure 3. As shown in Figure 3, different features from a different modal are fused to enable more accurate vehicle matching and tracking based on original tracking algorithm.Figure 3
The matching features in the tracking process.In the tracking process, three different features are utilized to be a representation of one vehicle. We extend a single trajectory by matching the end of the trajectory and the newly detected vehicle by comparing their feature distance. Here, the feature applied includes the appearance features, histogram features, and motion features. The appearing features are image similarity and histogram features are large granularity of similarity from the view of colour distribution. Nevertheless, the motion feature means the degree of position overlapping.


## 3.2. Occlusions at Urban Intersection

In urban intersections, occlusions commonly happen for the low shooting angle of cameras; an example is demonstrated in Figure4. Hence, a delayed matching model (DMM) is designed in the process of getting trajectories from videos. The diagram of DMM is shown in Figure 5.Figure 4
The occlusion in the urban intersection.Figure 5
The delayed matching model.As shown in Figure4, two vehicles are turning left in the intersection. In the middle of the red lane, due to the camera angle, the minibus prevents the car from being seen in this view. Fortunately, with the two vehicles going by, they appear in the video again. Therefore, it is possible to connect the broken trajectory again based on the delayed matching technology.As demonstrated in Figure5, two collections will be maintained: one is matched trajectory and the other is an unmatched trajectory. One unmatched trajectory could be changed to a matched trajectory if it is matched with any vehicle in the matching process. In the delayed matching model, we fully take advantage of feature matching, which recognizing the same vehicle even when it they reappear after some time.


## 3.3. Lane Detection at Intersections

As lane direction may change in the scene of tide traffic, it is necessary to detect fine-grained lanes at different times of the day. Therefore, an adaptive algorithm is designed to detect lanes in the intersection. The proposed method consists of two steps to estimate lane, coarse-grained partition of trajectory data, and trajectory clustering method. An algorithm is proposed to estimate the lane in the intersection, as shown in Figure6.Figure 6
The process of lane detection.As shown in Figure6, the flow data of the intersection could be computed by simply counting the number of vehicles at a certain time. The extreme values of the flow data indicate the switching time points of the traffic phase, and due to the change in traffic direction, it will change following the switching of the traffic phase. Hence, the trajectory of vehicles could be divided into several groups, as shown in Figure 6.


### 3.3.1. Coarse-Grained Partition of Trajectories

Usually, there are several phases at this intersection, as shown in Figure7.Figure 7
Recognized traffic phases.In different phases, vehicles of different directions will start to move orderly. It is possible to estimate the phase changes through traffic flow analysis. Furthermore, by tracking all the vehicles on the road, it is possible to get traffic flow data for all these lanes and directions. This peak and valley of flow data chart indicate the traffic phase changing time point, as shown in Figure8.Figure 8
Traffic flow distribution and phase switching diagram.As shown in Figure8, the horizontal axis is time; the vertical axis is the flow data of vehicles for all lanes. The phase switching time is the time when the flow changes drastically, so the simple extremum point of the flow can be used to judge the phase switching time. According to these switching times, trajectory data will be divided into several groups.


### 3.3.2. Lane Detection Based on Clustering

The problem of lane recognition changes to clustering different groups of trajectories. DTW (dynamic time warping) is a sequence alignment method to find an optimal matching between two trajectories and measure the similarity without considering lengths and time ordering [20]:(1)WA,B=fmin1n∑i=1nai−bfi2,where A has n points and B has m points; all mappings fi,n⟶1,m should satisfy the requirements that f1=1,fn=m and fi≤fj, for all 1≤i≤j≤n.The positions of the estimated lane are average positions of one group of trajectories, and they are shown with green colour in Figure9.Figure 9
Trajectory data in different phases.
(a)(b)(c)It can be seen that the phase switching point based on the traffic flow extreme value computing can accurately separate the trajectories of adjacent phases.All the lanes estimated in the intersection are shown in Figure10. Although these lanes are easily separated from each other, it is still an important task considering the dynamic traffic state and invertible lanes existing. These lanes in different intersections could be recognized adaptively.Figure 10
Recognized lanes in an intersection.


## 3.3.1. Coarse-Grained Partition of Trajectories

Usually, there are several phases at this intersection, as shown in Figure7.Figure 7
Recognized traffic phases.In different phases, vehicles of different directions will start to move orderly. It is possible to estimate the phase changes through traffic flow analysis. Furthermore, by tracking all the vehicles on the road, it is possible to get traffic flow data for all these lanes and directions. This peak and valley of flow data chart indicate the traffic phase changing time point, as shown in Figure8.Figure 8
Traffic flow distribution and phase switching diagram.As shown in Figure8, the horizontal axis is time; the vertical axis is the flow data of vehicles for all lanes. The phase switching time is the time when the flow changes drastically, so the simple extremum point of the flow can be used to judge the phase switching time. According to these switching times, trajectory data will be divided into several groups.


## 3.3.2. Lane Detection Based on Clustering

The problem of lane recognition changes to clustering different groups of trajectories. DTW (dynamic time warping) is a sequence alignment method to find an optimal matching between two trajectories and measure the similarity without considering lengths and time ordering [20]:(1)WA,B=fmin1n∑i=1nai−bfi2,where A has n points and B has m points; all mappings fi,n⟶1,m should satisfy the requirements that f1=1,fn=m and fi≤fj, for all 1≤i≤j≤n.The positions of the estimated lane are average positions of one group of trajectories, and they are shown with green colour in Figure9.Figure 9
Trajectory data in different phases.
(a)(b)(c)It can be seen that the phase switching point based on the traffic flow extreme value computing can accurately separate the trajectories of adjacent phases.All the lanes estimated in the intersection are shown in Figure10. Although these lanes are easily separated from each other, it is still an important task considering the dynamic traffic state and invertible lanes existing. These lanes in different intersections could be recognized adaptively.Figure 10
Recognized lanes in an intersection.


## 4. Generating Traffic Flow Data for Each Lane

Once these lanes are detected, the trajectory of each vehicle can be assigned to the closest lane. This lane metadata provide semantic information about the vehicle’s movement. For one lane, the corresponding lane metadata may need to be marked, as shown in Table3.Table 3
Description information for automatically detected lane.
Lane numberLane nameStart point of laneEnd point of laneTurn type1Junggar Road and Western Ring RoadJunggar Road and west ring road southboundJunggar Road and West Ring Road intersection northboundGoing straight2Junggar Road and Western Ring RoadJunggar Road and west ring road southboundWest Loop WestTurning leftWith accurate detection of lanes, accurately counting vehicles in each lane is possible. Meanwhile, the time for each vehicle passes through the intersection can be calculated. According to the vehicle trajectory similarity algorithm, the trajectory of each vehicle is accurately divided into an estimated lane. Figure11 shows the flow data of two lanes.Figure 11
Flow data for each lane.As shown in Figure11, the flow data in a certain lane mean the traffic flow data are fluctuating with time changes. Commonly, the flow data in different lanes could give indication for red-blue light time adjusting.When the lanes are determined by the system, the flow data for each lane are counted. In the beginning, there is an assumption that each vehicle passing the intersection will follow one of the detected lanes, so the flow data of fine-grained lane is the number of vehicles travelling in this lane in a certain time period. The assumption is reasonable and acceptable considering very few vehicles will break the traffic rules. However, the statistical data are convincible in the long run.Three intersections are chosen to validate our method; the time period is settled as five minutes which facilitates making the ground truth data. These intersections are shown in Figure12.Figure 12
Three intersections in our experiments.The flow data for each fine-grained lane are shown in Table4.Table 4
Fine-grained lane flow data in intersection.
Lane IDVolume in five minutesVolume in five minutes (ground truth)AccuracyScene 1 (Tahe road)125320.78226320.81323290.79462670.93549540.91667710.94720240.83826290.9968750.911036420.861134390.87Scene 2 (Youjian road)112140.86222260.85325270.93416190.84536390.92674760.97752560.93843460.93928310.91036370.97Scene 3 (Tianshan road)112150.8213170.76323260.88436390.92516180.89616190.84724270.89868740.92957680.841026290.91169790.871274860.861329340.851419250.76Through the results of our experiment, it is obvious that our system could achieve an average accuracy of 87% in all the lanes of the three intersections. The vehicle occlusion brings some errors in this data, but it is acceptable in such experiments.At the same time, we notice, in scene one, the average accuracy is higher than that in scenes two and three. The reason lies in the angles of cameras of intersection one which are higher than those of the other two intersections which make the occlusion lighter. This proves that the angles of cameras have a big influence on the effectiveness of our method.


## 5. Conclusion

This paper proposed a framework to generate traffic big data from video in the intersection based on vehicle tracking and lane detection. Due to no unified distribution of lanes in the intersection, manually setting the lane metadata is resource consuming.Each vehicle in the intersection is tracked, and trajectory data for all vehicles are computed. The phase switching time is estimated based on these trajectories and is applied to divide the whole trajectory into several small groups. The lane is estimated based on the individual group of trajectories.Additionally, the flow of each lane can be computed by adding the number of vehicles following this lane. The framework can work fine in the intersection and is adaptive for each intersection based on the experiments on different videos.Since the collected traffic data are based on cameras at intersections, therefore, when the video data are unclear or missing severely, then our experimental results will be affected. Future work may focus on generating traffic data, based on video analysis, from several connected intersections.



---

*Source: 4764174-2021-11-02.xml*