Multi‐camera multiple vehicle tracking in urban intersections based on multilayer graphs

IET Intelligent Transport SystemsVolume 14, Issue 12 p. 1673-1690 Research ArticleFree Access Multi-camera multiple vehicle tracking in urban intersections based on multilayer graphs Mohadeseh Delavarian, Corresponding Author Mohadeseh Delavarian delavarian.mohadeseh@shahroodut.ac.ir Faculty of Computer Engineering, Shahrood University of Technology, Shahrood, IranSearch for more papers by this authorOmid Reza Marouzi, Omid Reza Marouzi Faculty of Electrical & Robotic Engineering, Shahrood University of Technology, Shahrood, IranSearch for more papers by this authorHamid Hassanpour, Hamid Hassanpour orcid.org/0000-0002-5513-9822 Faculty of Computer Engineering, Shahrood University of Technology, Shahrood, IranSearch for more papers by this authorReza M. Parizi, Reza M. Parizi College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USASearch for more papers by this authorMohammad S. Khan, Mohammad S. Khan Department of Computer & Information Sciences, East Tennessee State University, Johnson City, USASearch for more papers by this author Mohadeseh Delavarian, Corresponding Author Mohadeseh Delavarian delavarian.mohadeseh@shahroodut.ac.ir Faculty of Computer Engineering, Shahrood University of Technology, Shahrood, IranSearch for more papers by this authorOmid Reza Marouzi, Omid Reza Marouzi Faculty of Electrical & Robotic Engineering, Shahrood University of Technology, Shahrood, IranSearch for more papers by this authorHamid Hassanpour, Hamid Hassanpour orcid.org/0000-0002-5513-9822 Faculty of Computer Engineering, Shahrood University of Technology, Shahrood, IranSearch for more papers by this authorReza M. Parizi, Reza M. Parizi College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USASearch for more papers by this authorMohammad S. Khan, Mohammad S. Khan Department of Computer & Information Sciences, East Tennessee State University, Johnson City, USASearch for more papers by this author First published: 26 October 2020 https://doi.org/10.1049/iet-its.2020.0086Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Vehicle visual tracking is a challenging issue in intelligent transportation systems. The tracking gets more challenging when vehicles change direction at intersections. Undetermined motion flows, occlusion, and congestion are the potential issues of vehicle tracking at intersections. In this study, a new method for tracking multiple vehicles from a multi-view is proposed to overcome occlusion caused at the intersections with undetermined motion flows. In the authors' method, a multilayer graph is presented that assigns motion flows to distinct layers with different neighbourhoods for each layer represented by the graph's edges. Hence, the vehicle trajectories are distributed among layers such that vehicles entering from the same side with similar motion flows are assigned to the same layer. All multilayer graphs of different views are mapped to the graph of the selected view. Then, tracking is performed on the distinct layers of the mapped multilayer graph by computing min-cost flows. In cases such as vehicle crossing, misdetection, or occlusion, the method can predict the vehicle's tracks by using history, layer neighbourhoods, and other views' information. Experimental results show a consistency of the ground truth and the analysis obtained using the proposed method in tracking vehicles in the inner part of the intersection. 1 Introduction Multiple vehicle tracking is a field of multiple object tracking (MOT) that plays an important role in computer vision and intelligent transportation systems. Visual vehicle tracking has attracted attention recently due to the growth of accidents. It is used in many applications such as safety management, accident avoidance, and automatic driving systems. The demand for fast, efficient, and accurate tracking systems is increasing due to the importance of efficient safety and traffic management. In particular, intersections are important targets for monitoring surveillance systems because of the high number of reported accidents. Consequently, monitoring at intersections has gained attention in recent years. In [1, 2], several challenging issues are discussed regarding vehicle tracking at intersection (VTI) such as occlusion, congestion, undetermined motion flows and conflicts that cause accidents (see Fig. 1). Therefore, it is of concern by many research studies in the literature to address such challenges. As a result, the main phase of intersection monitoring systems can be considered tracking vehicles at intersections, which is the main focus of this work. Fig. 1Open in figure viewerPowerPoint Intersection where admissible motion directions for each side is presented Vehicle monitoring systems consist of two main phases: (i) vehicle tracking, and (ii) vehicle behaviour and safety analysis. A variety of sensors have been used in tracking systems [2]. Cameras are commonly used for tracking road users at intersections since they are less expensive, easy to install, and have a wide field of view (FOV). There exists a lot of research and reports in object tracking using a single sensor [3]. However, there has been a shift in recent years from using a single sensor to multi-sensor methods [4]. The sensors can be from the same source or different ones. In visual tracking, multi-camera systems use the same source sensors but with different views. Studying vehicle tracking is ongoing research as the systems are not yet reliable and robust for vehicle tracking in different challenges. In this paper, a vehicle tracking method is proposed for tracking multiple vehicles from a multi-camera network at the intersection to overcome undetermined directions and the occlusion caused in VTI. The proposed method considers daytime tracking with cameras installed around the intersection and a few metres above the ground. A multilayer model is proposed to distinguish and distribute motion flows into different and distinct layers. For achieving this, for each layer two neighbourhoods based on allowed directions are introduced, Near and Far neighbourhoods. Each layer represents one side of an intersection that vehicles can enter and the neighbourhood is defined for that side motion flow. After constructing the multilayer model for each view, a multilayer graph [weighted layered DAG (directed acyclic graph)] is introduced and constructed. The vertices are a grid of cells on the ground plane over a time period, and edges connect vertices based on neighbourhoods on each layer for each allowed movement. In our method, tracking is performed on each layer separately, since the layers are disjoint, to avoid a collision. When a vehicle enters the intersection, a track record is created. During the time that it crosses over the intersection, the tracks are updated. As it moves from one cell to another, the weight edges of the graph of that view are also updated. Tracking is performed based on vehicle movements and neighbourhood prediction. After each time period, the multilayer graph of all views is mapped to the selected main view based on planar homography between camera pairs. So, one mapped multilayer graph containing the tracking information of all views is created. The final tracking phase is performed by finding the shortest paths as the tracks in the final mapped multilayer graph that is considered as a network flow. In the case of occlusion or misdetection, in the first phase of tracking in one view, the proposed method examines the tracking history and the route of the vehicle based on the neighbourhood of layers. In the second phase, since the trajectories of other views are mapped to one view, misdetections in other views can be managed. The specific contributions of this paper are as follows: A multilayer model with two neighbourhoods is introduced to distribute the motion flows into different layers to overcome collision. Based on the multilayer model, a multilayer graph for intersections-specific tracking is introduced to help linking trajectories using a graph-based approach to overcome false and misdetections. A mapping procedure to map different camera views' graphs without the need for camera calibration is introduced to associate tracks between views in multi-camera networks. Three trackers based on three-phase tracking are proposed. The three tracking phases are conducted by the multilayer model, multilayer graph, and mapped multilayer graph. The remainder of this paper is organised as follows. Section 2 reviews the related works. Section 3 describes the proposed tracking method. Experimental results are discussed in Section 4. Finally, Section 5 concludes the paper. 2 Related works With the rise of intelligent surveillance and automated driving systems, the attention to vision-based detection tracking and behaviour analysis has grown. Many research studies have been performed on detection, tracking, and traffic monitoring [3, 5-7]. Tracking can be used in intelligent vehicles to improve the performance and safety of different driving situations and conflicts [8, 9]. In the study by Sivaraman and Trivedi [7], they provided a review of the literature in on-road vision-based vehicle detection, tracking, and behaviour understanding. In [10], the authors discussed MOT methods, benchmarks, and evaluation metrics. Although pedestrian tracking and vehicle tracking at locations such as highways have been discussed over the years [7, 10], tracking at the intersection has gained attention over the past decade. Equipping intersections with multiple cameras are getting attention in the traffic systems, in order to obtain reliable tracks for better traffic management, control systems, and intelligent vehicles [11, 12]. Recently, there are research studies that solely focus on traffic monitoring at intersections [1, 2]. In the research performed by Datondji et al. [1], they reviewed vision-based systems about intersection monitoring including sensing technologies, data sets, vehicle detection, tracking and monitoring methods, and their challenges. In the intersection monitoring study proposed in [2], the authors focused on behaviour and safety analysis of vehicles, drivers, and pedestrians at intersections in addition to techniques for automating visual sensing. Tracking at the intersection has been discussed in the Bayesian framework. Among the Bayesian tracking methods, the Kalman filter is one of the most popular algorithms that has been used in many applications. Two-level tracker, low-level that tracks binary blobs and high-level that models the target, was introduced for VTI in [13]. They used the Kalman filter for predicting movements, however, it could not track motionless vehicles, also in crowded scenes. In another research by Lira et al. [14], the Kalman filter is applied to footage from a drone-floating camera. If there is prolonged shake, many frames are missed so their algorithm cannot track vehicles. In the research presented by Saunier and Sayed [15], they extended the feature-based tracking methods into intersections. Song and Nevatia [16] proposed a Markov chain Monte Carlo in order to segment multiple overlapped vehicles into separate ones with respective orientation. Their proposed method could track multiple vehicles with orientation changes. These methods suffer from lost measurements for tracking in case of occlusion. In some research studies, a video-based system is developed [17-21]. Nateghinia and Moradi [17] introduced a video-based system for vehicle detection and tracking. They used a point tracking method with weighted recursive least square. In the research done in [18], a method for extracting traffic information from a low-resolution camera is proposed. Their proposed method can handle perspective distortions to some extent. Wang et al. [19] proposed a real-time system for tracking and counting multiple vehicles. They used a fisheye camera based on simple feature points. The authors in [22] proposed an algorithm for detecting and tracking vehicles entering intersections in real-time. The algorithm is based on blob analysis for the main tracking and means shift kernel tracking when a blob merging occurred. Min et al. [23] introduced a method based on two classifiers for tracking multiple vehicles. Their method can be used at the intersection according to their experiments. In this method, they used an improved ViBe algorithm for accurate detection with the aid of the grey-scale spatial information. For tracking rectangle of vehicles, they used support vector machine and convolutional neural network classifiers to address the problem of failure when occlusion occurred. Some researchers have detected and tracked vehicles, however, in addition to that others used trajectories to count vehicles, classify them, and learn the behaviours. Kamijo et al. [24] developed a spatio–temporal Markov random field algorithm for tracking at intersections. The proposed algorithm is robust enough for tracking occluded vehicles. In addition, their proposed system can use the tracking results and learn behaviour patterns of vehicles in HMM (hidden Markov model). Then, it could recognise events such as reckless driving. However, it requires only greyscale images. A real-time vision system at urban crossroads was presented in [25]. They used monocular images from pole-mounted video cameras. Their proposed system consists of segmentation with robust background updating and feature-based motion tracking method. In addition, they used model-based matching for the classification of vehicles. Their system is not robust when dealing with occluded objects. The research in [26] aimed to detect, count, and track vehicles at an intersection. For vehicle detection, a combination of running Gaussian average and the local binary pattern is used. Then Kalman filtering is used for MOT. Their proposed method counts the number of vehicles in a particular area. Wu et al. [27] proposed traffic pattern modelling, trajectory classification, and vehicle tracker within the urban intersection. Trajectories are classified with HMM into various motion patterns after that tracklet prediction is performed based on them. A tracking framework by using a classification label from a deep learning detection approach is presented in [28]. Their main goal was to investigate the performance of a multi-class object detector in traffic scenes. They showed that labels from the detection phase improve tracking performances. The authors in [29] developed a vehicle counting system using motion estimation with the aid of Taylor series approximation. In the proposed method, a virtual entering or exiting box is applied to improve predicting the vehicle direction. Also, a video-based system for obtaining traffic-flow statistics is presented for road intersections [30]. For this purpose, vehicles are recognised by training a deep learning architecture from a pre-trained model and after that, it is used to classify them. They used CCTV footage under daylight conditions and no discrimination among vehicles. Their main purpose is to distinguish vehicles and non-vehicles. Different systems are developed to use trajectories for other phases such as classification and behaviour analysis. The performance of single-source object tracking can be deteriorated due to the complex environment and difficult situation. Combining multiple sources for tracking can enhance performance over a single one [4]. As mentioned, multiple sources can be from the same or different sensors. For vision-based systems, the focus is on using multi-camera as the same type multi-sensor. In the review done by Wang [31], he focused on intelligent multi-camera video surveillance. The survey presented in [32] focused on multi-view vision systems in various platforms. Walia and Kapoor [4] aimed at categorising multi-cue tracking methods into single-modal (by one type of source) and multi-modal (combining different types of sources). Some of the research studies that track vehicles from multi-camera with overlapping views are reviewed here. The authors in [33] presented a vehicle tracker for automatic data collection at intersections and highways. Their proposed measurement model supports measurements from any number of views. After that, they are fused by the presented estimation procedure. In the research done in [34], an automatic traffic data collection system from multiple camera tracking system is proposed. First, the cameras are calibrated. Then, tracking is performed from multiple cameras by applying the Kalman filter. After that, vehicle trajectories are analysed to extract traffic data. In their experiments, they just considered partial occlusion. Subedi and Tang [35] presented a multiple-camera three-dimensional (3D) vehicle tracking for intersections. First, each camera is calibrated, then the system takes synchronised images from the cameras and derives vehicle silhouettes, after that they are projected to 3D real world. They apply Kalman-based vehicle tracking to derive trajectories. Liu et al. [36] proposed a 3D particle filter tracking method. They improved the robustness of tracking by collaborative tracking via visual information complementation. Their proposed method is based on the joint observations of multiple visual sensors, and the traditional 2D visual particle filter target tracking algorithm. When reviewing the subject of MOT, data association between detections and trajectories of continuous frames or between two or more cameras is unfolded. One approach to solve this data association is graphs and flows [37]. Most research studies are on tracking pedestrians rather than vehicles. So, we reviewed some of the notable ones, here. The authors in [38] proposed a data association method based on undirected hierarchical relation hyper-graph for multi-target tracking. In another research by Leal-Taixé et al. [39], they developed an algorithm to track multiple objects in multi-view images. Tracking is formulated as a single global optimisation and a min-cost flow problem by defining a graph that expresses both temporal and spatial correlation between objects and cameras, respectively. Hofmann et al. [40] had done multi-object tracking by the generalisation of a network flow problem. They presented a combined maximum a posteriori formulation that models multi-camera reconstruction along with global temporal data association. The objects are tracked in the 3D world space by the aid of a flow graph. In the research done by Azevedo et al. [41], a graph for connection between consecutive candidate position is built by criteria based on vehicle's speeds. Also, the dual graph is built by acceleration-based criteria. After that, they used the k -shortest path algorithm for multi-object tracking by finding trajectories on the constructed graph. The authors in [42] proposed a method for MOT by detecting objects in frames separately and then linking them for acquiring the trajectories. The linking step results in an optimisation problem and it is reformulated as a constrained flow optimisation and solved using the k -shortest paths algorithm. They extended their method by making use of appearances to prevent identity switch (IDS) [43]. A graph matching method for multi-view pedestrian tracking is proposed by exploiting object characteristics and spatial/temporal relationships between them [44]. Another graph matching problem for vehicle tracking is presented in [45]. In the proposed method, geographic information and its combination with deep ReID features and motion are used. Some research studies track all road users at the intersection [46, 47]. Tracking all road users at the intersection can be very useful for transportation systems, as all trajectories can be used for monitoring and safety analysis. Recently, within the Ko-PER project, a data set for intersection monitoring has been released [48]. This data set contains laser-scanners and videos from multiple cameras. Also, they presented a method for detecting and tracking vehicles at the intersection from multiple camera views. At first background subtraction is done, the foreground segments are detected. Then, detected vehicles are tracked by a Gaussian-mixture approximation of the probability hypothesis density (PHD) filter [49]. In addition, they developed the system for tracking all road users by using all measurements of their data set. They presented a general-purpose multi-sensor tracking algorithm and classifying relevant road users. On account of different sensors and objects to track, an extension of PHD filter is proposed to handle them [47]. To summarise, VTI at the intersection by one camera can suffer from missing measurements. Clearly, more clues about vehicles are needed. The multi-sensor tracking methods can have better performance. Though there is growing attention to such methods in recent years [35, 36], generally speaking, in overlapping views at the intersection, there are limited studies. Using graphs in tracking and data association with respect to pedestrians has been widely studied but that is not the case with tracking vehicles. Due to the structure of intersections, a well-designed graph can be helpful that even in one-view tracking, accurate tracks are acquired. Inspired by this, we introduce the multilayer model and construct a multilayer graph for tracking vehicles at the intersection. 3 Proposed method VTI is a challenging task due to various involved parameters. Simultaneous crossing cars, buses, bicycles, and pedestrians with dynamic movement directions can be some of such parameters. Fig. 1 depicts different motion flows from each entry–exit side when vehicles crossing the intersection. As it can be seen in the figure, there exists several collision points between different flows. One standard approach applied in MOT algorithms is tracking-by-detection, in which a set of detections are extracted from frames and are used in the enabling tracking algorithm [50]. This approach emphasises on comparing tracking performance exclusively since the detection performance can thoroughly impact the tracking performance. The proposed tracking method falls into this approach and considers only vehicles (excluding bicycles and pedestrians). It is assumed that vehicles crossing the intersection only follow the predefined directions (see Fig. 1). 3.1 Multilayer model and neighbourhoods The multilayer model is proposed for the tracking concept in which different motion flows are distributed among distinct layers. The area of tracking, i.e. FOV is considered the inner part of the intersection and is indicated manually. The sides of FOV, that are associated with each entry–exit side and each layer, are labelled clockwise starting from the top side and one (see Fig. 2). Fig. 2Open in figure viewerPowerPoint FOV and grid cells of an intersection with labelled sides and layers Afterwards, a grid is constructed over the ground of the intersection as shown in Fig. 2. The size of the grid cell is an important parameter because each cell represents a part of an intersection. The cell size is determined by the distance between the grid lines in pixels that is user-defined and represented by a variable called step parameter. So, the y -intercept of the grid lines that is called y_intercept_change is modified as shown in (1) where a is the line slope (1) Then, a grid cell-matrix with the size of is constructed where each cell represents a part of the intersection. The smaller value of step results in more cells and vice versa. The effect of this aspect is discussed in the experimental results (Section 4). In the proposed multilayer model, each side of the intersection has a corresponding layer. A vehicle can only move in permissible directions. So, when vehicles entering the intersection, their tracks enter the layer that represents that side. Layers are disjoint, which means vehicles cannot move between them. To model the allowed directions in each layer, neighbourhoods are introduced. Neighbourhoods show motion directions in each layer, therefore the difference between layers is equal to cell neighbours in each layer. Suppose a vehicle is at of layer k at a time t, it can be located at the time at the same layer and cell or where the index is in the neighbourhood of according to the layer k. Two neighbourhoods are introduced: Near and Far. The cells in Near neighbourhood are the ones immediately beside according to the motion flow of that layer. Moreover, cells that are located immediately beside Near neighbourhood cells are called Far neighbourhood. The two neighbourhoods of layer 1 that is for the vehicles entering from the top side is defined in Table 1. In Table 1, the cell is the current location that a vehicle is detected. The bold values are Near neighbourhood cells and are along with motion directions that are shown in Fig. 1 (going straight, turning right, or turning left). The non-bold values are Far neighbourhoods. The neighbourhoods of other layers are computed the same way. Table 1. Near and Far neighbourhoods for top side layer (Layer 1) The different step parameter results in smaller or bigger grid cells. Also, frame rates can be high or low, in low frame rates a vehicle can move more than one neighbouring cell. Moreover, one grid cell can represent a part of different sizes of actual intersection due to camera view and placement. All of these result in introducing and applying the two neighbourhoods while tracking. It should be noted that a cell can only contain one vehicle at a time with any value of the step parameter. 3.2 Multilayer graph The concept of MOT can be represented by finding paths as tracks in a graph. Here, we introduce a multilayer graph that will have the tracking information for each camera view. The multilayer graph is defined as where V, E, and L denote sets of vertices, edges, and layers, respectively. The multilayer graph is a weighted DAG. A sample graph consisting of one layer is depicted in Fig. 3. Fig. 3Open in figure viewerPowerPoint Sample graph with one layer and imaginary detections Layers : Within VTI, usually each intersection has four sides. However, there is no limitation in the number of entry–exit sides and layers. Here, L, the total number of layers, is equal to 4. Each separate layer denoted by l, where , is for one entry side, has a grid cell-matrix , also for each cell Near and Far neighbourhoods are defined as before. Vertices : As mentioned in Section 3.1, grid cells of the intersection, matrix , is constructed. Based on the value of the step parameter, it has a different number of rows and columns that are denoted by N and M, respectively. Each vertex of the graph G is presented by where l represents the layer number, t represents time or frame number, and is cell index of the matrix . So, each vertex represents a part of the intersection in a specified frame and layer. The constraints for each vertex are . Two vertices named and are added before the first frame and after the last frame, respectively. They are added for the starting and ending of each track in a specified period of time. Note that the graph is constructed for each batch of T frames. The number of vertices is . Edges : Note that in each layer for each cell Near and Far neighbourhood cells are defined. Also, entry and exit cells for each layer are determined. Entry cells are from the side that a vehicle can enter the intersection, and exit ones are the other three sides of the intersection. So, directed edges are defined based on acceptable motion flows and possible occurring tracks. All edges have a predetermined weight of w when constructing the graph. They are defined and added as follows. Entrance edges o From to : Each track can be started from any location of the first frame. o From to : A vehicle can enter the intersection in each frame and layer from entry vertices. Exit edges o From to : Each track could end for a batch. o From to : Each track ends if a vehicle exits the intersection from other sides' exit vertices. Transition edges o From to : For remaining motionless or a vehicle's location does not change more than one grid cell. o From to : When a vehicle's location changes to its Near or Far neighbourhood cells. 3.3 Mapping multi-camera multilayer graphs The VTI proposed tracking method uses multilayer graphs from multi-camera (Fig. 4). In multi-camera tracking methods, camera views are classified as overlapping, partially overlapping, or non-overlapping. In our work, the overlapping or at least 80% partially overlapping camera views are considered. The camera placement is around the intersection and a few metres above the ground. Two or more cameras can be installed in different corners, e.g. two cameras on the opposite side of the intersection. Multi-camera vehicle tracking is usually addressed as a data association problem between cameras. Here, the data association is fulfilled by a mapping procedure. Therefore, no cameras' calibration is needed in the proposed tracking method, and all the correspondences are computed based on the mapping. Fi