An efficient multi‐sensor fusion and tracking protocol in a vehicle‐road collaborative system

IET CommunicationsVolume 15, Issue 18 p. 2330-2341 ORIGINAL RESEARCH PAPEROpen Access An efficient multi-sensor fusion and tracking protocol in a vehicle-road collaborative system Zhehan Xu, orcid.org/0000-0003-0382-3697 School of Software Engineering, Tongji University, Shanghai, ChinaSearch for more papers by this authorShengjie Zhao, orcid.org/0000-0002-4301-394X School of Software Engineering, Tongji University, Shanghai, ChinaSearch for more papers by this authorRongqing Zhang, Corresponding Author rongqingz@tongji.edu.cn orcid.org/0000-0003-3774-6247 School of Software Engineering, Tongji University, Shanghai, China National Mobile Communications Research Laboratory, SoutheastUniversity, China Correspondence Rongqing Zhang, School of Software Engineering, Tongji University, Shanghai, China. Email: rongqingz@tongji.edu.cnSearch for more papers by this author Zhehan Xu, orcid.org/0000-0003-0382-3697 School of Software Engineering, Tongji University, Shanghai, ChinaSearch for more papers by this authorShengjie Zhao, orcid.org/0000-0002-4301-394X School of Software Engineering, Tongji University, Shanghai, ChinaSearch for more papers by this authorRongqing Zhang, Corresponding Author rongqingz@tongji.edu.cn orcid.org/0000-0003-3774-6247 School of Software Engineering, Tongji University, Shanghai, China National Mobile Communications Research Laboratory, SoutheastUniversity, China Correspondence Rongqing Zhang, School of Software Engineering, Tongji University, Shanghai, China. Email: rongqingz@tongji.edu.cnSearch for more papers by this author First published: 15 September 2021 https://doi.org/10.1049/cmu2.12273AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Nowadays, driving safety has become an important topic in the field of intelligent driving. The key step in the intelligent transportation system (ITS) is to detect and track the vehicles on the road through various sensors equipped on the vehicle. While putting the sensors aside the road can have benefits in the vehicle-road collaborative scenario. In this paper, an efficient and applicable multi-sensor fusion protocol for a vehicle-road collaborative system is proposed where roadside units (RSUs) are equipped with cameras and radars. To fully use the history information and the multi-source heterogeneous sensing data to improve the reliability of multi-sensor fusion, a Bayesian-based two-layer fusion scheme is further proposed. Moreover, the efficiency and robustness of the proposed scheme are evaluated in a real highway environment in Changsha, China. 1 INTRODUCTION The technology of intelligent vehicles and the intelligent transportation system (ITS) is ever-increasing rapidly in recent years. The latest vehicles launched by several companies are equipped with excellent advanced driving assistance systems (ADAS), some of which can even cooperate with the roadside unit (RSU). These systems are designed to help drivers enjoy a more comfortable driving journey as well as help them make fewer mistakes during the driving process. In order to guarantee safe driving, it is necessary to detect the on-road vehicles and track them, which is also the key step in the ADAS or ITS. The designed system should be robust and reliable even though the sensors equipped are not always working correctly. Therefore, such sensors like the millimeter-wave (MMW) radar and the camera are used to be responsible for the detection and tracking task in intelligent vehicles [1]. The camera captures the picture at time T c a m then uses some algorithms to detect the vehicles inside its sight. The output of the camera is the position of an object on the picture and other information like the size and the type of such vehicle. Compared with the MMW radar, although the camera is sensitive to lighting conditions, it can provide richer information than the MMW radar can do. The MMW radar detects objects by emitting and receiving the millimeter-wave radio then analyzing the wave frequency changes to get the object information such as the position and the velocity. Therefore, the vehicles' speed provided by the radar is quite precise. Different from the camera's principle, the MMW radar can provide relatively reliable results even with poor light conditions such as at night and even on cloudy or rainy days. Because of its stability and reliability, the MMW radar is usually used for vehicle application and navigation. But the measurement of the MMW radar is noisy and will detect the false targets that are objects that do not exist [2]. The two kinds of sensors have their own advantages, but to use the data from both of the sensors have to take their disadvantages into consideration. The results provided by the MMW radar is noisy and there will generate many false objects because of the noise or the re-reflection between the different metal surface. The results provided by the camera will be significantly influenced by the light condition and the weather condition. Also, the results from a camera with high accuracy need plenty of time to calculate. One solution, which is the most adopted one, is to install the sensors on the vehicle [3-5]. By having these sensors, each vehicle can have a view in a certain space around, as is shown by Figure 1. Then the advanced driving assistance system in the vehicle can use a certain kind of fusion algorithm to process the data and give the driver more precise feedback. This kind of method can help a vehicle know the situation around it quickly and independently to help the driver make better decisions when facing some situations. FIGURE 1Open in figure viewerPowerPoint Two or more sensors are installed on the vehicle to enhance the robustness of the data. The data then can be shared with other vehicles. These vehicles get the unique description of the environment through a consensus algorithm However, this solution is not perfect, it has some disadvantages. First of all, the sensors are set on the vehicle, where the field of view is relatively low. Therefore, the problem of occlusion is serious, as shown in Figure 2. The orange vehicle may not be able to see the blue vehicles because the black vehicles occlude its sight. Another disadvantage is that if there are too many vehicles on the road at one moment, the description of road conditions from different vehicles may differ from each other because the algorithms they use may be different. Even though there are some researches in generating the unified description, the fusion of heterogeneous data of vehicles from different manufacturers to produce a unified description is still a problem to be solved. In addition, letting every vehicle on the road detect the surrounding environment independently wastes a lot of computation. FIGURE 2Open in figure viewerPowerPoint (a) An example of occlusion when sensors are set on the vehicles. The sight of the sensors on the orange vehicle A is occluded by two black vehicles B and C. Therefore, it cannot see the blue vehicles D and E. (b) Better vision can be achieved when sensors are set aside the road Another solution is to set the sensors (a camera and an MMW radar) aside from the road, as shown in Figure 3. We can set the sensors higher to get a better vision, where the situation of occlusion is fewer. What is more, the sensors can capture the road condition of a certain road section, which means it can provide a unique road condition and send them to each vehicle so that there will be no ambiguity of the environment. It also saves more computation compared with that each vehicle calculating the surrounding environment independently. FIGURE 3Open in figure viewerPowerPoint A picture from the camera set aside the road. The camera is set about 6 m high away from the ground in order to avoid most occlusion cases In this paper, we propose an efficient and applicable multi-sensor fusion scheme by fusing multi-source heterogeneous sensing data, which can handle the unreliable sensors and the considered sensors are set aside the road. Moreover, we evaluate the efficiency and robustness of our proposed scheme in a real highway environment in Changsha, China. The main contributions of this paper can be summarized below. We propose an efficient and applicable multi-sensor fusion protocol in a vehicle-road collaborative system. It fuses the heterogeneous data from both the camera and the radar set aside the road to get unified and precise vehicular information while meeting the real-time requirement. To fully use the history information to improve reliability, we propose a Bayesian-based two-layer fusion scheme. The provided fusion process not only fuses the camera and radar information at the current time instant (the first layer) but also takes the historical data into consideration (the second layer). We execute practical experiments by installing our designed system on a 20 km road in the expressway in Changsha, China. We post the RSUs equipped with cameras and MMW radars every 200 m along the road. The results obtained in this practical prototype test scenario verify the efficiency of our proposed multi-sensor fusion protocol in increasing the accuracy of vehicle tracking. The remaining part of the paper is organized as follows. In Section 2, we introduce the researches of the fusion of the multi-source heterogeneous data in recent years. Section 3 presents the system model we study in this paper. The proposed protocol is discussed in detail in Section 4. In Section 5, the results of the experiment using the data from reality are discussed and we briefly summarize our work in Section 6. 2 RELATED WORKS In recent years, quite a few researches have focused on multi-sensor data fusion. An efficient method for the localization is to fuse the in-vehicle sensors with a low-cost Global Positioning System (GPS) because of their complementary features [6-10]. Based on this idea, Kichun Jo et al. proposed to fuse the GPS data with the data from inertial navigation system (INS) [11]. They mainly focused on the positioning algorithm based on an interacting multiple model (IMM) filter, which is composed of a kinematic vehicle model and a dynamical vehicle model. Alessandretti et al. proposed the fusion of two types of sensors in the early stage [12]. They used the camera to capture the image to identify the region of interest while exploiting the MMW radar to detect objects. Through this fusion, the computational intensity can be significantly reduced so that it can satisfy a high real-time requirement. Kadow et al. and Dieterle et al. also did the fusion of radar and camera in the early stage to get the position, as well as the width information [13, 14]. Kadow et al. developed a new method by building a combined symmetry measure and finally achieved an accuracy of 90.3 % for the car class. While Thomas Dieterle et al. used 3D-processing techniques for frame-to-frame detection of dynamic objects. Experiments show that the robustness to occlusions and sensor failure is significantly improved. Qingquan Li tet al. fused LIDAR and vision data to do the lane detection, including the structured and unstructured roads [15]. Wang Xiao et al. tried to fuse the MMW radar and monocular camera for on-road detection and tracking [16]. They provided a collaborative fusion approach to achieve an optimal balance between vehicle detection accuracy and computational efficiency. The designed system is evaluated in a real-world data set collected by the intelligent vehicle and has a result of 92.36 % detection and 0 % false alarm rate. Bharanidhar Duraisamy et al. got a track-level fusion from heterogeneous sensors [17]. They mainly focused on deriving fused extended shapes and used Bayesian Fusion to do the shape fusion. Six combination possibilities of target shapes were taken into consideration in their paper. In real situations, the sensors combined in an automotive application are usually asynchronous to one another. Aeberhard et al. used the information matrix fusion (IMF) algorithm to get a stable global track [18]. They found that the IMF approach provides similar performance to a centralized Kalman filter in a low-layer fusion. Their work performs a good result in asynchronous and out-of-sequence sensor configurations. Yue Zhang et al. used the camera data which is set aside the road to conduct the fusion [19]. They fused the image information and the velocity obtained from motion sensors at a low layer. The color-faster R-CNN is chosen to process the image data to get the vehicle detection and the Kalman filter is employed to get the velocity information. It can meet the real-time requirement but the data they tried to fuse was limited. As we can see, the fusion of multiple sensors can make the result more precise than a single sensor. However, very few researchers have considered a usual scenario in current widely-proposed road-vehicle collaborative systems, where many sensors are set aside the road. In addition, most current works focused on the low-layer fusion, but in practical applications, sensors are sometimes not reliable. Therefore, the method should be highly robust to the high-mobility and complicated real environment. In this paper, we provide an efficient and applicable multi-sensor fusion scheme, which can handle the unreliable sensors and the considered sensors are set aside the road. Moreover, we evaluate the efficiency and robustness of our proposed scheme in a real highway environment in Changsha, China. 3 SYSTEM MODEL In this section, the investigated system model is described. We first introduce a general road-vehicle collaborative scenario that is suitable for most of the roads. And then we will discuss some practical restrictions in the system model. 3.1 System scenario We consider an on-road scenario where a set ν = { 1 , ⋯ , N ν } of N ν vehicles are moving on a road, which is considered to be a two dimensional space as illustrated in Figure 4. At time instant t, vehicle i ∈ ν can be described by the position of P i , t ( V ) = [ P x i , t ( V ) , P y i , t ( V ) ] ∈ R 2 × 1 and has the velocity of V i , t ( V ) = [ V x i , t ( V ) , V y i , t ( V ) ] ∈ R 2 × 1 . Therefore, the kinematics parameters of vehicle i ∈ ν can be represented by the state vector X i , t : X i , t = P i , t ( V ) V i , t ( V ) . (1) FIGURE 4Open in figure viewerPowerPoint An illustrated scenario with N ν = 5 vehicles on the road. The camera detects N C = 4 objects and the radar detects N R = 8 objects Each vehicle i ∈ ν is moving on a specific road segment which is equipped with the Road Side Unit (RSU) consisting of a camera and a radar. At the same time instant t, the camera captures a set C = { 1 , ⋯ , N C } of N C objects on the road as is shown in Figure 4. The position of the object j ∈ C projecting on the road is denoted as P j , t ( C ) = [ P x j , t ( C ) , P y j , t ( C ) ] ∈ R 2 × 1 . Each object is a potential mapping of a real vehicle on the road. The position of the camera itself is P ( C ) = [ P X ( C ) , P Y ( C ) ] ∈ R 2 × 1 . The direction of the road is D R = [ D X R , D Y R ] ∈ R 2 × 1 , which satisfies that D X R 2 + D Y R 2 = 1 . Because the farthest distance the radar can detect is 100 m, and the shortest distance is 20 m, we only take the objects that meet the following constraints into consideration: F ( C ) = { j ∈ C | 20 ⩽ ( P j , t ( C ) − P ( C ) ) · D R ⩽ 100 } . The velocity of the object j ∈ F ( C ) can be measured to be V j , t ( C ) = [ V x j , t ( C ) , V y j , t ( C ) ] ∈ R 2 × 1 . Then the kinematics parameters of the object j ∈ F ( C ) can be described by the state vector ρ j , t C : ρ j , t C = P j , t ( C ) V j , t ( C ) . (2) Similarly, at the time instant t, the radar catches a set R = { 1 , ⋯ , N R } of N R objects. Some cars have two or more large metal reflective surfaces (e.g., roof and rear), both of which reflect the waves emitted by the radar, as is shown in Figure 5. The radar will then receive multiple reflected waves from the same object and generate multiple targets. Therefore, the value of N R tends to be larger than that of N C . The position of the object k ∈ R projecting on the road can be described as P k , t ( R ) = [ P x k , t ( R ) , P y k , t ( R ) ] ∈ R 2 × 1 and the velocity is V k , t ( R ) = [ V x k , t ( R ) , V y k , t ( R ) ] ∈ R 2 × 1 . The subset of all the detected objects to be processed can be denoted as: F ( R ) = { j ∈ R | 20 ⩽ ( P k , t ( R ) − P ( R ) ) · D R ⩽ 100 } . The kinematics parameters of the object k ∈ F ( R ) is: ρ k , t R = P k , t ( R ) V k , t ( R ) . (3) FIGURE 5Open in figure viewerPowerPoint The two reflective surfaces of the vehicle (roof and rear) reflect the waves emitted from the radar separately Focus of this paper is on the prediction of the kinematics state X i , t , based on the measurements from two kinds of sensors in (2) and (3). The proposed protocol will be given in Section 4. 3.2 Practical restriction To reflect the practical vehicle sensing and tracking scenario, we take the constraints and influence that may happen in reality into consideration. There are many problems and challenges in fusing the camera and radar data for vehicle detection [20-22]. The noise problem is shown in Figure 6. It would make the vehicle's movement trajectory jagged a lot, which may cause some troubles in real practice. The problem of noise also causes false targets, as shown in Figures 7 and 8. The feature of the false targets is that the false targets are often captured by one sensor and cannot be observed in another sensor. Tracking is also a big problem, which is shown in Figure 9. When detecting vehicles with big sizes like trucks and buses, there will be more than one target detected for one real object on the road. FIGURE 6Open in figure viewerPowerPoint The trajectory of a vehicle captured by a camera. The black line is the road line. The blue line is the data provided by the camera FIGURE 7Open in figure viewerPowerPoint False Objects Detected by Camera FIGURE 8Open in figure viewerPowerPoint An Example of False Target FIGURE 9Open in figure viewerPowerPoint Track Problems (a) Magnified details of radar and camera data. The black line is the road line. The green and red lines are the data from radar with different ID. (b) The data in (a) is the trajectory of the red truck on the right detected by the sensors When the sensors of the camera and radar are set aside on the road, new challenges emerge. In the traditional methods where the sensors are on the vehicle, the targets detected by the sensors often have relatively low speeds because they are moving in the same direction. And thus the same target in two adjacent frames does not have a big sudden change in the relative position. But in this paper, the sensors are set aside on the road, not on the vehicles. They are stationary relative to the ground, which also means that the vehicles moving at a high speed on the ground have a higher relative speed with them. It is more difficult to handle the noise and the tracking problems because the same target has greater displacement in two adjacent frames. Based on the above objective situation, in practice, the number of vehicles on the road, the number of the objects detected by the camera, and the number of the objects detected by the radar are not necessarily equal to each other. And the mapping relations between each other are not known. That means we do not know which of the objects detected by the camera or the radar represents the specific vehicle on the road. And one vehicle on the road may generate more than one object detected by a sensor. The same object in two adjacent frames may have a longer distance compared with the scenario that the sensors are set on the vehicle. 4 MULTI-SENSOR FUSION PROTOCOL DESIGN In this section, we propose a developed sensor fusion protocol. The proposed protocol is designed for roadside equipment to detect traffic and vehicular situations in real-time. The equipment is set aside on the road with a radar and a camera. However, the camera and the radar collect the road information independently, which means the data do not come synchronously. 4.1 Protocol overview Figure 10 shows the designed protocol architecture. The radar and the camera modules collect information about the environment independently. Hence, the raw data from the two modules may arrive asynchronously. We use the interpolation method to align the timestamp of these two sensors. Then, the heterogeneous raw data are treated together by the proposed structure. We transform the coordinates to make the two kinds of data comparable so that they can be matched to get higher credibility. If the match of a pair of data is successful, then both of them are sent to the data fusion module to get one piece of fused data. The fused data will be the output to the vehicle drivers on the nearby roads to tell them whether there are any potential dangers on the road. To make full use of the vehicle information and to enhance the credibility of the data, the fused data are reused during the preprocessing and the fusion process. FIGURE 10Open in figure viewerPowerPoint Protocol structure The format of some data like the position information of each vehicle is given related to the sensors. For example, the vehicle position information in the data produced by the camera is the pixel coordinate of the middle point of the rear of the vehicle, and the one produced by the radar is displayed in polar coordinates. In order to compare the position of each point from the camera and the radar to execute the further process, the first step is to convert these different kinds of coordinates into one recognized coordinate system, that is, the world coordinate system. In this paper, we suppose that each vehicle moves on the ground, which means the Z value of the world coordinate of the vehicles ( X , Y , Z ) are set to 0. In order to reduce the impact of the noise, we employ Kalman filtering [23] on the location, speed, and other information of the data. During the process, we also make a judgment whether the detected object is a real object or not. We also reduce the impact of the noise by using Bayesian Fusion in the fusion process, leading to a significantly improved result of the jagged vehicle trajectory, which we will show by results in Section 5. In the real situation, as described in Section 1, the original tracking system in camera and radar is not stable. In order to achieve satisfactory performance in the match and fusion process, it is necessary to track the vehicle again. Our objective is not to present the results of two sensors but to fuse them into one target so that the drivers can get more precise and detailed information about the vehicles around them. Therefore, after retracking the objects from both sensors again, we need to match them and fuse the data information. The Kuhn–Munkres algorithm [24] is used to solve the task assignment problem in polynomial time, it can give a perfect match to two groups with the same number of goals. However, the noise and multi-reflective surface will generate some false targets, which means the number of objects captured by the two sensors may not necessarily be equal. We solve this problem by adding objects to the small group to make sure they have the same scale. The distance between any object detected by the sensors(including the real object and the false target) is considered to be infinity so that the match result will not affect the reliability. 4.2 Coordinate convert and preprocess As described in Section 3, we first have to unify the different kinds of coordinates into one recognized coordinate system, that is, the world coordinate system so that we can compare the data from each sensor in a uniform standard. We use the Newton algorithm to transform the two kinds of coordinates into geodetic coordinates. In the preprocessing, we keep the lists of objects of the last few frames from the history information pool. Then, for each new frame, we compare the position of each vehicle in this frame C F with the position of the vehicles in the list of objects of the last frame C L . If the ID of the vehicle v 1 in C F is the same as the ID of the vehicle v 2 in C L and the position of v 1 matches v 2 's prediction, then the two vehicles are regarded as the same object. If the ID of the vehicle v 1 in C F does not match anyone in C L , then the information of this vehicle is kept as a new object to be determined. If it can be matched to the objects in the following few frames, then it is considered to be a real object and will be observed by the users. ALGORITHM 1. Preprocess 4.3 Match process with Kuhn—Munkres algorithm The fusion part uses the information provided by the two sensors. Thus, before the two objects from the camera and radar are fused, we first need to match them so that we can prove the two objects provided by the two sensors corresponding to the same target in the practical system. The match part mainly uses the Kuhn–Munkres algorithm to decide which object in one sensor has the highest probability to be matched to the one in the other. The Kuhn–Munkres algorithm requires that the two data sets should have the same size. Hence, the first step is to fill in the data set of a smaller size with some virtual objects, whose values are set to the maximum value. Then, we create a N ∗ N matrix W M a p , and set the value of the matrix the score indicating how close the two points are judged by their distance, line number, and the velocity with different weights. For example, if a frame of the camera has m objects and the corresponding frame of the radar has n objects. We suppose that n > m (generally speaking, the number of detected objects in a radar frame is often more than that in a camera frame because of the false targets). Then, the size of W M a p is n ∗ n , and the value of the element in the matrix a i j means the adjacency score between the ith object in the camera frame and the jth object in the radar frame. According to WMap, we use Kuhn–Munkres algorithm to obtain the camera's match information M I C m a x ( n , m ) and the radar's match information M I R m a x ( n , m ) . If the values of the elements in M I C m a x ( n , m ) and M I R m a x ( n , m ) are greater than the threshold we set, then we regard the two objects as successfully matched. Note that the two provided sensors work independently, and the time to produce an output is not restricted to be the same. Therefore, if a list of target information is given by radar at time T, then the targets joining in the match process in the camera are those with the nearest timestamps but smaller than T. To achieve this, after the fusion process, we keep both the camera and radar information in the history pool for a period of time. The algorithm 2 shows the procedure of the matching algorithm. ALGORITHM 2. Matching 4.4 Two-layer fusion with Bayesian process The fusion process is awakened after the matching part with the output matched pairs are provided. The main work of this process is to complement the data that is not available in one sensor and to make the common part more precise. For the data that exists in both of the sensors like position information, we fuse them through Bayesian Fusion [25-27]. In this step, we have 3 pieces of data in total: the last time's fused position, the current position provided by the camera, and the one provided by the radar. The first one provides the information of the vehicle position in the last detection, which can be the prior information in Bayesian Fusion. Moreover, with the fusion of the last two pieces of data, the position of the vehicle in the current frame can be acquired more precisely. Therefore, in this paper, we do the Bayesian Fusion twice to handle this situation, as shown in Figure 11. In the first round, we treat the last time's position information P l a s t as the prior information and the current position provided by the camera P c a m e r a as the posterior information. We fuse P l a s t and P c a m e r a to get the temporary position P t e m p o r a r y . Then, in the second round, we treat the temporary position P t e m p o r a r y as the prior information and the current position provided by the radar P r a d a r as the posterior information. We fuse P t e m p o r a r y and P r a d a r to get the current fused position P. FIGURE 11Open in figure viewerPowerPoint FusionpProcess If the object is detected for the first time, which means there is no last time position information. In this situation, we treat the current position provided by the camera P c a m