Coarse‐to‐fine 3D road model registration for traffic video augmentation

IET Image ProcessingVolume 14, Issue 12 p. 2690-2700 Special Section: AI-Powered 3D VisionFree Access Coarse-to-fine 3D road model registration for traffic video augmentation Zhichao Cui, Zhichao Cui College of Artificial Intelligence, Xi'an Jiaotong University, Xi'an, Shaanxi, 710049 People's Republic of China School of Advanced Technology and Science, Tokushima University, Tokushima, JapanSearch for more papers by this authorYaochen Li, Corresponding Author Yaochen Li yaochenli@mail.xjtu.edu.cn School of Software Engineering, Xi'an Jiaotong University, Xi'an, People's Republic of ChinaSearch for more papers by this authorChi Zhang, Chi Zhang College of Artificial Intelligence, Xi'an Jiaotong University, Xi'an, Shaanxi, 710049 People's Republic of ChinaSearch for more papers by this authorYuehu Liu, Yuehu Liu College of Artificial Intelligence, Xi'an Jiaotong University, Xi'an, Shaanxi, 710049 People's Republic of ChinaSearch for more papers by this authorFuji Ren, Fuji Ren School of Advanced Technology and Science, Tokushima University, Tokushima, JapanSearch for more papers by this author Zhichao Cui, Zhichao Cui College of Artificial Intelligence, Xi'an Jiaotong University, Xi'an, Shaanxi, 710049 People's Republic of China School of Advanced Technology and Science, Tokushima University, Tokushima, JapanSearch for more papers by this authorYaochen Li, Corresponding Author Yaochen Li yaochenli@mail.xjtu.edu.cn School of Software Engineering, Xi'an Jiaotong University, Xi'an, People's Republic of ChinaSearch for more papers by this authorChi Zhang, Chi Zhang College of Artificial Intelligence, Xi'an Jiaotong University, Xi'an, Shaanxi, 710049 People's Republic of ChinaSearch for more papers by this authorYuehu Liu, Yuehu Liu College of Artificial Intelligence, Xi'an Jiaotong University, Xi'an, Shaanxi, 710049 People's Republic of ChinaSearch for more papers by this authorFuji Ren, Fuji Ren School of Advanced Technology and Science, Tokushima University, Tokushima, JapanSearch for more papers by this author First published: 14 September 2020 https://doi.org/10.1049/iet-ipr.2019.1036AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract This study addresses the problem of non-perspective pose estimation from line correspondences in the traffic scenarios. A coarse-to-fine 3D road registration method is proposed for this problem in two stages. Firstly, the iterative closest point algorithm is exploited to estimate the pose coarsely. An objective function is then established to incorporate the feature correspondences for refining the coarse pose. Besides, the framework including road registration is employed for traffic video augmentation. The framework begins with the inputs of traffic videos, road information from Geographic Information Systems and 3D models of traffic elements (e.g. vehicles, pedestrians). Subsequently, 3D road model generation and point-to-line correspondence establishment are achieved in the preprossessing stage. After road and viewpoint registration, the 3D graphic engine is employed to simulate the traffic scene with the road, viewpoints and traffic elements. The augmented videos are generated by fusing the original frames and newly projected traffic elements. The authors demonstrate the superiority of the proposed registration method by the comparison to state-of-the-arts in both quantitative and qualitative experiments. In addition, the frames of the augmented videos validate the proposed method in the application. 1 Introduction The problem of 3D registration aims to estimate the absolute pose (i.e. orientation and position) of cameras with respect to the 3D model, given a set of features from the 3D model and the corresponding 2D projection on the image. Solutions to the pose estimation problem has been widely applied in the fields of visual odometry [1–3], image-based localisation and navigation [4, 5], and augmented reality [6–9] etc. The pose estimation from a single image has been formulated two main types according to the feature correspondences. Ones solve the pose estimation by providing the n 2D–3D point correspondences, which is also called Perspective-n-Point (PnP) problem. Many efficient solutions [10–12] were put forward and succeeded in the PnP problem to some degree. With huge emergency of line feature extraction methods [13], the line correspondences are taken into account for this task. Many researchers [14, 15] refer to the pose estimation by line correspondences as Perspective-n-Line (PnL) problem. No matter what kinds of correspondences, the transformation estimation of a single camera is totally called perspective pose estimation. In recent years, with increasing utilisation of multi-camera systems in the intelligent robots, autonomous vehicles etc., the solutions for single camera pose estimation cannot be applied into the multi-camera systems. Thus, the perspective pose estimation has been extended and investigated for multi-camera system, which is called non-perspective pose estimation. The essential difference between the perspective and non-perspective pose estimation is that for the latter, there exists more than one camera centres [16]. Besides, the latter problem is converted to the former one, when multi-camera system degenerate to a single camera. Thus, the solutions to non-perspective pose estimation are generally efficient for the perspective pose estimation [16]. According to correspondences, these problems are still classified into point-based and line-based non-perspective pose estimation. The point-based non-perspective pose problem, which is also known as non-perspective-n-point (NPnP), was relatively fully studied in the recent decade. On the contrary, line-based non-perspective pose estimation, which is called non-perspective-n-line (NPnL), was first put forward by Lee [16] in 2016. Thus, the NPnL problem remains relatively un-investigated. In this paper, we aim to estimate the relative pose (i.e. position and orientation) between the cameras and the model of the traffic road, which also belongs to the problem of non-perceptive pose estimation. The main reason for traffic scenarios is that one important application is to utilise the augmentation reality to synthesise and render new traffic images according to individual demands. The newly synthesised traffic images can be supplemented to the open-sourced image data (such as KITTI [17], RobotCar [18], ApolloScape [19]) for testing the visual algorithms (such as vehicle detection [20–22]) of self-driving car. This task is still full of challenges in the following reasons. (1) The 3D road models of traffic scenes are hard to acquire in actual. Generally, the prevalent fashions utilise the sensors (such as lidar, camera) to capture the data of objects. The corresponding methods [23, 24] are proposed to generate 3D models. These fashions need expensive sensor equipment and lots of time-consuming manual processing, which is suitable for small objects. However, it is generally costly for modelling the traffic scenarios through such ways. Thus, we propose a 3D wire-frame road model from the Geographic Information Systems (GIS) information (Google Earth) instead. The processing method for the wire-frame model is relatively low-cost compared to that from images and lidar points [25, 26]. (2) Based on the aforementioned wire-frame road model, the existing non-perspective pose estimation solutions from line correspondences are not robust and accurate enough to achieve the augmentation reality. Thus, a two-stage coarse-to-fine registration method is proposed to estimate the poses of non-perspective cameras from point-to-line correspondences. Firstly, the proposed method adopts the iterative closest point (ICP) algorithm to match the road model and the viewpoints of multi cameras to estimate the coarse pose. Secondly, the objective function combines the point-to-line feature correspondences to refine the coarse pose. Besides, the framework of augmentation reality is employed in order to achieve the application of synthesising images. As depicted in Fig. 1, the framework consists of viewpoint registration, video augmentation etc. except for aforementioned wire-frame road and road registration (non-perspective pose estimation). Each module in the framework has been appointed as a specific method to achieve its function. In this framework, the road model and traffic images have been matched after road registration. We can add some new traffic elements (such as cyclists, vehicles) and plan their trajectories by actual requirements. Finally, the newly traffic images are generated after projection. The newly traffic images prove that this combination of modules with the specific method in this framework is effective. Fig. 1Open in figure viewerPowerPoint Pipeline of the proposed registration and augmented framework. Input includes two videos from binocular cameras, road information from GIS and camera parameters. Preprocessing obtains the road model and correspondence establishment, where blue points in the model are labelled from GIS; the labelled red lines in the image and the blue points from the model construct correspondences. In Registration, the poses of viewpoints are estimated first in viewpoint registration; the pose of the road is estimated in road registration. In Augmentation, the road, viewpoints and newly added traffic elements are added for simulation; the augmented frames are generated by projection and fusing The contributions of this paper are presented as follows: (1) To the best of our knowledge, it's the first time to introduce the non-perspective pose estimation in the traffic scenario. Besides, we propose a new coarse-to-fine two-stage scheme to estimate the road pose. Within the scheme, an objective function combining the 3D–2D pre-defined feature correspondences is designed for pose refinement. The comparison experiments verify the effectiveness and superiority of our method. (2) This paper provides a convenient method to make the wire-frame road model from GIS information. Generally, many GISs, such as Google Earth, contain the free and global road information. Through this way, the wire-frame model of any road can be extracted and made instead of field measurement. (3) This paper exploits AR methods to synthesise new traffic videos according to the test requirements of autonomous vehicles in a 3D graphic engine. In the application, two kinds of traffic elements (i.e. vehicles, cyclists) are supplemented and the new augmented traffic videos are generated by the proposed method. This paper builds on and extends our previous work [27], with a more detailed description of the approach, additional comparison results and analysis, and the application of traffic image augmentation. The remaining part of this paper is organised as follows: A review of the recently related works is given in Section 2. Section 3 provides an outline of the proposed method, including the problem formulation and the pipeline of the total framework. Section 4 introduces the details of preprocessing about the multi-modal data (traffic videos, 3D road model). As described in Section 5, the road and videos are registered in the traffic scene. Section 6 presents the details of the traffic video augmentation. Section 7 presents the qualitative and quantitative experimental results. Section 8 displays some frames of the augmented traffic videos. Finally, we close this paper with the conclusion in Section 9. 2 Related work This section will give a full discussion and introduction about the existing relevant solutions to the NPnL problem. As stated above, NPnL solutions aim to estimate the poses of multi-cameras with respective to a model measured in the world coordinate by providing the line correspondences. The NPnL problem was first proposed in 2016 by Lee [16], which utilises the Plücker line representation to provide a solution for a minimal configuration (three cameras and line correspondences). The method [28] studied two solvers for the newly proposed minimal configuration of a combination of points and lines. Subsequently, the approach [29] still employed the Plücker coordinate to estimate the pose in the urban environment. Employment of the IMU sensors simplifies the pose estimation to 4 DoF (Degree of Freedom) problem [29]. Besides the Plücker coordinate, there exist another representation form and the corresponding methods for the NPnL problem. Horanyi and Kato [30] first exploited the representation of a point and direction vector and proposed the cubic polynomial solutions for the problem given the vertical vector. Abdellali and Kato [31] provided two kinds solvers to estimate the absolute and relative pose simultaneously. Taking the outliers and noises into consideration, Abdellali et al. [32] put forward a minimal solvers with the RANSAC scheme to estimate the pose and eliminate the outlier simultaneously. Besides, the method [32] can be applied to not only perspective but also omnidirectional cameras. Abdellali et al. [33] utilised a direct least squares solver with a Grobner basis for both the minimal and general configuration. According to the various representations of 3D lines, the aforementioned methods can be classified and summarised in Table 1. The first category of Table 1 utilises the Plücker coordinate to represent a 3D line segment [16, 28, 29]. Concretely, any 3D line segment consists of two vectors: a direction unit vector of the line and the vector which is perpendicular to the plane consisting of the line and the origin. The methods of second categories [30–33] express a 3D line segment utilising a unit direction vector of the line and a 3D point located on the line. Besides, these two kinds of methods attempt to estimate the pose in 3D space. However, these two kinds of representations indicate that the length of the line segment is infinite. The parts located on the infinite line rather than the line segment will have a big influence on the pose estimation. Besides, utilisation of the direction vector for representation is often suitable for the scene with lots of intersecting lines (such as building). The lines from the road model in this paper are relatively parallel in the 3D space. Thus, these representations are sometimes inaccurate when the scene is large scale (such as traffic scenarios) and consists of some parallel lines. Different from these representations, a line segment is expressed as a set of points located on itself in this paper. Furthermore, the pose is estimated through minimising the distance between 2D lines and projected points in the image. Experiments prove our accuracy exceeds others in the real traffic scenario. Table 1. Description of the solutions to NPnL problem Papers Line Representation Space [16, 28, 29] Plücker coordinate 3D [30–33] a point and direction vector 3D Ours a set of points 2D(image) 3 Overviews 3.1 Problem formulation As mentioned in the introduction, finding the rigid transformation between multi-cameras and road model from line correspondences belongs to non-perspective pose estimation and non-perspective-n-lines (NPnL) problem. The solutions to these problems are also the 'Road Registration' module of the framework in Fig. 1. As defined in [16], we are given the 3D line features from the road model in the world coordinate , and its corresponding 2D projected lines by N cameras. The N cameras are fixed in the reference coordinate , and the conversion from reference to the ith camera coordinate is known and denoted as . The intrinsic parameters of the ith cameras () has been calibrated. Thus, the rigid transformation between cameras and road model can be determined by combination of the transformation and . In the 1, the point in the world coordinate is converted to in the ith camera coordinate through the transformation , . Fig. 2 gives an example to explain the problem, where the 3D red line in the model and 2D red line on the image establish a correspondence and can be converted to each other through the and . Utilisation of the relationship between line correspondences can estimate the transformation . As the black points located on the red line in Fig. 2, the 3D line is expressed as a set of 3D points in our method. Thus, it is equivalent to solve the NPnL problem utilising the point-to-line correspondences. (1) Fig. 2Open in figure viewerPowerPoint Instance of non-perspective pose estimation for traffic road 3.2 Framework As shown in Fig. 1, the pipeline of the augmented reality framework contains four stages: input, pre-processing, registration and video augmentation. (1) The input includes traffic videos, camera intrinsic matrix, road information from GIS and camera baseline . (2) The pre-processing stage contains wire-frame road model generation (Section 4.1) and point-to-line feature correspondence establishment (Section 4.2). (3) The registration contains two parts: viewpoint registration, road registration. The viewpoint registration (Section5.1) is to estimate the rigid transformation of all cameras with respective to aforementioned reference coordinate (i.e. ), which is the input of NPnL problem. Road registration takes the feature correspondences, camera poses with respective to reference coordinate (), intrinsic matrix of camera as the input to solve the NPnL problem mentioned in problem formulation section (Section 3.1). As shown in Fig. 1, the two-stage registration method estimates the transformation from initialisation (Section 5.2.1) to refinement (Section 5.2.3). (4) Finally, in the augmentation, the road and added traffic elements are put into a 3D graphic engine to synthesise the new videos (Section 6). 4 Multi-modal data preprocessing The prerequisite of non-perspective pose estimation is the 3D–2D feature correspondences. Furthermore, the 3D feature is generally derived from the 3D road model. Thus, this section mainly includes two aspects: (1) the wire-frame road model generation from road information. (2) Feature correspondence establishment between the traffic image and road model. 4.1 Road model Inspired by Li et al. [34], the corridor road model is employed in this paper. The road model consists of three parts: road surface, right and left walls. As shown in Fig. 3b, the road surface is wire-frame model consisted of triangle meshes. The left and right walls are composed of the triangle mesh planes, which are perpendicular to the road surface and pass the boundary lines. The reason why utilising this model is that this structure can provide a movement space for traffic elements. Specifically, the traffic elements move on the road surface and within the boundaries of left and right walls. Fig. 3Open in figure viewerPowerPoint Utilising the Google Earth software to extract the road model (a) Road in Google Earth. The red lines are the labelled lanes and boundaries, (b) The wire-firm road model composed of three parts: right wall, left wall and road surface, where the green points are labelled in Google Earth To make such a road model, the Google Earth Software containing abundant GIS information is employed to obtain the road information. The road information includes the boundary and lane lines of a road, which are manually labelled by points in the Google Earth software. As shown in Fig. 3a, the red lines including lanes and boundaries on the road surface are labelled by blue points. After acquiring the road information, the points on the lines are converted from the WGS84 (World Geodetic System 1984) [35] to ENU (local East, North, Up) coordinate. In terms of the points on left and right walls, the normal direction of the road surface is estimated based on the labelled road points by Torr and Zisserman [36]. The points on the walls can be calculated through boundary points and the normal direction. After obtaining the points of the road model relative to the ENU coordinate, the triangle meshes of the surface, left and right walls are generated by Lee and Schachter [37], respectively. 4.2 Feature correspondence establishment Accurate feature correspondences are crucial for 3D–2D road registration. As mentioned in the introduction, this paper utilises the point-to-line feature correspondences. In terms of the 3D point feature, the 3D points of road boundaries and lane lines have been labelled and calculated relative to the ENU coordinate. With respect to the 2D lines in the images, the straight road lines including lanes and boundaries are manually labelled by points. Subsequently, the parameters of each line can be calculated by linearly fitting the points. Without decreasing the accuracy of the registration algorithm, we only label lines from some images of each sequence for reducing workloads. The reasons are as follows: (1) The information of the road between consecutive frames is redundant. There is no need to label all frames of a sequence. (2) Since the road geometry is a strong constraint in the model, only labelling few frames can achieve good registration performances. These two reasons indicate the frames to be labelled have a long interval. Finally, the 3D points and the corresponding lines in the frames construct the feature correspondences. 5 Registration As shown in Fig. 1, the registration module consists of two parts: viewpoint registration and road registration. Viewpoint registration aims to estimate the camera poses with respect to the reference coordinate (i.e. , ). Road registration is to find the transformation between the reference and world coordinate, (i.e. , ). 5.1 Viewpoint registration Viewpoint registration is to recover the poses (i.e. , ) of the viewpoints of all frames in the reference coordinate. There exist plenty of perfect algorithms, such as structure from motion (SfM), stereo vision and SLAM (simultaneous localisation and mapping), to solve this problem. This paper utilises the ORB feature [38] based SLAM method (i.e. ORB-SLAM2 [2]) to estimate the viewpoints from image sequences by giving the camera intrinsic matrix and camera baseline . To extract the ORB features, we set a six-level pyramid and extract one thousand amount of multi-scale features [38] for an image. Through this method, the poses of viewpoints (i.e. , ) are estimated relative to the reference coordinate. 5.2 Road registration 5.2.1 Initialisation The aim of this paragraph is to estimate a rough pose () of the road, which provides a robust and suitable initial value for optimisation. First of all, the road is represented by points of a lane line in the model. The registered viewpoints of an image sequence can be also regarded as a 3D point cloud. Thus, the 'Point-to-Point' based ICP algorithm [39] can be employed to match the road and viewpoints. Meanwhile, the viewpoints can be registered in the road model by the estimated transformation parameters . 5.2.2 Point selection The feature correspondences cannot be utilised in the following objective function directly, since some points are not located in the image after projection in actual. Thus, the aim of this paragraph is to select the 3D point (the th feature point in the th line) belonging to the th frame. The principle to select points of the road model for a specific viewpoint is that the points of the road can be seen in this viewpoint. In other words, the points must locate in the region of the view frustum. Fig. 4 illustrates the view frustum, which is determined by five planes (up plane, down plane, left plane, right plane, image plane). Given the intrinsic matrix and image size, the five planes can be expressed as (2), where denote the four corner points (up-left, down-left, down-right, up-right point) in the pixel coordinate; denote the normal vectors of four planes (shown in Fig. 4); is the focal length of the camera and denote any 3D point located on the view frustum. Thus, the region of the view frustum can be expressed in (3). (2) (3) Fig. 4Open in figure viewerPowerPoint Utilising the view frustum of each frame to select points of the road. The blue region is the frustum determined by five planes. The red points on the road model are located in the frustum In terms of selecting the suitable points for the th frame, all the points of road model firstly are converted into the th camera coordinate by the rotation matrix and translation vector . Then the point satisfying the (3) is located in the view frustum (like the red points in Fig. 4). Finally, an amount of points near to the viewpoint of the th frame are selected to construct the feature correspondence and (). 5.2.3 Pose refinement The objective function is designed to refine the transformation matrix , of the road model in this paragraph. In terms of the th frame, there exist an amount of feature lines , an amount of 3D points (=1, …, ) corresponding to the feature line . The pose of the th viewpoint, i.e. , , has been obtained in Section 5.1. As shown in Fig. 5, the pixel point in the th frame is calculated after transformation and projection by providing the transformation parameters , , , and intrinsic matrix , as follows: (4) Fig. 5Open in figure viewerPowerPoint Explanation figure for objective function construction However, the is in homogeneous format. Thus, the vector is employed to extract the value of the projected point , and the final point in the pixel coordinate is . Then, the Euclidean distance between the projected point and corresponding line is calculated by the following equation: (5)In terms of all viewpoints, the summation of all the Euclidean distances forms the objective function as follows: (6)In the ideal situation, the projected point should be located on the corresponding line after a series of transformations. Thus, the accurate pose () of the road model is determined through minimising the objective function. The matrix has only 3 DoF and satisfies the constraint in 6. Thus, the matrix is replaced by the rotation axis and angle , namely Rodrigues [40] equation . After the substitution, the final format of the objective function is shown in (7), where the rotation axis is the unit vector; the axis degree is in the range of and radians; and the components of the translation vector , i.e. , , , are in the range of and 5 metres. (7)For optimisation, it is hard to get closed-form solution for the objective function (7). Thus, the gradient descent method based iterative scheme is employed for optimisation. Meanwhile, a good initial value and suitable step size is must. In actual, the initial values of the rotation axis , rotation angle and translation vector are the normal vectors of the road surface, 0 degree and zero vector, respectively. The interior algorithm [41] is exploited for iterative updating. To avoid obtaining the local minimum by single initial value, we exploit the scatter-search mechanism [42] to acquire multiple initial values in the scope of constraints. The global minimum can be found by comparing all the local optimums. After estimating the accurate pose of the road model, the road and viewpoints have been registered in the world coordinate. Besides, they can be converted into the global coordinate (WGS84). The conversion is achieved by two steps: from ENU to ECEF and from ECEF to WGS84, which is the inverse process of that in Section 4.1. Algorithm: 1 (see Fig. 6). Fig. 6Open in figure viewerPowerPoint Algorithm 1: Coarse-to-fine registration for road pose estimation 6 Video augmentation One of the major applications is to utilise augmented reality method to synthesise new traffic videos in a 3D graphic engine. The process includes traffic scene simulation and traffic image augmentation. Traffic scene simulation utilises traffic elements including road model and newly added elements to construct the dynamic scenario. Image augmentation is to project the newly supplemented elements to generate new traffic images with time order. Before these, the capturing time of all the frames of the video is imported for scene simulation and image augmentation. To construct the traffic scene, the road model is placed in the traffic scene with the estimated pose first. The models of new traffic elements (e.g. vehicles, pedestrians) are put on the road surface. Besides, the paths of moved elements are planned in advance. In order to ensure all the elements moving within the road region, the 'gravity' and 'collision detection' function are added for the new traffic elements in the 3D graphic engine. After constructing the traffic scene, a virtual camera is exploited for capturing newly added traffic elements in the graphic engine. To ensure the quality of augmented image, the parameters (i.e. extrinsic and intrinsic matrices) of the virtual camera are same with those of the real one. Through fusing the projected traffic elements and the original images, the new frames can be generated with time. 7 Experiments In this section, two kinds of experiments are conducted to validate the effectiveness of the proposed method. One is the quantitative experiments to evaluate and analyse the 3D–2D registration accuracy. Another is the qualitative experiments to illustrate the 3D registration res