Feature‐based detection and classification of moving objects using LiDAR sensor

IET Intelligent Transport SystemsVolume 13, Issue 7 p. 1088-1096 Research ArticleFree Access Feature-based detection and classification of moving objects using LiDAR sensor Ziming Guo, Ziming Guo School of Electronic and Information Engineering, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China Beijing Engineering Research Center of EMC and GNSS Technology for Rail Transportation, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of ChinaSearch for more papers by this authorBaigen Cai, Baigen Cai School of Electronic and Information Engineering, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China Beijing Engineering Research Center of EMC and GNSS Technology for Rail Transportation, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of ChinaSearch for more papers by this authorWei Jiang, Corresponding Author Wei Jiang weijiang@bjtu.edu.cn School of Electronic and Information Engineering, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China Beijing Engineering Research Center of EMC and GNSS Technology for Rail Transportation, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of ChinaSearch for more papers by this authorJian Wang, Jian Wang School of Electronic and Information Engineering, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China Beijing Engineering Research Center of EMC and GNSS Technology for Rail Transportation, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of ChinaSearch for more papers by this author Ziming Guo, Ziming Guo School of Electronic and Information Engineering, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China Beijing Engineering Research Center of EMC and GNSS Technology for Rail Transportation, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of ChinaSearch for more papers by this authorBaigen Cai, Baigen Cai School of Electronic and Information Engineering, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China Beijing Engineering Research Center of EMC and GNSS Technology for Rail Transportation, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of ChinaSearch for more papers by this authorWei Jiang, Corresponding Author Wei Jiang weijiang@bjtu.edu.cn School of Electronic and Information Engineering, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China Beijing Engineering Research Center of EMC and GNSS Technology for Rail Transportation, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of ChinaSearch for more papers by this authorJian Wang, Jian Wang School of Electronic and Information Engineering, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of China Beijing Engineering Research Center of EMC and GNSS Technology for Rail Transportation, Beijing Jiaotong University, No.3, Shangyuancun, Haidian district, Beijing, People's Republic of ChinaSearch for more papers by this author First published: 26 March 2019 https://doi.org/10.1049/iet-its.2018.5291Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Detection and classification of moving objects is essential for autonomous driving. To tackle this problem, this paper proposes an object classification method at detection level using a single LiDAR sensor. The aim is to extract and classify all the moving vehicles, bicyclists, and pedestrians in front of the sensor. First, the point clouds are segmented to produce distinct groups of points representing different objects, where the line segments are extracted. A segmentation combination strategy is conducted to address the over-segmentation caused by occlusion. Then, considering the object geometry and reflection intensity, several features for classification are defined and extracted from different hand-labelled object classes. Reference ranges of all the features are generated on a set of experiment samples. Finally, the object class can be decided by checking if its features match with the existing feature reference ranges of a certain class. The proposed method was evaluated using datasets gathered by the vehicle demonstrator, and the experiment results show considerable improvement of classification performance compared to an existing object classification method based on the object matching framework. 1 Introduction Autonomous driving has moved to a current area of research and development of intelligent vehicle. Accurate detection and classification of moving objects surrounding the vehicle are central objectives in many prominent applications, such as simultaneous localisation and mapping (SLAM), path planning, and event monitoring [1, 2] for the purpose of enhancing driving safety. Correct detection of moving objects is the necessary prerequisite to object tracking system. Research efforts have shown the feasibility of detecting moving objects around the car using various sensors, such as video camera [3, 4], light detection and ranging (LiDAR) [5, 6], and radar [7], or through multi-sensor fusion [8-11]. Camera is easy to be applied and operated in low cost; however, it is hard to measure the distance of the identified objects and camera-based object detection approaches have lower efficiency under complex shadow and bad weather. Radar-based approaches can provide range and velocity of objects, which are commonly used in an adaptive cruise control (ACC) system due to the longer detection range and narrow field of view (FOV) [12]. LiDAR sensors provide range data with the high accuracy and resolution in a wide FOV. Additionally, they perform satisfactorily in bad weather conditions. LiDAR is commonly considered as the main sensor to detect objects in extensive research work. Classification of the objects around the ego-vehicle presents a better perception of driving environment. Understanding the classes of moving objects is helpful to object detection and tracking, and improve the behaviour prediction based on their nature [13]. For the weakness of LiDAR that it cannot provide information about object shape and colour, fusing LiDAR measurements with vision sensors has shown good performance for object classification. Zhang F. et al. [14] used the stereo camera to generate positions that represent vehicles. Verification was be achieved to classify the objects by LiDAR sensor. The complementary advantages of two sensors lead to an enhanced detection performance. However, the LiDAR/camera fusion method requires high computation cost despite the good results. Here, an object detection and classification method using single LiDAR sensor is presented. A modified clustering algorithm based on adaptive distance threshold is adopted to group the raw laser data collected by the LiDAR sensor into a set of candidate object classes. Features related to object geometry are extracted from point clouds as the input of classification in LiDAR space. A segmentation combination strategy is made to solve the problem that laser points belonging to the same object split into several parts due to occlusion and confusion. In the classification step, several reference ranges of the features corresponding to different object classes (e.g. vehicle, pedestrian, bicyclists etc.) are obtained given a set of hand-labelled examples of point clouds. After that, features are extracted from objects to be classified and compared with the feature reference ranges to solve the classification task. This classification based on multi-feature matching is considered to be the contribution of this paper. The rest of this paper is organised as follows. After a survey of related works in the next section, the sensor setup and overall method architecture is introduced in Section 3. In Section 4, the point cloud segmentation and line segment extraction are presented. Section 5 details the object classification based on feature matching. In Section 6, the experiment results are discussed. Finally, the conclusion and future work are stated in Section 7. 2 Related works Object detection based on LiDAR sensors mainly includes two methods: feature-based [15] and grid-based methods [5]. Feature-based methods process object detection by means of clustering, geometric feature fitting, and feature extraction etc. [6]. Different geometric models need to be built for different objects. Grid-based methods use discrete two-dimensional (2-D) occupancy grid maps that are divided into regular grid cells to represent the environment. In outdoor environments, complex geometries of objects are hard to describe by simple features like lines, boxes, or circles. Grid maps can describe complex geometries of objects if there is a sufficiently high resolution of the grid which requires higher computational performance. Usually, two types of LiDAR (i.e. 3-D and 2-D LiDAR) sensors are deployed for object detection and classification application. The 3-D LiDAR, such as Velodyne HDL-64, can produce accurate and large amount of point clouds from environment. Börcs A. et al. [16] adopted a grid-based point cloud segmentation method for producing distinct groups of points which represent different objects, and performed object classification with jointly utilising deep learning-based object appearance models and contextual scene analysis. Wang D.Z. et al. [17] explored the use of a Euclidean Minimum Spanning Tree for 3-D point cloud segmentation and constructed a 120-dimentional feature vector for each group of points. For classification task support vector machine (SVM) classifiers with radial basis function (RBF) was employed. Wang H. et al. [18] used Hash table in the segmentation step for searching and comparison. Curb information was taken into account to generate alarms if pedestrians were detected to be on road or close to curbs. An SVM classifier was trained to recognise pedestrians, and the performance was improved by object tracking. As for 2-D LiDAR application, Zhang H.J. et al. [19] extracted feature vectors related to object geometry and velocity during the tracking process to train the classifier to obtain the likelihood measures between each feature vector and object class. Han J. et al. [20] proposed a breakpoint detector to segment the raw LiDAR data and line segments were extracted in polar coordinates. The extracted line segments were classified into road boundary and object line segments according to their geometric characteristics. Huang R.L. et al. [21] used a distance threshold-based segmentation method to detect discontinuities in the point clouds. Three SVM classifiers were trained by spatio-temporal features extracted from objects of interest. Classification using 2-D LiDAR is always achieved at tracking level. Several sensors (e.g. GPS, speed sensors etc.) for positioning are adopted to provide global information of ego-vehicle, which helps produce complete trajectories of every object. Nevertheless, classification should be immediately accomplished within a single frame once an object suddenly appears in complicated situations. The work presented here only employs a 2-D LiDAR sensor without the use of positioning sensors. Detection and classification of moving objects are carried out in LiDAR coordinates. By taking into account the intensity of reflected laser pulse, the proposed approach can classify objects of interest at detection level, which has low time cost. 3 System overview 3.1 LiDAR setup The ibeo LUX LiDAR sensor with four scan layers is chosen to obtain datasets from different driving scenarios. The multi-layer technology of the ibeo LUX enables better perception of the objects surrounding the vehicle. Table 1 details the specifications of the chosen sensor. Table 1. LiDAR specifications FOV (deg) Angular resolution, deg Measurement rate, scans/s Maximum range, m 85∼110 0.125/0.25/0.5 12.5/25/50 200 As shown in Fig. 1, the LiDAR sensor is mounted on the bracket which has 55-centimeter elevation from the ground in front of the vehicle demonstrator so as to achieve proper measurement. Fig. 1Open in figure viewerPowerPoint Image of the vehicle demonstrator equipped with the ibeo LUX The laser pulses transmitted by the ibeo LUX are reflected by the objects in the surrounding area. The range is calculated from the time-of-flight of the laser pulse and the speed of light. This paper assumes that the LiDAR sensor provides raw laser data in the form of , where are the LiDAR coordinates of the point, is the reflection intensity of the point, N is the number of laser points within one scan. The configuration of the LiDAR coordinates is shown in Fig. 2. Fig. 2Open in figure viewerPowerPoint LiDAR coordinates 3.2 Method design The method architecture to detect and classify objects of interest is divided into three parts, as shown in Fig. 3. First, the LiDAR sensor gives raw point clouds in LiDAR coordinates. Then, for object detection, environment clutter and noise peaks are removed by data pre-processing as the first step. The segmentation task is performed using a clustering algorithm based on adaptive distance threshold, followed by the use of the iterative-end-point-fit (IEPF) algorithm [22] for line segment extraction. Accordingly, features related to object geometry and reflection intensity are obtained. The reference ranges of features corresponding to object classes of interest (e.g. car, pedestrian, bicyclists etc.) are generated using a set of hand-labelled samples from real experiment datasets. After an object movement recognition step, classification decision can be made by comparing the feature of the object to be classified with the feature reference ranges of different object classes. As a result, the list of object classes is generated. Fig. 3Open in figure viewerPowerPoint Method architecture 4 Object detection Oftentimes an unknown number of objects of interest exist in a single time frame. The ability to segment the raw point clouds into groups is required for a successful classification of these object candidates, under no conditions. This problem is formulated as a segmentation task and a line segment extraction task for following classification in this section. A data pre-processing is adopted to preclude the outliers existing in the raw point clouds. An adaptive distance threshold-based clustering algorithm is employed to obtain groups of point clouds of candidate object classes. Line segments are extracted within each segmentation. However, bad observation conditions (e.g. occlusion from other objects) lead to laser points of one object splitting into several parts, which has bad effect on estimation of an object size and position. Therefore, a segmentation combination strategy is made to deal with this problem. 4.1 Data pre-processing In the process of data collecting, there exist environment clutter and occasional noise peaks caused by vehicle bumps in raw point clouds. These points do not belong to any entity corresponding to objects of interest. The existence of the unwanted points results in more difficulties for data segmentation. The removal of the outliers must be carried out as the first step of data processing. A median filter is typically adopted to remove noise points. Y Peng et al. used a 3 × 3 median filtering window in the distance map [23], taking into consideration both space and time. The values of the distance of current point and its two neighbours are sorted using the quicksort method and the value of the current distance point is replaced by the median. Similar to this method, a three-neighbour median filter is used to eliminate noise points from the raw point clouds here. 4.2 Segmentation Consider a set of points collected by a 2-D LiDAR sensor. The information of each point contains its position in Cartesian coordinates where LiDAR is taken as its origin, as shown in Fig. 2. Han J. et al. presented a distance-threshold-based clustering algorithm to detect the discontinuity between the consecutive laser points [20]. The fundament of the method is that the two points do not belong to one same object when (1) where represent the distance between two consecutive points and within the raw point cloud, and is the distance threshold. The larger the distance from the object to the LiDAR is, the larger the distance between consecutive points corresponding to the object is due to the fixed angular resolution of LiDAR sensor. Therefore, as shown in Fig. 4, the distance threshold is determined according to the range of point by the following equation. (2) where is the range distance for point , is the angular resolution of the LiDAR sensor, is an angle threshold to extrapolate the maximum acceptable range point , is an offset relative to sensor noise. Fig. 4Open in figure viewerPowerPoint Distance-threshold-based clustering algorithm However, the points which belong to the surface relatively perpendicular to the laser ray and the surface relatively parallel to the laser ray, may not be segmented into one same object. Hence the modification of the calculation of is introduced on the basis of the existing algorithm. Let represent the positions of the two consecutive points relative to the laser ray (see Fig. 4). If is smaller than an angle threshold (i.e. the two (Fig. 5) Fig. 5Open in figure viewerPowerPoint Algorithm 1: Clustering algorithm based on adaptive distance threshold points belong to the surface relatively perpendicular to the laser ray), set a big value for to obtain a smaller . If not, set a small value for to obtain a larger . The implementation of the modified clustering algorithm based on adaptive distance threshold is shown in algorithm 1. 4.3 Line segment extraction Note that all the raw data points are recorded in the order as received. To extract line segments, the first and last laser points are connected as a line segment in a segmentation. The other laser points between the two points need to be checked whether they are included in the same line segment, as shown in Fig. 6. The maximum distance of the laser points relative to the line segment is searched for. If the maximum distance is larger than a threshold value , the line segment is divided into two line segments then check iteratively. Otherwise, the line segment represents all the laser points between the two endpoints of the line segment. The overall line segment extraction algorithm for each segmentation is detailed by algorithm 2, where is the minimum number of point elements in a line segment, s and e are the indices of the first and last laser points in a continuous sequence. Fig. 6Open in figure viewerPowerPoint IEPF algorithm 4.4 Segmentation combination The results of segmentation are often interfered with occlusion from other objects in real driving environment. This leads to wrong estimation of object size and shape, which causes mis-classification based on objects' geometry. Consider a common situation where an object is occluded by other objects closer to the LiDAR sensor. In order to combine segmentations belonging to an object into a complete one, a slope comparison method is implemented to detect the continuities between segmentations. For three adjacent point segmentations, if the distance between the first and the third segmentations is smaller than a maximum threshold value , and the change of the slope of the line segment which connect the last point of the first segmentation and the first point of the third segmentation compared with the slope of the line segments of the first or the third segmentation is smaller than a threshold value , then the first and the third segmentations become one. An example of segmentation combination is shown in Figs. 7 and 8. Fig. 7Open in figure viewerPowerPoint Results of segmentation and line segment extraction, from left to right: reference image, results before and after segmentation combination. Three objects (two cars and a person) are outlined in the camera image by hand. The farther car is partly occluded by a person. The two separated segmentations caused by occlusion are combined after applying segmentation combination strategy. Line segments extracted within each segmentation are marked in red Fig. 8Open in figure viewerPowerPoint Algorithm 2: Line segment extraction algorithm (IEPF) 5 Moving object classification In this section, a feature-based object classification method at detection level is proposed. Intensity information is taken into consideration to enhance object representation. 5.1 Object model Fig. 9 shows typical laser point clouds that correspond to different classes of objects and models are built for each of them. Fig. 9Open in figure viewerPowerPoint Some typical laser point clouds that correspond to different classes of objects. The extracted line segments are marked in red. The LiDAR drawn at the bottom is for a better understanding The data appearance of a vehicle varies as the relative direction and position to the LiDAR sensor changes. A vehicle might be scanned on one or two neighbouring sides, so the relative laser points look like one or two orthogonal edges. Accordingly, either a one-line model or a two-line (L-shape) model is built for the vehicle. In the case of a pedestrian, its point cloud appearance do not change remarkably. One line segment can be extracted if the relative laser points are sufficient enough. The pedestrian is characterised as a one-line object. As for a bicyclist, when it is scanned from its rear or head, two-line segments could be extracted from the point cloud. If the bicyclist is measured on the side, the laser points can be fit on a line segment (see Fig. 9). Hence, the bicyclist can be modelled as either a one-line object or a two-line one. The number of the line segments is the most fundamental feature to describe an object. On the basis of the object models, other features are defined in the following. 5.2 Feature definition Suppose that line segments are extracted within the point clouds of an object. For a one-line object, l denotes the length of the line segment. and are used to indicate the length of the two-line segments, respectively, of a two-line object. represents the intensity variance of all the laser points belonging to the object, the range variance, both of which depict the shape and medium characteristics of the object. All the features defined for classification are listed in Table 2. Table 2. Defined features for classification Feature Description number of the line segments l length of the line segment for one-line objects , length of the two line segments for two-line objects range variance intensity variance 5.3 Object classification A feature-matching method is implemented to classify objects of interest. Reference ranges of the features corresponding to different objects need to be generated at first. After detecting the moving state of the object, the class of it can be obtained by checking which reference ranges its features fit. 5.3.1 Feature reference range generation A point segmentation can be hand-labelled as a certain object, so the features of it are categorised as well. On the basis of large amount of point clouds from real experiment datasets, a set of feature samples can be collected for each of the defined features (see Table 2) of different classes of objects. The reference ranges of the features are determined by searching the maximum and minimum value within the feature samples. All the feature reference ranges are represented as shown in Table 3 Table 3. Feature reference ranges Reference ranges of features Description Vehicle range of line segment length for two-line vehicle range of object size for two-line vehicle range of intensity variance for two-line vehicle range of line segment length for one-line vehicle range of intensity variance for one-line vehicle Bicyclist range of line segment length for two-line bicyclist range of range variance for two-line bicyclist range of intensity variance for two-line bicyclist range of line segment length for one-line bicyclist range of range variance for one-line bicyclist range of intensity variance for one-line bicyclist Pedestrian range of line segment length for pedestrian range of range variance for pedestrian range of intensity variance for pedestrian Confusion between bicyclist and pedestrian range of line segment length for one-line object range of range variance for one-line bicyclist range of intensity variance for one-line bicyclist As one or two segments might be extracted from a possible vehicle or bicyclist, two sets of feature reference ranges are obtained for each of these two objects. Note that for one-line object, there is a range of line segment length which could lead to confusion, that it may be a bicyclist, a pedestrian or something else. A threshold for range variance is designed based on experiment data to tackle this problem. 5.3.2 Movement recognition As it is described in many research works [8, 9, 17, 19], classification mainly focuses on moving objects. The object association between two adjacent frames of LiDAR scan is made by the nearest-neighbour approach [9]. Moving objects can be recognised after searching for the displacement conflict information among all associated objects. 5.3.3 Classification Given a set of features extracted from moving objects within a single time frame, the classes of the objects are determined through comparing its features with relative reference ranges. The classification task is carried out using such a processing flow shown in Fig. 10. Fig. 10Open in figure viewerPowerPoint Object classification method based on feature matching 6 Experiment results and analysis 6.1 Experiment setup The experiment datasets were gathered from the campus of Beijing Jiaotong University using the vehicle demonstrator depicted in Section 3. The configuration of LiDAR sensor was a measurement rate of 12.5 Hz with a scan angular resolution of 0.25° considering the method performance and computational cost. Besides, a video camera was mounted to verify the results of LiDAR point cloud processing. The design parameters for the segmentation and line segment extraction algorithm were determined based on a qualitative analysis of performance on a small number of point cloud samples, as shown in Table 4. The objects extracted from the experiment datasets were manually annotated with labels vehicle, bicyclist, pedestrian by comparison with video images in order to provide a ground truth reference. The detection and classification of moving objects were performed in the LiDAR coordinates, and no positioning sensor was used. The goal of the experiment is to analyse the performance of object classification achieved at detection level. Table 4. Design parameters for the segmentation and line segment extraction Parameter Description Value Segmentation big threshold value 8deg small threshold value 5deg angular resolution 0.25deg offset 0.09 m angle threshold value 105deg line segment extraction minimum number of point elements in a line segment 2 distance threshold 0.3 m segmentation combination maximum distance value between segmentations 1.5 m slope threshold value 0.3 6.2 Classification results The method architecture from Section 3 is followed and evaluated with the gathered datasets. There are 632 objects (143 vehicles, 222 bicyclists, 267 pedestrians) extracted from the datasets according to the ground truth. Among the 632 objects, about 60% of randomly selected objects are used to determine the reference ranges of the features corresponding to different object class. The reference ranges of different features are shown in