GPNET: MONOCULAR 3D VEHICLE DETECTION BASED ON LIGHTWEIGHT WHEEL GROUNDING POINT DETECTION NETWORK

1 INTRODUCTION . 3D location and orientation detection is a basic but challenging problem in computer vision , which focuses on the prediction accuracy of visible and invisible points . It has been applied in many ways , including human action recognition , human-computer interaction , recently popular object detection and so on . In our application scenario , we define the point of wheel contacting with the ground as the keypoint in the vehicle instance . This paper mainly solves the problem of non-fixed number of vehicle keypoint detection , which is the basis of vehicle automatic driving perception technology . Recent researches have shown that deep convolutional network has powerful ability in information acquisition and image processing . Advanced network structures , such as Hourglass ( Newell et al. , 2016 ) , HRNet ( Sun et al. , 2019 ) etc. , usually have multi-scale architectures in critical point detection tasks . Location and orientation estimation tasks based on above networks with efficient transposed convolution structure can effectively solve the problem of invisible points in inference . Because they effectively combine the context information in different receptive field to ensure the highlevel semantic information and high resolution information fusion at the same time . The fusion in inference process provides a rich multi-level information . This is also an important method to improve the detection accuracy of keypoints of fixed quantity . Different from the current keypoint detection tasks , we aim to solve the problem of non-fixed number of keypoint detection . Due to the influence of shooting angle and distance , the purpose of keypoints of vehicles visible in the sample is not fixed and fluctuates violently . To adapt to the application scenes and avoid the disastrous consequences caused by the potential wrong inference of invisible points , we only forecast visible points in the image . We adopt the top-down keypoint detection strategy and put forward a novel detection process , constraining the location information of the keypoint through wheel detection . Meanwhile , the wheel area also provides abundant pixel information for keypoint detection , including visual identifiable geometric and location information . After obtaining the grounding point information , we project the 2D point to 3D coordinate . Finally , we fuse and process multi-frame vehicle ’ s location and orientation information to complete visual-only vehicle trajectory description . The inference process of vehicle trajectory proposed by us has the following innovation points : • We obtain the location and orientation in the 3D coordinate information of the vehicle by the way of keypoint detection , which doesn ’ t need radar and 3D detection technology . It reduces the time cost greatly and ensures the real-time performance at the same time . • We directly constrain the quantitative changes of keypoints through auxiliary target information , ensuring the continuity of updating the model gradient . • We combine the information of multiple frames and infer the vehicle trajectory which contains location and orientation in the 3D coordinate information of the vehicle , with high information richness . 2 RELATED WORK . The deep convolution neural network has powerful information processing capability that makes it dominate the keypoint detection task ( Belagiannis & Zisserman , 2017 ; Bulat & Tzimiropoulos , 2016 ; G. Papandreou & Murphy , 2017 ; Haoshu Fang , 2016 ; Papandreou et al. , 2018 ; X . Sun & Wei , 2017 ) .The current popular keypoint detection methods are usually based on the deep convolution network ( Belagiannis & Zisserman , 2017 ; Bulat & Tzimiropoulos , 2016 ) . From the form of the keypoint label , there are two excellent methods : the one-hot vector ( He et al. , 2018 ) and the heatmap ( F. Xia & Yuille , 2017 ; Girshick et al. , 2014 ; Pfister et al. , 2015 ; Ren et al. , 2017 ; Kaiming He , 2015 ) . One-hot mask . In Mask R-CNN ( He et al. , 2018 ) , the labels are encoded as a one-hot mask where each class of keypoint corresponds a mask . The mask is predicted by Mask R-CNN . For each of the K keypoints of an instance , the network outputs a one-hot binary mask with only one pixel is marked as the foreground . Heatmap regression and optimizing strategy . The concept of heatmap is first presented by Pfister et al . ( 2015 ) . The heatmap has local correlation that is similar to the feature map output by deep convolution neural network . There are many approaches proposed aiming to accelerate the network training based on heatmap , such as G-RMI ( G. Papandreou & Murphy , 2017 ) and DSNT ( Nibali et al. , 2018 ) , etc . These works divide heatmap regression problems into different sub-tasks or create techniques to decrease the regression difficulty . For instance , the heatmap is decomposed in a probability map and an offset map in G-RMI . The probability of a point being a keypoint , and the offset map describes the relative offset of the keypoint . The DSNT compresses the heatmap in size M×N that satisfies Gaussian distribution into a 2× 1 vector , which decrease the difficulty of regression . Our Approach . Our model is based on the top-down strategy ( Chen et al. , 2018 ; Newell et al. , 2017 ; G. Papandreou & Murphy , 2017 ) in two phases . However , our approach is different from most existing works . The difculty encountered here is the number of visible keypoints varies . We utilize feature maps as the input of the second stage instead of initial image inspired by intermediate supervision ( Szegedy et al. , 2015 ) . Compared with the initial image , feature maps contain richer information which can improve the model performance of keypoint detection . The two phase detection is shown as Figure1 , we locate the grounding point in the second stage through the wheel location information obtained in the rst stage . This method takes less time/This method more efficient obtains the accurate location and orientation information in the real 3D world coordinate . Fully utilizing the geometric constraints between the vehicle and wheel ensures the gradient of the keypoints is continuous , and completely avoids the disturbance from truncate . GPNet . Our approach focuses on learning the size information at the pixel level , therefore we propose a dual-branch prediction network . With the same amount of parameters , dual-branch structure can achieve higher accuracy than single-branch structure.As shown in Figure 1 . Gradient-based OHKM . We propose a novel online hard keypoints mining method based on the current gradients . ˆdiff = { scale× ˆdiff , ˆdiff , ( 1 ) where scale is the scaling coefficient of the gradient , ˆdiff is the initial gradient of the network , and dt is the lower limit of the gradient . In this experiment , we select dt as 0:05 and scale as 0:1 . We adjust the weights instead of truncating the gradient , which is different from OHKM ( Cao et al. , 2017 ) . Our approach ensures the continuity of gradient updates while mining hard keypoints . As a result , our method has better performance , as shown in Table 1 Fixed Range of Softmax Inputs . Just like the principle in argsoftmax , softmax has different corresponding to different range inputs , in order to improve the performance of the proposed network , we normalize the range of input in [ 0 , 20 ] before softmax operation . As shown in Table2 , it can greatly speeds up the training process and obtains the optimal gradient response . And experiments proved that this process can accelerate the training process and improve the performance of the proposed network 3 METHOD INTRODUCTION . 3.1 TOP - DOWN STRATEGIES . Vehicle keypoint detection requires high accuracy for close vehicles . However , for the distant object only location information is required . Therefore , we follow the top-down keypoint detection strategy , which only performs keypoint detection on a single instance at a time to achieve higher accuracy . 3.2 WHEEL DETECTION . The network structure of wheel detection is shown in the Figure 1 . The total network of keypoint detection is shown as Figure 3 . We cite the RoI Align module in mask R-CNN to cut down the pooling quantization error . And for further improving the recall rate of small targets , especially the wheel recall rate , residual structure is applied here as backbone to provide rich and fine-grained information for the model . 3.3 KEYPOINT DETECTION . In order to accelerate the convergence of the model and ensure the accuracy , we use DSNT to compress the spatial continuous spatial information of the network feature map . The normalized feature map of each channel maps to a two-dimensional mean point through two mapping matrices by DSNT . The mean value represents the position information of the peak point satisfying the distribution of heat map . The mapping matrices is as Equation ( 2 ) and Equation ( 3 ) separately . Xi , j = 2i− ( H + 1 ) H , i ∈ [ 1 , H ] ( 2 ) Yi , j = 2j − ( W + 1 ) W , j ∈ [ 1 , W ] ( 3 ) where H and W are the width and height of the feature map separately , Xi , j and Yi , j represents the value at ( i , j ) on the matching matrix X and Y separately.So the mapping process is as follows : x̂ = 〈 Ẑ , X 〉 F ( 4 ) ŷ = 〈 Ẑ , Y 〉 F ( 5 ) where < , > is the sum of the two matrices after the dot product . Ẑ represents the predicted value of the network , X and Y are the matrices described in Equation ( 2 ) and Equation ( 3 ) . x̂ , ŷ represents the horizontal index value and the vertical index value of the peak point coordinates on Ẑ . Therefore , the regression of H × W heatmap can be converted into the regression of 2 × 1 vector , and the transformation is spatially continuous . We also perform the same compression operation on the real heatmap . x = 〈Z , X〉F ( 6 ) y = 〈Z , Y 〉F ( 7 ) Where x , y represent the values of the real heatmap Z through the matrix X , Y compression.We choose the Euclidean distance between the label compression tensor Lxy and the predicted compression tensor Dxy as the loss function : loss = 1 N H∑ y=1 W∑ x=1 ( Dxy − Lxy ) , N = H ×W ( 8 ) Where W , H represent the high and width of the heatmap . 3.4 INFERENCE OF LOCATION . In the process of inference , the position of each point should be determined . After inference through the network , the output result is handled like Equation ? ? and Equation ? ? : locŷ = argmin i , j ( ŷ − Yij ) ( 9 ) locx̂ = argmin i , j ( x̂−Xij ) ( 10 ) Where locx̂ , locŷ represents the x , y coordinate position of the peak point on the heatmap . i , j represent the coordinate index , as described in Equation ( ? ? ) and Equation ( ? ? ) . In conclusion , our method can ensure the accuracy of vehicle yaw angle information while providing real-time performance . At the same time , it has strong robustness , especially when the model capacity is limited by storage space .

Summary: This paper introduces a method to detect cars from a single image. The method imposes several handcrafted constraints specific to the dataset in order to achieve higher improvement and efficiency. These constraints are quite strong and they not generalize to new situations (eg. a car in the sky, a car upside down a car with multiple wheels). The results do not seem particularly strong because the dataset seems easy and the improvements over previous works is small.