Patent Publication Number: US-11397242-B1

Title: 3D object detection method based on multi-view feature fusion of 4D RaDAR and LiDAR point clouds

Description:
CROSS REFERENCE TO THE RELATED APPLICATIONS 
     This application is based upon and claims priority to Chinese Patent Application No. 202110200544.2, filed on Feb. 23, 2021, the entire contents of which are incorporated herein by reference. 
     TECHNICAL FIELD 
     The present invention relates to the field of automatic driving, and in particular to a 3D object detection method based on multi-view feature fusion of 4D RaDAR and LiDAR point clouds. 
     BACKGROUND 
     The four core technologies of autonomous driving are environment perception, precise localization, path planning, and actuation by wire. Environmental information is acquired by a camera, an ultrasonic RaDAR, a LiDAR, a RaDAR, an inertial navigation system and a global satellite navigation system, and a high-precision map is built for vehicle localization, thereby predicting conditions of surrounding static and dynamic scenes. Environmental perception information and vehicle localization information are fused to provide an input to an anomaly processing system, to generate path planning. Finally, in actuation by wire, steering control, drive control and brake control signals are sent to a decision system by trajectory tracking, and a specific drive program and an execution component in an execution layer executes instructions sent by a decision layer. 
     Currently, the key to autonomous driving is still environmental perception, such as obstacle detection, lane line detection, road information reading, map building and assisted positioning, and detection and identification of traffic participants (vehicles, pedestrians, animals, etc.). Main sensors are cameras, LiDARs and RaDARs. Cameras have the advantages of a low cost and mature development of 2D algorithms, and obvious disadvantages such as low adaptability to scenes with obvious changes in light illumination, and rainy, snowy and foggy weather, and bad functional performance in distance and speed measurements. LiDARs have the advantages of high-level distance measurement and resolution, good directionality, and high anti-interference ability, and the disadvantages of a high cost and influence by rainy, snowy and foggy weather. Advantages of RaDARs include strong environmental adaptability, all-weather and all-time working characteristics, strong ability of longitudinal object distance and speed detection, long-distance perception and detection, and high-precision measurement for both static and dynamic objects. 
     In typical 3D object detection methods, a point cloud is snowy, used as an input, and a directed 3D bounding box is generated for each detected object. These methods can be grouped into two main categories: region proposal methods and single-stage methods. In a region proposal method, several region proposals are provided indicating regions which contain objects, and then region features are extracted to determine a class label for each proposal. The single-stage method uses a single-stage network to directly predict a class probability and regress a 3D bounding box of an object, without region proposal generation and post-processing, and is faster in operation. Both methods localize a 3D bounding box by extracting features of point clouds and inputting the same to a network, so the number and precision of the point clouds on an object determine whether the object can be detected effectively. For example, in the range of more than 70 meters, a 16-line LiDAR often only irradiates a single beam of point cloud on the object, making it impossible to describe the class and size of the object. In rainy, snowy and foggy weather, the point cloud quality is further deteriorated as the LiDAR sensor is blocked by environmental factors such as raindrops and snowflakes. A RaDAR produces point clouds within 100 meters, does not attenuate with distance, and is less affected by rainy, snowy and foggy weather. 
     Most open-source datasets contain 3D RaDAR data. For example, the nuScenes dataset contains only about more than 100 3D RaDAR points, each containing horizontal position information and velocity information (x, y, v). The Astyx dataset used in the present invention has about more than 1000 4D RaDAR point clouds, each having 3D position information and velocity information (x, y, z, v). Compared to other datasets, Astyx is more suitable for extracting features using a 3D object detection algorithm and fusing them effectively with those of a LiDAR. 
     In summary, to achieve a perception system adapted to special weather and long-distance high-resolution detection, 4D RaDAR and LiDAR data need to be fused to perceive the surrounding environment. A LiDAR can provide high-precision point cloud data at a close distance due to high density and precision of point clouds thereof, but such data needs to be complemented by RaDAR data at a longer distance. However, the RaDAR data has low vertical accuracy and low point cloud density and high noise, and needs deep fusion with the lidar data at an eigenvalue extraction stage. 
     SUMMARY 
     The present invention aims to overcome the above technical shortcomings and provides a 3D object detection method based on multi-view feature fusion of 4D RaDAR and LiDAR point clouds, which can learn the correlation between two modalities and intra-modal correlation through multiple fields of view and an attention mechanism, and thereby extract more effective features. 
     To achieve the above object, Embodiment 1 of the present invention provides a 3D object detection method based on multi-view feature fusion of 4D RaDAR and LiDAR point clouds, the method including: 
     simultaneously acquiring RaDAR point cloud data and LiDAR point cloud data; and 
     inputting the RaDAR point cloud data and LiDAR point cloud data into a pre-established and trained RaDAR and LiDAR fusion network and outputting a 3D object detection result, 
     wherein the RaDAR and LiDAR fusion network is configured to learn interaction information of a LiDAR and a RaDAR from a bird&#39;s eye view and a perspective view, respectively, and concatenate the interaction information to achieve fusion of the RaDAR point cloud data and LiDAR point cloud data. 
     As an improvement of the above method, the RaDAR and LiDAR fusion network includes: a voxelization module, a feature fusion module, a RaDAR and LiDAR feature interaction module, a pseudo-image processing module, a 2D convolutional neural network, and a detection head; 
     the voxelization module is configured to voxelize the RaDAR point cloud data and LiDAR point cloud data in the bird&#39;s eye view, respectively, and output pillar features of the RaDAR point cloud data and pillar features of the lidar point cloud data; and voxelize the RaDAR point cloud data and the lidar point cloud data in a perspective view, respectively, and output pyramid features of the RaDAR point cloud data and pyramid features of the lidar point cloud data; 
     the feature fusion module is configured to concatenate the pillar features of the LiDAR point cloud data and the pyramid features of the RaDAR point cloud data, concatenate the pillar features of the RaDAR point cloud data and the pyramid features of the LiDAR point cloud data, and input the two types of stitched features into the RaDAR and LiDAR feature interaction module; 
     the RaDAR and LiDAR feature interaction module is configured to learn the interaction information of the LiDAR and the RaDAR from a bird&#39;s eye view, and learn the interaction information of the LiDAR and the RaDAR from a perspective view to obtain a LiDAR feature with RaDAR interaction information and a RaDAR feature with LiDAR interaction information; and concatenate the features in a channel dimension to obtain a feature F, which is input to the pseudo-image processing module; 
     the pseudo-image processing module is configured to encode, by location, the feature F output by the RaDAR and LiDAR feature interaction module into an x-y plane according to coordinates of each voxel generated in the voxelization module to form a 128-channel pseudo-image; 
     the 2D convolutional neural network is configured to extract multi-scale feature information from the 128-channel pseudo-image and output the same to the detection head; and 
     the detection head is configured to process the feature output by the 2D convolutional neural network and output a object detection result. 
     As an improvement of the above method, the voxelization module includes a RaDAR point cloud data pillar feature extraction unit, a LiDAR point cloud data pillar feature extraction unit, a RaDAR point cloud data pyramid feature extraction unit, and a LiDAR point cloud data pyramid feature extraction unit; 
     the RaDAR point cloud data pillar feature extraction unit includes a first fully connected layer, a first bird&#39;s-eye view, a first fully connected layer and a first maximum pooling layer; the RaDAR point cloud data is dimensionally expanded to 64 dimensions through the first fully connected layer, and voxelized in the BEV field of view through the first bird&#39;s-eye view, and then enters the first fully connected layer and the first maximum pooling layer to output the pillar features of the RaDAR point cloud data; 
     the LiDAR point cloud data pillar feature extraction unit includes a second fully connected layer, a second bird&#39;s eye view, a second fully connected layer and a second maximum pooling layer; the LiDAR point cloud data is dimensionally expanded to 64 dimensions through the second fully connected layer, and voxelized in the BEV field of view through the second bird&#39;s eye view, and enters the second fully connected layer and the second maximum pooling layer to output the pillar features of the LiDAR point cloud data; 
     the RaDAR point cloud data pyramid feature extraction unit includes a third fully connected layer, a first perspective view, a third fully connected layer and a third maximum pooling layer; the RaDAR point cloud data is dimensionally expanded to 64 dimensions through the third fully connected layer, and voxelized in the PV view through the first perspective view, and enters the third fully connected layer and the third maximum pooling layer to output the pyramid features of the RaDAR point cloud data; and 
     the LiDAR point cloud data pyramid feature extraction unit includes a fourth fully connected layer, a second perspective view, a fourth fully connected layer and a fourth maximum pooling layer; and the LiDAR point cloud data is dimensionally expanded to 64 dimensions through the fourth fully connected layer, and voxelized in the PV view through the second perspective view, and then enters the fourth fully connected layer and the fourth maximum pooling layer to output the pyramid features of the LiDAR point cloud data. 
     As an improvement of the above method, the RaDAR point cloud data pillar feature extraction unit is specifically implemented in the following process: 
     projecting the RaDAR point cloud data onto the x-y plane to form a grid of H×W, thereby dividing the same into H×W pillars of a volume of 0.16×0.16×4, 
     wherein each point of an original 4D RaDAR point cloud has 4 dimensions (x,y,z,r), where (x, y, z) is 3D coordinates, and r is reflectivity; there are many points within each pillar, and each point is expanded to 10 dimensions (x,y,z,x c ,y c ,z c ,x p ,y p ,z p ), which are calculated as: 
     
       
         
           
             
               { 
               
                 
                   
                     
                       
                         [ 
                         
                           
                             x 
                             c 
                           
                           , 
                           
                             y 
                             c 
                           
                           , 
                           
                             z 
                             c 
                           
                         
                         ] 
                       
                       = 
                       
                         
                           [ 
                           
                             x 
                             , 
                             y 
                             , 
                             z 
                           
                           ] 
                         
                         - 
                         
                           [ 
                           
                             
                               x 
                               m 
                             
                             , 
                             
                               y 
                               m 
                             
                             , 
                             
                               z 
                               m 
                             
                           
                           ] 
                         
                       
                     
                   
                 
                 
                   
                     
                       
                         [ 
                         
                           
                             x 
                             p 
                           
                           , 
                           
                             y 
                             p 
                           
                           , 
                           
                             z 
                             p 
                           
                         
                         ] 
                       
                       = 
                       
                         
                           [ 
                           
                             x 
                             , 
                             y 
                             , 
                             z 
                           
                           ] 
                         
                         - 
                         
                           [ 
                           
                             
                               x 
                               g 
                             
                             , 
                             
                               y 
                               g 
                             
                             , 
                             
                               z 
                               g 
                             
                           
                           ] 
                         
                       
                     
                   
                 
               
             
               
           
         
       
     
     in the formula, (x c ,y c ,z c ) is a deviation of a point within the pillar relative to a pillar central point, (x m ,y m ,z m ) is pillar central point coordinates, (x p ,y p ,z p ) is a deviation of the point relative to a grid central point, and (x g ,y g ,z g ) is grid central point coordinate; 
     each frame of RaDAR point cloud data forms a tensor of dimensions (D p , N, P), where D p  is dimensions of the point, D p =10, N is the number of points sampled for each pillar, N=32, P=H×W; each pillar with more than N points is randomly downsampled, and each pillar with less than N points is filled with 0 values; and 
     the tensor is the pillar feature of the RaDAR point cloud data. 
     As an improvement of the above method, the RaDAR point cloud data pyramid feature extraction unit is specifically implemented in the following process: 
     projecting the RaDAR point cloud data onto the x-y plane to form a grid of H×W, thereby dividing point clouds within a pyramid with a vertical angle θ of [−26°, 6°] and a horizontal angle φ of [−90, 90°] into H×W pyramids, 
     wherein a maximum of N points are randomly sampled within each pyramid, and each pyramid with less than N points is filled with 0; and 
     each frame of RaDAR point cloud data forms a tensor of dimensions (D L , N, P), where D L =4, P is the number of pyramids H×W, N is the number of points within each pyramid, N=32, and the tensor is the pyramid feature of the RaDAR point cloud data. 
     As an improvement of the above-mentioned method, a specific implementation process of the feature fusion module is as follows: 
     concatenating the pillar feature from the LiDAR and the pyramid feature from the RaDAR to form a 14-dimensional feature vector F L :
 
 F   L =Concat( F   Lpi   ,F   Rpy )
 
     where F Lpi  is the pillar feature from the LiDAR, and F Rpy  is the pyramid feature from the RaDAR; and Concat represents a feature stitching operation; 
     concatenating the pillar feature from the RaDAR and the pyramid feature from the LiDAR to form a 14-dimensional feature vector F R ,
 
 F   R =Concat( F   Rpi   ,F   Lpy )
 
     where F Rpi  is the pillar feature from the RaDAR, and F Lpy  is the pyramid feature from the LiDAR; and 
     inputting the feature vectors F L  and F R  into the RaDAR and LiDAR feature interaction module, respectively. 
     As an improvement of the above-mentioned method, a specific implementation process of the RaDAR and LiDAR feature interaction module is as follows: 
     expanding the feature vector F L  into a 64-dimensional feature F L     64    through an FC layer and a Maxpool layer, and performing a convolution operation to dimensionally reduce the feature F L     64    to a 16-dimensional feature F L     16   :
 
 F   L     64   =Maxpool(Linear( F   L ))
 
 F   L     16   =Conv(Maxpool(Linear( F   L     64   )))
 
     expanding the feature vector F R  into a 64-dimensional feature F R     64    through a FC layer and a Maxpool layer, and performing a convolution operation to dimensionally reduce the feature F R     64    to a 16-dimensional feature F R     16   :
 
 F   R     64   =Maxpool(Linear( F   R ))
 
 F   R     16   Conv(Maxpool(Linear( F   R     64   )))
 
     where Conv represents a convolutional layer, Maxpool represents a maximum pooling layer, and Linear represents a fully connected layer; 
     transposing the 16-dimensional feature F L     16    and then multiplying the same with the 16-dimensional feature F R     16   , and performing a Softmax normalization operation to generate a weight matrix F Lw  of size M×N:
 
 F   Lw =Sfot max(( F   L     16   ) T   F   R     16   )
 
     wherein in the formula, Softmax represents the normalization operation; 
     transposing the 16-dimensional feature F R     16    and then multiplying the same with the 16-dimensional feature F L     16   , and performing a Softmax normalization operation to generate a weight matrix F Rw  of size N×M:
 
 F   Rw =Sfotmax(( F   R     16   ) T   F   L     16   )
 
     multiplying the weight matrix F Rw  with F L     64    to obtain a new 64-dimensional feature vector, subtracting F R     64    from the feature vector, and after processing by a linear layer, a normalization layer and an ReLU activation function, adding F R     64    thereto, to obtain a RaDAR feature F Rt  with LiDAR interaction information:
 
 F   Rt =ReLU( BN (linear( F   Rw   F   L     64     −F   R     64   )))+ F   R     64    
 
     where ReLU is the activation function, BN is the normalization layer, and linear is the linear layer; 
     multiplying the weight matrix F Lw  with the feature F L     64   , subtracting F L     64    therefrom, and after processing by a linear layer, a normalization layer and an ReLU activation function, adding F L     64    thereto, to obtain a LiDAR feature F Lt  with RaDAR interaction information:
 
 F   Lt =ReLU( BN (linear( F   Lw   F   R     64     −F   L     64   )))+ F   L     64    
 
     concatenating the features F Rt  and F Lt  of the two modalities by dimensions to accomplish an interaction of the two modalities:
 
 F =Conca( tF   Rt   ,F   Lt )
 
     wherein in the formula, F is a concatenated feature, and Concat represents a concatenating operation. 
     As an improvement of the above method, the method further includes a step of training the RaDAR and LiDAR fusion network, specifically including: 
     normalizing an Astyx dataset used here to the format of a standard KITTI dataset, and aligning the LiDAR data and a 3D bounding box to a RaDAR coordinate system by using a calibration file to generate a training set; and 
     training the RaDAR and LiDAR fusion network by using the training set to obtain a trained RaDAR and LiDAR fusion network. 
     Embodiment 2 of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the above method. 
     Embodiment 3 of the present invention provides a storage medium, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein when executing the computer program, the processor executes the above method. 
     The present invention has the following advantages: 
     The 3D object detection method based on 4D RaDAR and LiDAR point cloud multi-view feature fusion of the present invention is intended to combine advantages of RaDAR and LiDAR, while avoiding disadvantages of the two modalities as much as possible to obtain a better 3D object detection result. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       To more clearly describe the present invention, drawings for using in the present invention will be introduced briefly below. Obviously, the drawings in the following description illustrate some embodiments of the present invention, and for some embodiments of the present invention, other drawings may also be obtained by those of ordinary skill in the art based on these drawings without creative work. 
         FIG. 1  is a flow diagram of a 3D object detection method based on 4D RaDAR and LiDAR point cloud multi-view feature fusion provided in Embodiment 1 of the present invention; 
         FIG. 2  is a schematic diagram of the structure of a RaDAR and LiDAR fusion network provided in Embodiment 1 of the present invention; and 
         FIG. 3  is a structure diagram of a RaDAR and LiDAR feature interaction module (interRAL) provided in Embodiment 1 of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
     To make the objectives, technical solutions and advantages of the present invention clearer and more apparent, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used for explaining the present invention, rather than limiting the present invention. 
     As shown in  FIG. 1 , Embodiment 1 of the present invention provides a 3D object detection method based on multi-view feature fusion of 4D RaDAR and LiDAR point clouds, the method including the following steps: 
     Step 1) simultaneously acquiring RaDAR point clouds and LiDAR point clouds. 
     Step 2) reading the radar point clouds and the LiDAR point clouds and voxelizing the two types of point clouds in a BEV field of view, respectively, projecting the point clouds onto an x-y plane to form a grid of H×W, and dividing the same into H×W pillars of a volume of 0.16×0.16×4(m). 
     There are many point clouds within each pillar, and an original point cloud data point has 4 dimensions (x, y, z, r), where r represents reflectivity; and the point cloud is expanded to 10 dimensions, which are calculated according to formula (1) (x,y,z,x c ,y c ,z c ,x p ,y p ,z p ) 
     
       
         
           
             
               
                 
                   
                     { 
                     
                       
                         
                           
                             
                               [ 
                               
                                 
                                   x 
                                   c 
                                 
                                 , 
                                 
                                   y 
                                   c 
                                 
                                 , 
                                 
                                   z 
                                   c 
                                 
                               
                               ] 
                             
                             = 
                             
                               
                                 [ 
                                 
                                   x 
                                   , 
                                   y 
                                   , 
                                   z 
                                 
                                 ] 
                               
                               - 
                               
                                 [ 
                                 
                                   
                                     x 
                                     m 
                                   
                                   , 
                                   
                                     y 
                                     m 
                                   
                                   , 
                                   
                                     z 
                                     m 
                                   
                                 
                                 ] 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               [ 
                               
                                 
                                   x 
                                   p 
                                 
                                 , 
                                 
                                   y 
                                   p 
                                 
                                 , 
                                 
                                   z 
                                   p 
                                 
                               
                               ] 
                             
                             = 
                             
                               
                                 [ 
                                 
                                   x 
                                   , 
                                   y 
                                   , 
                                   z 
                                 
                                 ] 
                               
                               - 
                               
                                 [ 
                                 
                                   
                                     x 
                                     g 
                                   
                                   , 
                                   
                                     y 
                                     g 
                                   
                                   , 
                                   
                                     z 
                                     g 
                                   
                                 
                                 ] 
                               
                             
                           
                         
                       
                     
                   
                     
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     In the formula, (x c ,y c ,z c ) is a deviation of each point cloud within the pillar relative to a pillar central point, (x m ,y m ,z m ) is pillar central point coordinates, (x p ,y p ,z p ) is a deviation of each point cloud relative to a grid central point, and (x p ,y g ,z g ) is grid central point coordinate. Each column with more than N points is randomly downsampled, and each column with less than N points is filled with 0. Hence, a tensor of dimensions (D, P, N) is formed, where D is 10, N is the number of samples for each pillar, which is 32, and P is the total number of pillars H×W. 
     Step 3) reading the RaDAR point clouds and the LiDAR point clouds and voxelizing the two types of point clouds in a PV view, respectively, and dividing point clouds within a pyramid with a vertical angle θ of [−26°, 6°] and a horizontal angle φ of [−90°, 90°] into H×W small pyramids. 
     A maximum of 32 points are randomly sampled within each pyramid, and each pyramid with less than 32 points is filled with 0. Each frame of point cloud is also processed to form a tensor of dimensions (D, P, N), and points in each pyramid are not processed like those in each pillar, so D is 4, P is the number of pyramids, which is H×W, and N is the number of points in each pyramid, which is 32. 
     Step 4) after the point clouds are processed into low-latitude features by pillar and pyramid methods, concatenating a pillar feature of a LiDAR and a pyramid feature of a radar, and concatenating a pillar feature of the radar and a pyramid feature of a LiDAR to form two 14-dimensional feature vectors, respectively: 
     
       
         
           
             
               
                 
                   
                     { 
                     
                       
                         
                           
                             
                               F 
                               L 
                             
                             = 
                             
                               Concat 
                               ⁡ 
                               
                                 ( 
                                 
                                   
                                     F 
                                     Lpi 
                                   
                                   , 
                                   
                                     F 
                                     Rpy 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               F 
                               R 
                             
                             = 
                             
                               Concat 
                               ⁡ 
                               
                                 ( 
                                 
                                   
                                     F 
                                     Rpi 
                                   
                                   , 
                                   
                                     F 
                                     Lpy 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                     
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
     In the formula, F Lpi  is the pillar feature of the LiDAR, F Rpy  is the pyramid feature of the radar, F Rpi  is the pillar feature of the radar, F Lpy  is the pyramid feature of the LiDAR, F L  is a feature formed by adding L Lpi  and F Rpy , and F R  is a feature formed by adding F Rpt  and F Lpi , and Concat represents a feature concatenating operation. 
       FIG. 2  shows a RaDAR and LiDAR fusion network structure. 
     Step 5) inputting the two 14-dimensional features obtained in step 4) into the RaDAR and LiDAR feature interaction module interRAL, respectively. 
     As shown in  FIG. 3 , interRAL uses a self-attention mechanism, in which after transposed feature matrices are introduced between two modalities, the correlation between the modalities is learned from the other modalities, and more effective features are selected in the process. 
     Specific steps of network implementation are as follows: 
     1) expanding the 14-dimensional feature of the LiDAR point cloud into a 64-dimensional feature through an FC layer and a Maxpool layer, and performing a convolution operation to dimensionally reduce the feature to form a 16-dimensional feature, and expanding the 14-dimensional feature of the radar point cloud into a 64-dimensional feature through an FC layer and a Maxpool layer, and performing a convolution operation to dimensionally reduce the feature to form a 16-dimensional feature: 
     
       
         
           
             
               
                 
                   
                     { 
                     
                       
                         
                           
                             
                               F 
                               
                                 L 
                                 64 
                               
                             
                             = 
                             
                               Maxpool 
                               ⁡ 
                               
                                 ( 
                                 
                                   Linear 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       F 
                                       L 
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               F 
                               
                                 R 
                                 64 
                               
                             
                             = 
                             
                               Maxpool 
                               ⁡ 
                               
                                 ( 
                                 
                                   Linear 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       F 
                                       R 
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                     
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     
       
         
           
             
               
                 
                   
                     { 
                     
                       
                         
                           
                             
                               F 
                               
                                 L 
                                 16 
                               
                             
                             = 
                             
                               Conv 
                               ⁡ 
                               
                                 ( 
                                 
                                   Maxpool 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       Linear 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           F 
                                           L 
                                         
                                         ) 
                                       
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               F 
                               
                                 R 
                                 16 
                               
                             
                             = 
                             
                               Conv 
                               ⁡ 
                               
                                 ( 
                                 
                                   Maxpool 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       Linear 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           F 
                                           R 
                                         
                                         ) 
                                       
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                     
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
     In the formula, F L     64    and F R     64    are features formed by expanding F L  and F R  to 64 dimensions, respectively, and F L     16    and F R     16    are features formed by dimensionally reducing F L  and F R  to 16 dimensions, respectively, Conv represents a convolutional layer, Maxpool represents a maximum pooling layer, and Linear represents a fully connected layer; 
     2) transposing the 16-dimensional feature of each modality and then multiplying the same with the 16-dimensional feature of the other modality, and performing a Softmax normalization operation to generate weight matrices of size M×N and N×M, respectively: 
     
       
         
           
             
               
                 
                   
                     { 
                     
                       
                         
                           
                             
                               F 
                               Lw 
                             
                             = 
                             
                               Sfot 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 max 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     
                                       
                                         ( 
                                         
                                           F 
                                           
                                             
                                               L 
                                               ⁢ 
                                               
                                                   
                                               
                                             
                                             16 
                                           
                                         
                                         ) 
                                       
                                       T 
                                     
                                     ⁢ 
                                     
                                       F 
                                       
                                         R 
                                         16 
                                       
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               F 
                               Rw 
                             
                             = 
                             
                               
                                 Sfot 
                                 ⁢ 
                                 max 
                               
                               ⁡ 
                               
                                 ( 
                                 
                                   
                                     
                                       ( 
                                       
                                         F 
                                         
                                           R 
                                           16 
                                         
                                       
                                       ) 
                                     
                                     T 
                                   
                                   ⁢ 
                                   
                                     F 
                                     
                                       L 
                                       16 
                                     
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                     
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     In the formula, F Lw  is the weight matrix generated by multiplying a transpose of F L     16    with F R     16   , and F Rw  is the weight matrix generated by multiplying a transpose of F R     16    with F L     16   , and Softmax represents the normalization operation; 
     3) multiplying F Lw  with F Rw  to obtain a new 64-dimensional feature vector, subtracting F R     64    from the feature vector, and after processing by a linear layer, a normalization layer and an activation function, adding F R     64    thereto, and finally concatenating the features of the two modalities by dimensions to accomplish an interaction of the two modalities: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             
                               F 
                               Rt 
                             
                             = 
                             
                               
                                 Re 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   LU 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       BN 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           linear 
                                           ⁡ 
                                           
                                             ( 
                                             
                                               
                                                 
                                                   F 
                                                   Rw 
                                                 
                                                 ⁢ 
                                                 
                                                   F 
                                                   
                                                     L 
                                                     64 
                                                   
                                                 
                                               
                                               - 
                                               
                                                 F 
                                                 
                                                   R 
                                                   64 
                                                 
                                               
                                             
                                             ) 
                                           
                                         
                                         ) 
                                       
                                     
                                     ) 
                                   
                                 
                               
                               + 
                               
                                 F 
                                 
                                   R 
                                   64 
                                 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               F 
                               Lt 
                             
                             = 
                             
                               
                                 Re 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   LU 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       BN 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           linear 
                                           ⁡ 
                                           
                                             ( 
                                             
                                               
                                                 
                                                   F 
                                                   Lw 
                                                 
                                                 ⁢ 
                                                 
                                                   F 
                                                   
                                                     R 
                                                     64 
                                                   
                                                 
                                               
                                               - 
                                               
                                                 F 
                                                 
                                                   L 
                                                   64 
                                                 
                                               
                                             
                                             ) 
                                           
                                         
                                         ) 
                                       
                                     
                                     ) 
                                   
                                 
                               
                               + 
                               
                                 F 
                                 
                                   L 
                                   64 
                                 
                               
                             
                           
                         
                       
                     
                       
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
             
               
                 
                   F 
                   = 
                   
                     Concat 
                     ⁡ 
                     
                       ( 
                       
                         
                           F 
                           Rt 
                         
                         , 
                         
                           F 
                           Lt 
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     In the formula, F Rt  is a RaDAR feature with LiDAR interaction information, F Lt  is a LiDAR feature with a radar interaction information, F is a concatenated feature, ReLU is the activation function, BN is the normalization layer, linear is the linear layer, and Concat represents a concatenating operation. 
     Step 6) encoding the interacted features F into the x-y plane according to coordinates of each voxel retained previously during voxelization, to form a 128-channel pseudo-image. 
     Step 7) inputting the 128-channel pseudo-image into a 2D convolutional neural network (2DCNN) for further feature extraction, wherein the 2DCNN uses a mature pyramidal structure CNN to extract multi-scale feature information. Step 8) inputting features output from the 2DCNN to a detection head, and outputting an object detection result, wherein the detection head uses is a mature RPN Head. 
     An Astyx dataset used in the present invention is normalize to the format of a standard KITTI dataset, and the LiDAR data is aligned to a RaDAR coordinate system by using a calibration file; and the RaDAR and LiDAR fusion network is trained. 
     Embodiment 2 
     Embodiment 2 of the present invention may also provide a computer device, including a processor, a memory, at least one network interface and a user interface. Components of the device are coupled together via a bus system. It may be understood that the bus system is configured to implement connection and communication between these components. The bus system includes a power bus, a control bus, and a status signal bus in addition to a data bus. 
     The user interface may include a display, a keyboard, or a clicking device (e.g., a mouse, a track ball, a touch pad, or a touch screen). 
     It may be understood that the memory in embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAMs may be used, such as a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDRSDRAM), an enhanced SDRAM (ESDRAM), a Synchlink DRAM (SLDRAM), and a direct Rambus RAM (DRRAM). The memory described herein is intended to include, but is not limited to, these and any other suitable types of memory. 
     In some implementations, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and an application. 
     The operating system contains various system programs, such as a framework layer, a core library layer, and a driver layer, for implementing various basic services and performing hardware-based tasks. The application contains various applications, such as a media player, and a browser, for implementing various application services. A program for implementing the method of embodiments of the present disclosure may be included in the application. 
     In the above embodiments, by calling a program or instructions stored in the memory, which may specifically be a program or instructions stored in the application, the processor is configured to: 
     execute the steps of the method of Embodiment 1. 
     The method of Embodiment 1 may be applied in the processor or implemented by the processor. The processor may be an integrated circuit chip with signal processing capability. During implementation, the steps of the above-mentioned method may be accomplished by an integrated logic circuit in the form of hardware or instructions in the form of software in the processor. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps and logical block diagrams disclosed in Embodiment 1 may be implemented or executed. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor or the like. The steps of the method disclosed in conjunction with Embodiment 1 may be directly embodied in hardware and executed by a decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be in a storage medium mature in the art, such as a random memory, a flash memory, a read-only memory, a programmable read-only memory or electrically erasable programmable memory, or a register. The storage medium is in the memory, and the processor reads information in the memory and accomplishes the steps of the above-mentioned method in conjunction with hardware thereof. 
     It may be understood that these embodiments described in the present invention may be implemented with hardware, software, firmware, middleware, microcodes, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP Devices, DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microprocessors, microcontrollers, other electronic units for performing the functions described in the present application, or a combination thereof. 
     For software implementation, the technology of the present invention may be implemented by executing functional modules (e.g. processes, and functions) of the present invention. Software codes may be stored in the memory and executed by the processor. The memory may be implemented in the processor or outside the processor. 
     Embodiment 3 
     Embodiment 3 of the present invention provides a non-volatile storage medium configured to store a computer program. When the computer program is executed by the processor, the steps in the method in embodiment 1 may be implemented. 
     Finally, it should be noted that the above embodiments are only used for describing instead of limiting the technical solutions of the present invention. Although the present invention is described in detail with reference to the embodiments, persons of ordinary skill in the art should understand that modifications or equivalent substitutions of the technical solutions of the present invention should be encompassed within the scope of the claims of the present invention so long as they do not depart from the spirit and scope of the technical solutions of the present invention.