Patent Publication Number: US-2023147480-A1

Title: Radar-lidar extrinsic calibration in unstructured environments

Description:
BACKGROUND 
     1. Technical Field 
     The subject technology provides solutions for autonomous vehicle systems, and in particular, for performing radar-to-lidar extrinsic calibration with on-road drive data. 
     2. Introduction 
     Autonomous vehicles are vehicles having computers and control systems that perform driving and navigation tasks that are conventionally performed by a human driver. As autonomous vehicle technologies continue to advance, ride-sharing services will increasingly utilize autonomous vehicles to improve service efficiency and safety. However, autonomous vehicles will be required to perform many of the functions that are conventionally performed by human drivers, such as avoiding dangerous or difficult routes, and performing other navigation and routing tasks necessary to provide a safe and efficient transportation. Such tasks may require the collection and processing of large quantities of data disposed on the autonomous vehicle. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Certain features of the subject technology are set forth in the appended claims. However, the accompanying drawings, which are included to provide further understanding, illustrate disclosed aspects and together with the description serve to explain the principles of the subject technology. In the drawings: 
         FIG.  1    illustrates an example system environment that can be used to facilitate autonomous vehicle navigation and routing operations, according to some aspects of the disclosed technology. 
         FIGS.  2 A and  2 B  illustrate example reflectors for radar-lidar calibration, according to some aspects of the disclosed technology. 
         FIGS.  3 A and  3 B  illustrate an example lidar calibration process that utilizes a turntable to rotate a vehicle, according to some aspects of the disclosed technology. 
         FIG.  4    illustrates an example aggregated radar point cloud, according to some aspects of the disclosed technology. 
         FIGS.  5 A- 5 F  illustrate example graphs of cost functions utilizing radar, according to some aspects of the disclosed technology. 
         FIGS.  6 A- 6 F  illustrate example graphs of cost functions utilizing radar and lidar, according to some aspects of the disclosed technology. 
         FIGS.  7 A and  7 B  illustrate example misaligned and calibrated images, according to some aspects of the disclosed technology. 
         FIG.  8    illustrates an example calibrated image, according to some aspects of the disclosed technology. 
         FIGS.  9 A and  9 B  illustrate example radar images, according to some aspects of the disclosed technology. 
         FIGS.  10 A- 10 C  illustrate example calibration tools and methods, according to some aspects of the disclosed technology. 
         FIGS.  11 A and  11 B  illustrate example before and after radar images, according to some aspects of the disclosed technology. 
         FIGS.  12 A and  12 B  illustrate example before and after radar images, according to some aspects of the disclosed technology. 
         FIG.  13    illustrates an example radar image, according to some aspects of the disclosed technology. 
         FIGS.  14 A- 14 L  illustrate example graphs of cost functions of an embodiment, according to some aspects of the disclosed technology. 
         FIGS.  15 A- 15 L  illustrate example graphs of cost functions of the embodiment of  FIGS.  14 A- 14 L , according to some aspects of the disclosed technology. 
         FIGS.  16 A- 16 L  illustrate example graphs of cost functions of an embodiment, according to some aspects of the disclosed technology. 
         FIGS.  17 A- 17 L  illustrate example graphs of cost functions of the embodiment of  FIGS.  16 A- 16 L , according to some aspects of the disclosed technology. 
         FIGS.  18 A- 18 L  illustrate example graphs of cost functions of an embodiment, according to some aspects of the disclosed technology. 
         FIGS.  19 A- 19 L  illustrate example graphs of cost functions of the embodiment of  FIGS.  18 A- 18 L , according to some aspects of the disclosed technology. 
         FIG.  20    illustrates an example process of performing radar-to-lidar extrinsic calibration with on-road drive data, according to some aspects of the disclosed technology. 
         FIG.  21    illustrates an example processor-based system with which some aspects of the subject technology can be implemented. 
     
    
    
     DETAILED DESCRIPTION 
     The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology can be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a more thorough understanding of the subject technology. However, it will be clear and apparent that the subject technology is not limited to the specific details set forth herein and may be practiced without these details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology. 
       FIG.  1    illustrates an example system environment  100  that can be used to facilitate AV dispatch and operations, according to some aspects of the disclosed technology. Autonomous vehicle  102  can navigate about roadways without a human driver based upon sensor signals output by sensor systems  104 - 106  of autonomous vehicle  102 . Autonomous vehicle  102  includes a plurality of sensor systems  104 - 106  (a first sensor system  104  through an Nth sensor system  106 ). Sensor systems  104 - 106  are of different types and are arranged about the autonomous vehicle  102 . For example, first sensor system  104  may be a camera sensor system and the Nth sensor system  106  may be a Light Detection and Ranging (LIDAR) sensor system. Other exemplary sensor systems include radio detection and ranging (RADAR) sensor systems, Electromagnetic Detection and Ranging (EmDAR) sensor systems, Sound Navigation and Ranging (SONAR) sensor systems, Sound Detection and Ranging (SODAR) sensor systems, Global Navigation Satellite System (GNSS) receiver systems such as Global Positioning System (GPS) receiver systems, accelerometers, gyroscopes, inertial measurement units (IMU), infrared sensor systems, laser rangefinder systems, ultrasonic sensor systems, infrasonic sensor systems, microphones, or a combination thereof. While four sensors  180  are illustrated coupled to the autonomous vehicle  102 , it is understood that more or fewer sensors may be coupled to the autonomous vehicle  102 . 
     Autonomous vehicle  102  further includes several mechanical systems that are used to effectuate appropriate motion of the autonomous vehicle  102 . For instance, the mechanical systems can include but are not limited to, vehicle propulsion system  130 , braking system  132 , and steering system  134 . Vehicle propulsion system  130  may include an electric motor, an internal combustion engine, or both. The braking system  132  can include an engine brake, brake pads, actuators, and/or any other suitable componentry that is configured to assist in decelerating autonomous vehicle  102 . In some cases, braking system  132  may charge a battery of the vehicle through regenerative braking. Steering system  134  includes suitable componentry that is configured to control the direction of movement of the autonomous vehicle  102  during navigation. 
     Autonomous vehicle  102  further includes a safety system  136  that can include various lights and signal indicators, parking brake, airbags, etc. Autonomous vehicle  102  further includes a cabin system  138  that can include cabin temperature control systems, in-cabin entertainment systems, etc. 
     Autonomous vehicle  102  additionally comprises an internal computing system  110  that is in communication with sensor systems  180  and systems  130 ,  132 ,  134 ,  136 , and  138 . Internal computing system  110  includes at least one processor and at least one memory having computer-executable instructions that are executed by the processor. The computer-executable instructions can make up one or more services responsible for controlling autonomous vehicle  102 , communicating with remote computing system  150 , receiving inputs from passengers or human co-pilots, logging metrics regarding data collected by sensor systems  180  and human co-pilots, etc. 
     Internal computing system  110  can include a control service  112  that is configured to control operation of vehicle propulsion system  130 , braking system  132 , steering system  134 , safety system  136 , and cabin system  138 . Control service  112  receives sensor signals from sensor systems  180  as well communicates with other services of internal computing system  110  to effectuate operation of autonomous vehicle  102 . In some embodiments, control service  112  may carry out operations in concert one or more other systems of autonomous vehicle  102 . 
     Internal computing system  110  can also include constraint service  114  to facilitate safe propulsion of autonomous vehicle  102 . Constraint service  116  includes instructions for activating a constraint based on a rule-based restriction upon operation of autonomous vehicle  102 . For example, the constraint may be a restriction upon navigation that is activated in accordance with protocols configured to avoid occupying the same space as other objects, abide by traffic laws, circumvent avoidance areas, etc. In some embodiments, the constraint service can be part of control service  112 . 
     The internal computing system  110  can also include communication service  116 . The communication service  116  can include both software and hardware elements for transmitting and receiving signals from/to the remote computing system  150 . Communication service  116  is configured to transmit information wirelessly over a network, for example, through an antenna array that provides connectivity using one or more cellular transmission standards, such as long-term evolution (LTE), 3G, 5G, or the like. 
     In some embodiments, one or more services of the internal computing system  110  are configured to send and receive communications to remote computing system  150  for such reasons as reporting data for training and evaluating machine learning algorithms, requesting assistance from remoting computing system or a human operator via remote computing system  150 , software service updates, ridesharing pickup and drop off instructions etc. 
     Internal computing system  110  can also include latency service  118 . Latency service  118  can utilize timestamps on communications to and from remote computing system  150  to determine if a communication has been received from the remote computing system  150  in time to be useful. For example, when a service of the internal computing system  110  requests feedback from remote computing system  150  on a time-sensitive process, the latency service  118  can determine if a response was timely received from remote computing system  150  as information can quickly become too stale to be actionable. When the latency service  118  determines that a response has not been received within a threshold, latency service  118  can enable other systems of autonomous vehicle  102  or a passenger to make necessary decisions or to provide the needed feedback. 
     Internal computing system  110  can also include a user interface service  120  that can communicate with cabin system  138  in order to provide information or receive information to a human co-pilot or human passenger. In some embodiments, a human co-pilot or human passenger may be required to evaluate and override a constraint from constraint service  114 , or the human co-pilot or human passenger may wish to provide an instruction to the autonomous vehicle  102  regarding destinations, requested routes, or other requested operations. 
     As described above, the remote computing system  150  is configured to send/receive a signal from the autonomous vehicle  140  regarding reporting data for training and evaluating machine learning algorithms, requesting assistance from remote computing system  150  or a human operator via the remote computing system  150 , software service updates, rideshare pickup and drop off instructions, etc. 
     Remote computing system  150  includes an analysis service  152  that is configured to receive data from autonomous vehicle  102  and analyze the data to train or evaluate machine learning algorithms for operating the autonomous vehicle  102 . The analysis service  152  can also perform analysis pertaining to data associated with one or more errors or constraints reported by autonomous vehicle  102 . 
     Remote computing system  150  can also include a user interface service  154  configured to present metrics, video, pictures, sounds reported from the autonomous vehicle  102  to an operator of remote computing system  150 . User interface service  154  can further receive input instructions from an operator that can be sent to the autonomous vehicle  102 . 
     Remote computing system  150  can also include an instruction service  156  for sending instructions regarding the operation of the autonomous vehicle  102 . For example, in response to an output of the analysis service  152  or user interface service  154 , instructions service  156  can prepare instructions to one or more services of the autonomous vehicle  102  or a co-pilot or passenger of the autonomous vehicle  102 . 
     Remote computing system  150  can also include rideshare service  158  configured to interact with ridesharing applications  170  operating on (potential) passenger computing devices. The rideshare service  158  can receive requests to be picked up or dropped off from passenger ridesharing app  170  and can dispatch autonomous vehicle  102  for the trip. The rideshare service  158  can also act as an intermediary between the ridesharing app  170  and the autonomous vehicle wherein a passenger might provide instructions to the autonomous vehicle to  102  go around an obstacle, change routes, honk the horn, etc. 
       FIGS.  2 A and  2 B  illustrate example reflectors  200 ,  202  for radar-lidar calibration, according to some aspects of the disclosed technology. 
     In some implementations, methods of radar-lidar calibration can include utilizing calibration targets  200 ,  202  as shown in  FIGS.  2 A and  2 B .  FIG.  2 A  illustrates a target board  200  with apertures distributed throughout, and  FIG.  2 B  illustrates a target cross  202 . Both targets  200 ,  202  can include reflectors that may be detected by radar and lidar for calibration purposes. For example, corner reflectors may be associated in lidar and radar to optimize relative pose to minimize distance. In some examples, accurate detection of the same target (e.g., targets  200 ,  202  as shown in  FIGS.  2 A and  2 B ) by both radar and lidar may require special targets, which may include a long lead time, be expensive, and require additional resources. Moreover, accurate detections with radar may be tricky due to sensor noise and sparseness. 
       FIGS.  3 A and  3 B  illustrate an example lidar calibration process that utilizes a turntable  302  to rotate a vehicle  300 , according to some aspects of the disclosed technology. 
     In some implementations, the vehicle  300  can be rotated on the turntable  302  in a controlled scene with targets  304 . A lidar calibration process can build a high resolution map  306  of the scene and provide vehicle pose at each position on the turntable  302 . In some examples, hole target detection (e.g., by utilizing targets  304 ) in a lidar map can provide a pose of each target  304  relative to the vehicle  300  to aid target-based camera-lidar and radar-lidar calibration. 
     In other implementations, corner reflectors can be fixed and/or at a known pose relative to hole targets  304  to facilitate radar-lidar calibration. In some examples, a large number of associations (e.g., in the hundreds) is possible as the vehicle  300  can be rotated on the turntable  302 . A root-mean-square error (RMSE) can also be utilized to gauge calibration quality. 
     In other examples, radar-lidar targets  304  can take time to design, fabricate, and maintain. There may also be stack up of errors from hole target detection, frame construction, and radar target detections. Moreover, having a lot of metal in the scene (e.g., stands, overhead structures, walls, etc.) may decrease the reliability of the radar target detection. 
     Traditional target-based radar-lidar calibration is prone to target detection errors and requires a high degree of control on the fixture design and the environment around a vehicle. This type of process takes a significant amount of time and effort to design and develop a stable and scalable process for a vehicle fleet. Moreover, target-based methods cannot be usable on-road after the vehicle has been deployed. Calibration tests conducted within a warehouse (e.g., metallic roof, pillars, etc.) with many other fixtures for other calibrations make it difficult to obtain quality target returns necessary for a good calibration. 
     As such, a need exists to perform radar-radar and radar-lidar extrinsic calibration in unstructured environments (e.g., without special calibration specific targets). 
     As discussed herein, in some implementations, systems and methods can include utilizing point cloud registration and entropy minimization to align radar and lidar point clouds gathered from different vehicle poses. The systems and methods as described herein provide for a more robust and accurate process of calibration. 
     In other implementations, the systems and methods as described herein can generate aggregated maps of a scene with multiple radar and lidar point clouds by using entropy minimization as a cost function to globally optimize six degrees of freedom calibration between sensors. 
       FIG.  4    illustrates an example aggregated radar point cloud  400 , according to some aspects of the disclosed technology. 
     In some implementations, purposes of the aggregated radar point cloud  400  generated from different poses of the vehicle (e.g., the vehicle  300  of  FIG.  3 A ) may be to provide “crisp” target detections as possible (e.g., walls, targets, etc. may have reasonable noise levels) and to be aligned with a lidar map. Minimizing “fuzziness” can be a result of good calibration with/without targets (e.g., unstructured environments). 
     Entropy Minimization: 
     
       
         
           
             
               
                 
                   
                     
                       H 
                       RQE 
                     
                     [ 
                     
                       X 
                       ^ 
                     
                     ] 
                   
                   = 
                   
                     - 
                     
                       log 
                       ( 
                       
                         
                           1 
                           
                             N 
                             2 
                           
                         
                         ⁢ 
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             N 
                           
                           
                             
                               ∑ 
                               
                                 j 
                                 = 
                                 1 
                               
                               N 
                             
                             
                               G 
                               ⁡ 
                               ( 
                               
                                 
                                   
                                     
                                       x 
                                       ^ 
                                     
                                     i 
                                   
                                   - 
                                   
                                     
                                       x 
                                       ^ 
                                     
                                     j 
                                   
                                 
                                 , 
                                 
                                   2 
                                   ⁢ 
                                   
                                     σ 
                                     2 
                                   
                                   ⁢ 
                                   I 
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   Eq 
                   . 
                       
                   1 
                 
               
             
           
         
       
     
     Equation 1 can be a measure of compactness of points in X with an information-theoretic provenance, for which a free parameter is u. For purposes of optimization, the logarithm may be a monotonic operator and the scale factor may be unnecessary, so these terms may be dropped to produce a cost function, as provided below in Equation 2. 
     
       
         
           
             
               
                 
                   
                     
                       E 
                       ⁡ 
                       ( 
                       
                         X 
                         ^ 
                       
                       ) 
                     
                     = 
                     
                       - 
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             1 
                           
                           N 
                         
                         
                           
                             ∑ 
                             
                               j 
                               = 
                               1 
                             
                             N 
                           
                           
                             G 
                             ⁡ 
                             ( 
                             
                               
                                 
                                   
                                     x 
                                     ^ 
                                   
                                   i 
                                 
                                 - 
                                 
                                   
                                     x 
                                     ^ 
                                   
                                   j 
                                 
                               
                               , 
                               
                                 2 
                                 ⁢ 
                                 
                                   σ 
                                   2 
                                 
                                 ⁢ 
                                 I 
                               
                             
                             ) 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   Eq 
                   . 
                       
                   2 
                 
               
             
           
         
       
     
     The cost function of Equation 2 may depend on pairwise distances between measured points in {circumflex over (X)}. For Equations 1 and 2, x i  can refer to a 3D position vector for a return. σ can refer to a standard deviation of a gaussian kernel (e.g., an adjustable parameter). I can refer to an identity matrix. N can refer to a number of points. E(X) can refer to a cost function as described herein. 
     In some implementations, the cost function of Equation 2 can be utilized to measure “crispness” of a point cloud based on radar data. The cost value may be scene specific, but location of minima may not be. In other implementations, the radar scans can be aggregated together from different positions of a vehicle in a common frame of reference. The cost function (e.g., as described in Equation 2) can be calculated based on the data from the radar scans. The aggregated process can assume a lidar to radar extrinsic pose (e.g., six degrees of freedom (DoF)) to transform a lidar point cloud from a radar frame of reference to a lidar frame of reference (e.g., a lidar_T_radar). In some examples, the systems and methods as described herein can utilize the transformation from a position of the vehicle to the initial starting position in the lidar frame (e.g., a starting_pos_T_initial_pos). For example, the transformation can be an outcome of a lidar-lidar calibration process or can be generated based on aligning lidar point clouds independent of the radar (e.g., using iterative closest point (ICP) techniques). In some on-road examples, the transformation can be replaced with vehicle odometry or localization outputs. In other examples, x i  (e.g., at a lidar frame at an initial starting position)=starting_pos_T_initial_pos*lidar_T_radar*x i _radar_frame. After the transformation, Equation 2 can be utilized to determine an equivalent cost function. 
       FIGS.  5 A- 5 F  illustrate example graphs of cost functions utilizing radar  500 , according to some aspects of the disclosed technology.  FIGS.  5 A- 5 F  further illustrate cost functions of six degrees of freedom (DoF):  FIG.  5 A —“x;”  FIG.  5 B —“y;”  FIG.  5 C —“z;”  FIG.  5 D —“roll;”  FIG.  5 E —“pitch;” and  FIG.  5 F —“yaw.” 
     In some implementations, the cost functions of  FIGS.  5 A- 5 F  may be the cost functions determined for a single radar (e.g., self-alignment). The entropy can be calculated for radar extrinsic, aggregate radar scans from the various stops (e.g., poses/six DoF) using lidar calibration. However, a single radar cost function may not be sufficient because of a strong signal in the x, y, roll, and pitch direction, a weak signal in the z direction (e.g., shift of point cloud may be change crispness), and a weak signal in the yaw direction (e.g., sparse data). 
       FIGS.  6 A- 6 F  illustrate example graphs of cost functions utilizing radar and lidar  600 , according to some aspects of the disclosed technology.  FIGS.  6 A- 6 F  further illustrate cost functions of six degrees of freedom (DoF):  FIG.  6 A —“x;”  FIG.  6 B —“y;”  FIG.  6 C —“z;”  FIG.  6 D —“roll;”  FIG.  6 E —“pitch;” and  FIG.  6 F —“yaw.” 
     In some implementations, radar scans and a lidar map (e.g., downsampled) can be utilized to force alignment of radar with lidar. For example, cost functions may be convex in six DoF including x, y, z, roll, pitch, and yaw. The z-signal may be unreliable as radar and lidar features may not match, partly due to elevation noise. In other implementations, the cost function can include a signal global minima proximate to an expected solution that can indicate whether a signal is miscalibrated. For example, if increasing a radar-to-lidar offset increases a value in either direction away from the solution that visually appears to be calibrated (e.g., best calibration), then for another vehicle, convex optimization can be performed to search for a global minima that may be the most optimal setting (e.g., lidar_T_radar), without user intervention to perform visual checks. 
     In other implementations, the z value may be fixed to a computer aided design (CAD) value. In some examples, the CAD value can refer to a radar pose relative to a vehicle chassis. A physical sensor mounting process can include some errors. As such, the actual mounting process can be different from what was intended by design. The systems and processes as described herein can provide a calibration process as each car uniquely differs from an intended design. Moreover, ten DoF optimization (e.g., x, y, roll, pitch, yaw) times two radars may utilize solvers (e.g., a Ceres solver) to generate a result. In one example, six runs in total were performed, three runs with turntable 1 (“TT1”) and three runs with turntable 2 (“TT2”). TT1 included many radar reflectors for target-based calibration. TT2 is a larger turntable, but included less reflectors for conti-radars. Some reflectors were not in the field of view. 
     
       
         
           
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                 (max-min) pose relative to roof-left lidar 
               
            
           
           
               
               
               
            
               
                   
                 Roof_front_left_45 
                 Roof_front_right_45 
               
               
                   
                   
               
            
           
           
               
               
               
               
            
               
                   
                 X [m] 
                 0.019859 
                 0.021434 
               
               
                   
                 Y [m] 
                 0.011280 
                 0.007827 
               
               
                   
                 Z [m] 
                 0.007486 
                 0.008039 
               
               
                   
                 Roll [deg] 
                 0.232350 
                 0.370030 
               
               
                   
                 Pitch [deg] 
                 0.242064 
                 0.209772 
               
               
                   
                 Yaw [deg] 
                 0.147191 
                 0.083011 
               
               
                   
                   
               
            
           
         
       
     
     For roof-right, roll change may be high, but is consistent for a given scene. TT1 included a roll max-min of 0.1 degrees (roof-right), and TT2 included a roll max-min of 0.06 degrees (roof-right). 
       FIGS.  7 A and  7 B  illustrate example misaligned and calibrated images, according to some aspects of the disclosed technology. For example,  FIG.  7 A  illustrates a misalignment in yaw, while  FIG.  7 B  illustrates a calibrated radar-lidar system. 
       FIG.  8    illustrates an example calibrated image, according to some aspects of the disclosed technology. For example,  FIG.  8    illustrates a top view of a calibrated image using the radar-lidar calibration process as described herein. 
     In some implementations, a vehicle was brought back to TT1 to confirm scene dependence and ensure that the sensor did not shift. Parameter and algorithm tuning can be performed intermittently. The z value may be fixed to a CAD value. Moreover, ten DoF optimization (e.g., x, y, roll, pitch, yaw) times two radars may utilize solvers (e.g., a Ceres solver) to generate a result. In one example, eight runs in total were performed, five runs with TT1 and three runs with TT2. 
     
       
         
           
               
             
               
                 TABLE 2 
               
             
            
               
                   
               
               
                 (max-min) pose relative to roof-left lidar 
               
            
           
           
               
               
               
               
               
               
               
            
               
                   
                 Roof left 
                 Roof right 
                 Roof left 
                 Roof right 
                 Roof left 
                 Roof right 
               
               
                   
                 (both scenes) 
                 (both scenes) 
                 (TT1) 
                 (TT1) 
                 (TT2) 
                 (TT2) 
               
               
                   
                   
               
            
           
           
               
               
               
               
               
               
               
            
               
                 X [m] 
                 0.017476 
                 0.019229 
                 0.006310 
                 0.007847 
                 0.001377 
                 0.002475 
               
               
                 Y [m] 
                 0.011152 
                 0.005844 
                 0.003404 
                 0.004718 
                 0.000757 
                 0.000843 
               
               
                 Z [m] 
                 0.007517 
                 0.008054 
                 0.004717 
                 0.008054 
                 0.000564 
                 0.000317 
               
               
                 Roll [deg] 
                 0.291802 
                 0.400420 
                 0.213418 
                 0.118074 
                 0.119291 
                 0.060538 
               
               
                 Pitch [deg] 
                 0.234209 
                 0.264259 
                 0.233605 
                 0.189628 
                 0.037206 
                 0.008818 
               
               
                 Yaw [deg] 
                 0.093825 
                 0.068758 
                 0.088975 
                 0.064607 
                 0.055528 
                 0.047507 
               
               
                   
               
            
           
         
       
     
     In some implementations, yaw repeatability was within less than 0.1 degrees, and roll and pitch were within less than 0.4 degrees, for a translation of less than two centimeters. Repeatability was acceptable for TT1 and TT2, with some scene dependence for roof-right radar pitch. There also appeared to be no apparent sensor shift as the TT1 run was two days after the TT2 run, which produced similar results as the initial TT1 results. 
     In another implementation, eight TT1 runs were performed while fixing z to a CAD value. 
     
       
         
           
               
             
               
                 TABLE 3 
               
             
            
               
                   
               
               
                 (max-min) pose relative to roof-left lidar 
               
            
           
           
               
               
               
            
               
                   
                 Roof left (TT1) 
                 Roof right (TT1) 
               
               
                   
                   
               
            
           
           
               
               
               
               
            
               
                   
                 X [m] 
                 0.005089 
                 0.004156 
               
               
                   
                 Y [m] 
                 0.006364 
                 0.011487 
               
               
                   
                 Z [m] 
                 0.016452 
                 0.018081 
               
               
                   
                 Roll [deg] 
                 0.173807 
                 0.353290 
               
               
                   
                 Pitch [deg] 
                 0.188681 
                 0.215788 
               
               
                   
                 Yaw [deg] 
                 0.093217 
                 0.061581 
               
               
                   
                   
               
            
           
         
       
     
     In some implementations, the lidar calibrations passed with a 100% pass rate. 
       FIGS.  9 A and  9 B  illustrate example radar images, according to some aspects of the disclosed technology.  FIG.  9 A  illustrates a top view radar image of a radar on TT1. For example, TT1 includes many corner reflectors in the field of vision of the radar, smaller in size than TT2, and includes a concrete wall on one side.  FIG.  9 B  illustrates a top view radar image of a radar on TT2. Most TT2 returns include returns from partition walls and a few corner reflectors close to the height of the radar. 
     In some implementations, the z value can be calibrated. For example, calibrating z values can be difficult as feature overlap between lidar and radar is not very high in a scene. In some examples, one radar z (roof-left) can be locked, while allowing the other to be calibrated (e.g., eleven DoF optimization at a time). In one example, eight runs in total were performed, five runs with TT1 and three runs with TT2. 
     
       
         
           
               
             
               
                 TABLE 4 
               
             
            
               
                   
               
               
                 (max-min) pose relative to roof-left lidar 
               
            
           
           
               
               
               
               
               
               
               
            
               
                   
                 Roof left 
                 Roof right 
                 Roof left 
                 Roof right 
                 Roof left 
                 Roof right 
               
               
                   
                 (both scenes) 
                 (both scenes) 
                 (TT1) 
                 (TT1) 
                 (TT2) 
                 (TT2) 
               
               
                   
                   
               
            
           
           
               
               
               
               
               
               
               
            
               
                 X [m] 
                 0.016783 
                 0.019800 
                 0.006095 
                 0.007731 
                 0.001734 
                 0.002080 
               
               
                 Y [m] 
                 0.010686 
                 0.004765 
                 0.003369 
                 0.004658 
                 0.000948 
                 0.001134 
               
               
                 Z [m] 
                 0.007519 
                 0.052514 
                 0.004749 
                 0.015947 
                 0.000565 
                 0.025862 
               
               
                 Roll [deg] 
                 0.289261 
                 0.398610 
                 0.216612 
                 0.113912 
                 0.117266 
                 0.056400 
               
               
                 Pitch [deg] 
                 0.404252 
                 0.193672 
                 0.306376 
                 0.180303 
                 0.092834 
                 0.141986 
               
               
                 Yaw [deg] 
                 0.096646 
                 0.063923 
                 0.078115 
                 0.058540 
                 0.057447 
                 0.046609 
               
               
                   
               
            
           
         
       
     
     Table 4 illustrates some scene dependence and roll/pitch issues, which may be transferred to other roll/pitch dimensions or radar extrinsic. For example, 5 cm of z change may indicate that elevation noise may limit the calibration of the z value. 
       FIGS.  10 A- 10 C  illustrate example calibration tools and methods, according to some aspects of the disclosed technology.  FIG.  10 A  illustrates an example calibration tool (e.g., a target with high reflectivity) that can be utilized for the calibration process described herein.  FIG.  10 B  illustrates misalignment via manual calibration processes as demonstrated by the misalignment of the target.  FIG.  10 C  illustrates a correctly calibrated radar utilizing an unstructured calibration method as described herein. As shown in  FIG.  10 C , the radar detection of the target (also enlarged) is correctly aligned by utilizing the unstructured calibration method as described herein. 
       FIGS.  11 A and  11 B  illustrate example before and after radar images, according to some aspects of the disclosed technology.  FIG.  11 A  illustrates before radar images that utilize a manual calibration, while  FIG.  11 B  illustrates a calibrated radar utilizing an unstructured calibration method as described herein (e.g., calibration between radars and roof-left lidar). 
       FIGS.  12 A and  12 B  illustrate example before and after radar images, according to some aspects of the disclosed technology.  FIG.  12 A  illustrates before radar images that utilize a manual calibration, while  FIG.  12 B  illustrates a calibrated radar utilizing an unstructured calibration method as described herein. In some examples, elevation changes may be hard to observe visually. In response, the system as described herein may further include utilizing filters to remove noise and detect features. 
     In some implementations, a comparison with target detection may be utilized by the method and system described herein. For example, a stack up of errors can include: hole target detection in a lidar point cloud that may have small pose errors; a lidar board and radar target that may not be attached to a rigid frame; and radar target detection uncertainty due to sensor noise. Repeatability may indicate that target detection are adequate to be utilized as a check. 
     
       
         
           
               
             
               
                 TABLE 5 
               
             
            
               
                   
               
               
                 (max-min) pose relative to roof-left lidar 
               
            
           
           
               
               
               
            
               
                   
                 Roof left (TT1) 
                 Roof right (TT1) 
               
               
                   
                   
               
            
           
           
               
               
               
               
            
               
                   
                 X [m] 
                 0.028987 
                 0.027554 
               
               
                   
                 Y [m] 
                 0.031328 
                 0.011166 
               
               
                   
                 Z [m] 
                 0.100583 
                 0.075382 
               
               
                   
                 Roll [deg] 
                 0.144283 
                 0.140538 
               
               
                   
                 Pitch [deg] 
                 1.052281 
                 0.476451 
               
               
                   
                 Yaw [deg] 
                 0.433902 
                 0.403891 
               
               
                   
                   
               
            
           
         
       
     
     In some implementations, a yaw repeatability of approximately 0.4 degrees and roll/pitch of approximately 1 degree. 
     In other implementations, transforms can be applied to incoming radar scans, and computed extrinsics can be utilized to recover the transform. Results converged to expected values and tested to within 5 cm translation and three degrees of rotation offset. The calibration process as described herein can properly calibrate misalignments with less dependence on targets. 
       FIG.  13    illustrates an example radar image, according to some aspects of the disclosed technology. As shown in  FIG.  13   , an aggregated cloud is illustrated for a radar front. In some implementations, eight long range radars (LRR) can be utilized with no/unusable elevation data being equal to or greater than x, y, and yaw calibrations. In some examples, short range may also utilize a “near scan” mode. Filters may further be utilized and based on RCS, range, and azimuth data. 
     Repeatability of Current Target-Based Method: 
     
       
         
           
               
             
               
                 TABLE 6 
               
             
            
               
                   
               
               
                 (max-min) pose relative to roof-left lidar 
               
            
           
           
               
               
               
               
            
               
                   
                 X_rel (m) 
                 Y_rel (m) 
                 Yaw_rel (deg) 
               
               
                   
                   
               
            
           
           
               
               
               
               
            
               
                 Radar_front 
                 0.155517 
                 0.058835 
                 0.792577 
               
               
                 Radar_rear 
                 0.154509 
                 0.073017 
                 0.876164 
               
               
                 Radar_front_left_45 
                 0.057715 
                 0.152050 
                 0.873768 
               
               
                 Radar_front_right_45 
                 0.115960 
                 0.033294 
                 0.616749 
               
               
                 Radar_front_left_90 
                 0.165568 
                 0.058682 
                 0.727369 
               
               
                 Radar_front_right_90 
                 0.087597 
                 0.031060 
                 1.955809 
               
               
                 Radar_back_left_45 
                 0.125271 
                 0.032563 
                 0.626507 
               
               
                 Radar_back_right_45 
                 0.044588 
                 0.115123 
                 0.545004 
               
               
                   
               
            
           
         
       
     
     Table 6 may illustrate a two degree of change in yaw with a translation of 15-16 cm. In some examples, targets may have been too close to the vehicle and the returns may have been of poor quality. There may also have been a stack up of detection errors. 
       FIGS.  14 A- 14 L  illustrate example graphs of cost functions of an embodiment, according to some aspects of the disclosed technology.  FIGS.  14 A- 14 L  further illustrate example unstructured cost functions of TT1 including:  FIG.  14 A —“radar_front: x;”  FIG.  14 B —“radar_front: y;”  FIG.  14 C —“radar_front: yaw;”  FIG.  14 D —“radar_rear: x;”  FIG.  14 E —“radar_rear: y;”  FIG.  14 F —“radar_rear: yaw;”  FIG.  14 G —“radar_front_left_45: x;”  FIG.  14 H —“radar_front_left_45: y;”  FIG.  14 I —“radar_front_left_45: yaw;”  FIG.  14 J —“radar_back_left_45: x;”  FIG.  14 K —“radar_back_left_45: y;” and  FIG.  14 L — “radar_back_left_45: yaw.” 
       FIGS.  15 A- 15 L  illustrate example graphs of cost functions of the embodiment of  FIGS.  14 A- 14 L , according to some aspects of the disclosed technology.  FIGS.  15 A- 15 L  further illustrate example unstructured cost functions of TT1 including:  FIG.  15 A —“radar_back_right: x;”  FIG.  15 B —“radar_back_right_45: y;”  FIG.  15 C — “radar_back_right_45: yaw;”  FIG.  15 D —“radar_front_left_90: x;”  FIG.  15 E — “radar_front_left_90: y;”  FIG.  15 F —“radar_front_left_90: yaw;”  FIG.  15 G — “radar_front_right_90: x;”  FIG.  15 H —“radar_front_right_90: y;”  FIG.  15 I — “radar_front_right_90: yaw;”  FIG.  15 J —“radar_back_right_45: x;”  FIG.  15 K — “radar_back_right_45: y;” and  FIG.  15 L —“radar_back_right_45: yaw.” 
       FIGS.  16 A- 16 L  illustrate example graphs of cost functions of an embodiment, according to some aspects of the disclosed technology.  FIGS.  16 A- 16 L  further illustrate example unstructured cost functions of TT2 including:  FIG.  16 A —“radar_back_left_45: x;”  FIG.  16 B —“radar_back_left_45: y;”  FIG.  16 C —“radar_back_left_45: yaw;”  FIG.  16 D — “radar_back_right_45: x;”  FIG.  16 E —“radar_back_right_45: y;”  FIG.  16 F — “radar_back_right_45: yaw;”  FIG.  16 G —“radar_front_left_45: x;”  FIG.  16 H — “radar_front_left_45: y;”  FIG.  16 I —“radar_front_left_45: yaw;”  FIG.  16 J —“radar_rear: x;”  FIG.  16 K —“radar_rear: y;” and  FIG.  16 L —“radar_rear: yaw.” 
       FIGS.  17 A- 17 L  illustrate example graphs of cost functions of the embodiment of  FIGS.  16 A- 16 L , according to some aspects of the disclosed technology.  FIGS.  17 A- 17 L  further illustrate example unstructured cost functions of TT2 including:  FIG.  17 A —“radar_front_left_90: x;”  FIG.  17 B —“radar_front_left_90: y;”  FIG.  17 C — “radar_front_left_90: yaw;”  FIG.  17 D —“radar_front_right_45: x;”  FIG.  17 E — “radar_front_right_45: y;”  FIG.  17 F —“radar_front_right_45: yaw;”  FIG.  17 G — “radar_front_right_90: x;”  FIG.  17 H —“radar_front_right_90: y;”  FIG.  17 I — “radar_front_right_90: yaw;”  FIG.  17 J —“radar_front: x;”  FIG.  17 K —“radar_front: y;” and  FIG.  17 L —“radar_front: yaw.” 
       FIGS.  18 A- 18 L  illustrate example graphs of cost functions of an embodiment, according to some aspects of the disclosed technology.  FIGS.  18 A- 18 L  further illustrate example cost functions including:  FIG.  18 A —“radar_back_left_45: x;”  FIG.  18 B —“radar_back_left_45: y;”  FIG.  18 C —“radar_back_left_45: yaw;”  FIG.  18 D —“radar_rear: x;”  FIG.  18 E —“radar_rear: y;”  FIG.  18 F —“radar_rear: yaw;”  FIG.  18 G —“radar_back_right_45: x;”  FIG.  18 H —“radar_back_right_45: y;”  FIG.  18 I —“radar_back_right_45: yaw;”  FIG.  18 J —“radar_front_left_45: x;”  FIG.  18 K —“radar_front_left_45: y;” and  FIG.  18 L —“radar_front_left_45: yaw.” 
       FIGS.  19 A- 19 L  illustrate example graphs of cost functions of the embodiment of  FIGS.  18 A- 18 L , according to some aspects of the disclosed technology.  FIGS.  19 A- 19 L  further illustrate example cost functions including:  FIG.  19 A —“radar_front_right_45: x;”  FIG.  19 B —“radar_front_right_45: y;”  FIG.  19 C —“radar_front_right_45: yaw;”  FIG.  19 D —“radar_front_right_90: x;”  FIG.  19 E —“radar_front_right_90: y;”  FIG.  19 F —“radar_front_right_90: yaw;”  FIG.  19 G —“radar_front: x;”  FIG.  19 H —“radar_front: y;”  FIG.  19 I —“radar_front: yaw;”  FIG.  19 J —“radar_front_left_90: x;”  FIG.  19 K —“radar_front_left_90: y;” and  FIG.  19 L —“radar_front_left_90: yaw.” 
     Combined Optimization: 
     
       
         
           
               
             
               
                 TABLE 7 
               
             
            
               
                   
               
               
                 Combined optimization 
               
            
           
           
               
               
               
               
            
               
                   
                 X_rel (m) 
                 Y_rel (m) 
                 Yaw_rel (deg) 
               
               
                   
                   
               
            
           
           
               
               
               
               
            
               
                 Radar_front 
                 0.109414 
                 0.080543 
                 0.943750 
               
               
                 Radar_rear 
                 0.151159 
                 0.069371 
                 1.444306 
               
               
                 Radar_front_left_45 
                 0.050794 
                 0.101874 
                 0.906070 
               
               
                 Radar_front_right_45 
                 0.131451 
                 0.143884 
                 1.289863 
               
               
                 Radar_front_left_90 
                 0.059457 
                 0.068720 
                 0.827719 
               
               
                 Radar_front_right_90 
                 0.141650 
                 0.054079 
                 1.174027 
               
               
                 Radar_back_left_45 
                 0.103500 
                 0.110067 
                 0.817190 
               
               
                 Radar_back_right_45 
                 0.098085 
                 0.089360 
                 0.855968 
               
               
                   
               
            
           
         
       
     
     In some implementations, a combined optimization may include fixed z, roll, and pitch values for all of the radars (e.g., eight radars). This may include 24 DoF optimization (e.g., x, y, and yaw) times eight radars using a solver (e.g., a Ceres solver). A total of eight runs were performed including five in TT1 and three in TT2. For example, over a degree of variability in yaw may be equal to or greater than a similar performance as the target-based method. In some examples, the solver can take an extremely long time. As such, convergence may be difficult. 
     One Radar at a Time: 
     
       
         
           
               
             
               
                 TABLE 8 
               
             
            
               
                   
               
               
                 One radar at a time 
               
            
           
           
               
               
               
               
            
               
                   
                 X_rel (m) 
                 Y_rel (m) 
                 Yaw_rel (deg) 
               
               
                   
                   
               
            
           
           
               
               
               
               
            
               
                 Radar_front 
                 0.081241 
                 0.022952 
                 1.363927 
               
               
                 Radar_rear 
                 0.113057 
                 0.100160 
                 1.596260 
               
               
                 Radar_front_left_45 
                 0.043537 
                 0.081900 
                 0.815448 
               
               
                 Radar_front_right_45 
                 0.062602 
                 0.096582 
                 0.869589 
               
               
                 Radar_front_left_90 
                 0.098853 
                 0.081124 
                 1.182841 
               
               
                 Radar_front_right_90 
                 0.093716 
                 0.064431 
                 1.564575 
               
               
                 Radar_back_left_45 
                 0.061015 
                 0.053671 
                 0.815764 
               
               
                 Radar_back_right_45 
                 0.051912 
                 0.122894 
                 0.699521 
               
               
                   
               
            
           
         
       
     
     In some implementations, a process of utilizing one radar at a time may include fixed z, roll, and pitch values for all of the radars (e.g., eight radars). This may include 24 DoF optimization (e.g., x, y, and yaw) times eight radars using a solver (e.g., a Ceres solver). A total of eight runs were performed including five in TT1 and three in TT2. For example, while the results of Table 8 (e.g., one radar at a time) are similar to the results of Table 7 (e.g., combination optimization), runtime for analyzing one radar at a time is much shorter (e.g., 2-3 minutes) as all of the radars can be easily calibrated in parallel. 
     In other implementations, the processes and systems as described herein may further utilize on-road drive data (e.g., in real-time) to calibrate the radars and lidars of the vehicle. For example, a cloud radar-lidar calibration can include aggregating radar and lidar point clouds using odom/SLP (e.g., forLoc for reference maps), and minimizing entropy over multiple drive segments. 
     Having disclosed some example system components and concepts, the disclosure now turns to  FIG.  20   , which illustrates an example method  2000  for performing radar-to-lidar extrinsic calibration with on-road drive data. The steps outlined herein are exemplary and can be implemented in any combination thereof, including combinations that exclude, add, or modify certain steps. 
     At step  2002 , method  2000  can include receiving, at an autonomous vehicle system, radar data from a radar of an object. 
     At step  2004 , method  2000  can include receiving, at the autonomous vehicle system, lidar data from a lidar of the object. The radar data can include radar point clouds and the lidar data can include lidar point clouds. The receiving of the radar data and the lidar data can be received from a plurality of autonomous vehicle poses. 
     At step  2006 , method  2000  can include generating, by the autonomous vehicle system, a plurality of cost functions based on the radar data and the lidar data of the object. The generating of the plurality of cost functions can include minimizing an entropy of a plurality of drive segments. The method  2000  can further include generating an aggregated map of a scene based on the radar data, the lidar data, and the minimizing of the entropy as the plurality of cost functions. 
     At step  2008 , method  2000  can include adjusting, by the autonomous vehicle system, at least one setting based on the plurality of cost functions of the radar data and the lidar data of the object. The adjusting of the at least one setting can include aligning the radar point clouds and the lidar point clouds based on the plurality of cost functions. 
     The method  2000  can further include optimizing six degrees of freedom of the radar based on the plurality of cost functions. 
       FIG.  21    illustrates an example processor-based system with which some aspects of the subject technology can be implemented. For example, processor-based system  2100  that can be any computing device making up internal computing system  110 , remote computing system  150 , a passenger device executing the rideshare app  170 , internal computing device  130 , or any component thereof in which the components of the system are in communication with each other using connection  2105 . Connection  2105  can be a physical connection via a bus, or a direct connection into processor  2110 , such as in a chipset architecture. Connection  2105  can also be a virtual connection, networked connection, or logical connection. 
     In some embodiments, computing system  2100  is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple data centers, a peer network, etc. In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some embodiments, the components can be physical or virtual devices. 
     Example system  2100  includes at least one processing unit (CPU or processor)  2110  and connection  2105  that couples various system components including system memory  2115 , such as read-only memory (ROM)  2120  and random-access memory (RAM)  2125  to processor  2110 . Computing system  2100  can include a cache of high-speed memory  2112  connected directly with, in close proximity to, and/or integrated as part of processor  2110 . 
     Processor  2110  can include any general-purpose processor and a hardware service or software service, such as services  2132 ,  2134 , and  2136  stored in storage device  2130 , configured to control processor  2110  as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor  2110  may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric. 
     To enable user interaction, computing system  2100  includes an input device  2145 , which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system  2100  can also include output device  2135 , which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input/output to communicate with computing system  2100 . Computing system  2100  can include communications interface  2140 , which can generally govern and manage the user input and system output. The communication interface may perform or facilitate receipt and/or transmission wired or wireless communications via wired and/or wireless transceivers, including those making use of an audio jack/plug, a microphone jack/plug, a universal serial bus (USB) port/plug, an Apple® Lightning® port/plug, an Ethernet port/plug, a fiber optic port/plug, a proprietary wired port/plug, a BLUETOOTH® wireless signal transfer, a BLUETOOTH® low energy (BLE) wireless signal transfer, an IBEACON® wireless signal transfer, a radio-frequency identification (RFID) wireless signal transfer, near-field communications (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), Infrared (IR) communication wireless signal transfer, Public Switched Telephone Network (PSTN) signal transfer, Integrated Services Digital Network (ISDN) signal transfer, 3G/4G/5G/LTE cellular data network wireless signal transfer, ad-hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. 
     Communications interface  2140  may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers that are used to determine a location of the computing system  2100  based on receipt of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russia-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the Europe-based Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed. 
     Storage device  2130  can be a non-volatile and/or non-transitory computer-readable memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, a floppy disk, a flexible disk, a hard disk, magnetic tape, a magnetic strip/stripe, any other magnetic storage medium, flash memory, memristor memory, any other solid-state memory, a compact disc read only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick® card, a smartcard chip, a EMV chip, a subscriber identity module (SIM) card, a mini/micro/nano/pico SIM card, another integrated circuit (IC) chip/card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1/L2/L3/L4/L5/L#), resistive random-access memory (RRAM/ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and/or a combination thereof. 
     Storage device  2130  can include software services, servers, services, etc., that when the code that defines such software is executed by the processor  2110 , it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor  2110 , connection  2105 , output device  2135 , etc., to carry out the function. 
     As understood by those of skill in the art, machine-learning based classification techniques can vary depending on the desired implementation. For example, machine-learning classification schemes can utilize one or more of the following, alone or in combination: hidden Markov models; recurrent neural networks; convolutional neural networks (CNNs); deep learning; Bayesian symbolic methods; general adversarial networks (GANs); support vector machines; image registration methods; applicable rule-based system. Where regression algorithms are used, they may include including but are not limited to: a Stochastic Gradient Descent Regressor, and/or a Passive Aggressive Regressor, etc. 
     Machine learning classification models can also be based on clustering algorithms (e.g., a Mini-batch K-means clustering algorithm), a recommendation algorithm (e.g., a Miniwise Hashing algorithm, or Euclidean Locality-Sensitive Hashing (LSH) algorithm), and/or an anomaly detection algorithm, such as a Local outlier factor. Additionally, machine-learning models can employ a dimensionality reduction approach, such as, one or more of: a Mini-batch Dictionary Learning algorithm, an Incremental Principal Component Analysis (PCA) algorithm, a Latent Dirichlet Allocation algorithm, and/or a Mini-batch K-means algorithm, etc. 
     Embodiments within the scope of the present disclosure may also include tangible and/or non-transitory computer-readable storage media or devices for carrying or having computer-executable instructions or data structures stored thereon. Such tangible computer-readable storage devices can be any available device that can be accessed by a general purpose or special purpose computer, including the functional design of any special purpose processor as described above. By way of example, and not limitation, such tangible computer-readable devices can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other device which can be used to carry or store desired program code in the form of computer-executable instructions, data structures, or processor chip design. When information or instructions are provided via a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable storage devices. 
     Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. By way of example computer-executable instructions can be used to implement perception system functionality for determining when sensor cleaning operations are needed or should begin. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform tasks or implement abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps. 
     Other embodiments of the disclosure may be practiced in network computing environments with many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices. 
     The various embodiments described above are provided by way of illustration only and should not be construed to limit the scope of the disclosure. For example, the principles herein apply equally to optimization as well as general improvements. Various modifications and changes may be made to the principles described herein without following the example embodiments and applications illustrated and described herein, and without departing from the spirit and scope of the disclosure. Claim language reciting “at least one of” a set indicates that one member of the set or multiple members of the set satisfy the claim.