Patent Description:
The development of a vision-based autonomous driving system has been critical to providing safety for driving vehicles. The system may require information about camera location and orientation relative to the vehicle coordinates. Typical applications involve mapping lane lines into a bird-eye view to achieve goals like keeping the vehicle in the lane, estimating distances of detected moving objects, stitching a surround view image from front-view, side-view and rear view cameras, etc. As the location of camera to vehicle is usually originally fixed but minor drifts may occur during later operation, the calibration of camera orientation is crucial for driver assistance systems.

Traditional method of vehicle camera calibration can be performed with assistance from controlled visual targets and tools in a controlled environment, namely, offline calibration. This is typically used in either factory production line or service station where a well-controlled environment with professional equipment such as photographed patterns, vehicle centering platform, lighting control device, etc. can be set up.

Offline calibration has several limitations. First, it is inconvenient because the equipment placement and size are regulated by the camera location, resolution, and field of view. Second, the cost of station setup is high, and it often requires intensive human labor to assist the process.

There are some circumstances that the vehicle cameras are off their calibrated orientation and therefore need an immediate re-calibration. One possible case is that the cameras are off its original position after service, repairing, or replacement at service station. Another possible case is the drift of the cameras over time due to vibrations, mounting, or deformations in the vehicle body after driving a certain amount of time. The first case can be handled by offline calibration by setting up a camera calibration room at the high cost of equipment and human labor. The second case, however, has no solution and leaves the risk in driver assistance perception due to using inaccurate camera orientation to monitor neighboring environments. Online calibration, in contrast, provides a solution that completes the whole calibration in an uncontrolled environment without help from any targets and tools. It not only saves great amount of cost for the first case, but also automatically monitors and completes the calibration for the second case by driving on the road for a certain amount of time. Online solution has no cost but driving and has no human participation that every car owners do it without even feelings.

Document <CIT> relates to online calibration of multiple cameras attached to vehicle. The proposed approach uses vanishing points (of identified lane markings) for forward and rearward camera calibration, considering the vehicle speed and the steering angle.

Document <CIT> relates to online calibration of forward-looking camera attached to vehicle. The proposed approach uses vanishing points (of identified lane markings) for camera calibration and taking into account vehicle speed (calibration is only effected if specific speed is exceeded).

Document <CIT> relates to the calibration of a multi-camera RGB-D system using movable calibration target and involving a pose-graph approach.

Some existing online calibration methods have several limitations when applied to a complete vehicle camera system that consists of front, side and rear cameras to cover a surround field of view for vehicle perception. Odometry-based methods require high accuracy of the location of other sensors and their measurements, and often contain accumulated error due to bias or measurement inaccuracy. From the perspective of vehicle setup, if using Lidar as the anchor for camera calibration, it limits its application in vehicles without such sensor. Pure-vision based methods often are not general solutions for a complete camera system, they either calibrate one front camera or calibrate surround cameras with prior camera setup of one or two cameras.

To overcome these limitations from prior disclosed patents and literature, the combination of vehicle sensor measurements and camera perception system is important. Instead of relying mostly on odometry sensor measurements, these measures can be utilized as supplemental signals for vision solutions to avoid accuracy issues. Vision measurements, including lane line and object detection, can be easily obtained from camera perception system of a vehicle with driving assistance package. Therefore, the online calibration system can be tightened to the whole driving system instead of being an independent system that occupies a large amount of resources.

The following presents a simplified summary of one or more aspects to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects.

One aspect of the present disclosure provides a method for calibrating cameras on an autonomous driving system. The method includes receiving a current speed of a moving vehicle from a wheel odometer; determining that the current speed of the moving vehicle is greater than a first predetermined speed threshold; calibrating, based on the determination that the current speed is greater than the predetermined speed threshold, at least one front camera of the moving vehicle according to a position of a vanishing point of two lane lines; determining that the current speed of the moving vehicle is less than a second predetermined speed threshold and is greater than zero; and calibrating, based on the determination that the current speed is less than the predetermined speed threshold and is greater than zero, multiple side and rear cameras of the moving vehicle according to a pose graph that includes relative poses of the multiple side and rear cameras.

Another aspect of the present disclosure provides a system for calibrating cameras on an autonomous driving system. The system includes a wheel odometer configured to measure a current speed of a moving vehicle; a speed monitor configured to determine that the current speed of the moving vehicle is greater than a first predetermined speed threshold; and a camera calibrator configured to calibrate, based on the determination that the current speed is greater than the predetermined speed threshold, at least one front camera of the moving vehicle according to a position of a vanishing point of two lane lines. The speed monitor is further configured to determine that the current speed of the moving vehicle is less than a second predetermined speed threshold and is greater than zero. The camera calibrator is further configured to calibrate, based on the determination that the current speed is less than the predetermined speed threshold and is greater than zero, multiple side and rear cameras of the moving vehicle according to a pose graph that includes relative poses of the multiple side and rear cameras.

To the accomplishment of the foregoing and related ends, the one or more aspects comprise the features herein after fully described and particularly pointed out in the claims.

In the following description, for purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects.

In the present disclosure, the term "comprising" and "including" as well as their derivatives mean to contain rather than limit; the term "or," which is also inclusive, means and/or.

In this specification, the following various embodiments used to illustrate principles of the present disclosure are only for illustrative purpose, and thus should not be understood as limiting the scope of the present disclosure by any means. The following description taken in conjunction with the accompanying drawings is to facilitate a thorough understanding of the illustrative embodiments of the present disclosure defined by the claims and its equivalent. There are specific details in the following description to facilitate understanding. However, these details are only for illustrative purpose.

In addition, for clear and concise purpose, some known functionality and structure are not described. Besides, identical reference numbers refer to identical function and operation throughout the accompanying drawings.

An online calibration system for a moving vehicle is described herein to calibrate yaw and pitch orientations of multiple (e.g., <NUM>) cameras with overlapping field-of-view covering <NUM> degree monitoring. In some examples, two of the multiple cameras are mounted on the moving vehicle facing the front, four cameras are mounted on the side, and one camera on the rear.

Calibration for the front two cameras may be based on a calculated vanishing point. The vanishing point is estimated by calculating an intersection of lane lines detected by deep learning models from a built-in camera perception system of the autonomous driving system.

Calibration for the side and rear cameras may be based on a pose graph optimization algorithm. A pose graph may be built by set camera absolute orientation as vertices and relative orientation between two cameras as edges. The relative orientation is estimated with feature detection and matching. The pose graph is optimized accordingly with the front cameras fixed to balance the errors among all edges.

<FIG> illustrates a diagram showing an example autonomous driving system <NUM> equipped with multiple cameras in accordance with the disclosure. As depicted, the example autonomous driving system <NUM> may be implemented on a land vehicle <NUM>.

The example autonomous driving system <NUM> may include multiple cameras mounted on different locations of the vehicle <NUM>. As described above, the orientations of the cameras may drift due to vibrations, mounting, deformation in the vehicle body. Such drift may occur in three different principal axes or directions, i.e., yaw, pitch, and roll. Contrary to conventional "offline" camera calibration, the present disclosure provides systems and methods for calibrating the cameras while the vehicle is moving. It is noted that the term "online" refers to systems including working components in communication with each other on a traveling vehicle and updating configuration including camera orientations, rather than requiring the support of Internet. In some examples, the term "online" may be used interchangeably with "on-the-fly.

In some examples, the multiple cameras are mounted facing different directions from the perspective of the vehicle <NUM> and may cover different field-of-view (FOV). For example, a front main camera <NUM> and a front narrow camera <NUM> (collectively, "the front cameras") may be both mounted on top of the vehicle <NUM> facing front. The FOV of the front main camera <NUM> is typically wider than the FOV of the front narrow camera <NUM>.

Other cameras may be mounted on the side and rear of the vehicle <NUM> (collectively, "side and rear cameras"). For example, a front right camera <NUM> facing front right and a rear right camera <NUM> facing rear right may be mounted the right side of the vehicle <NUM>. A front left camera <NUM> facing front left and a rear left camera <NUM> facing rear left may be mounted on the left side of the vehicle <NUM>. A rear main camera <NUM> facing rear may be mounted on the rear part of the vehicle <NUM>.

The overlapping FOV between two adjacent cameras may be manually adjusted and set to a predetermined degree. For example, the overlapping FOV between the front main camera <NUM> and the front left camera <NUM> or the front right camera <NUM> may be around <NUM> degrees. The overlapping FOV between the front right camera <NUM> and the rear right camera <NUM> may be the same as the overlapping FOV between the front left camera <NUM> and the rear left camera <NUM>. Both may be set to around <NUM> degrees. The overlapping FOV between the rear main camera <NUM> and the rear left camera <NUM> or the rear right camera <NUM> may be around <NUM> degrees.

The example autonomous driving system <NUM> may further include a wheel odometer <NUM> configured to measure a current speed of the vehicle <NUM>. As described in greater detail below, the online calibration in accordance with the present disclosure may include two parts of calibration, i.e., calibrating the front cameras when the vehicle <NUM> travels at a high speed and calibrating the side and rear cameras when the vehicle <NUM> travels at a low speed. Thus, example autonomous driving system <NUM> may further include a speed monitor <NUM> configured to determine whether the measured current speed is "high" or "low" according to predetermined speed thresholds. For example, when the current speed is greater than <NUM> per hour, the current speed may be determined as "high" and when the current speed is less than <NUM> per hour, the speed may be determined as "low. " The result of the speed determination may be sent to a camera calibrator <NUM> to trigger the calibration process.

<FIG> illustrates a diagram showing an example camera calibrator <NUM> for calibrating the multiple cameras of the example autonomous driving system in accordance with the disclosure.

In the case where the speed monitor <NUM> determines that the current speed of the vehicle <NUM> is high, the camera calibrator <NUM> may be configured to start the calibration process for the front cameras first.

In some examples, the camera calibrator <NUM> may first check if one or more additional conditions are met prior to the calibration process to ensure the environment surrounding the vehicle <NUM> is suitable for the calibration process. For example, the camera calibrator <NUM> may check signals transmitted from other modules of the vehicle <NUM> via controller area network (CAN) service. The camera calibrator <NUM> may check the signal from a wiper indicator to determine if the wipers are working, which may indicate whether it is raining. The camera calibrator <NUM> may check the signal from an angular rater monitor to determine if the angular rate is less than one degree per second, which may indicate whether the vehicle <NUM> is turning.

Further, the example autonomous driving system <NUM> may also include a visual perception system configured to detect lane lines and surrounding objects based on deep learning or artificial intelligence algorithms. Results of the detection may be sent to the camera calibrator <NUM>. The camera calibrator <NUM> may be configured to determine whether the lane lines are straight and whether the vehicle <NUM> is traveling forward in the middle of the lane lines, e.g., polynomial fitting second order coefficient <α.

In some examples where the additional conditions are determined to be met, i.e., no rain, traveling forward, lane lines are straight, a vanishing point determiner <NUM> of the camera calibrator <NUM> may be configured to calculate a convergence of the two lane lines and determine the convergence as the vanishing point of the two lane lines. As the vehicle <NUM> is moving, the vanishing point determiner <NUM> may be configured to periodically calculate multiple vanishing points. Once a count of the multiple vanishing points reaches a preset threshold, the multiple vanishing points may be sent to an orientation calculator <NUM>.

The orientation calculator <NUM> may be configured to determine an offset between a predetermined principal point and the vanishing point and further determine a yaw orientation and a pitch orientation based on the offset. The yaw and pitch orientations may indicate the amount of the drift of the corresponding camera. The example autonomous driving system <NUM> may be configured to utilize new camera orientation parameters in autonomous driving algorithms accordingly to compensate for the drift.

The calculation of the vanishing point and the yaw and pitch orientations is described in more detail in accordance with <FIG>. As shown in <FIG>, the yaw and pitch orientations may be sent to a pose graph generator <NUM> for further processing.

In the case where the speed monitor <NUM> determines that the current speed of the vehicle <NUM> is low, the camera calibrator <NUM> may be configured to start the calibration process for the side and rear cameras subsequent to the calibration of the front cameras if a group of additional conditions are met. The camera calibrator <NUM> may be configured to check the signal from a wiper indicator to determine if the wipers are working, which may indicate whether it is raining. The camera calibrator <NUM> may check the signal from an angular rater monitor to determine if the angular rate is less than one degree per second. The camera calibrator <NUM> may further check the status of low/high/fog beam and the lux of the environment to determine the environmental light is sufficient.

For example, a pose determiner <NUM> of the camera calibrator <NUM> may be configured to determine the relative poses of the multiple side and rear cameras. The relative poses of the multiple side and rear cameras may refer to the poses of each camera relative to a reference camera. The reference camera may be one of two adjacent cameras having a connecting edge in the pose graph.

In more detail, the pose determiner <NUM> may further include a feature detector <NUM> configured to detect features of surrounding static objects in images captured by the side and rear cameras. A feature point matcher <NUM> of the pose determiner <NUM> may be configured to match feature points of a same static object in different images based on ranked similarities. A pose calculator <NUM> of the pose determiner <NUM> may then be configured to determine the relative poses.

Based on the relative poses of the multiple side and rear cameras and the yaw and pitch orientations of the front cameras, the pose graph generator <NUM> may be configured to generate a pose graph with one of the front cameras being set as a reference or an anchor point. A loss calculator <NUM> may be configured to calculate an optimization loss for each of the side and rear cameras based on the pose graph. The optimization loss may refer to the orientations that indicate the drifts of the side and rear cameras. Similarly, the example autonomous driving system <NUM> may be configured to adjust parameters in the autonomous driving algorithms accordingly to compensate for the drifts.

<FIG> illustrates a flow chart showing an example method <NUM> for calibrating cameras on the example autonomous driving system in accordance with the disclosure. The process of performing example method <NUM> may start from block <NUM>.

At block <NUM>, the operations of method <NUM> may include receiving a current speed of a moving vehicle from a wheel odometer. For example, the speed monitor <NUM> may be configured to receive a current speed of vehicle <NUM> from the wheel odometer <NUM>.

At block <NUM>, the operations of method <NUM> may include determining that the current speed of the moving vehicle is greater than a first predetermined speed threshold or is less than a second predetermined speed threshold. If the result of the determination of the current speed is greater than the first predetermined speed threshold, the process of the method <NUM> may continue to block <NUM>. If the result of the determination of the current speed is less than the second predetermined speed threshold but still greater than zero, the process of the method <NUM> may continue to block <NUM>.

At block <NUM>, the operations of method <NUM> may include checking if a group of first additional conditions are met based on signals fed from a wiper indicator <NUM>, an angular rate monitor <NUM>, and a lane detector <NUM> via CAN service. In some examples, the camera calibrator <NUM> may be configured to determine whether it is raining based on the signals from the wiper indicator <NUM>, whether the vehicle <NUM> is turning based on the signals from the angular rate monitor <NUM>, and whether the lane lines are straight based on the output from the lane detector <NUM> in the visual perception system.

At block <NUM>, the operations of method <NUM> may include determining the vanishing point by calculating a convergence of the two lane lines on an image plane. For example, the vanishing point determiner <NUM> of the camera calibrator <NUM> may be configured to calculate a convergence of the two lane lines and determine the convergence as the vanishing point of the two lane lines. As the vehicle <NUM> is moving, the vanishing point determiner <NUM> may be configured to periodically calculate multiple vanishing points. Once a count of the multiple vanishing points reaches a preset threshold, the multiple vanishing points may be sent to the orientation calculator <NUM>.

At block <NUM>, the operations of method <NUM> may include calculating a yaw orientation and a pitch orientation respectively for the at least one front camera based on a difference between the vanishing point and a principal point and calibrating the front cameras. For example, the orientation calculator <NUM> may be configured to determine an offset between a predetermined principal point and the vanishing point and further determine a yaw orientation and a pitch orientation based on the offset. The process of method <NUM> may continue to block <NUM>.

At block <NUM>, the operations of method <NUM> may include determining a group of second additional conditions are met. For example, the camera calibrator <NUM> may be configured to check the signal from a wiper indicator to determine if the wipers are working and if it is raining. The camera calibrator <NUM> may check the signal from an angular rater monitor to determine if the angular rate is less than one degree per second. The camera calibrator <NUM> may further check the status of low/high/fog beam and the lux of the environment to determine the environmental light is sufficient.

At block <NUM>, the operations of method <NUM> may include determining the relative poses of the multiple side and rear cameras. For example, a pose determiner <NUM> of the camera calibrator <NUM> may be configured to determine the relative poses of the multiple side and rear cameras.

At block <NUM>, the operations of method <NUM> may include generating the pose graph that includes the relative poses of the multiple side and rear cameras. For example, based on the relative poses of the multiple side and rear cameras and the yaw and pitch orientations of the front cameras, the pose graph generator <NUM> may be configured to generate a pose graph with one of the front cameras being set as a reference or an anchor point.

At block <NUM>, the operations of method <NUM> may include calculating an optimization loss for each of the multiple side and rear cameras based on the pose graph. For example, the loss calculator <NUM> may be configured to calculate an optimization loss for each of the side and rear cameras based on the pose graph.

<FIG> illustrates a diagram showing an example vanishing point in accordance with the disclosure.

In general, given a 3D point in the real-world, the corresponding 2D image point can be expressed as: x2d = K[R|T]X3d, where K is an intrinsic matrix of the camera that captured the 2D image, R and T respectively refer to the 3D rotation and translation. In the present disclosure, the vanishing point may be defined as a point on the image plane representing the converge of 2D perspective projections of mutually parallel lines in 3D space. Specifically, the vanishing point is an infinity point independent of translation T in the 3D coordinate. Thus, the vanishing point on a 2D image can be expressed as: <MAT> where VP2d refers to the position of the vanishing point in the 2D image and VP3d refers to the 3D coordinate of the vanishing point. The vanishing point may correspond to the yaw and pitch orientation of the front camera. To estimate the vanishing point, we take the assumption that when the vehicle is moving straight in the center of two lanes, the two ego lane lines can be considered to be parallel to its moving direction.

Considering the image plane with the principal point (cx, cy), and estimated vanishing point (x, y). The principal point is typically provided in camera intrinsic information and is determined when a camera is produced. As shown in <FIG>, the vanishing point in plane offset by the principal point may be expressed as: (uvp, vvp) = (xvp, yvp) - (cx, cy),.

The yaw and pitch orientation can be computed as: uvp = fxtan(yaw) and vvp = -fytan(pitch) respectively.

<FIG> illustrate a flow chart showing an example method <NUM> for determining relative poses of multiple side and rear cameras in accordance with the disclosure.

As a part of calibrating the side and rear cameras, the process of determining or estimating relative poses of the side and rear cameras may be performed based the overlapping FOV between two adjacent cameras, for example, the overlapping FOV between the front main camera <NUM> and the front right camera <NUM>. Images captured by the two adjacent cameras may be provided at the beginning of the process of method <NUM> for determining the relative poses. Dash-lined blocks may indicate optional operations of method <NUM>.

At block <NUM>, the operations of method <NUM> may include detecting features of objects in images captured by the multiple side and rear cameras. For example, the feature detector <NUM> may be configured to detect the features of surrounding static objects in accordance with some existing algorithm, e.g., a scale-invariant feature transform (SIFT) algorithm. As feature points of moving objects may cause errors in later feature matching process, the feature points of moving objects may be removed. The moving objects in the images may then be marked with bounding boxes. In other words, features points in the bounding boxes may not be considered in later process.

At block <NUM>, the operations of method <NUM> may include matching feature points in the images based on ranked similarities. For example, the feature point matcher <NUM> may be configured to match the feature points in different images captured by the two adjacent cameras and rank the feature points based on the similarities of the feature points. The ranking may be performed by the feature point matcher <NUM> in accordance with a k-nearest-neighbor (KNN) algorithm.

In some examples, the feature point matcher <NUM> may be further configured to filter and discard unqualified feature points. The filtering may include one or more phases, e.g., distance filtering, symmetric filtering, and nominal extrinsic filtering. In distance filtering, the feature point matcher <NUM> may be configured to discard two similar points that are too close. In symmetric filtering, the feature point matcher <NUM> may be configured to discard feature points that only can be projected from one image to the other, but not vice versa. In nominal extrinsic filtering, the feature point matcher <NUM> may be configured to discard feature points based on point correspondences between two cameras that are constrained by initial camera setup including camera translation relative to vehicle and camera ideal installation orientation angles. Feature points that meet the criteria will be kept. The criteria can be formulated as <MAT>, where x<NUM> and x<NUM> are normalized corresponding points from two images captured by two adjacent cameras, t and R are the ideal installation translation and rotation, and δ is a predetermined threshold value.

At block <NUM>, the operations of method <NUM> may include determining relative poses based on the matched feature points. For example, the pose calculator <NUM> may be configured to estimate or determine the relative pose of each camera taking an adjacent camera as a reference point. In more detail, the pose calculator <NUM> may be configured to calculate an essential matrix between the two adjacent cameras and decompose the essential matrix into a rotation and a translation matrix as the relative pose.

At block <NUM>, the operations of method <NUM> may include rejecting unqualified relative poses. For example, when the relative poses appear to be the result of reprojection error and depth distribution, the relative poses may be discarded and rejected as invalid by the pose calculator <NUM>. A reprojection error may refer to a geometric error when projecting a point from one image to another image using estimated relative pose. The average error of all matching points should be smaller than a predetermined threshold. Otherwise, this estimated relative pose may be rejected. Further, each point depth (distance) can be calculated based on estimated relative pose. The standard deviation of the depth distribution formed by all matching points should be larger than a predefined threshold. Otherwise, this estimated relative pose is rejected.

At block <NUM>, the operations of method <NUM> may include outputting valid relative poses. For example, the pose calculator <NUM> may be configured to output the valid relative poses to the pose graph generator <NUM>.

<FIG> illustrates a diagram showing an example process for determining optimization losses of the multiple side and rear cameras in accordance with the disclosure.

In some examples, the loss calculator <NUM> may be configured to generate a pose graph with each of the six cameras being vertices and the relative poses (e.g., R<NUM> to R<NUM> as shown) between any two adjacent cameras being edges. Adjacent cameras may refer to a pair of cameras that are connected by an edge on the pose graph (e.g., the front right camera <NUM> and the rear right camera <NUM>). Since the front main camera <NUM> has been calibrated prior to other side and rear cameras, the front main camera <NUM> may be set as the anchor point. The graph optimization loss can be formulated to minimize the discrepancy for the each of the side and rear cameras as follows: <MAT> <MAT> <MAT> <MAT> <MAT>.

The pose graph may also be applied to calibrate one or two cameras if other cameras are well calibrated. For example, if only the rear left and front right needs to be calibrated, the optimization loss can be formulated as: <MAT> <MAT>.

<FIG> illustrates a diagram showing another example process for calibrating the multiple cameras in accordance with the disclosure.

As depicted, a camera image stream <NUM> may be fed to one or more memory ring buffers <NUM> respectively for each camera. The memory ring buffers <NUM> may temporarily store the camera image stream <NUM> and feed the camera image stream <NUM> to a vehicle perception system <NUM> (alternatively, "visual perception system" as described above). The vehicle perception system <NUM> may be configured to detect or recognize static objects in the camera image stream <NUM> including lane lines. The detected static objects may be transmitted to the pose determiner <NUM>, e.g., block <NUM>. To reduce the possibility of later errors, moving objects may be detected but marked with bounding boxes such that the feature points within the bounding boxes will not be considered as input for the feature point matcher <NUM>.

When lane lines are detected in the camera images, the camera calibrator <NUM> may be configured to check lane lines straightness (block <NUM>) and other signals via CAN service <NUM>. For example, the camera calibrator <NUM> may be configured to determine whether it is raining based on the signals from the wiper indicator <NUM>, whether the vehicle <NUM> is turning based on the signals from the angular rate monitor <NUM>, and whether the lane lines are straight based on the output from the lane detector <NUM> in the visual perception system.

At block <NUM>, the vanishing point determiner <NUM> of the camera calibrator <NUM> may be configured to calculate a convergence of the two lane lines and determine the convergence as the vanishing point of the two lane lines. Further, at block <NUM>, when a count of collected vanishing points, e.g., stored by a vanishing point counter <NUM>, is greater than a preset threshold, e.g., NH, a histogram may be created and the peak value of the histogram may be selected as the rotation angle for the final calibration results as indicated by block <NUM>. The calibration results may be sent to a calibration file management system <NUM> that manages the calibration configuration files of cameras of the example autonomous driving system <NUM>. In some examples, when camera calibrator <NUM> sends the latest calibrated camera orientations to the file management system <NUM>, the file management system <NUM> may update the camera configuration files and notify other modules of the autonomous driving system <NUM>.

At block <NUM>, the camera calibrator <NUM> may similarly check if additional conditions are met based on signals transmitted via CAN service <NUM>. For example, the camera calibrator <NUM> may be configured to check the signal from a wiper indicator to determine if the wipers are working and if it is raining. The camera calibrator <NUM> may check the signal from an angular rater monitor to determine if the angular rate is less than one degree per second. The camera calibrator <NUM> may further check the status of low/high/fog beam and the lux of the environment to determine the environmental light is sufficient.

When the addition conditions are met, the camera calibrator <NUM> may be configured to identify pairs of images taken by two adjacent cameras from the memory ring buffer <NUM>.

At block <NUM>, the feature detector <NUM> may be configured to detect the features of surrounding static objects in accordance with some existing algorithm, e.g., a scale-invariant feature transform (SIFT) algorithm. The feature point matcher <NUM> may be configured to match the feature points in different images captured by the two adjacent cameras and rank the feature points based on the similarities of the feature points.

Similarly, at block <NUM>, the pose calculator <NUM> may be configured to check an edge counter <NUM> that stores a count of estimated relative poses. When the count of estimated relative poses exceeds a predetermined threshold, e.g., NL, the pose graph generator <NUM> may be configured to generate the pose graph. At block <NUM>, the loss calculator <NUM> may be configured to calculate an optimization loss for each of the side and rear cameras based on the pose graph. The calibration results may also be sent to a calibration file management system <NUM>.

The process and method as depicted in the foregoing drawings may be executed through processing logics including hardware (e.g., circuit, special logic, etc.), firmware, software (e.g., a software embodied in a non-transient computer readable medium), or combination of each two. Although the above describes the process or method in light of certain sequential operation, it should be understood that certain operation described herein may be executed in different orders. Additionally, some operations may be executed concurrently rather than sequentially.

In the above description, each embodiment of the present disclosure is illustrated with reference to certain illustrative embodiments. Any of the above-mentioned components or devices may be implemented by a hardware circuit (e.g., application specific integrated circuit (ASIC)). Correspondingly, the description and accompanying figures should be understood as illustration only rather than limitation. It is understood that the specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged. Further, some steps may be combined or omitted. The accompanying method claims present elements of the various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

Thus, the claims are not intended to be limited to the aspects shown herein but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more. No claim element is to be construed as a means plus function unless the element is expressly recited using the phrase "means for.

Claim 1:
A method for calibrating cameras on an autonomous driving system, comprising:
receiving a current speed of a moving vehicle from a wheel odometer;
determining that the current speed of the moving vehicle is greater than a first predetermined speed threshold;
calibrating, based on the determination that the current speed is greater than the predetermined speed threshold, at least one front camera of the moving vehicle according to a position of a vanishing point of two lane lines;
determining that the current speed of the moving vehicle is less than a second predetermined speed threshold and is greater than zero; and
calibrating, based on the determination that the current speed is less than the predetermined speed threshold and is greater than zero, multiple side and rear cameras of the moving vehicle according to a pose graph that includes relative poses of the multiple side and rear cameras.