Patent ID: 12249100

DETAILED DESCRIPTION

FIGS.1through5, described below, and the various embodiments used to describe the principles of this disclosure are by way of illustration only and should not be construed in any way to limit the scope of this disclosure. Those skilled in the art will understand that the principles of this disclosure may be implemented in any type of suitably arranged device or system.

As noted above, advanced driver assistance system (ADAS) features are being incorporated into more and more vehicles over time. ADAS systems generally support features implemented using various technologies that assist drivers in performing driving and parking functions. Among other things, ADAS systems often use cameras or other imaging sensors to capture images and identify information related to vehicles during travel. For example, ADAS systems can process images captured by imaging sensors on vehicles in order to identify lane marking lines associated with traffic lanes in which the vehicles are traveling. The identification of lane marking lines associated with a traffic lane in which a vehicle is traveling can be used for various purposes, such as to trigger an audible or other warning if the vehicle appears to be straying from its current traffic lane while a turn signal indicator (blinker) of the vehicle is not activated. In some cases, this type of information or other information obtained by processing images associated with a vehicle may be used to generate an estimated measure of the attentiveness or inattentiveness of a vehicle's driver.

Cameras and other imaging sensors used in ADAS systems and other in-vehicle systems generally need to be calibrated, which often involves identifying characteristics of the imaging sensors as installed on the vehicles. For example, the actual location of an imaging sensor on a vehicle and the orientation of the imaging sensor on the vehicle (such as its pitch and yaw angles) may need to be identified so that this information can be used in various mathematical calculations during use. As a particular example, the pitch and yaw angles of an imaging sensor may need to be determined so that images from the imaging sensor can be processed in order to identify distances to lane marking lines, other vehicles, or other objects. When imaging sensors are installed on vehicles during the manufacture of the vehicles on an assembly line, it may be possible to install the imaging sensors at substantially the same location and at substantially the same orientation on multiple vehicles of the same type (such as the same make and model). In those cases, it may be possible to calibrate the imaging sensors using the same information, meaning each imaging sensor itself may not require a unique calibration (at least in relation to its position and orientation on a vehicle).

Imaging sensors can also be installed on vehicles after their manufacture, which is typically referred to as “aftermarket” installation. During these or other types of installations, imaging sensors may be installed on different vehicles at different locations and at different orientations. Even if installation locations are similar to one another and installation orientations are similar to one another, each of the imaging sensors typically needs to undergo its own calibration process in order to precisely identify its pitch and yaw angles, and the results of the calibration process depend on the specific location and the specific orientation of each imaging sensor. An effective calibration process allows information to be obtained so that images from each imaging sensor can be processed in order to accurately identify information such as distances to lane marking lines and distances to objects (such as other vehicles) around the associated vehicle. However, performing an effective calibration process can be difficult to due to various circumstances.

In one prior approach, an imaging sensor is installed on a vehicle, and manual measurements identifying the location of the imaging sensor on the vehicle are obtained. Also, a large vinyl banner or other banner is placed on the ground in front of the vehicle. The banner includes multiple lines or other known markings, and the banner is placed at a known distance in front of the vehicle. The imaging sensor on the vehicle is used to capture images of the banner, and the images and the manual measurements are processed in order to estimate extrinsic calibration parameters associated with the imaging sensor. These extrinsic calibration parameters depend (among other things) on the location of the imaging sensor on the vehicle and the orientation of the imaging sensor on the vehicle.

While this approach may provide acceptable accuracy in the identification of certain extrinsic calibration parameters of an imaging sensor (such as its pitch and yaw angles), this approach suffers from a number of shortcomings. For example, this approach can be difficult, time-consuming, and costly to perform, and this approach typically cannot be performed during inclement weather (such as during windy or rainy conditions). Also, this approach assumes that a vehicle and a banner are in the same plane, which may not always be the case (such as on sloped parking lots). Further, this approach typically requires an installer to position a banner exactly in the mid-line of a vehicle at a large distance (such as twenty to thirty feet away) from the vehicle, which may be difficult to ensure. In addition, if the calibration process is not performed correctly, there may be no way to correct the extrinsic calibration parameters without taking a vehicle out of service, and the incorrect extrinsic calibration parameters can lead to a high rate of false-positive triggers related to tailgating, lane departure, forward collision, or other warnings generated by a vehicle based on images from its incorrectly-calibrated imaging sensor.

This disclosure provides techniques for performing highly-accurate and self-adjusting imaging sensor auto-calibration for an in-vehicle ADAS system or other system. As described in more detail below, measurements can be obtained that identify the position of an imaging sensor on a vehicle, such as measurements identifying the distance of the imaging sensor from one or more specified points or axes of the vehicle. The imaging sensor is also used to capture images of a road or other surface in front of the vehicle while the vehicle is being driven. The images are processed to identify lane marking lines for a traffic lane in which the vehicle is traveling, and the identified lane marking lines are used to identify vanishing points within the images. The locations of the identified vanishing points can be averaged in order to identify an average position of the vanishing points within the images. The average position of the vanishing points in the images is used, along with one or more intrinsic calibration parameters of the imaging sensor (such as its focal lengths and sensor centers), to identify one or more extrinsic calibration parameters of the imaging sensor (such as its pitch and yaw angles).

The one or more extrinsic calibration parameters of the imaging sensor that are identified as described above can then be used in any suitable manner. For example, the one or more extrinsic calibration parameters of the imaging sensor can be stored and used to identify distances to lane marking lines, other vehicles, or other objects during travel. The identified distances may be used to trigger audible or other warnings, such as a warning that the vehicle is tailgating another vehicle, has departed from its current traffic lane without signaling, or is about to impact another vehicle or other object. The identified distances to lane marking lines, other vehicles, or other objects may also or alternatively be used to generate an estimated measure of the attentiveness or inattentiveness of the vehicle's driver or to identify specific instances of driver inattentiveness. For instance, the driver may be determined to be less attentive when the vehicle departs from its current traffic lane without signaling more often.

These techniques for performing highly-accurate and self-adjusting imaging sensor auto-calibration for an in-vehicle ADAS system or other system can operate under the assumption that lane marking lines for a traffic lane in which a vehicle is traveling will appear generally parallel to one another (at least to within some desired threshold). This can be particularly true when the vehicle is traveling at a relatively fast speed. Given this, it is possible for certain images captured by an imaging sensor on the vehicle to be processed assuming that the lane marking lines are parallel to one another. This allows the vanishing point defined by the identified lane marking lines in each image to be determined, and the vanishing points can be averaged and used in various calculations to identify the one or more extrinsic calibration parameters for the imaging sensor.

In this way, the disclosed techniques allow for effective calibration of an imaging sensor used in a vehicle's ADAS system or other system. The calibration can be performed easily and without requiring the manual placement of a banner or other object in front of the imaging sensor. The calibration can therefore be performed much faster, at lower cost, and with higher accuracy. In some cases, for instance, these techniques may allow the orientation of an imaging sensor on a vehicle to be estimated to within one tenth of a degree or less in terms of both pitch angle and yaw angle. Moreover, since the orientation of the imaging sensor is used to estimate distances to lane marking lines, other vehicles, or other objects, these techniques allow for more accurate distances to be obtained. This can help to increase the reliability of the distance measurements themselves and to increase the reliability of tailgating, lane departure, forward collision, or other warnings generated based on the distances. This can also or alternatively help to increase the accuracy of estimated measurements regarding the attentiveness or inattentiveness of the vehicle's driver (as based on the distances). In addition, these techniques can be repeated as needed or desired, such as daily, periodically, intermittently, or at any other suitable times. This may allow, for example, one or more updated extrinsic calibration parameters associated with the imaging sensor to be obtained and used over time.

FIG.1illustrates an example system100supporting highly-accurate and self-adjusting imaging sensor auto-calibration for an in-vehicle ADAS system or other system according to this disclosure. As shown inFIG.1, the system100includes or is used in conjunction with a vehicle102. In this particular example, the vehicle102represents a tractor-trailer truck, which includes a tractor having a cab in which a driver sits and a trailer pulled by the tractor. Note, however, that this type of vehicle102is for illustration only and that the system100may be used with any other suitable type of vehicle. Other example types of vehicles that can be used here may include passenger sedans, pickup trucks, sport utility vehicles, passenger vans, box trucks, and buses.

The vehicle102has an imaging system104, which in this example may be mounted on the interior surface of a front windshield of the vehicle102. Note, however, that the actual position of the imaging system104can vary as needed or desired. The imaging system104includes one or more cameras or other imaging sensors106that are used to capture images or other image-related data associated with the vehicle102. For example, the imaging system104may include one or more forward-facing imaging sensors that are used to capture images of scenes in front of the vehicle102, such as images of the road or other surface in front of the vehicle102. These images may capture lane marking lines that identify the current traffic lane in which the vehicle102is traveling and one or more other traffic lanes supporting traffic flow in the same direction and/or in different direction(s). The images may also capture one or more other vehicles traveling in the same direction as the vehicle102and/or in other direction(s). The images may further capture other content, such as pedestrians, light poles, buildings, or background (such as the ground, sky, hills, mountains, etc.). In some cases, the imaging system104may also include at least one driver-facing imaging sensor, which may be used to capture images of the driver of the vehicle102. These images may be used, for instance, to determine if the driver is looking at his or her mobile phone, is drowsy, or otherwise might be inattentive. However, the use of any driver-facing imaging sensor is optional here.

The vehicle102also includes at least one processing device108, which can process one or more types of information in the vehicle102and perform one or more operations (where the specific information and operations can vary depending on the specific implementation). In this example, the processing device108can receive images from the imaging sensor(s)106of the imaging system104and process the images. For instance, the processing device108can analyze the images captured by the forward-facing imaging sensor(s)106of the imaging system104in order to detect lane marking lines, other vehicles, or other objects near the vehicle102. The processing device108can use this information to estimate distances to the lane marking lines, other vehicles, or other objects near the vehicle102, and the estimated distances can be used in any suitable manner. The processing device108can also support the techniques described below to identify one or more extrinsic calibration parameters associated with the forward-facing imaging sensor(s)106, which helps in the calibration of the forward-facing imaging sensor(s)106. Optionally, the processing device108can further process images captured by the driver-facing imaging sensor(s)106, such as to identify any indicators or instances of the vehicle's driver becoming drowsy or otherwise being inattentive. The processing device108includes any suitable number(s) and type(s) of processors or other processing devices in any suitable arrangement. Example types of processing devices108include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuitry.

The processing device108here is coupled to at least one memory110, which can store any suitable instructions and data used, generated, or collected by the processing device108. The memory110represents any suitable structure(s) capable of storing and facilitating retrieval of information, such as data, program code, and/or other suitable information on a temporary or permanent basis. For example, the memory110may represent at least one random access memory, read only memory, hard drive, Flash memory, optical disc, or any other suitable volatile or non-volatile storage device(s).

In this example, the processing device108is coupled to or can interact with one or more indicators112, which may represent at least one audible, visual, tactile, or other indicator of the vehicle102. In response to identifying that a specified condition exists, the processing device108may trigger at least one indicator112in order to notify the driver of the vehicle102of the specified condition. For example, if the processing device108detects that the vehicle102is crossing a lane marking line while a turn signal indicator (blinker) of the vehicle102is not activated, the processing device108may trigger an indicator112informing the driver of the lane departure. If the processing device108detects that the vehicle102is approaching another vehicle or other object at a high rate of speed, the processing device108may trigger an indicator112informing the driver of the potential collision. Note that the specified conditions sensed by the processing device108can vary and that the type(s) of indicator(s)112triggered by the processing device108can vary based on a number of factors.

The processing device108here can also communicate via at least one communication interface114. The communication interface114may allow, for example, the processing device108to communicate with other components of the vehicle102itself, such as sensors, internal communication buses, engine components, or other components of the vehicle102. The communication interface114may also or alternatively allow the processing device108to communicate with one or more external components outside the vehicle102, such as one or more databases or analysis systems that store or process information from the vehicle102. As a particular example, the processing device108may identify instances that are indicative of inattentiveness of the driver of the vehicle102, and the processing device108may communicate information identifying or related to those instances of inattentiveness (such as timestamps, vehicle actions, driver actions, etc.) to an external system for processing or storage.

AlthoughFIG.1illustrates one example of a system100supporting highly-accurate and self-adjusting imaging sensor auto-calibration for an in-vehicle ADAS system or other system, various changes may be made toFIG.1. For example, various components shown inFIG.1may be combined, further subdivided, replicated, omitted, or rearranged and additional components may be added according to particular needs. Also, as noted above, the system100may involve the use of any other suitable type of vehicle102. In addition, the use of the driver-facing imaging sensor(s)106and the identification of driver inattentiveness are optional here.

FIG.2illustrates an example arrangement of imaging sensors106for use in an in-vehicle ADAS system or other system according to this disclosure. In this example, the imaging system104for the vehicle102is shown as being mounted to the interior surface of the front windshield of the vehicle102, although the actual position of the imaging system104can vary from the position shown here. Note that any suitable mechanism may be used here to mount the imaging system104to the front windshield of the vehicle102. Also note that any suitable mechanism may be used to facilitate communications between the imaging system104and the processing device108or other component(s), such as physical or wireless communications.

In this particular example, the imaging system104includes at least one forward-facing imaging sensor106aand at least one driver-facing imaging sensor106b. The at least one forward-facing imaging sensor106acan be used to capture images or other image-related data associated with scenes in front of the vehicle102. In the example shown inFIG.2, for instance, the at least one forward-facing imaging sensor106amay capture images of the traffic lane occupied by the vehicle102and possibly one or more adjacent traffic lanes. Here, the images may include lane marking lines202associated with the traffic lane(s) and other vehicles204. In some embodiments, the imaging system104can be installed such that images captured by the forward-facing imaging sensor106acontain a generally horizontal horizon and primarily capture the road or other surface ahead of the vehicle102while containing some lesser amount of sky, mountains, or other background. In particular embodiments, for example, the forward-facing imaging sensor106amay capture images in which the upper quarter to upper third of the images contain sky, mountains, or other background. However, this is for illustration only, and any individual image can vary based on (among other things) the slope of the road or other surface on which the vehicle102is traveling.

The at least one driver-facing imaging sensor106bis optional in the vehicle102and may be used to capture images or other image-related data associated with a driver206of the vehicle102. The processing device108may process the images or other data from the at least one driver-facing imaging sensor106bin order to estimate a measure of the attentiveness or inattentiveness of the driver206. For example, the processing device108may support the use of the SMARTSENSE FOR INATTENTIVE DRIVING technology from OMNITRACS, LLC in order to detect and quantify driver inattentiveness.

As described above, the actual location of the forward-facing imaging sensor106aon the vehicle102and the orientation of the forward-facing imaging sensor106aon the vehicle102(such as its pitch and yaw angles) impact how distances to objects like lane marking lines202and other vehicles204from the vehicle102are calculated. Inaccurate extrinsic calibration parameters associated with the forward-facing imaging sensor106amay therefore lead to inaccurate distance estimations to objects like the lane marking lines202and the other vehicles204. This may lead to false triggers of one or more indicators112, inaccurate measurements of the attentiveness of the driver206, or other issues. The processing device108can therefore use the calibration process described below to help identify extrinsic calibration parameters associated with the forward-facing imaging sensor106a. The extrinsic calibration parameters can include or be based on the pitch angle and the yaw angle of the forward-facing imaging sensor106a, which can be used in other calculations (such as to identify the distances to objects like the lane marking lines202and the other vehicles204). The calibration process described below can be extremely accurate, such as accurate to within one tenth of a degree or less for both the pitch angle and the yaw angle of the forward-facing imaging sensor106a. Moreover, the calibration process can be repeated as needed or desired.

AlthoughFIG.2illustrates one example of an arrangement of imaging sensors106a-106bfor use in an in-vehicle ADAS system or other system, various changes may be made toFIG.2. For example, the actual position(s) of the imaging sensors106a-106bin the vehicle102may vary from the positions that are shown here. Also, the imaging sensors106a-106bmay have any other suitable form factor. In addition, while both forward-facing and driver-facing imaging sensors106a-106bare shown here, the driver-facing imaging sensor(s)106bmay be omitted if desired.

FIGS.3A and3Billustrate example coordinate systems associated with in an in-vehicle ADAS system or other system according to this disclosure. As shown inFIG.3A, an image300of a scene is shown and may represent an image captured using the forward-facing imaging sensor(s)106aof the vehicle102. The image300here is associated with a coordinate system302, which includes a horizontal axis (denoted the u axis) and a vertical axis (denoted the v axis). In this example, the coordinate system302has an origin at the upper left corner of the image300, although this is for illustration only. Each pixel within the image300may therefore be defined as having a discrete position using an integer-valued tuple (u, v).

The vehicle102itself is associated with a coordinate system304, which includes a horizontal axis (denoted the X axis) extending across the width of the vehicle102, a vertical axis (denoted the Y axis) extending across the height of the vehicle102, and a horizontal axis (denoted the Z axis) extending along the length of the vehicle102. Here, coordinates of lane marking lines, other vehicles, or other objects around the vehicle102may be expressed using a continuous-valued tuple (X, Y, Z) (where X, Y, Z E Each of the (X, Y, Z) tuples corresponds to a location in three-dimensional (3D) world coordinates, where the origin of the coordinate system304is located at the center of the vehicle's front bumper at ground level. The orientation of the coordinate system304is such that the X-Z plane lies in a plane of the (assumed flat) road.

The forward-facing imaging sensor106aof the vehicle102has a similar coordinate system306with x, y, and z axes. In the coordinate system306, coordinates of lane marking lines, other vehicles, or other objects around the vehicle102may be expressed using a continuous-valued tuple (x, y, z) (where x, y, z∈R). Each of the (x, y, z) tuples corresponds to a location in 3D world coordinates, where the origin of the coordinate system306is located at the center of the forward-facing imaging sensor106a(which may be considered to function as a pinhole camera). The x axis of the coordinate system306extends in the same direction as the u axis of the coordinate system302, and they axis of the coordinate system306extends in the same direction as the −v axis of the coordinate system302. It should be noted here that the forward-facing imaging sensor106amay not actually represent a pinhole camera since it can include a lens that focuses light onto a sensor. However, treating the forward-facing imaging sensor106aas if it is a pinhole camera can be justified since the distance between the lens and the sensor in the forward-facing imaging sensor106ais much smaller than the distance between the lens and an object far away from the forward-facing imaging sensor106abeing viewed (this is called “pinhole approximation”).

In general, effectively determining the distance between the vehicle102and lane marking lines, other vehicles, or other objects depends on an accurate transformation between the coordinate system306and the coordinate system304. However, there may be both translational and rotational differences between the coordinate systems304and306themselves. Even if attempts are made to ensure that the forward-facing imaging sensor106ais installed on the vehicle102with as little rotational offset as possible between its coordinate system306and the vehicle's coordinate system304, some rotational differences typically still exist after installation of the forward-facing imaging sensor106a. Accurate knowledge of how the coordinate system306of the forward-facing imaging sensor106ais offset in terms of both translation distances and rotational angles relative to the coordinate system304of the vehicle102may be necessary or desirable to calculate distances from the vehicle102to other objects or to perform other functions. The calibration process described below helps to identify the extrinsic calibration parameters needed for transformations between the coordinate systems304and306.

In the discussion below, the following nomenclature is used. The terms Tx, Ty, and Tz, refer to translational distances from the origin of the coordinate system306to the origin of the coordinate system304along the x, y, and z axes. The terms Rx, Ry, and Rzrefer to rotations about the x, y, and z axes needed to transform coordinates from the coordinate system306to the coordinate system304. The calibration process described below can be used to identify at least values for Rx(which identifies the pitch of the forward-facing imaging sensor106a) and Ry(which identifies the yaw of the forward-facing imaging sensor106a). Distance estimates involving the vehicle102may be most sensitive to these two extrinsic calibration parameters, so more accurate estimates of the Rxand Ryvalues may have the largest impact on the accuracy of the distance measurements. Note that, in some cases, it may be assumed that Rz=0, which can simplify distance calculations while having minimal impact on the accuracy of the distance calculations. This is because rotating the forward-facing imaging sensor106aabout its z axis can have far less impact on the distance calculations compared to rotating the forward-facing imaging sensor106aabout its x and y axes. However, in other cases, the value of Rzmay be determined and used during distance calculations.

As described below, the calibration process can use manual or other measurements identifying distances of the forward-facing imaging sensor106afrom one or more specified points or axes of the vehicle102. These measurements can be used to define the Tx, Ty, and Tz, translational distances between the origins of the coordinate systems304and306. Note that these measurements may or may not have high accuracy. In some cases, for example, the calibration process may operate effectively even when the translational distances Tx, Ty, and Tz, have an accuracy of one or several inches. In some cases, the Tytranslational distance may only need to be accurate to within about six inches. The calibration process can also process images captured by the forward-facing imaging sensor106awhile the vehicle102is being driven. The processing device108may analyze the images as described below to identify images in which multiple lane marking lines can be identified and used to calculate the locations of vanishing points in the images. The locations of the vanishing points can be averaged, and the average position of the vanishing points in the images can be used to calculate the Rxand Ryrotational values. The extrinsic calibration parameters of the forward-facing imaging sensor106a(such as Tx, Ty, Tz, Rx, Ry, and Rz) can then be used in various other computations, such as to estimate distances from the vehicle102to lane marking lines, other vehicles, or other objects in 3D space based on two-dimensional (2D) images captured by the forward-facing imaging sensor106a.

AlthoughFIGS.3A and3Billustrate examples of coordinate systems302,304, and306associated with in an in-vehicle ADAS system or other system, various changes may be made toFIGS.3A and3B. For example, the forward-facing imaging sensor106amay be positioned elsewhere on the vehicle102, so there may be different translational distances between the origins of the coordinate systems304and306. Also, the coordinate system306of the forward-facing imaging sensor106acan vary based on the orientation of the forward-facing imaging sensor106aon the vehicle102, so there may be different rotational angles between the coordinate systems304and306.

FIGS.4A and4Billustrate an example method400for performing highly-accurate and self-adjusting imaging sensor auto-calibration for an in-vehicle ADAS system or other system according to this disclosure. For ease of explanation, the method400is described as being performed using the forward-facing imaging sensor106aand the processing device108in the vehicle102ofFIG.1. However, the method400may be performed using any other suitable device(s) and in any other suitable vehicle(s).

As shown inFIGS.4A and4B, measurements identifying a location of a forward-facing imaging sensor on a vehicle are obtained at step402. This may include, for example, the processing device108receiving manual or other measurements identifying the position of the forward-facing imaging sensor106aon the vehicle102relative to one or more specified points or axes of the vehicle102. In some cases, for instance, the measurements may include a measure of the horizontal distance from the front of the vehicle's bumper to the location of the forward-facing imaging sensor106a(which represents the Tz, translational distance), a measure of the vertical distance from the ground to the location of the forward-facing imaging sensor106a(which represents the Tytranslational distance), and a measure of the horizontal distance from a centerline of the vehicle102to the location of the forward-facing imaging sensor106a(which represents the Txtranslational distance). Note that other measurements of the vehicle102itself might also be obtained here, such as a measure of the total horizontal width of the vehicle102(like a measure of outside bumper-to-outside bumper distance or a measure of outside tire-to-outside tire distance of the vehicle102).

Images of the road or other surface in front of the vehicle are obtained during travel of the vehicle at step404. This may include, for example, the processing device108obtaining multiple images captured by the forward-facing imaging sensor106aduring a time period when the vehicle102is being driven. Depending on the implementation, this may involve the collection of a large number of images, such as one thousand images or more. In some cases, the vehicle102may be driven at a relatively high rate of speed during the image capture, and the road on which the vehicle102is being driven may be relatively straight. In particular embodiments, for instance, the vehicle102may be driven at a speed of at least about 45 miles per hour (about 72.4 kilometers per hour). However, the vehicle102may be driven at faster or slower speeds as needed or desired. Also, in some cases, the forward-facing imaging sensor106amay capture the images in rapid succession, such as when the images are captured by the forward-facing imaging sensor106aat a rate of fifteen frames per second or some other rate. A determination is made whether the obtained images are suitable for further processing at step406. This may include, for example, the processing device108determining whether at least a threshold number of images that contain two lane marking lines and that have a suitable image quality have been obtained. If not, the process returns to step404to obtain additional images.

Otherwise, for each of at least some of the obtained images, pixel locations of lane marking lines in the image are identified at step408. This may include, for example, the processing device108performing object recognition or other image processing operations to identify lane marking lines in the obtained images. The pixel locations of the lane marking lines may be saved in memory (such as the memory110) for further processing. A determination is made whether enough data has been collected at step410. This may include, for example, the processing device108determining whether at least a threshold number of images with identifiable lane marking lines have been identified. If not, the process returns to step404to obtain and process additional images. In some cases, the lane marking lines that are identified in the images may each have an associated confidence level, where the confidence level identifies the strength of (or the level of confidence in) the identified pixel locations of that lane marking line. In those cases, step410may involve determining whether at least a threshold number of images with identifiable lane marking lines having a minimum confidence level (such as at least an 80% confidence) have been identified.

For at least some of the images in which the pixel locations of the lane marking lines are identified, the pixel locations of the lane marking lines are fit to multi-order polynomials within pixel space at step412. This may include, for example, the processing device108using a curve-fitting algorithm or other logic to fit a second-order polynomial or other multi-order polynomial to the pixel locations of each lane marking line in each image being processed. Each multi-order polynomial identifies the approximate path of the associated lane marking line in the image containing that lane marking line. Images having multi-order polynomials with less than a threshold amount of curvature are selected at step414. This may include, for example, the processing device108excluding images having lane marking lines with multi-order polynomials that diverge from straight lines by more than a specified amount or percentage. For those selected images, the pixel locations of the lane marking lines are fit to first-order polynomials (straight lines) within the pixel space at step416. This may include, for example, the processing device108using a curve-fitting algorithm or other logic to fit a first-order polynomial to the pixel locations of each lane marking line in each selected image. Each first-order polynomial identifies the approximate path of the associated lane marking line in the selected image containing that lane marking line.

Vanishing points in the selected images are identified based on the first-order polynomials for the lane marking lines in those selected images at step418. This may include, for example, the processing device108identifying the vanishing point in each selected image as the pixel where the first-order polynomials for the lane marking lines in that selected image intersect. One example of this is shown inFIG.5, which illustrates example processing results for an image500captured by an imaging sensor106ain an in-vehicle ADAS system or other system according to this disclosure. As shown inFIG.5, the image500captures a scene with two lane marking lines502and504. The processing device108has identified first-order polynomials506and508for these lane marking lines502and504. The processing device108can then determine a vanishing point510as the pixel at which the first-order polynomials506and508intersect one another in the image500.

The locations of the vanishing points determined for the selected images are averaged. In this example, the averaging is performed as a two-step process. First, the locations of the vanishing points determined for all of the selected images are averaged to produce a first average at step420. This may include, for example, the processing device108averaging the (u, v) coordinates of the identified vanishing points in the selected images to identify a first average of the (u, v) coordinates. The first average may be said to represent a rough estimate of the average position of the vanishing points in the selected images. Second, the locations of the vanishing points identified as being within some threshold range of the first average are averaged to produce a second average at step422. This may include, for example, the processing device108averaging the (u, v) coordinates of the identified vanishing points in the selected images that are within a twenty-pixel circle (or other area) around the first average to identify a second average of these (u, v) coordinates. This average may represent a more precise estimate of the average position of the vanishing points in the selected images. This type of averaging may be done to exclude outliers that are not within a specified distance of the rough estimate of the average vanishing point. Note, however, that an average vanishing point may be determined in any other suitable manner.

One or more extrinsic calibration parameters associated with the forward-facing imaging sensor are identified based on the average position of the vanishing points at step424. This may include, for example, the processing device108using the average position of the vanishing points and one or more intrinsic calibration parameters of the forward-facing imaging sensor106a(such as its focal lengths and sensor centers) to calculate pitch and yaw angles of the forward-facing imaging sensor106a. Example calculations that may be used to identify the pitch and yaw angles of the forward-facing imaging sensor106aare described below. However, the one or more extrinsic calibration parameters of the forward-facing imaging sensor106amay be determined in any other suitable manner. Note that the intrinsic calibration parameters of the forward-facing imaging sensor106ahere may represent universal constants.

The one or more extrinsic calibration parameters of the forward-facing imaging sensor are stored, output, or used in some manner at step426. This may include, for example, the processing device108receiving additional images from the forward-facing imaging sensor106aduring travel of the vehicle102and using the pitch and yaw angles of the forward-facing imaging sensor106ato identify distances to lane marking lines, other vehicles, or other objects around the vehicle102. This may also include the processing device108triggering one or more indicators112in response to one or more detected conditions, such as when the processing device108detects that the vehicle102is crossing a lane marking line while a turn signal indicator of the vehicle102is not activated or that the vehicle102may impact another vehicle or other object. This may further include the processing device108identifying an instance of driver inattentiveness in response to detecting that the vehicle102is crossing a lane marking line while the turn signal indicator of the vehicle102is not activated or in response to detecting some other condition associated with driver attentiveness.

A determination is made whether to repeat this process at step428. If so, the process returns to step404to collect additional images for processing. Note that the process may instead return to a different step, such as when the process returns to step402in order to obtain more accurate measurements of the location of the forward-facing imaging sensor106aon the vehicle102. The process can be repeated based on any suitable criteria, such as in response to the passage of a specified amount of time or in response to excessive false triggering of one or more indicators112.

AlthoughFIGS.4A and4Billustrate one example of a method400for performing highly-accurate and self-adjusting imaging sensor auto-calibration for an in-vehicle ADAS system or other system, various changes may be made toFIGS.4A and4B. For example, while shown as a series of steps, various steps inFIGS.4A and4Bmay overlap, occur in parallel, occur in a different order, or occur any number of times. AlthoughFIG.5illustrates examples of processing results for an image500captured by an imaging sensor106ain an in-vehicle ADAS system or other system, various changes may be made toFIG.5. For instance, images of scenes in front of the vehicle102may vary widely, and the contents of the images that are obtained and processed may similarly vary widely.

Note that various functions described above as being performed in or by the vehicle102may be implemented in any suitable manner in the system100. For example, in some embodiments, various functions described above as being performed in or by the vehicle102may be implemented or supported using one or more software applications or other software/firmware instructions that are executed by at least one processor or other processing device. In other embodiments, at least some of the functions described above as being performed in or by the vehicle102can be implemented or supported using dedicated hardware components. In general, the functions described above as being performed in or by the vehicle102may be performed using any suitable hardware or any suitable combination of hardware and software/firmware instructions. Any suitable hardware may also be used here, such as one or more microprocessors, microcontrollers, DSPs, ASICs, FPGAs, or discrete circuitry.

As described above, once the pitch and yaw angles (Rxand Ryvalues) associated with the forward-facing imaging sensor106aare identified, these values can be used to perform transformations between different coordinate systems. For example, the pitch and yaw angles can be used to process image data expressed within the coordinate system302(which is captured using the forward-facing imaging sensor106ahaving the coordinate system306) and identify distances or other measurements within the vehicle's coordinate system304. As noted above with reference toFIGS.3A and3B, the extrinsic calibration parameters that are identified for the forward-facing imaging sensor106ainclude three translation distances (Tx, Ty, and Tz) and three rotation angles (Rx, Ry, and Rz). These variables parameterize the transformation between the (x, y, z) coordinate system306of the forward-facing imaging sensor106aand the (X, Y, Z) coordinate system304of the vehicle102. In some embodiments, the relationship between these coordinate systems304and306may be expressed using homogeneous coordinates as follows.

(xyz)=(100Tx010Ty001Tz)⁢(10000cos⁢Rxsin⁢Rx00-sin⁢Rxcos⁢Rx00001)⁢(cos⁢Ry0-sin⁢Ry00100sin⁢Ry0cos⁢Ry00001)⁢(XYZ1)(1)
Note that the rotations here are passive transformations and that it is explicitly assumed Rz=0 (meaning the roll of the forward-facing imaging sensor106ais zero), which as noted above can have a minimal effect on estimating distances to objects.

The intrinsic calibration parameters for the forward-facing imaging sensor106amay include four parameters that describe the geometry of a camera with a mostly-spherical lens. More specifically, the intrinsic calibration parameters for the forward-facing imaging sensor106amay include two focal lengths (fxand fy) and two focal centers (cxand cy). Using these parameters, it is possible to transform image data between an image's coordinate system302and the (x, y, z) coordinate system306. For example, projective geometry (which involves a transformation called a “homography” in computer vision) may be used and can be expressed as follows.

(-z)⁢(uv1)=(fx0cx0fycy001)⁢(1000-1000-1)⁢(xyz)(2)
In Equation (2), the tuple (x, y, z) represents the same tuple used in Equation (1).

Within the coordinate system304, the location of an infinite distance ahead in (X, Y, Z) coordinates can be defined by taking any point (X, Y, Z) and determining the limit as Z→∞. At this vanishing point, the homography has a very simple expression, which can be defined as follows.
u*=cx+fxsec(Ry)tan(Ry)  (3)
v*=cy+fysec(Ry)sin(Rx)  (4)
Here, the position (u*, v*) represents the location of the vanishing point in an image. Note that there is no dependence on the Tx, Ty, Tz, X, and Y values here in the expressions for u* and v*. Equations (3) and (4) can be inverted and solved analytically as follows.

Rx=arc⁢sin[βfy⁢fx⁢γ2⁢α2](5)Ry=arctan[2⁢fx⁢γα2,-γα](6)
In Equations (5) and (6):
α=u*−cx(7)
β=v*−cy(8)
γ=−fx+√{square root over (fx2+4α2)}  (9)
Note that the values of Rxand Ryfrom these equations will be in radians (not degrees) but can easily be converted. Also note that the arctan function here has the form arctan(x, y), which identifies the angle between the positive x axis and a point (x, y) in Cartesian coordinates.

It is therefore possible, using the equations above, to identify the Rxand Ryextrinsic calibration parameters of the forward-facing imaging sensor106abased on (i) the intrinsic calibration parameters of the forward-facing imaging sensor106aand (ii) the (u*, v*) location of the vanishing point. Thus, the calibration process described above may be used when a vehicle102is traveling, such as on a generally straight road, where the lane marking lines for the road are generally parallel and point straight ahead. When capturing an image of such lane marking lines, it is possible to extrapolate these lane marking lines to an infinite distance ahead, where the lane marking lines intersect at the vanishing point of the image. The location of the vanishing point in the image can be used to solve for Rxand Rygiven prior knowledge of fx, fy, cx, and cy(and assuming Rz=0).

In some cases, the following operations can be performed as part of the calibration process described above with respect toFIGS.4A and4B. Images showing two (high confidence) lane marking lines can be obtained, which ideally may include images showing minimal road curvature and with the lane marking lines extending out in front of the vehicle. These lane marking lines can be identified, and polynomials can be identified for at least some of the lane marking lines. For each of at least some of the images, the intersection of the polynomials representing the lane marking lines in that image is identified as the vanishing point (u*, v*) for that image. The locations of the vanishing points in multiple images can be averaged (such as by using the two-step approach described above or another suitable approach), and the average position of the vanishing point can be used in Equations (5) and (6) above. This allows for the identification of the extrinsic calibration parameters Rxand Ryfor the forward-facing imaging sensor106a. At that point, the extrinsic calibration parameters can be stored and used during normal operation of the system100.

As described above, the identification of the one or more extrinsic calibration parameters of the imaging sensor106acan be repeated periodically or at any other suitable times. For example, during the method400, the measurements identifying the location of the imaging sensor106aon the vehicle102may be obtained at step402. Assuming the imaging sensor106ais fixed in its location on the vehicle102(such as on the windshield of the vehicle102), there may be no need to repeat these measurements during subsequent recalibrations of the imaging sensor106a. As a result, the one or more extrinsic calibration parameters of the imaging sensor106acan be updated over time by capturing images in front of the vehicle102and determining updated Rxand Ryvalues based on those images. This might be repeated daily, for instance, in order to keep the overall system accurate in case the imaging sensor106amoves slightly. Movement of the imaging sensor106amight be possible, for example, due to shifting of the adhesive contacting the windshield of the vehicle102or shifting of the imaging sensor106a. However, if the windshield of the vehicle102is replaced, the imaging sensor106awould typically be reinstalled on the new windshield. In that case, the measurements at step402may be repeated to identify the current location of the imaging sensor106aon the vehicle102, and additional images can be captured and used to determine updated Rxand Ryvalues based on those images.

In some embodiments, various functions described in this patent document are implemented or supported by a computer program that is formed from computer readable program code and that is embodied in a computer readable medium. The phrase “computer readable program code” includes any type of computer code, including source code, object code, and executable code. The phrase “computer readable medium” includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or any other type of memory. A “non-transitory” computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. A non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable storage device.

It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms “application” and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer code (including source code, object code, or executable code). The term “communicate,” as well as derivatives thereof, encompasses both direct and indirect communication. The terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation. The term “or” is inclusive, meaning and/or. The phrase “associated with,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, have a relationship to or with, or the like. The phrase “at least one of,” when used with a list of items, means that different combinations of one or more of the listed items may be used, and only one item in the list may be needed. For example, “at least one of: A, B, and C” includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.

The description in the present application should not be read as implying that any particular element, step, or function is an essential or critical element that must be included in the claim scope. The scope of patented subject matter is defined only by the allowed claims. Moreover, none of the claims invokes 35 U.S.C. § 112(f) with respect to any of the appended claims or claim elements unless the exact words “means for” or “step for” are explicitly used in the particular claim, followed by a participle phrase identifying a function. Use of terms such as (but not limited to) “mechanism,” “module,” “device,” “unit,” “component,” “element,” “member,” “apparatus,” “machine,” “system,” “processor,” or “controller” within a claim is understood and intended to refer to structures known to those skilled in the relevant art, as further modified or enhanced by the features of the claims themselves, and is not intended to invoke 35 U.S.C. § 112(f).

While this disclosure has described certain embodiments and generally associated methods, alterations and permutations of these embodiments and methods will be apparent to those skilled in the art. Accordingly, the above description of example embodiments does not define or constrain this disclosure. Other changes, substitutions, and alterations are also possible without departing from the spirit and scope of this disclosure, as defined by the following claims.