SYSTEMS AND METHODS FOR CALIBRATING IMAGE SENSORS OF A VEHICLE

A method includes obtaining infrastructure data from one or more infrastructure sensors, determining an infrastructure-based positional characteristic of the vehicle based on the infrastructure data, obtaining onboard image data from the onboard image sensor, determining an image-based positional characteristic of the vehicle based on the onboard image data and a digital twin of the manufacturing environment, generating an offset matrix based on the infrastructure-based positional characteristic and the image-based positional characteristic, and selectively adjusting a rotation matrix of the onboard image sensor based on the offset matrix and one or more additional offset matrices.

FIELD

The present disclosure relates to systems and methods for calibrating image sensors of a vehicle, such as a vehicle camera.

BACKGROUND

A vehicle manufacturing environment may include one or more end-of-line (EOL) testing stations that are configured to calibrate and verify the functionality of various components of a vehicle. As an example and after the vehicle is assembled, a camera calibration EOL station may calibrate a forward-facing camera that is employed during an automatic emergency braking routine performed by the vehicle. To calibrate the forward-facing camera, the camera calibration EOL station may include a fiducial marker (e.g., a checkerboard) that is placed at a known position in front of the vehicle, and a control module of the vehicle may calculate a distance/pose relative to the fiducial marker to thereby define an offset roll value, an offset pitch value, and an offset yaw value of a rotation matrix of the camera.

However, the camera calibration EOL station inhibits the efficiency of the vehicle inspection/verification process due to the increased labor, time, and infrastructure resources needed to perform the calibration. Additionally, the camera calibration EOL station may not account for dynamic conditions inhibiting the accuracy of the calibration routine, such as lighting, off-axis maneuvers, signal noise, and other factors that inhibit the accuracy of the offset roll, pitch, and yaw determinations. These issues with camera calibration EOL stations, among other issues, are addressed by the present disclosure.

SUMMARY

The present disclosure provides a method for calibrating an onboard image sensor of a vehicle in a manufacturing environment. The method includes obtaining infrastructure data from one or more infrastructure sensors, determining an infrastructure-based positional characteristic of the vehicle based on the infrastructure data, obtaining onboard image data from the onboard image sensor, determining an image-based positional characteristic of the vehicle based on the onboard image data and a digital twin of the manufacturing environment, generating an offset matrix based on the infrastructure-based positional characteristic and the image-based positional characteristic, and selectively adjusting a rotation matrix of the onboard image sensor based on the offset matrix and one or more additional offset matrices.

The following paragraph includes variations of the method of the above paragraph, and the variations may be implemented individually or in any combination.

In one form, selectively adjusting the rotation matrix based on the offset matrix and the one or more additional offset matrices further comprises performing a recursive updating routine based on the rotation matrix based on the offset matrix and the one or more additional offset matrices; the recursive updating routine is a recursive least-squares estimation routine; the recursive updating routine is one of an exponential weighted moving average routine, a gradient descent routine, and a root-finding routine; the rotation matrix defines an offset roll value of the onboard image sensor, an offset pitch value of the onboard image sensor, and an offset yaw value of the onboard image sensor; the infrastructure-based positional characteristic of the vehicle includes an infrastructure-based location of the vehicle, an infrastructure-based pose of the vehicle, or a combination thereof; the image-based positional characteristic of the vehicle includes an image-based location of the vehicle, an image-based pose of the vehicle, or a combination thereof; the method further includes determining whether a calibration condition is satisfied based on the infrastructure-based positional characteristic, the image-based positional characteristic, or a combination thereof, and generating the offset matrix in response to the calibration condition being satisfied; the calibration condition is satisfied in response to a difference between the infrastructure-based positional characteristic and the image-based positional characteristic being less than a threshold difference; the calibration condition is not satisfied in response to the infrastructure-based positional characteristic being associated with an undetectable area of the manufacturing environment; the calibration condition is not satisfied in response to one of the infrastructure-based positional characteristic and the image-based positional characteristic being associated with a predefined noise area of the manufacturing environment; and/or the one or more additional offset matrices are based on one or more additional infrastructure-based positional characteristics and one or more additional image-based positional characteristics.

The present disclosure provides a system for calibrating an onboard image sensor of a vehicle in a manufacturing environment. The system includes one or more processors and one or more nontransitory computer-readable mediums comprising instructions that are executable by the one or more processors. The instructions include obtaining infrastructure data from one or more infrastructure sensors, determining an infrastructure-based positional characteristic of the vehicle based on the infrastructure data, obtaining onboard image data from the onboard image sensor, determining an image-based positional characteristic of the vehicle based on the onboard image data and a digital twin of the manufacturing environment, determining whether a calibration condition is satisfied based on the infrastructure-based positional characteristic, the image-based positional characteristic, or a combination thereof, and in response to the calibration condition being satisfied: generating an offset matrix based on the infrastructure-based positional characteristic and the image-based positional characteristic, and selectively adjusting a rotation matrix of the onboard image sensor based on the offset matrix and one or more additional offset matrices.

The following paragraph includes variations of the system of the above paragraph, and the variations may be implemented individually or in any combination.

In one form, selectively adjusting the rotation matrix based on the offset matrix and the one or more additional offset matrices further comprises performing a recursive least-squares estimation routine based on the rotation matrix based on the offset matrix and the one or more additional offset matrices; the rotation matrix defines an offset roll value of the onboard image sensor, an offset pitch value of the onboard image sensor, and an offset yaw value of the onboard image sensor; the infrastructure-based positional characteristic of the vehicle includes an infrastructure-based location of the vehicle, an infrastructure-based pose of the vehicle, or a combination thereof, and the image-based positional characteristic of the vehicle includes an image-based location of the vehicle, an image-based pose of the vehicle, or a combination thereof; the calibration condition is satisfied in response to a difference between the infrastructure-based positional characteristic and the image-based positional characteristic being less than a threshold difference; the calibration condition is not satisfied in response to the infrastructure-based positional characteristic being associated with an undetectable area of the manufacturing environment; the calibration condition is not satisfied in response to one of the infrastructure-based positional characteristic and the image-based positional characteristic being associated with a predefined noise area of the manufacturing environment.

The present disclosure provides another method for calibrating an onboard image sensor of a vehicle in a manufacturing environment. The method includes obtaining infrastructure data from one or more infrastructure sensors, determining an infrastructure-based positional characteristic of the vehicle based on the infrastructure data, obtaining onboard image data from the onboard image sensor, determining an image-based positional characteristic of the vehicle based on the onboard image data and a digital twin of the manufacturing environment, determining whether a calibration condition is satisfied based on the infrastructure-based positional characteristic, the image-based positional characteristic, or a combination thereof, and in response to the calibration condition being satisfied: generating an offset matrix based on the infrastructure-based positional characteristic and the image-based positional characteristic, and selectively adjusting a rotation matrix of the onboard image sensor based on the offset matrix and one or more additional offset matrices.

DETAILED DESCRIPTION

The present disclosure provides systems and methods for calibrating an onboard image sensor of a vehicle as it navigates within a manufacturing environment without a dedicated camera calibration EOL station. Specifically, a vehicle control module determines an infrastructure-based positional characteristic of the vehicle and an image-based positional characteristic of the vehicle and generates an offset matrix based on the infrastructure-based positional characteristic and the image-based positional characteristic. The vehicle control module further selectively adjusts a rotation matrix of the onboard image sensor based on the offset matrix and one or more additional offset matrices based on, for example, a recursive estimation updating routine. Accordingly, the accuracy of the camera calibration routines is improved, as the offset roll value, offset pitch value, and offset yaw value of the rotation matrix of the onboard image sensors are independent of the dynamic conditions of the manufacturing environment that inhibit the accuracy of static camera calibration EOL testing stations. Additionally, the efficiency of the camera calibration routines is improved by dynamically performing the calibration as the vehicle navigates within the manufacturing environment as opposed to having a dedicated camera calibration EOL testing station.

Referring toFIG.1, a manufacturing environment5is shown and generally includes an infrastructure system100, a fleet management system200, and a plurality of vehicles300. In one form, the infrastructure system100includes a communication module110and one or more infrastructure sensors120, and the fleet management system200includes a communication module210, and a navigation instruction module220. In one form, the vehicles300each include a communication module310, one or more onboard image sensors320, a vehicle control module330, a digital twin database340, and a calibration module350. It should be understood that any one of the components of the infrastructure system100, the fleet management system200, and the vehicles300can be provided at the same location or distributed at different locations (e.g., via one or more edge computing devices) and communicably coupled accordingly.

In one form, the infrastructure system100, the fleet management system200, and the vehicles300are communicably coupled via the communication modules110,210,310. As an example, the communication modules110,210,310may employ known wireless communication protocols to communicate, such as a cellular protocol, a wireless fidelity (Wi-Fi)-type protocol, a Bluetooth®-type protocol, a near-field communication (NFC) protocol, and/or an ultra-wideband (UWB) protocol. Additionally, or alternatively, communication modules110,210,310may employ a vehicle-to-vehicle, a vehicle-to-infrastructure, a vehicle-to-network, a vehicle-to-grid communication system, an infrastructure-to-vehicle, an infrastructure-to-infrastructure, an infrastructure-to-network, and/or an infrastructure-to-grid, communication system to communicate (e.g., a CV2X and/or dedicated short range communication (DSRC) protocol). Accordingly, the communication modules110,210,310may include one or more transceivers, modulation/demodulation circuits, controllers, routers, and/or input/output interface hardware to perform the functionality described herein.

In one form, the one or more infrastructure sensors120are configured to obtain infrastructure sensor data associated with manufacturing environment5, such as the vehicles300. As an example, the one or more infrastructure sensors120include image sensors (e.g., a two-dimensional camera, a three-dimensional camera, an infrared sensor, a radar scanner, a laser scanner, a LIDAR sensor, and/or an ultrasonic sensor) that obtain image data (as the infrastructure sensor data) of the vehicles300. As described below in further detail, the fleet management system200is configured to control a movement of the vehicles300based on the image data. In one form, the infrastructure sensors120are disposed on an infrastructure element within the manufacturing environment5, such as a tower, a light pole, a building, a sign, an automated guided vehicle, among others fixed and/or moveable elements. As an example, the one or more infrastructure sensors120may be attached or secured to a drone that is configured to autonomously navigate within the manufacturing environment5to obtain the infrastructure sensor data. In one form, the one or more infrastructure sensors120broadcast the sensor data to the fleet management system200via the communication module110.

In one form, the navigation instruction module220is configured to control a movement of the vehicles300within the manufacturing environment5based on the image data obtained from the one or more infrastructure sensors120and status data obtained from the vehicle control module330. As an example, the navigation instruction module220is configured to perform one or more known path planning routines to define a path for the vehicles300and broadcasts one or more commands to autonomously control the vehicles300along the defined paths by employing known autonomous navigation routines. As a more specific example, the navigation instruction module220instructs the vehicles300to autonomously navigate between one or more validation/inspection stations (e.g., one or vehicle end-of-line (EOL) testing stations) of the manufacturing environment5based on the infrastructure data generated by the one or more infrastructure sensors120.

In one form, the one or more onboard image sensors320are disposed at the vehicle300and are configured to obtain image data of the manufacturing environment5. The onboard image sensors320may include, but are not limited to, a two-dimensional (2D) camera, a 3D camera, a red-green-blue (RGB)-camera, a stereo vision camera, an infrared sensor, a radar scanner, a laser scanner, a light detection and ranging (LIDAR) sensor, and/or an ultrasonic sensor. As a specific example, the one or more onboard image sensors320are provided by a forward-facing camera that is employed during an automatic emergency braking routine performed by the vehicle300. In one form, the one or more onboard image sensors320have various image sensor characteristics, such as a pixel density, focal length, height, width, and/or geometry. Additionally, the image sensor characteristics may define a translation vector and/or a rotation matrix that are collectively employed to transform coordinates between a coordinate system of the onboard image sensor320to a coordinate system of the manufacturing environment5.

As an example, the rotation matrix defines an offset roll value, offset pitch value, and offset yaw value of the onboard image sensor320. As used herein, the “offset roll value” refers to a deviation of the roll value relative to a ground plane of the vehicle300or the manufacturing environment5, such as a surface traversed by the vehicle300as it operates within the manufacturing environment5(e.g., a parking lot, a floor, etc.). As used herein, the “offset pitch value” refers to a deviation of the pitch value relative to the ground plane of the vehicle300or the manufacturing environment5. As used herein, the “offset yaw value” refers to a deviation of the yaw value relative to a vertical plane of the vehicle300or the manufacturing environment5.

Referring toFIGS.1-2, the vehicle control module330includes an image-based positional characteristic module332and an infrastructure-based positional characteristic module334, and the calibration module350includes a calibration condition module352, an offset matrix module354, an offset matrix database356, and a rotation matrix adjustment module358. In one form, the image-based positional characteristic module332determines an image-based positional characteristic of the vehicle300based on the image data obtained by the onboard image sensor320and a digital twin of the manufacturing environment5stored in the digital twin database340. As used herein, “digital twin” refers to a three-dimensional (3D) digital model that includes digital representations of various elements within the manufacturing environment5and the corresponding dimensions, 3D position coordinates, and orientations. Example digital representations of the systems/components include virtual sensors (e.g., digital representations of the infrastructure sensors120), virtual robots, virtual poles, virtual beams, virtual conveyors, virtual workstations, among other elements of the manufacturing environment5. In one form, the digital twin is a computer-aided design (CAD) file, a standard tessellation language (STL) file, and/or any other file type configured to provide a 3D digital model of the manufacturing environment5.

Additionally, the digital twin may further include one or more radio frequency (RF) heat maps that define RF signal characteristics associated within one or more current or previous RF signals broadcasted within the manufacturing environment5, such as RF signal magnitudes and/or RF interference. In one form, the digital twin may further define undetectable areas within the manufacturing environment5(e.g., occluded areas). As described below in further detail, the calibration module350may determine whether a calibration condition is satisfied based on the RF heat maps and/or the presence of undetectable areas within the manufacturing environment5.

In one form, the image-based positional characteristic of the vehicle300includes an image-based location of the vehicle300, an image-based pose of the vehicle300, or a combination thereof. As used herein, “image-based location of the vehicle300” refers to a location of the vehicle300that is determined based on the image data obtained from the onboard image sensor320, and “image-based pose of the vehicle300” refers to a pose of the vehicle300that is determined based on the image data obtained from the one or more onboard image sensors320.

As an example, the image-based positional characteristic module332performs known image processing routines (e.g., a difference-based image processing routine, a semantic-based image processing routine, among others) on the image data to detect objects within the manufacturing environment5. As a more specific example, the image-based positional characteristic module332detects the objects by comparing the image data to the digital twin during a difference-based image processing routine. As another more specific example, the image-based positional characteristic module332detects the objects by performing a semantic-based image processing routine on the image data and comparing the classified objects to the digital twin. As yet another more specific example, the image-based positional characteristic module332detects the objects by decoding fiducial markers disposed thereon and correlating the decoded fiducial markers to one or more objects of the digital twin. Example routines for decoding fiducial markers provided within image data are disclosed in U.S. Pat. No. 11,417,015 titled “DECENTRALIZED LOCATION DETERMINATION SYSTEMS AND METHODS,” which is commonly owned with the present application and the contents of which are incorporated herein by reference in its entirety.

In response to detecting the objects, the image-based positional characteristic module332identifies the 3D position coordinates and pose of the detected object defined by the digital twin and performs known image-based processing routines to determine an image-based distance and pose between the vehicle300and the detected object. Specifically, the image-based positional characteristic module332employs known image-based distance and posed determination routines to determine the image-based distance and pose, respectively, based on the image data and the image sensor characteristics (e.g., a pixel density, focal length, height, width, and/or geometry). Subsequently, the image-based positional characteristic module332determines the location of the vehicle300(as the image-based location of the vehicle300) based on the image-based distances and the position coordinate of the detected object. Additionally, the image-based positional characteristic module332determines the pose of the vehicle300(as the image-based pose of the vehicle300) based on the image-based pose and the position coordinate of the detected object.

In one form, the infrastructure-based positional characteristic module334determines an infrastructure-based positional characteristic of the vehicle300based on the infrastructure data obtained from the one or more infrastructure sensors120. In one form, the infrastructure-based positional characteristic of the vehicle300includes an infrastructure-based location of the vehicle300, an infrastructure-based pose of the vehicle300, or a combination thereof. As used herein, “infrastructure-based location of the vehicle300” refers to a location of the vehicle300that is determined based on the infrastructure data obtained from the one or more infrastructure sensors120, and “infrastructure-based pose of the vehicle300” refers to a pose of the vehicle300that is determined based on the infrastructure data obtained from the one or more infrastructure sensors120. As an example, the infrastructure-based positional characteristic module334is configured to perform known location and/or pose determination routines to determine the positional characteristic of the vehicle300based on the infrastructure data (e.g., image data of the manufacturing environment5).

In one form, the calibration condition module352determines whether a calibration condition is satisfied based on the infrastructure-based positional characteristic, the image-based positional characteristic, or a combination thereof. That is, the calibration condition module352determines whether the one or more onboard image sensors320need to be calibrated based on the infrastructure-based positional characteristic and/or the image-based positional characteristic. When the calibration condition is satisfied, the calibration condition module352instructs the offset matrix module354and the rotation matrix adjustment module358to collectively calibrate a rotation matrix of the onboard image sensor320. Additional details regarding the calibration routine are provided below.

As an example, the calibration condition may be satisfied in response to a difference between the infrastructure-based positional characteristic and the image-based positional characteristic being less than a threshold difference. That is, when the infrastructure-based positional characteristic and the image-based positional characteristic deviate beyond the threshold difference, the offset matrix module354and the rotation matrix adjustment module358refrain from calibrating the rotation matrix of the onboard image sensor320due to, for example, a potential error associated with at least one of the infrastructure-based positional characteristic and the image-based positional characteristic. When the infrastructure-based positional characteristic and the image-based positional characteristic do not deviate beyond the threshold difference (i.e., the calibration condition is satisfied), the positional characteristics may be sufficiently accurate. Accordingly, the offset matrix module354and the rotation matrix adjustment module358may subsequently calibrate the rotation matrix of the onboard image sensor320when the calibration condition is satisfied.

As another example, the calibration condition may not be satisfied in response to the infrastructure-based positional characteristic or the image-based positional characteristic being associated with a predefined noise area of the manufacturing environment5. That is, the digital twin may further include a plurality of RF heat maps that overlay the 3D digital model of the manufacturing environment5, where each RF heat map indicates RF signal magnitudes and/or noise/interference for a given wireless communication channel. As a specific example, the digital twin may include eleven overlayed digital RF heat maps for each channel of a 2.4 GHz band Wi-Fi communication protocol, forty-five overlayed digital RF heat maps for each channel of the 5 GHz band Wi-Fi communication protocol, and so on. Example RF heat maps are disclosed in U.S. patent application Ser. No. 17/122,413 titled “RADIO FREQUENCY SPECTRUM MANAGEMENT SYSTEMS AND METHODS IN A MANUFACTURING ENVIRONMENT,” which is commonly owned with the present application and the contents of which are incorporated herein by reference in its entirety. Accordingly, when the infrastructure-based or image-based positional characteristics are associated with a noisy or high interference area (as defined by the one or more overlayed RF heat maps of the digital twin), the offset matrix module354and the rotation matrix adjustment module358refrain from calibrating the rotation matrix of the onboard image sensor320to inhibit noise-induced errors.

As yet another example, the calibration condition may not be satisfied in response to the infrastructure-based positional characteristic being associated with an undetectable area of the manufacturing environment5. Specifically, the placement and orientation of the infrastructure sensors120may be determined based on a sensor placement computing routine that optimizes sensor coverage of the manufacturing environment5by selectively designating the sizes and locations of detectable areas and undetectable areas of the manufacturing environment5. Example sensor placement routines are disclosed in U.S. patent application Ser. No. 17/143,634 titled “METHOD AND SYSTEM FOR DETERMINING SENSOR PLACEMENT FOR A WORKSPACE BASED ON ROBOT POSE SCENARIOS,” which is commonly owned with the present application and the contents of which are incorporated herein by reference in its entirety. Accordingly, when the infrastructure-based or image-based positional characteristics are associated with an undetectable area of the manufacturing environment5, the offset matrix module354and the rotation matrix adjustment module358refrain from calibrating the rotation matrix of the onboard image sensor320due to, for example, a potential error associated with at least one of the infrastructure-based positional characteristic and the image-based positional characteristic.

When the calibration condition module352determines that the calibration condition is satisfied, the offset matrix module354generates an offset matrix based on the infrastructure-based positional characteristic and the image-based positional characteristic. In one form, the offset matrix is further based on an image-based rotation matrix of the onboard image sensor320and an infrastructure-based rotation matrix of the onboard image sensor320, where each of the image-based rotation matrix and the infrastructure-based rotation matrices define an offset roll, offset yaw, and offset pitch value.

As an example, the offset matrix module354calculates the image-based rotation matrix based on the image-based positional characteristic of the vehicle300and a known positional characteristic of an object detected by the onboard image sensor320(as indicated by the digital twin). Furthermore, the offset matrix module354calculates the infrastructure-based rotation matrix based on the infrastructure-based positional characteristic of the vehicle300and a known positional characteristic of an object detected by the onboard image sensor320(as indicated by the digital twin). Accordingly, the offset matrix module354may generate the offset matrix based on various arithmetic representations of the image-based rotation matrix and the infrastructure-based rotation matrix, such as an average, minimum, or maximum of the offset roll, offset pitch, and/or offset yaw values of the image-based and infrastructure-based rotation matrices.

In one form, the rotation matrix adjustment module358selectively adjusts the rotation matrix of the one or more onboard image sensors320based on the offset matrix generated by the offset matrix module354and one or more additional offset matrices stored in the offset matrix database356. The one or more additional offset matrices may be offset matrices that were previously generated by the offset matrix module354and are based on one or more additional infrastructure-based positional characteristics and one or more additional image-based positional characteristics of the vehicle300. In one form, the rotation matrix adjustment module358selectively adjusts the rotation matrix of the one or more onboard image sensors320in response to the calibration condition being satisfied, as described above.

As an example, the rotation matrix adjustment module358selectively updates the rotation matrix by performing a recursive updating routine based on the offset matrix generated by the offset matrix module354and the one or more additional offset matrices stored in the offset matrix database356. Example recursive updating routines include, but are not limited to, a recursive least-squares estimation routine, an exponential weighted moving average routine, a gradient descent routine, a root-finding routine, among others. By performing the recursive updating routine, the rotation matrix adjustment module358generates a recursively calibrated rotation matrix for the one or more onboard image sensors320. Accordingly, the rotation matrix adjustment module358may update the current rotation matrix of the onboard image sensor320in response to the recursively calibrated rotation matrix not matching the current matrix. That is, the rotation matrix adjustment module358updates the current offset roll value, current offset pitch value, and current offset yaw value to match the recursively updated offset roll value, the recursively updated offset pitch value, and the recursively updated offset yaw value, respectively.

As a specific example, the rotation matrix adjustment module358performs a recursive least-squares estimation (RLSE) routine to fit a linear model to the offset matrix generated by the offset matrix module354and the one or more additional offset matrices stored in the offset matrix database356to predict the calibrated offset roll, offset pitch, and/or offset yaw values. That is, the RLSE routine generates an initial offset roll value, an initial offset pitch, and/or an initial offset yaw value based on the one or more additional offset matrices stored in the offset matrix database356. Subsequently, the rotation matrix adjustment module358corrects the initial offset roll, pitch and yaw values based on the corresponding values of the offset matrix generated by the offset matrix module354to generate the recursively updated offset roll, pitch, and yaw values. To perform the functionality described herein, the rotation matrix adjustment module358may recursively calculate filter coefficients based on an algebraic Riccati equation that minimizes a weighted linear least squares cost function related to the rotation matrices. In one form, the filter coefficients may be based on a filter order parameter, a forgetting factor parameter, and an initialization parameter.

As another specific example, the rotation matrix adjustment module358performs an exponential weighted moving average (EWMA) routine to determine an EWMA associated with the offset matrix generated by the offset matrix module354and the one or more additional offset matrices stored in the offset matrix database356. As an example, the rotation matrix adjustment module358defines the offset roll, offset pitch, and offset yaw values for the one or more additional matrices stored in the offset matrix database356(e.g., rotation matrices R1-R4) and the offset matrix generated by the offset matrix module354(e.g., rotation matrix R5). Furthermore, the rotation matrix adjustment module358generates the recursively updated offset roll, pitch, and yaw values (vectors Cal__R1-Cal_R5) by determining, for each new rotation matrix, an exponential moving average (e.g., vectors EWMA1-EWMA4), as shown below in Table 1.

In Table 1, β is a weighting coefficient given to previous EWMAs (i.e., larger values of β correspond to giving less weight to previous EWMAs). In some forms, the EWMAs may be a function of each preceding rotation matrix or a set of each preceding rotation matrix (e.g., the exponential moving average is calculated based on the EWMA of the ten preceding rotation matrices). While the EWMA is disclosed, it should be understood that other types of moving averages may be employed and is not limited to the example described herein, such as simple moving averages, smoothed moving averages, linear weighted moving averages, among other moving average types.

As an additional specific example, the rotation matrix adjustment module358performs a gradient descent routine to identify a local minimum of a differentiable function associated with the offset matrix generated by the offset matrix module354and the one or more additional offset matrices stored in the offset matrix database356. That is, the local minimum of the differentiable function representing the offset roll, offset pitch, and offset yaw values may correspond to the recursively updated offset roll, pitch, and yaw values. As yet another example, the rotation matrix adjustment module358performs a root-finding routine to identify zero values of a continuous function associated with the offset matrix generated by the offset matrix module354and the one or more additional offset matrices stored in the offset matrix database356. That is, the zero values of the continuous function representing the offset roll, offset pitch, and offset yaw values may correspond to the recursively updated offset roll, pitch, and yaw values.

Referring toFIG.3, a flowchart illustrating an example routine300for calibrating the onboard image sensor320is shown. At304, the vehicle control module330obtains the infrastructure data and the onboard image data. At308, the vehicle control module330determines an infrastructure-based positional characteristic based on the infrastructure data and an image-based positional characteristic based on the image data. At312, the calibration module350generates an offset matrix based on the infrastructure-based positional characteristic and the image-based positional characteristic. At316, the calibration module350selectively adjust the rotation matrix of the onboard image sensor320based on the offset matrix and one or more additional offset matrices stored in the offset matrix database356.

Referring toFIG.4, a flowchart illustrating another example routine400for calibrating the onboard image sensor320is shown. At404, the vehicle control module330obtains the infrastructure data and the onboard image data. At408, the vehicle control module330determines an infrastructure-based positional characteristic based on the infrastructure data and an image-based positional characteristic based on the image data. At412, the calibration module350determines whether the calibration condition is satisfied. If so, the routine400proceeds to416. Otherwise, the routine400proceeds to404. At416, the calibration module350generates an offset matrix based on the infrastructure-based positional characteristic and the image-based positional characteristic. At420, the calibration module350selectively adjust the rotation matrix of the onboard image sensor320based on the offset matrix and one or more additional offset matrices stored in the offset matrix database356.