Methods and apparatus for an imaging system

Various embodiments of the present technology may comprise a method and apparatus for an on-vehicle camera system capable of calibrating the camera's orientation (i.e. pitch, yaw, roll angles) in relation to the vehicle's coordinate system utilizing normally-encountered imagery. The method and apparatus may comprise utilizing an image processor to detect features from pixel data of image frames, match features, provide an estimated orientation, and validate the orientation. A system utilizing the method and apparatus may transmit the camera's orientation with respect to the vehicle coordinate system to a peripheral system.

BACKGROUND OF THE TECHNOLOGY

An accurate estimate of camera orientation with respect to a specific reference, such as the vehicle supporting the camera, facilitates many image processing and computer vision functionalities. Such functionalities include, but are not limited to, view synthesis (e.g., generating images from different viewing perspectives using a single backup camera, stitching images from surround-view cameras to form a panoramic image, etc.), object detection (e.g., detecting lanes, detecting obstacles on the road, etc.), and three-dimensional (3D) reconstruction (e.g., recovering the 3D structure of the scene using stereo matching from a pair of cameras or using structure from motion from a single camera), which are common components in advanced driver assistance systems (ADAS). A camera's orientation may be represented using several interchangeable representations, which include but are not limited to Euler angle representation (i.e., pitch, yaw, and roll angles following a predefined rotation order), rotation matrix representation, quaternion representation, and axis-angle representation.

Conventional camera orientation estimation uses specially-designed patterns, such as grid patterns, and time-consuming and error-prone techniques. Additionally, if the camera position changes, such as when the vehicle supporting the camera gets into an accident, the camera must be re-calibrated with special equipment and procedures to re-estimate the camera orientation.

Installing inertial measurement units (IMUs) on a camera has limited applicability due to the diminished precision and durability of consumer-grade IMUs and the high cost of higher-grade IMUs.

Other solutions for automatic camera orientation estimation without specially-designed patterns generally employ conventional feature detectors and feature matchers that either directly use local image patches or rely on high-dimensional feature vectors extracted from those patches to describe and match features. These feature detectors and feature matchers generally require the use of a frame buffer to hold an image, which consumes a great deal of power and adds a substantial cost to design. Techniques employing model fitting methods that heavily rely on linear system solvers also require more storage and power, which lead to a higher cost calibration system. Techniques estimating only one or two rotation angles generally assume that the rest of the rotation angles are known and remain constant. This assumption limits the applicability of such techniques.

Given the limitations of existing calibration methods and the limitations of IMUs, a low cost, low-power consumption calibration method with the capability of automatic camera orientation estimation without specially-designed patterns is desirable. Such capability is herein referred to as self-calibration.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The present technology may be described in terms of functional block components and various processing steps. Such functional blocks may be realized by any number of components configured to perform the specified functions and achieve the various results. For example, the present technology may employ various imaging sensors, image processing units, computations, algorithms, model fitting techniques, data partitioning, and the like, which may carry out a variety of functions. In addition, the present technology may be practiced in conjunction with any number of systems such as automotive and aviation systems, and the systems described are merely exemplary applications for the technology. Further, the present technology may employ any number of conventional techniques for detecting corners, measuring distances, calculating angles, and the like.

Methods and apparatus for an imaging system according to various aspects of the present technology may operate in conjunction with any suitable imaging system, such as a camera located on a vehicle. Referring now toFIGS. 1 and 6, in an exemplary embodiment of the present technology, methods and apparatus for an imaging system600may operate in conjunction with a self-calibrating imaging system including an image sensor665, an image processing unit145, and peripheral systems650, and may implement a self-calibration method100. The image processing unit145may detect features110in pixel data105received from the image sensor665, match the detected features115, perform orientation estimation120using the matched features, and perform validation of the estimated orientation130. In various embodiments, supplemental information705(FIG. 7), such as vehicle speed and camera height, may also be used by the imaging system600and incorporated into the method100.

In an exemplary embodiment, the image processing unit145may utilize pixel data105from the image sensor665comprising an array of pixels, wherein the pixels450(FIG. 4A) are arranged in rows and columns. The image processing unit145may utilize the pixel data105of a first image frame405(FIG. 4A) and a second image frame410(FIG. 4B), for example from the image sensor665. In another embodiment, pixel data105of more than two image frames400may be utilized when more than two image frames400are available.

In various embodiments, the image processing unit145may comprise a calibration unit165configured to perform various image processing techniques and utilize algorithms to determine the orientation of the camera565in relation to a reference coordinate system505(FIG. 5). The calibration unit165may perform feature detection110, feature matching115, orientation estimation120, and orientation validation130to determine the camera orientation.

In various embodiments, the image processing unit145and calibration unit165may be implemented with a programmable logic device, such as a field programmable gate array (FPGA) or any other device with reconfigurable digital circuits. In other embodiments, the image processing unit145and calibration unit165may be implemented in hardware using non-programmable devices. In alternative embodiments, the image processing unit145and calibration unit165may be formed partially or entirely within an integrated circuit in silicon using any suitable complementary metal-oxide semiconductor (CMOS) techniques or fabrication processes, in an ASIC, using a processor and memory system, or other suitable implementation.

The calibration unit165may perform feature detection110on the image data, such as data from each of the first and second image frames405,410(FIGS. 4A-4B). In the present exemplary embodiment, feature detection110may comprise detecting at least two features within each of the first image frame405and the second image frame410. In various embodiments, the calibration unit165may initially detect many features, for example more than 10 features. Once the features have been detected, the calibration unit165may provide image processing techniques to reject invalid features and keep valid features for further image processing.

The calibration unit165may, however, determine that there is not sufficient information to proceed to feature matching115, such as fewer than two features are detected from pixel data105of the current image frame400, in which case the calibration unit165may initiate a new round of feature detection110using the pixel data105of a new image frame400. In various embodiments, the calibration unit165may not perform feature detection110on pixel data105belonging to the entire image frame400, but on pixel data105belonging to part of the image frame400. For example, the calibration unit165may skip pixel data105belonging to multiple rows at the top of an image frame400.

Referring toFIG. 2, feature detection110, via the calibration unit165, may comprise detecting a set of candidate corners210by utilizing corner detection methods, such as the Harris corner detector and the Features from Accelerated Segment Test (FAST) detector, or any other suitable corner detection method. In an exemplary embodiment, the calibration unit165may perform feature detection110on normally encountered imagery, such as images containing brightness or color variations in the imaged road region that form corners, and may not require the use of specially designed targets, such as specific road surface signs, repeated textures, or symmetric dashed lane markers. For example, the calibration unit165may detect corners from a tiled surface, a newly patched road area, a shadow, and/or from any road markings with corners, such as lines to indicate a pedestrian walkway, a center lane marker, a speed bump marker, a turn lane marker, and the like.

In various embodiments, the calibration unit165may generate a set of candidate corners230for each image frame400(FIGS. 4A-4B). The set of candidate corners230may comprise a plurality of detected features, such as a plurality of candidate corners. For example,FIGS. 4C and 4Dillustrate the first image frame405and the second image frame410, respectively, where the first image frame405comprises a set of detected features420a-j(i.e. set of candidate corners230), and the second image frame410comprises a set of detected features420a′,420k′ (i.e. set of candidate corners230).

Once the sets of candidate corners230have been generated, the calibration unit165may provide additional processing of pixel data105forming each candidate corner230and the pixel data105surrounding each candidate corner230. For example, for each candidate corner in the set of candidate corners230, the calibration unit165may perform estimation of incident-edge orientations215, which determines the orientations of edges incident to the candidate corner230to distinguish a valid corner235from an invalid corner and reject the invalid corner. Such additional processing may be performed on each candidate corner230when that candidate corner230becomes available, or when all or part of the set of candidate corners230becomes available.

In the exemplary embodiment, for each candidate corner230, the calibration unit165may determine the orientations of the incident edges215. The calibration unit165may compute edge orientations for pixels forming the candidate corner230and/or in the neighborhood of the candidate corner230, for example, through computed gradients in at least two distinct directions, where two of the directions may be orthogonal to each other. The calibration unit165may utilize the edge orientations to compute a set of dominant edge orientations. For example, the calibration unit165may compute the set of dominant edge orientations, by assembling a histogram of the edge orientations and then selecting the main peaks in the histogram. The calibration unit165then tests the validity of each pair in the set of dominant edge orientations, for example, by determining the number of pixels450(FIG. 4A) along each of the two edge orientations originating from the corner, and determining the agreement between the intensity/color transition across one edge and the intensity/color transition across the other edge.

Referring toFIGS. 4E-4F, in various embodiments, a candidate corner230(FIG. 2) which is determined to comprise at least one valid pair of dominant edge orientations may be defined as a valid corner235. The calibration unit165may generate a set of valid corners235for each image frame400comprising a plurality of valid corners (i.e. valid features420,420′). For example,FIGS. 4E and 4Fillustrate the first image frame405and the second image frame410, where the first image frame405comprises a set of valid features420b,420d-j(i.e. set of valid corners235), and the second image frame410comprises a set of valid features420b′-d′,420f′,420k′ (i.e. set of valid corners235).

Referring again toFIG. 2, for each valid corner (i.e. valid feature420,420′), the calibration unit165may perform corner descriptor generation220, from which a corner descriptor240may be generated comprising information related to the valid corner's properties, such as a position, an orientation, a subtended angle, and a contrast, wherein the contrast is defined with respect to the intensity/color transition between the pixels inside the corner and the pixels outside the corner. In other embodiments, the corner descriptor240may also comprise information related to the valid corner's neighborhood, such as a list of indices of neighboring corners, where the neighborhood may be defined according to a predefined distance metric. The predefined distance metric may be a spatial distance metric, a spectral distance metric, a combined spatial and spectral distance metric, and the like. A valid corner's neighborhood information in its corner descriptor240may be generated when the set of valid corners235for an image frame400becomes available or before neighborhood matching315(explained below,FIG. 3) is performed. In an exemplary embodiment, the method100may not require that the entire image frame400, a downscaled frame, or an upscaled frame be stored in a storage unit625(FIG. 6) to perform feature detection110. For example, the calibration unit165may transmit pixel data105belonging to part of an image frame400to the storage unit625, such that the pixel data105may be stored without the use of a frame buffer, when performing feature detection110. In various embodiments, the calibration unit165rectifies the valid features420,420′ to account for lens distortion prior to feature matching115. For example, the position of a valid feature420,420′ may be rectified to the undistorted image coordinates. The subtended angle and the orientation of the valid feature420,420′ may also be rectified to the undistorted image coordinates.

The calibration unit165may perform feature matching115between the first image frame405and the second image frame410. Feature matching115may comprise comparing the valid features, for example420b,420d-jof the first image frame405with the valid features, for example420b′-d′,420f′,420k′ of the second image frame410, and identifying pairs of valid features that have similar properties, wherein each pair comprises one valid feature420from the first image frame405and the corresponding valid feature420′ from the second image frame410. A pair of valid features that have similar properties is referred to as a pair of matched features455hereafter. In an exemplary embodiment, the valid feature may be the valid corner, however, other embodiments may detect features other than corners.

In an exemplary embodiment, feature matching115may comprise comparing each valid corner from the first image frame405with each valid corner from the second image frame410. A pair of valid corners with similar properties may be determined as a pair of matched features455. In an exemplary embodiment, referring now toFIG. 3, feature matching115may comprise single-feature matching305to form the candidate set of matched features325and neighborhood matching315to form the final set of matched features330. Single-feature matching305may comprise matching the valid features420,420′ based on individual properties of each valid feature, such as the position, orientation, subtended angle, and contrast. For example, referring toFIGS. 4G-4H, the valid features420of the first image frame405may be compared with the valid features420′ of the second image frame410and then be determined as pairs of matched features455(1)-(7). Conversely, neighborhood matching315, described below, may comprise matching the valid features420,420′ based on the neighborhood properties of each valid feature.

Valid features420of the first image frame405may or may not have a match from the valid features420′ of the second image frame410. For example, as illustrated inFIG. 4G, demonstrating superimposing the features ofFIGS. 4E and 4F, the valid feature420bof the first image frame405and the valid feature420b′ of the second image frame410may be determined as a pair of matched features455(2) because they have similar properties; however, the valid feature420iof the first image frame405and the valid feature420b′ of the second frame410may not be determined as a pair of matched features because they have different properties.

In various embodiments, the calibration unit165may determine that there is not sufficient information to proceed to performing orientation estimation120, such as less than two pairs of matched features455are determined, where in such case, the calibration unit165may initiate a new round of feature detection110using the pixel data105of a new image frame400.

The pairs of matched features455may be included in a candidate set of matched features325. The pairs of matched features455may be also included in a final set of matched features330.

For each pair of matched features455(1)-(7) in the candidate set of matched features325, the calibration unit165may perform neighborhood matching315. Neighborhood matching315may comprise, for each pair of matched features in the candidate set of matched features325, determining whether their neighborhoods match315with each other based on the candidate set of matched features325and the topologies of the neighborhoods. For each valid feature420,420′ in each pair of matched features in the candidate set of matched features325, the topology of its neighborhood may be determined from that feature's descriptor240and its neighboring features' descriptors240. In an exemplary embodiment, for each valid feature420,420′ in each pair of matched features in the candidate set of matched features325, the topology of its neighborhood may be a spatial topology, which represents the relative position of each neighboring feature with respect to that feature. Each pair of matched features455in the candidate set325that also has matched neighborhoods is included in the final set of matched features330.

Two neighborhoods may be determined as matched neighborhoods if they have at least one pair of features that is in the candidate set of matched features325and if at least one pair of features has similar neighborhood topologies. The pairs of matched features455(1)-(7) may not all be included in the final set of matched features330. For example, only a subset of the pairs of matched features455may be included in the final set of matched features330. For example, as illustrated inFIG. 4H, demonstrating superimposed features from two image frames, the final set of matched features330may only include valid features420i,420i′,420f,420f′,420g,420g′.

For each pair of matched features455(1)-(7) within the final set of matched features330, the calibration unit165may generate match descriptors320, such that each pair of matched features455(1)-(7) has an associated match descriptor335The match descriptor335may comprise information related to the pair of matched features within the final set of matched features330. For example, the match descriptor335may comprise all or part of the information in each of the matched features' descriptors240. Conversely, the match descriptor335may comprise an index of each of the matched features indicating their locations within their respective sets of feature descriptors240. The match descriptor335may also comprise a similarity score, which indicates the degree of similarity between the two matched features. The match descriptor335may also comprise a scale factor, which indicates the change of scales between the two matched features.

In an exemplary embodiment, the match descriptor335may comprise the positions of each matched feature and the scale factor. In an exemplary embodiment, for each pair of matched features within the final set of matched features330, its scale factor may be determined as a function of the matched neighborhoods of the two features in that pair. For example, the function may be a function computing the ratio between measurements of the sizes of the matched neighborhoods. In an exemplary embodiment, the calibration unit165may not require that the entire image frame400, a downscaled frame, or an upscaled frame be stored in the storage unit625to perform feature matching115. For example, the calibration unit165may store the set of corner descriptors240in the storage unit625without the use of a frame buffer, when performing feature matching115.

Referring now toFIGS. 1 and 5, the calibration unit165may perform orientation estimation120of the camera565in relation to the reference coordinate system505. In the exemplary embodiment, only those features420,420′ within the final set of matched features330may be used to perform orientation estimation120of the camera's rotation angles, comprising a pitch angle515, a yaw angle520, and a row angle525(where the pitch angle515, the yaw angle520, and the row angle525may be collectively defined as a camera coordinate system560) with respect to the vehicle coordinate system505. The final set of matched features330may be identified through the set of match descriptors335and the set of corner descriptors240of the matched features in the final set of matched features330.

Referring toFIGS. 4-5, the orientation estimation120may comprise estimating an epipole425utilizing the final set of matched features330. In various embodiments, epipole estimation430may comprise estimating a general moving direction550of the camera565using the change of scales between the matched features330. For example, if the feature420from the first image frame405, for example as illustrated inFIG. 4E, is of smaller scale than the matched feature420′ of the second image frame415, for example as illustrated inFIG. 4I, then the camera565is moving toward535the structures in the scene that form the features. Conversely, if the feature420from the first frame405, for example as illustrated inFIG. 4E, is of larger scale than the matched feature420′ of the second image frame410, for example as illustrated inFIG. 4F, then the camera565is moving away555from the structures in the scene that form the features. The camera's565general moving direction550may be determined as the direction535or555that a majority of the pairs of matched features455in the final set of matched features330agree upon. Those pairs of matched features455that do not agree with the camera's565general moving direction550may be removed from the final set of matched features330.

Referring toFIGS. 4J and 5, when the camera565undergoes linear motion (i.e., translation) between two image frames and the camera's565linear motion direction570is not parallel to the image plane400, the epipole425may then be computed as the point on the image plane that minimizes the distance to the lines formed by each pair of matched features455in the final set of matched features330(i.e. those pairs of matched features330that also agree with the camera's565general moving direction550). Various light-weight model fitting techniques, such as Random Sample Consensus (RANSAC) and RANSAC-like methods, may be applied to computation of the epipole425.

Referring toFIG. 5, estimating the orientation120may further comprise extracting the pitch angle515, the yaw angle520, and the roll angle525(i.e. rotation angles) from the epipole425. For example, if the roll angle525is assigned to zero or a predetermined number, then the pitch and yaw angles515,520may be calculated from the epipole425. Conversely, any rotation angle may be initially assigned to zero or a predetermined number with subsequent calculation of the remaining angles from the epipole425.

In a linear motion (i.e., translation) of the vehicle510between two image frames, if the geometric relationship between the camera565and the vehicle510does not change in the linear motion, and if the reference coordinate system505is defined as having the vehicle's linear motion direction570along one of the reference coordinate system's505axes but not parallel to the image plane400, then the epipole425and the three rotation angles515,520, and525may be related as: g*e=K*R(u, v, w)*t=K*[R(u, v, w)]:,*[t]i, where g denotes a scaling factor; e denotes the epipole425; K denotes the camera intrinsic matrix; R(u, v, w) denotes the rotation matrix representing the rotation from the reference coordinate system505to the camera coordinate system560and is defined by the pitch angle u515, the yaw angle v520, and the roll angle w525following a predefined rotation order; t is the translation vector, the sign of which is determined by the camera's general moving direction550; [ ]:, idenotes i-th column of a matrix; [ ]idenotes i-th element of a vector; * denotes multiplication. For example, the i-th axis may be the z-axis, if the rotation order is: first, rotation about x-axis; second, rotation about y-axis; third, rotation about z-axis. The camera intrinsic matrix may be either derived from nominal values of the camera's565intrinsic parameters or obtained from intrinsic parameter calibration of the camera565, wherein the intrinsic parameters may comprise camera focal length, principal point, pixel aspect ratio, pixel skew, and lens distortion coefficients. Once the epipole425is computed, the rotation angles515,520, and525may be calculated using this relationship. The reference coordinate system's505origin may be defined to be coincident with the camera coordinate system's560origin. If not, the reference coordinate system's505origin may be assumed to be coincident with the camera coordinate system's560origin in all the calculations.

Referring toFIGS. 1, 4H and 5A-C, orientation estimation120may further comprise computing the three-dimensional (3D) coordinates for each valid feature420,420′ within the final set of matched features330. For example, the 3D coordinates for each valid feature420,420′ within the final set of matched features330may be computed by projecting the final set of matched features330from each of the first and second image planes405,410into the reference coordinate system505, for example, the vehicle coordinate system by utilizing the current estimate of the rotation angles515,520, and525and triangulation techniques.

Orientation estimation120may further comprise projecting the 3D coordinates of the final set of matched features330onto a two-dimensional (2D) plane540which is perpendicular to the vehicle's moving direction570, where the projection of a 3D coordinate onto the 2D plane540forms a point on the 2D plane540corresponding to each valid feature420,420′ within the final set of matched features330. The projection of a 3D coordinate onto the 2D plane540may be an orthographic projection.

Orientation estimation120may further comprise determining whether the valid features420,420′ within the final set of matched features330are positioned on a reference plane545, for example a ground plane. In the exemplary embodiment, determining the position of the final set of matched features330with respect to the reference plane545comprises fitting a line through the points corresponding to each valid feature420,420′ within the final set of matched features330on the 2D plane540, and calculating a slope from the fitted line. Fitting a line may be achieved by various light-weight model fitting techniques, such as RANSAC and RANSAC-like methods.

In one exemplary embodiment, orientation estimation120may comprise an iterative process, which may comprise repeating the steps: 1) computing the 3D coordinates for each valid feature420,420′ within the final set of matched features330; 2) projecting the 3D coordinates of the set of matched features onto a 2D plane540; 3) fitting a line through the points corresponding to each valid feature420,420′ within the final set of matched features330on the 2D plane540; 4) adjusting the roll angle525from the calculated slope of the fitted line to form an adjusted roll angle525′; and 5) updating the pitch and yaw angles515,520using the epipole425and the adjusted roll angle525′ until the absolute value of the slope of the fitted line is within a predetermined tolerance level. In the exemplary embodiment, the predetermined tolerance level may be a number smaller than a desired accuracy of the calibrated rotation angles515,520,525. For example, if the calibrated rotation angles515,520,525of the camera565are to be within 1 degree error with respect to the reference coordinate system505, then the tolerance level may be defined to be at most tan(1 degree), wherein tan( ) is the trigonometric tangent function.

In other embodiments, orientation estimation120may conclude once the number of iterations reaches a predetermined limit. For example, the predetermined limit may be less than 10 iterations, or any other number to achieve a desired result. If the orientation estimation120cannot converge within the predetermined number of iterations (i.e. the absolute value of the slope of the fitted line is not within a predetermined tolerance level), the calibration method100and calibration unit165may fetch pixel data105of a new image frame400and perform calibration using the pixel data105of the new image frame400and the set of feature descriptors240from one of the previous frames. When there are more than one set of feature descriptors240from previous frames, each set may be used in the new round of calibration.

In various embodiments, when performing orientation estimation120using the matched features120, the calibration unit165may determine that there is not sufficient information to proceed to validating the estimated orientation130, such as no epipole425is found or orientation estimation120does not converge to a camera orientation at termination. In such a case, the calibration unit165may initiate a new round of feature detection110using the pixel data105of a new image frame400.

In various embodiments, the calibration unit165may be configured to validate the estimated orientation130, such as using homography techniques. For example, validating the estimated orientation130using homography techniques may comprise computing a reference-plane-induced inter-frame homography matrix in the reference coordinate system505from the estimated rotation angles515,520,525. The reference-plane-induced inter-frame homography matrix in the reference coordinate system505may be a composition of the camera565intrinsic matrix, the camera565rotation matrix in the reference coordinate system505defined by the estimated rotation angles515,520,525, and the camera translation vector (i.e., movement amount) normalized by the camera565height in the reference coordinate system505. The only unknown may be the camera565translation vector normalized by the camera565height, which may be estimated from the matched features455in the final set of matched features330that were determined to be on the reference plane545.

Validating the estimated orientation130using homography techniques may further comprise determining the quality of the estimated orientation135from the current pair of image frames. For example, determining the quality of the estimated orientation135may comprise computing statistics from a homography matrix and computing a reprojection error of the final set of matched features330that were determined to be on the reference plane545using the homography matrix. The quality of the calibration may be represented as a numeric quality score defined using the statistics and the reprojection error.

In an exemplary embodiment, the statistics from a homography matrix may be metrics that measure the consistency of multiple estimates of the translation vector of the camera565in the reference coordinate system505between two image frames, wherein each estimate may be obtained from one pair of matched features in the final set of matched features330that were determined to be on the reference plane545. In an exemplary embodiment, the reference plane545comprises the ground plane.

In various embodiments, when validating the estimated orientation130, the calibration unit165may not be able to complete orientation validation130. In such a case, a new round of feature detection110may be initiated using the pixel data105of a new image frame400.

Once the camera rotation angles515,520,525have been validated, the calibration unit165may transmit a validated camera orientation135, comprising the pitch, yaw, and roll angles515,520,525with respect to the reference coordinate system, to an output unit655.

Referring now toFIGS. 1, 4, and 6, a system600, utilizing the method100as described above, may comprise the image sensor665and the image processing unit145.

The image processing unit145may further comprise the storage unit625. The storage unit625may temporarily store pixel data105and intermediate data. In various embodiments, the entire image frame400, a downscaled frame, or an upscaled frame may not be stored to a frame buffer located within the storage unit625, but may instead store pixel data105belonging to part of an image frame400, such as pixel data105belonging to a few rows of an image frame400. In various embodiments, information about the detected features420,420′ for each of the first and second image frames405,410such as the set of corner descriptors240, the set of match descriptors335, may be stored in the storage unit625. Estimates of camera orientation135may also be stored in the storage unit625.

In various embodiments, the storage unit625may comprise random-access memory, non-volatile memory or any other memory device suitable for the particular application. The storage unit625may comprise one or more memory cells, such as dynamic random access memory cells (DRAM), static random access memory cells (SRAM), or bistable flip-flops. The storage device may be implemented using transistors or any other suitable semiconductor devices.

In the exemplary embodiment, the image processing unit may further comprise the output unit655. The output unit655may transmit the camera orientation135to a peripheral system650. The output unit655may convert the camera orientation135from Euler angle representation515,520,525(FIG. 5) to other representations, such as rotation matrix, quaternion, or axis-angle, before transmitting the camera orientation135to the peripheral systems650, such as an ADAS. The output unit655may aggregate multiple estimates of the camera orientation135computed from multiple frame pairs before transmitting the aggregated camera orientation135to the peripheral systems650. For example, multiple estimates of the camera orientation135may be averaged using their respective quality scores as weights, wherein the average may be taken on the Euler angles, on the corresponding rotation matrices, on the corresponding quaternions, or on the corresponding axis-angle representations.

In an exemplary embodiment, the peripheral system650may use the camera orientation135to compensate for the camera565's rotational deviation from the reference coordinate system505. For example, a surround view system may use the transmitted camera orientation135to further process pixel data105from at least two image frames400, such as to perform image stitching, to perform 3D reconstruction, and/or to determine one or more regions of interest in an image, such as the region in an image that corresponds to the reference plane545, to produce a surround view output. The surround view output may then be transmitted to other peripheral systems650, such as other systems in ADAS, such as collision avoidance system and lane departure warning system, where these other systems in the ADAS may alert the driver using visual, auditory, or sensory alerts. For example, if the ADAS detects that the driver is approaching another vehicle, the ADAS may sound an alarm, flash a light, and/or vibrate the seat. The ADAS may also be used to signal an alert to a driver of the vehicle to alert the driver to a particular road condition, such as objects or pedestrians in the path of the vehicle, departure from the current lane, and the like.

In other embodiments, the camera orientation135may be used to physically realign the camera565to a desired orientation with respect to the reference coordinate system505. For example, physical realignment of the camera565may be achieved by mechanically rotating the camera565by an amount determined as the difference between its calibrated orientation135and the desired orientation within the reference coordinate system505.

Referring now toFIGS. 1, 6 and 7, in an exemplary embodiment of the present technology, the self-calibration method100and system600may be implemented by the on-vehicle camera565capturing sequential image frames400(FIGS. 4A-B), for example the first and second image frames405,410, and streaming the pixel data105from these frames400to the storage unit625. The image processing unit145, having access to the storage unit625, may receive the pixel data105(710) and perform feature detection110(715). Once features are detected from pixel data105belonging to at least two image frames, the image processing unit145may then perform feature matching115(745).

Once features within each of the first and second image frames405,410have been detected and matched, the image processing unit145may then estimate the camera orientation515,520,525(720). If a predetermined tolerance level is reached (725), the process continues where the estimated orientation may be validated (730) and the quality of the estimated orientation may be calculated (735). If the predetermined tolerance level is not reached, a third image frame (not shown), or a new subsequent image frame, may be utilized where feature detection110(715) and feature matching115(745) are then performed on the pixel data105from the second image frame410and the pixel data105from the third image frame, and/or on the pixel data105from the first image frame405and the pixel data105from the third image frame. If the quality of the estimated orientation is beyond a predetermined threshold (735), the camera orientation515,520,525with respect to the reference coordinate system505may then be utilized by the peripheral systems650(740). If the quality of the estimated orientation is not beyond a predetermined threshold (735), a new subsequent image frame400may be utilized to restart the process.

In the foregoing description, the technology has been described with reference to specific exemplary embodiments. The particular implementations shown and described are illustrative of the technology and its best mode and are not intended to otherwise limit the scope of the present technology in any way. Indeed, for the sake of brevity, conventional manufacturing, connection, preparation, and other functional aspects of the method and system may not be described in detail. Furthermore, the connecting lines shown in the various figures are intended to represent exemplary functional relationships and/or steps between the various elements. Many alternative or additional functional relationships or physical connections may be present in a practical system.

The technology has been described with reference to specific exemplary embodiments. Various modifications and changes, however, may be made without departing from the scope of the present technology. The description and figures are to be regarded in an illustrative manner, rather than a restrictive one and all such modifications are intended to be included within the scope of the present technology. Accordingly, the scope of the technology should be determined by the generic embodiments described and their legal equivalents rather than by merely the specific examples described above. For example, the steps recited in any method or process embodiment may be executed in any order, unless otherwise expressly specified, and are not limited to the explicit order presented in the specific examples. Additionally, the components and/or elements recited in any apparatus embodiment may be assembled or otherwise operationally configured in a variety of permutations to produce substantially the same result as the present technology and are accordingly not limited to the specific configuration recited in the specific examples.

Benefits, other advantages and solutions to problems have been described above with regard to particular embodiments. Any benefit, advantage, solution to problems or any element that may cause any particular benefit, advantage or solution to occur or to become more pronounced, however, is not to be construed as a critical, required or essential feature or component.

The present technology has been described above with reference to an exemplary embodiment. However, changes and modifications may be made to the exemplary embodiment without departing from the scope of the present technology. These and other changes or modifications are intended to be included within the scope of the present technology, as expressed in the following claims.