CONTROL ALGORITHM TO PRECISELY EVALUATE EDGE QUALITY FROM IMAGERY

A method for evaluating edge quality in an image includes receiving an image from a camera of a vehicle, detecting an edge in the image captured by the camera, fitting a polynomial curve to the edge in the image captured by the camera, executing a numerical optimizer to determine a minimum distance from each pixel in the image of the edge to the polynomial curve, determining a modulation transfer function (MTF) value using the minimum distance, determining whether the MTF value is greater than predetermined threshold, and providing an alert in response to determining that the MTF value is not greater than the predetermined threshold.

The present disclosure relates to images captured by cameras and, more particularly, to systems and methods for precisely evaluating edge quality from imagery.

This introduction generally presents the context of the disclosure. Work of the presently named inventors, to the extent it is described in this introduction, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against this disclosure.

Wide field of view cameras have a higher amount of fisheye distortion than other cameras. As a consequence, measuring modulation transfer function (MTF) on fisheye images is extremely complex and requires significant computational resources. Some methods of evaluating MTF use a simple algorithm that often sacrifices accuracy, particularly when measuring the accuracy of the curved edges, which are introduced because of the fisheye distortion. It is therefore desirable to develop a method that accurately determines MTF. The presently disclosed method involves a thorough calculation that fits the detected edge closer than other methods. Therefore, presently disclosed method is capable of an accurate MTF calculation on any shaped edges with a greater precision than other methods. Specifically, the presently disclosed method uses a numerical optimizer that allows this method to be generalized for any angle and/or curved edges.

SUMMARY

The present disclosure describes a method for evaluating edge quality in an image. In an aspect of the present disclosure, the method includes receiving an image from a camera of a vehicle, detecting an edge in the image captured by the camera, fitting a polynomial curve to the edge in the image captured by the camera, executing a numerical optimizer to determine a minimum distance om each pixel in the image of the edge to the polynomial curve, determining a modulation transfer function (MTF) value using the minimum distance, determining whether the MTF value is greater than predetermined threshold, and providing an alert in response to determining that the MTF value is not greater than the predetermined threshold. The method described in this paragraph improves imaging technology by more accurately determining the MTF value and therefore determining which images accurately depict an edge.

In an aspect of the present disclosure, the method further includes pixel binning a plurality of pixels of the edge in the image to generate a plurality of super pixels.

In an aspect of the present disclosure, the method further includes determining an average luminance value for each of the plurality of super pixels.

In an aspect of the present disclosure, the method further includes executing an adjusted pixel binning using the average luminance value for each of the plurality of super pixels.

In an aspect of the present disclosure, the image includes image data. The method further includes storing the image data in response to determining that the MTF value is greater than predetermined threshold.

In an aspect of the present disclosure, the method further includes determining an edge spread function of the edge using the plurality of super pixels.

In an aspect of the present disclosure, the method further includes determining a line spread function of the edge using the edge spread function. The line spread function is a first derivative of the edge spread function.

In an aspect of the present disclosure, determining the MTF value using the minimum distance includes determining the MTF value using the line spread function. The MTF is a Fast Fourier transform of the line spread function.

In an aspect of the present disclosure, detecting the edge in the image captured by the camera includes using a Canny edge detector or other appropriate edge detector to detect the edge in the image.

In an aspect of the present disclosure, detecting the edge in the image captured by the camera includes using a Sobel edge detector to detect the edge in the image.

In an aspect of the present disclosure, the method further includes cropping the image such that the edge is centered in the image.

The present disclosure also describes a tangible, non-transitory, machine-readable medium, including machine-readable instructions, that when executed by one or more processors, cause one or more processors to execute the method described above.

Further areas of applicability of the present disclosure will become apparent from the detailed description provided below. It should be understood that the detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.

The above features and advantages, and other features and advantages, of the presently disclosed system and method are readily apparent from the detailed description, including the claims, and exemplary embodiments when taken in connection with the accompanying drawings.

DETAILED DESCRIPTION

Reference will now be made in detail to several examples of the disclosure that are illustrated in accompanying drawings. Whenever possible, the same or similar reference numerals are used in the drawings and the description to refer to the same or like parts or steps.

With reference toFIG.1, a vehicle10includes (or is in communication with) a system11for evaluating edge quality from imagery. While the system11is shown inside the vehicle10, it is contemplated that the system11may be outside of the vehicle10. As a non-limiting example, the system11may be a cloud-based system in wireless communication with the vehicle10. Although the vehicle10is shown as a sedan, it is envisioned that that vehicle10may be another type of vehicle, such as a pickup truck, a coupe, a sport utility vehicle (SUVs), a recreational vehicle (RVs), etc. The vehicle10may be an autonomous vehicle configured to drive autonomously.

The system11includes a controller34and one or more cameras40in communication with the controller34. The cameras40have a field of view large enough to capture images in front, in the rear, and to the sides of the vehicle10.

The system11further includes a controller34in communication with the cameras40. The controller34includes at least one processor44and a non-transitory computer readable storage device or media46. The processor44may be a custom-made processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller34, a semiconductor-based microprocessor (in the form of a microchip or chip set), a macroprocessor, a combination thereof, or generally a device for executing instructions. The computer readable storage device or media46may include volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processor44is powered down. The computer-readable storage device or media of the controller34may be implemented using a number of memory devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or another electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable instructions, used by the controller34in controlling the vehicle10.

The instructions may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. The instructions, when executed by the processor44, receive and process signals from the cameras40, perform logic, calculations, methods and/or algorithms for automatically controlling the components of the vehicle10, and generate control signals to the actuators to automatically control the components of the vehicle10based on the logic, calculations, methods, and/or algorithms. Although a single controller34is shown inFIG.1, the system11may include a plurality of controllers34that communicate over a suitable communication medium or a combination of communication mediums and that cooperate to process the sensor signals, perform logic, calculations, methods, and/or algorithms, and generate control signals to automatically control features of the system11. In various embodiments, one or more instructions of the controller34are embodied in the system11. The non-transitory computer readable storage device or media46includes machine-readable instructions (shown, for example, inFIG.2), that when executed by the one or more processors, cause the processors44to execute the method100(FIG.2).

The vehicle10includes a user interface23in communication with the controller34. The user interface23may be, for example, a touchscreen in the dashboard and may include, but is not limited to, an alarm, such as one or more speakers to provide an audible sound, haptic feedback in a vehicle seat or other object, one or more displays, one or more microphones and/or other devices suitable to provide a notification or alert to the vehicle user of the vehicle10. The user interface23is in electronic communication with the controller34and is configured to receive inputs by a vehicle user (e.g., a vehicle user or a vehicle passenger). For example, the user interface23may include a touch screen and/or buttons configured to receive inputs from a vehicle user. Accordingly, the vehicle controller34is configured to receive inputs from the vehicle user via the user interface23and to provide an output (i.e., an alert) to the vehicle user.

As discussed in detail below, the system11is configured to precisely and objectively evaluate an edge in an image. In the present disclosure, the term “edge” means a region in an image where there is a sharp change intensity or a sharp change in color. In the presently disclosed system11, the camera evaluation for clarity, sharpness, and edge enhancement is determined by using the contrast at a particular resolution from the object to the imager (i.e., camera), typically by using a slanted edge process that is called Modulation Transfer Function (MTF). Currently, MTF varies depending on many aspects of the camera40, such as different pixel sizes and the types of object planes that are manufactured with distance from focal imager. Currently, the minimum distance of every pixel is assumed by averaging a set of distances. However, the presently disclosed system11and method100(FIG.2) improves accuracy by ensuring that the center of every pixel is aligned to obtain the true minimum distance between the center of a pixel and the projected polynomial curve edge, thereby obtaining a more consistent result among various cameras40. As a result, the MTF accuracy is improved. The system11uses a numerical optimizer to dynamically verify the accuracy of the minimum distance from the projected slant edge on the imager to further possibly correct the correlation between the lens and the imager to a precise value. The presently disclosed method100(FIG.2) may be executed in real time, on different possible scenarios, to verify the accuracy of the camera's radiometric elements (i.e., lens assembly), especially when placed behind glass/windshield.

FIG.2is a flowchart of a method100for evaluating edge quality from imagery. The method100begins at block102. At block102, one or more cameras40of the vehicle10captures an image. The image includes image data, which is indicative of the image. The image (and therefore the image data) is sent from the camera40to the controller34of the system11. The controller34of the system11then receives the image (and therefore the image data) from the camera40of the vehicle10. Then, the method100continues to block104.

At block104, the controller34acquires one or more frames from the image captured by the camera40. Then, the method100continues to block106.

At block106, the controller34determines (e.g., calculate) the luminance of the image captured by the camera40of the vehicle10. Then, the method100proceeds to block108.

At block108, the controller34determines (e.g., calculates) the contrast of the image captured by the camera40of the vehicle10. Next, the method100continues to block110.

At block110, the controller34detects one or more edges in the image captured by the camera40of the vehicle10. In the present disclosure, the term “edge” means a region in an image where there is a sharp change intensity or a sharp change in color. To detect an edge in the image, the controller34may use a Canny edge detector or a Sobel edge detector. After detecting the edge in the image, the method100proceeds to block112.

At block112, the controller34crops the image captured by the camera40. Specifically, the controller34crops a region of interest (ROI) such that the edge previously detected is centered. As a result, the previously detected edge is centered in the cropped image. Then, the method100continues to block114.

At block114, the controller34fits a polynomial curve to match the edge detected at block110. In other words, the controller34uses a polynomial curve fitting process to fit a polynomial curve to match the edge detected at block110. As a non-limiting example, the controller34may use a polynomial curve regression, a least squares method, among others, to fit a polynomial curve or other representative function to match the edge in the image. Then, the method100continues to block116.

At block116, the controller34executes a numerical optimizer116to find the minimum distance from each pixel in the image of the edge to the polynomial curve created at block114. As a non-limiting example, the numerical optimizer may be a minimum-distance estimator using the Chi-square criterion, the Cramér-von Mises criterion, the Kolmogorov-Smirnov criterion, and Anderson-Darling criterion. For example, the numerical optimizer may be a Levenberg-Marquardt algorithm, which is also known as the damped least-squares (DLS) method. The numerical optimizer116may include substep116and substep118. At substep116, the controller34determines (e.g., calculates) the distance from each pixel in the image of the edge to the polynomial curve. This calculated distance is ∥Current Pixel Location-Polynomial Fit(Current Pixel Location)∥2. Current Pixel Location is a value assigned to the current pixel location. Polynomial Fit(Current Pixel Location) is a value of the polynomial curve at the current pixel location. Substep120represents a stopping criteria check. When the stopping criteria is achieved, an acceptable minimum distance approximation is found. Thus, if the minimum distance is not found, then the method100returns to substep118. However, if the minimum distance is found, then the method100continues to block122. The minimum distance may be found after a certain number of iterations or when the iterative minimum distances found drop more than a predetermined percentage threshold. It is envisioned that the method100may proceed to substep118directly from block112. After finding the minimum distance from the edge of the image to the polynomial curve, the method100continues to block124.

Before executing block124, the controller34, at block122, receives a calibratable sub-sampling bins. Then, the method100continues to block124. At block124, the controller34executes a pixel binning process using the calibratable sub-sampling bins to generate a plurality of super pixels. In the present disclosure, the term “pixel binning” means a process of combining adjacent pixels throughout an image, by summing or averaging their values, during or after readout. Then, the method100continues to block126.

At block126, the controller34executes a binning optimizer process126. The binning optimizer process126includes substep128and substep130. At substep128, the controller34determines (e.g., calculates) an average luminance value for each of the plurality of super pixels (i.e., bins). Then, at substep130, the controller34executes an adjusted pixel binning process using the average luminance value for each of the plurality of super pixels (i.e., bins). Thus, the binning is adjusted dynamically based on pixel's luminance value and the superimpose contrast. It is envisioned that the method100may execute substep130directly after executing block108. Then, the method100continues to block132.

At block132, the controller34determines the edge spread function of the detected edge using the plurality of super pixels. Then, the method100continues to block134. At block134, the controller34determines a line spread function of the edge using the edge spread function. The line spread function is a first derivative of the edge spread function of the edge. Then, the method100continues to block136.

At block136, the controller34determines the modulation transfer function (MTF) value of the edge in the image using the linear spread function (LSF) of the edge and indirectly using the minimum distance determined at substep120. The Fast Fourier transform of the line spread function results in the MTF value of the edge in the image.

At block138, the controller34receives predetermined threshold for the MTF. Then, at block140, the controller34compares the MTF value of the edge in the image with the predetermined threshold to determine whether the MTF value is greater than the predetermined threshold. If the MTF value is not greater than the predetermined threshold, then the method100continues to block142. At block142, the controller34commands the user interface23to provide an alert to the vehicle user. The alert is indicative that the camera40is not accurately depicting the edge. The alert may be a visual notification and/or an audible sound. If the MTF value is greater than the predetermined threshold, then the method100continues to block144.

At block144, the controller34stores the image and the image data on the non-transitory computer readable storage device or media46. Then, the method100continues to block146. At block146, the controller34sends the image data to the perception system of the vehicle10. At this point, the image data is processed by the perception system of the vehicle10.

The drawings are in simplified form and are not to precise scale. For purposes of convenience and clarity only, directional terms such as top, bottom, left, right, up, over, above, below, beneath, rear, and front, may be used with respect to the drawings. These and similar directional terms are not to be construed to limit the scope of the disclosure in any manner.