Vehicle vision system with enhanced camera brightness control

A vision system for a vehicle includes a camera disposed at a vehicle and having a field of view exterior of the vehicle. A brightness control, responsive to processing of a frame of image data captured by the camera, interpolates towards an expected brightness for a next frame of captured image data by calculating a set of at least three brightness values using three different control coefficients derived from a previous frame of image data captured by said camera. The brightness control interpolates toward the expected brightness of the next frame of captured image data using the current expected brightness value and two of the three brightness values derived from the three control coefficients derived from the previous frame of captured image data.

FIELD OF THE INVENTION

BACKGROUND OF THE INVENTION

SUMMARY OF THE INVENTION

The present invention provides a driver assistance system or vision system or imaging system for a vehicle that utilizes one or more cameras (preferably one or more CMOS cameras) to capture image data representative of images exterior of the vehicle, and provides a brightness control that controls brightness parameters of the camera or image sensor. An image processor is operable to process image data captured by the camera. The brightness control, responsive to brightness values of a previous frame of image data, interpolates towards an expected brightness by calculating a set of at least three brightness values from the previous frame of image data with three different control coefficients, from which two are chosen in between set points (of expected or desired brightnesses) for further processing. The system repeats this process over multiple frames of captured image data, adjusting the coefficients based the previous frame and on an error between the previous frame brightness and the expected brightness, to adjust the camera brightness until it is within a threshold level from the expected brightness.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A vehicle vision system and/or driver assist system and/or object detection system and/or alert system operates to capture images exterior of the vehicle and may process the captured image data to display images and to detect objects at or near the vehicle and in the predicted path of the vehicle, such as to assist a driver of the vehicle in maneuvering the vehicle in a rearward direction. The vision system includes an image processor or image processing system that is operable to receive image data from one or more cameras and provide an output to a display device for displaying images representative of the captured image data. Optionally, the vision system may provide display, such as a rearview display or a top down or bird's eye or surround view display or the like.

Referring now to the drawings and the illustrative embodiments depicted therein, a vehicle10includes an imaging system or vision system12that includes at least one exterior viewing imaging sensor or camera, such as a rearward facing imaging sensor or camera14a(and the system may optionally include multiple exterior viewing imaging sensors or cameras, such as a forward viewing camera14bat the front (or at the windshield) of the vehicle, and a sideward/rearward viewing camera14c,14dat respective sides of the vehicle), which captures images exterior of the vehicle, with the camera having a lens for focusing images at or onto an imaging array or imaging plane or imager of the camera (FIG. 1). Optionally, a forward viewing camera may be disposed at the windshield of the vehicle and view through the windshield and forward of the vehicle, such as for a machine vision system (such as for traffic sign recognition, headlamp control, pedestrian detection, collision avoidance, lane marker detection and/or the like). The vision system12includes a control or electronic control unit (ECU) or processor18that is operable to process image data captured by the camera or cameras and may detect objects or the like and/or provide displayed images at a display device16for viewing by the driver of the vehicle (although shown inFIG. 1as being part of or incorporated in or at an interior rearview mirror assembly20of the vehicle, the control and/or the display device may be disposed elsewhere at or in the vehicle). The data transfer or signal communication from the camera to the ECU may comprise any suitable data or communication link, such as a vehicle network bus or the like of the equipped vehicle.

Conventional vehicle camera brightness controls typically employ a loop control. The brightness output is the feedback for the deviation from the desired brightness set point. The brightness can be calculated under use of a cumulative histogram, where, for example, the 80 percent bin may be the indicator of the brightness. To decrease or increase the brightness, there may be a coefficient which is a gain factor to the pixels before running the cumulative histogram. The coefficient can be a controlling parameter in a non-linear function, for example, a hyperbolic tangent function. By that the coefficient can be used as control for the brightness. Since the histogram's profile varies, the histogram cannot be expressed as a continuous mathematical function, and the single pixel's brightness is not linearly linked to the cumulative brightness result, the optimal coefficient cannot be calculated a priori, but must be estimated or assumed as best as possible in advance of each frame. A typical way is to predict the best coefficient for the next frame from the knowledge of the previous frames, so by loop (or feedback) control, is illustrated inFIG. 2.

The brightness (loop) control is typically a PID-, PD-, DI-, or PI-, or just a P-type. The parameters of the proportional, the integral and the differential control part are typically fixedly set. In case the overall dampening is below one, the system adapts to the (user desired) set point faster, but overshoots more and oscillates longer, and in the case the overall system's dampening above one, there is no or less over swing, but the desired set point or image brightness changes are reached later. Over swing, oscillation and slow brightness adaption are not desired in brightness control. The parameters of proportional-integral-derivative (PID) controls are typically set (optimized) in a way that the system dampening is close to 1, but since the control system is not continuous, there is no optimal PID setting. A PID controlled brightness control setting may be a compromise between the lackings of being slow or over swinging. A brightness always varies (needs to be adapted to) when the light condition in the capture image changes and the dynamic range of the camera, display and/or processing system is not wide enough.

Brightness Control in Practice:

Brightness control approaches mainly fall into 2 categories:(1) Within the image sensor. The brightness parameters, such as target brightness, integration time and/or global gain, are sent via I2C from the control unit to the image sensor. Then the image sensor updates those parameters for next frames and then the brightness has new values. A benefit of this approach is that the image sensor takes over the brightness adaption natively. However, I2C communication is slow and the reaction for the parameter change is slow.(2) After the image sensor. The image sensor has the constant parameters related to brightness. The control unit processes the raw image and outputs a new image with expected brightness. A benefit of this approach is that the image sensor keeps in the same state and the user can elaborate specific brightness control strategy. However, additional effort may be needed for the control unit design.

For the second approach, PID control is popular, such as shown inFIG. 3. PID is also available for the first approach.

Instead of using an comparably slow PID control for adapting the output brightness to a varying input brightness, the camera brightness control of the present invention interpolates towards the set point (expected brightness Bexp) by calculating a set of three (or five or optionally more) brightnesses from three (or five or optionally more) different control coefficients, from which two are chosen in between the set points, for further processing (seeFIG. 6). This may be done in a control of category two (after the image sensor).

For example, for every current frame of image or image data, there are two hidden frames running parallel (instantaneously) with it. These three frames of image data have their own brightness coefficients and brightness result. Due to the nonlinearity between the coefficient and the brightness (seeFIGS. 4 and 5in comparison) and the discontinuous behavior, this solution gives a more stable and faster brightness tracing without overshooting (seeFIGS. 11 and 12) compared to a PID controlled brightness control. Switching between the brightness coefficients according the present invention essentially equates to a switching between I (Integral) coefficients. Making the coefficient a function of the error makes the I-component adaptive to the deviation. The adaption may be proportional, exponential or any other function or may be a discontinuous relationship. By that the control of the present invention is an I-type with optional deviation (or error) adaptive I-parameter (or coefficient). The varying I parameter determines the slope at which the actual brightness value conveys towards the expected brightness value (set point).

Note that inFIG. 7, ƒ(e) is a function of the current difference (or error) e (as inFIG. 3), which means that, for larger brightness variation between frames, the trying scope ƒ(e) will also be larger, and vice versa. The function ƒ(e) can be linear, non-linear (such as, for example, an exponential curve, a saturation curve, hyperbolic, or a polygon) or discontinuous. In practice, it may be sufficient to set this function as a constant to simplify the implementation, for example, ƒ(e)=Δ (an according slope diagram is shown inFIG. 11).FIG. 12shows an example of a trace property of an algorithm, where the function of the difference “e” (for error) is proportional (and continuous) (ƒ(e)=k*e; with k as a constant).

The present invention provides enhanced fast online brightness control through coefficient interpolation (seeFIGS. 7 and 8):

under use of the minimal brightness deviation e:
min(e)=min(|B(Ci)−Bexp|),B−≤B≤B+.

Image brightness (B) is controlled by a coefficient (C), in the solution of the present invention (seeFIG. 9), by a knee-point parameter for the global brightness adjustment. The coefficient has good correlation (for example, tan h) to the image brightness and does not impact the original image sensor behavior. This means that the coefficient is calculated in a shorter time period and can be used at once for the next image.

Conventionally, a brightness control requirement may be fulfilled by PID control. But for real-time purposes, the PID introduces complexity of parameter selection. The PID control has at least 3 parameters to adjust and it is difficult to find an optimal combination for all lighting conditions.

The system or control of the present invention takes advantage of field-programmable gate array (FPGA) parallel processing on the original image and has 3 versions of brightness values out of 3 different previous coefficients simultaneously. The 3 coefficients are chosen as the previous coefficient (Coefi-1), a larger coefficient (+) and a smaller coefficient (−), so that there are correspondingly 3 brightness value results (B, B+and B−). The next coefficient (Coefi) is calculated by interpolation among these 3 coefficients with respect to smallest error to the 3 brightness values. Since the FPGA cannot hold the current frame in memory, the new coefficient may be used for the next frame's brightness determination. Optionally, with very fast systems or systems with long frame pause time and a frame memory, the brightness of the current frame may be calculated out of the new found coefficient instantaneously.

The interpolation works as in the formula below, whereBdenotes expected brightness and ε the dead-zone. In the strip of ∥ε∥ the brightness Coefiwill not change.

A dead-zone ∥ε∥ is needed to have a stable video (nun pumping or flickering). The dead zone ∥ε∥ may be a tolerance band above and below the error e (seeFIGS. 11-14).

In case the calculated coefficient correction (as a function of the current difference or error e) is within the dead zone boundary, the correction will be ignored so the new brightness coefficient will be identical to the former, seeFIG. 14. Optionally, inside the dead-zone, an accumulative deviation may be used for a fine tuning.

FIG. 11shows a trace property of this algorithm, where the actual brightness steps are proportional to the Δ value (ƒ(e)=Δ). On a brightness jump stimuli the control is ramping towards the set point in identical step width. A litte delta remains when the actual value is close to the set point, hitting the dead zone.

When an algorithm according the invention uses a function of the difference e which is linear such as being ƒ(e)=k*e, with k being a scaling factor coefficient or non-linear such as being ƒ(e)=2k*e, with k being a scaling factor coefficient in the exponent the convergence to the actual brightness may be significantly faster.

FIG. 12shows a trace property of an algorithm, where the function of the difference e is proportional (and continuous), such as being ƒ(e)=k*e, with k being a scaling factor coefficient. On a brightness jump stimuli the control is ramping towards the set point in larger steps at the beginning and smaller steps at the end. A litte delta remains when the actual value is close to the set point, hitting the dead zone. No overshooting appears. The factors may be chosen properly. Attention should be paid with this variable correction algorithm that the acceleration factor is not too large to get a stable brightness tracing (swinging) (seeFIG. 13). For the fastest control setting, the factor may always be aligned with ∥ε∥ so that the ∥ε∥ band may be reached past one over swing in maximum.

Optionally, the coefficient may be limited to a maximal value, shown in the graph ofFIG. 15. A benefit of this solution is a better and faster convergence together with a high stability. This approach avoids over-tuning even under quickly alternating lighting conditions. The computational operations are limited to multiplications and summations easy to implement in FPGAs.

With extreme lighting conditions (too bright and too dark), the control module can change the global gain or integration time of the image sensor in very low possibility.

This brightness control can also target on a sub region, such as a region of interest (ROI).

Optionally, the system of the present invention may incorporate image noise filtering methods and algorithms such as by utilizing aspects of the vision systems described in U.S. Publication No. US-2015-0042806, which is hereby incorporated herein by reference in its entirety.

Optionally, the system of the present invention may incorporate enhanced low light capability methods and algorithms such as by utilizing aspects of the vision systems described in U.S. Publication No. US-2014-0354811, which is hereby incorporated herein by reference in its entirety.

Optionally, the system of the present invention may incorporate shading correction methods and algorithms such as by utilizing aspects of the vision systems described in U.S. provisional application Ser. No. 62/448,091, filed Jan. 19, 2017, which is hereby incorporated herein by reference in its entirety.

Optionally, the system of the present invention may incorporate test methods and devices such as by utilizing aspects of the vision systems described in U.S. provisional application Ser. No. 62/486,072, filed Apr. 17, 2017, which is hereby incorporated herein by reference in its entirety.

The camera or sensor may comprise any suitable camera or sensor. Optionally, the camera may comprise a “smart camera” that includes the imaging sensor array and associated circuitry and image processing circuitry and electrical connectors and the like as part of a camera module, such as by utilizing aspects of the vision systems described in International Publication Nos. WO 2013/081984 and/or WO 2013/081985, which are hereby incorporated herein by reference in their entireties.

Optionally, the vision system may include a display for displaying images captured by one or more of the imaging sensors for viewing by the driver of the vehicle while the driver is normally operating the vehicle. Optionally, for example, the vision system may include a video display device, such as by utilizing aspects of the video display systems described in U.S. Pat. Nos. 5,530,240; 6,329,925; 7,855,755; 7,626,749; 7,581,859; 7,446,650; 7,338,177; 7,274,501; 7,255,451; 7,195,381; 7,184,190; 5,668,663; 5,724,187; 6,690,268; 7,370,983; 7,329,013; 7,308,341; 7,289,037; 7,249,860; 7,004,593; 4,546,551; 5,699,044; 4,953,305; 5,576,687; 5,632,092; 5,677,851; 5,708,410; 5,737,226; 5,802,727; 5,878,370; 6,087,953; 6,173,508; 6,222,460; 6,513,252 and/or 6,642,851, and/or U.S. Publication Nos. US-2012-0162427; US-2006-0050018 and/or US-2006-0061008, which are all hereby incorporated herein by reference in their entireties. Optionally, the vision system (utilizing the forward facing camera and a rearward facing camera and other cameras disposed at the vehicle with exterior fields of view) may be part of or may provide a display of a top-down view or birds-eye view system of the vehicle or a surround view at the vehicle, such as by utilizing aspects of the vision systems described in International Publication Nos. WO 2010/099416; WO 2011/028686; WO 2012/075250; WO 2013/019795; WO 2012/075250; WO 2012/145822; WO 2013/081985; WO 2013/086249 and/or WO 2013/109869, and/or U.S. Publication No. US-2012-0162427, which are hereby incorporated herein by reference in their entireties.