Methods and systems for detecting bright objects and/or adaptively modifying video camera exposure

Methods and systems for automatically detecting the presence or absence of a bright object in the field of view of a video camera and/or for adaptively modifying video camera exposure level. A video camera system may be configured to enter and exit an adaptive exposure modification mode upon detection of the presence of a bright object in the field of view of a video camera.

FIELD OF THE INVENTION

This invention relates generally to video cameras, and more particularly to detection of bright objects and/or modification of video camera exposure.

BACKGROUND OF THE INVENTION

In the field of video surveillance, it is often desirable to capture an image of a vehicle license plate with a video camera to allow reading of the license plate information, for example, to facilitate identification or verification of the vehicle or its occupants. Under some conditions, it is necessary to capture an image of a vehicle license plate under relatively low light conditions. Automatic exposure control during license plate capture in low-light is made difficult by the very large dynamic range of lighting that typically exist under such conditions. Specifically, car headlights and taillights that may be significantly brighter than the license plate are typically combined with a poorly illuminated background that may be many times darker. In addition, motion of the license plate relative to the camera requires an auto-exposure algorithm to react quickly and complicates the exposure process since an exposure level must be selected to minimize image blur (a particular problem in low-light when a camera's exposure time might otherwise be longer). Although auto-exposure algorithms may allow for capture of license plate images during the day, under low light conditions they tend to overexpose headlights, taillights, and license plates that have been illuminated by an external infrared (IR) or visible light source.

Video surveillance cameras are currently available that are dedicated for license plate capture. These dedicated cameras make use of multiple snapshots, each with a different exposure time, to capture images of license plates. Out of the multiple snapshots captured by the dedicated surveillance camera, the hope is that one snapshot will have the correct balance of exposure and motion blur for capture of the license plate image. These cameras are dedicated license plate image capture devices and do not allow for non-license plate image capture related surveillance. Thus, it is necessary to add other video cameras to handle more general surveillance tasks.

SUMMARY OF THE INVENTION

Disclosed herein are methods and systems for automatically detecting the presence or absence of a bright object (e.g., license plate candidate) in the field of view of a video camera and/or for adaptively modifying video camera exposure level to improve ability to capture information or other details from the bright object. Such information or other details may include, for example, alpha or numeric characteristics depicted on the object, graphic designs depicted on the object, one or more physical characteristics of the object (e.g., borders, raised features, size, etc.), one or more color/s of the object, etc.

A bright object may be any object in the field of view of a video camera which is brightly lit relative to the background of the video camera field of view. In some cases, such a bright object may not be as brightly lit as other objects in the field of view. For example, a license plate on an automobile may be brightly illuminated by an external visible or IR light source as compared to a relatively dimly lit background (e.g., night scene background) in the field of view of a video camera, but the same license plate may not be as brightly lit (in some cases significantly less brightly lit) as internally illuminated automobile headlights or taillights that are simultaneously present in the same field of view of the video camera. These bright exposure conditions may result, for example, from the relative dimness of the background. Such a bright object may have information or other detail/s which are not discernable in the displayed video due to bright exposure conditions that obscure (e.g., “wash out”) the information or other detail/s of the object.

In one embodiment of the disclosed methods and systems, a video camera system may be configured to enter and exit (respectively) an adaptive exposure modification mode upon detection of the presence of a bright object in the field of view of a video camera. In this embodiment, the adaptive exposure modification algorithm may be configured to be capable of modifying an exposure level for an object that is brightly illuminated by an external IR or visible light source in a low-light environment (e.g., night) so that the modified exposure level improves ability to capture information from the object, e.g., to improve legibility of text and/or numbers of a brightly lit license plate.

In one embodiment, a video camera system may be configured to adaptively modify exposure level of a bright object that is present in the field of view of a video camera. In one exemplary embodiment, such a bright object may be a license plate candidate, i.e., an object in the field of view of a video camera that requires modification of video exposure level in order to determine whether it is or is not a license plate.

In another embodiment, a video camera system may be configured to automatically detect the presence of a bright object (e.g., license plate candidate) in the field of view of the video camera and to enter an adaptive exposure modification mode that optimizes capture of information from the bright object (e.g., improves legibility of the license plate information under the lighting conditions). The video camera system may also be configured to automatically exit the adaptive exposure modification mode when the bright object is no longer detected in the field of view of the video camera, and/or after some other criteria is satisfied (e.g., after a given length of time operating in adaptive exposure modification mode, etc.). The disclosed adaptive exposure modification capability may be advantageously implemented to enable a single video camera to both capture information from brightly lit objects (e.g., license plates) over a wide variety of lighting conditions, as well as to perform more general surveillance tasks or duties under a wide range of lighting scenarios.

In one embodiment, the disclosed methods and systems may be implemented using a three stage adaptive exposure modification algorithm operating, for example, as part of a video camera system. In the first stage, the algorithm automatically identifies license plate candidates by analysis of characteristics of a video image, e.g., that represents a scene that is illuminated with IR or visible light in the presence of headlights, taillights, and in combination with a poorly illuminated background. Upon detection of a license plate candidate in the video image, the adaptive exposure modification algorithm modifies exposure of the video image to enhance capture of information from the license plate. In the third stage, the algorithm automatically determines when the license plate candidate has left the field of view, at which time the adaptive exposure modification algorithm exits the modification mode and no longer modifies the exposure of the video image.

In one exemplary embodiment, an adaptive exposure modification algorithm may be implemented on a video camera system and employ a histogram function to detect when a bright object event occurs. As used herein, a bright object event refers to an event in which one or more bright object/s (e.g., headlights and/or taillights in combination with a license plate) enter the field of view of the video camera system that is generating a video image having a default video exposure level. Upon detecting the bright object event, the adaptive exposure modification algorithm may be configured to automatically switch into an exposure modification mode and modify the default exposure of the video image generated by the video camera system to more properly expose the license plate as opposed to (and despite the presence of) the headlight or taillights. After the adaptive exposure modification algorithm detects that the bright object/s have left the video camera system field of view and the light within the system field of view has returned to level that existed before the occurrence of the pre-bright object event, the adaptive exposure modification algorithm may be configured to automatically exit the exposure modification mode so that it no longer modifies the default exposure of the video image.

In one respect, disclosed herein is a method for detecting bright objects in a video image, including determining the number of bright luma samples in the video image, and then determining if the number of bright luma samples in the video image is indicative of the presence of at least one bright object in the video image.

In another respect, disclose herein is a method for modifying a default exposure level of a video image, including determining the number of bright luma samples in the video image having a default exposure, and modifying the default exposure level of the video image based on a comparison of the number of bright luma samples in the video image to an exposure modification threshold number of luma samples.

In another respect, disclosed herein is a system configured to detect bright objects in a video image, the system including logic configured to determine the number of bright luma samples in the video image, and then determine if the number of bright luma samples in the video image is indicative of the presence of at least one bright object in the video image.

In another respect, disclosed herein is a system configured to modify a default exposure level of a video image, the system including logic configured to determine the number of bright luma samples in the video image having a default exposure, and modify the default exposure level of the video image based on a comparison of the number of bright luma samples in the video image to an exposure modification threshold number of luma samples.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

FIG. 1illustrates a color video camera system100(e.g., color security or surveillance camera) as it may be configured according to one exemplary embodiment of the disclosed methods and systems. In this embodiment, camera system100is configured with an optical path that includes optics in the form of lens102that is provided to collect light120received from a light source (e.g., a scene under surveillance by camera system100) and to transmit this collected light energy along the optical path as collected light stream122to an image sensor104where collected light122is sensed. Image sensor104may be any type of image sensor (e.g., CCD or CMOS) capable of sensing light energy in collected light stream122and capable of providing image signal110that includes information representative of color characteristics of light energy within collected light stream122, e.g., bayer pattern red-green-blue (“RGB”), cyan-magenta-yellow (“CMY”). Specific examples of suitable image sensors include, but are not limited to, AltaSens ProCamHD 246x and 256x series, Micron MT9x series, etc. Although a color video camera system100is described and illustrated in the exemplary embodiment ofFIG. 1, it will be understood that the disclosed methods and systems may also be implemented with a black and white video camera system.

As further illustrated inFIG. 1, one or more processors106(e.g., multiprocessor, DSP, or other suitable processor/s) may be present in camera100for implementing one or more tasks (e.g., logic, algorithms, etc.) such as image sensor signal conditioning, image processing, digital to analog conversion, etc. A video output signal130(e.g., digital video output signal, analog video output signal, etc.) is shown provided by color video camera system100. In one possible embodiment, color video camera system100may provide a digital video output signal to a video access component (e.g., stream server) for delivery as a coded video stream across an IP network medium. In another possible embodiment, color video camera system100may provide a digital video output signal suitable for delivery to a digital video recorder and/or for display on a video display device. It will be understood that these embodiments are exemplary only, and that color video camera system100may provide any other type of digital and/or analog video output signal that is suitable for transmission, display, recording, etc.

As shown inFIG. 1, a movable IR block filter114(e.g., IR filter sled) is provided that is configured so that it is capable of being selectably inserted into and retracted from the optical path between lens102and image sensor104. An actuator108(e.g., motor, solenoid, etc.) is provided to move IR block filter114from a first position (represented in solid outline by114a) that is retracted out of the optical path between lens102and image sensor104to a second inserted position (represented in dashed outline by114b) that is inserted into the optical path between lens102and image sensor104. In one exemplary embodiment, IR block filter114may be any material that is at least partially transparent to visible light (e.g., light having a wavelength from about 400 nanometers to about 770 nanometers), while at the same time being substantially opaque or substantially non-transmissive to near IR light (e.g., light having a wavelength from greater than about 770 nanometers to about 1200 nanometers).

Still referring toFIG. 1, when deployed in its second inserted position114b, IR block filter114acts to filter out or remove IR light energy from collected light stream122before it reaches image sensor104, while at the same time allowing visible light energy from collected light stream122to reach image sensor114. When deployed in its second retracted position114a, substantially all of visible and IR light energy of collected light stream122is allowed to reach image sensor104. Although not shown in this embodiment, it will be understood that other optional filter components (e.g. ultra-violet (UV) filter component, etc.) may also be present in the light path between lens102and video sensor104to filter out other wavelengths from collected light stream122. As further shown inFIG. 1, processor/s106(e.g., one or more Central Processing Units, CPUs) may be coupled to control actuator108using IR block filter control signals112to selectably position IR block filter114between first and second positions114aand114b, although IR block filter114may be selectably positioned using any other suitable combination of actuator/s and/or processor/s. In this regard, processor/s106may be configured to control actuator108using variable threshold logic and/or multiple color difference logic described further herein, e.g., implemented as algorithm/s executing on processor/s106. Further information on methodology and algorithms for controlling insertion and retraction of IR block filter114may be found in U.S. patent application Ser. No. 11/416,787 entitled “METHODS AND SYSTEMS FOR ESTIMATION OF VISIBLE LIGHT IN A LIGHT SOURCE” by Horowitz, et al. that is concurrently filed herewith on the same day as this patent application, and which is incorporated herein by reference.

In one embodiment of the disclosed methods and systems, the presence of a license plate in the field of view of a video camera system (e.g., such as color video camera system100ofFIG. 1) may be detected using an algorithm such as Equation 1 below to provide a value for Bc. In this regard, the algorithm of Equation 1 may be executing, for example, on processor/s106of the color video camera system100ofFIG. 1. In Equation 1, Bcrepresents bright luma samples (e.g., samples of black/gray/white information in a video signal) which are the number of luma samples with values larger than some threshold value τ. This threshold value τ may be selected, for example, based on the smallest luma sample value that will appear overexposed (i.e., the video sensor element corresponding to the luma sample has been exposed to so much light that all meaningful image detail is lost) when processed by the camera and then reproduced by a particular display monitor. An example of a threshold value τ for video with eight-bit luma samples (i.e., each luma sample may have a value between 0 and 255 inclusive) is 250, although greater or lesser τ values are also possible.

Bc≡∫τ∞⁢h⁡(i)⁢ⅆiEquation⁢⁢(1)
where:Bcis the number of bright luma samples,τ is the lower bound of brightness that defines an overexposed luma sample andh(i) is the luminance histogram.

FIG. 2illustrates one embodiment of a methodology200that may employ Bcof Equation 1 (e.g., using processor/s106of a color video camera system100) to detect the presence of a bright object and to modify exposure based thereupon. As shown in step202, video camera system100may be configured to generate a default exposure under normal conditions (e.g., no license plate detected in the field of view), for example, using any suitable default exposure determination logic executing on processor/s106. Such a default exposure may be a fixed exposure for a given set of default light conditions, may be a variable exposure determined by an auto-exposure algorithm (e.g., a histogram-based algorithm that measures the amount of bright and dark luma samples in a frame and adjusts exposure to achieve some prescribed balance of bright and dark samples), may be other logic and/or hardware capable of adjusting exposure, etc.

As shown inFIG. 2, methodology200enters an adaptive exposure modification algorithm210at step204, in which it is determined whether video camera system100is viewing a low-light scene. In this regard, existence of a low light scene may be determined using any methodology suitable for identifying light conditions under which exposure modification is desired or needed to enhance the ability to capture information or other details from a bright object. In one exemplary embodiment, default exposure level may be compared to a specified light threshold to determine if the default exposure level is below the specified light threshold, and a low light scene may be determined to exist if the default exposure level is below the specified light threshold. In one particular example, such a specified light threshold may be the same light threshold (e.g., gain factor threshold) at which an IR block filter114is retracted from an optical path of the camera system100. In another alternative exemplary embodiment, a low-light scene may be considered to exist when a retractable IR block filter114is in retracted position (i.e., position114b). Further information on retraction of an IR block filter based on light conditions may be found in U.S. patent application Ser. No. 11/416,787 entitled “METHODS AND SYSTEMS FOR ESTIMATION OF VISIBLE LIGHT IN A LIGHT SOURCE” by Horowitz, et al. which has been incorporated herein by reference.

If a low light scene is not determined to exist in step204, the adaptive exposure modification algorithm210exits without modifying the default exposure as shown. However, if a low light scene is determined to exist in step204, then the adaptive exposure modification algorithm210proceeds to step205where Bc, the number of bright luma samples, is determined using Equation (1), and then to step206where Inequality (2) is evaluated.
Bc≧teInequality (2)
where:Bcis the number of bright luma samples andTeis the minimum number of bright luma samples required to detect a bright event.

If Bc, the number of bright luma samples determined in step205using Equation (1), is greater than or equal to the value of Te, it indicates that a bright even has occurred, i.e., a bright object (e.g., license plate) has entered the field of view of camera system100under low light scene conditions.FIG. 5Aillustrates an occurrence of such a bright event when an automobile502is driven into a bank drive-through stall504opposite a bank teller window506under low light scene conditions, e.g., at night. In this situation, the default video camera exposure level that is suited for the low light bank drive-through scene does not allow information on license plate510to be captured. Instead, information on license plate510is overexposed and not legible. When such a bright event is detected in step206the removable IR block filter is locked in retracted position114ain step208, timer ttimeris started, and adaptive exposure modification algorithm210proceeds to step214to further evaluate the need for exposure modification.

With regard to Inequality (2), a value for Temay be determined in one exemplary embodiment, for example, by setting up a test scene typical of the intended application (e.g., the bank drive through inFIGS. 5A and 5B). In such a case, the value of Temay be set to the largest value that will reliably cause Bcto exceed Tewhen an automobile502enters the camera's field of view. In this regard, a value of Tethat is too large would result in Bcnot exceeding Tein the presence of an automobile502(i.e., resulting in a missed bright event), while a value of Tethat is too small may result in Bcexceeding Tewhen an automobile502is not present (i.e., resulting in a false alarm). It will be understood that a determined value of Tealso depends in part on the total number luma samples employed for a given application. In one exemplary system embodiment employing 1280×720 luma samples, an exemplary value of Temay be about 0.1% of the total luma samples, although lesser or greater values of Teare also possible.

Next, the value of Bcis then compared in step214to Tb, a threshold that controls the number of overexposed samples as shown in Inequality (3).
Bc≧TbInequality (3)
where:Tbis the maximum number of bright luma samples that are allowed without taking action.

With regard to Inequality (3), the value of an exposure modification threshold number of luma samples such as Tbmay be chosen at system setup time to maximize the probability of selecting the correct exposure level for a license plate. This may be done using any suitable methodology, e.g., based on empirical measurement, etc. For example, in one exemplary embodiment used to determine Tba test scene may be set up that consists of an automobile with headlights on, an IR illuminated license plate and dark background. If the value of Tbis too small, it will cause the modified exposure to darken the scene such that the license plate is underexposed (i.e., too dark to be legible), indicating that the value of Tbshould be increased to allow more bright luma samples. At some value of Tb, all luma samples associated with the headlights will be categorized as bright (e.g., overexposed) while the less-bright license plate will be correctly exposed. If however the value of Tbis too large, luma samples associated with the license plate will be categorized as bright and the plate will be unreadable due to overexposure. In this case, the value of Tbshould be decreased until step216of the modified exposure algorithm210darkens the scene enough to make the license plate legible. It will be understood that a determined value of Tbalso depends in part on the total number luma samples employed for a given application. In one exemplary system embodiment employing 1280×720 luma samples, an exemplary value of Tbmay be about 0.05% of the total luma samples, although lesser or greater values of Tbare also possible.

If in step214Bc, the number of bright luma samples determined using Equation (1), is less than Tb, then adaptive exposure modification algorithm210does not modify the default exposure, and step214proceeds to step218and220(described further below) which determine whether adaptive exposure modification algorithm210should be exited and methodology200should return to step202, or whether adaptive exposure modification algorithm210should proceed to step212for recalculation of Bcand then step214repeated as shown. However, if Bcis found in step214to be greater than or equal to the value of Tb, then in step216adaptive exposure modification algorithm210modifies the default exposure of step202to darken the scene viewed by color video camera system100until Inequality (3) is no longer satisfied.

FIG. 5Bis an illustration representing the low light drive-through scene ofFIG. 5A, after it has been darkened by adaptive exposure modification algorithm210so that information on license plate510is now legible. At the same time, the remainder of the scene is now darker. Darkening of the scene in step216may be accomplished using any suitable methodology and/or in any suitable fixed or variable size exposure increment. However, in one exemplary embodiment, adaptive exposure modification algorithm210may darken the default exposure of the scene e.g., by providing command or control signal to exposure control circuitry and/or exposure control logic executing on processor/s106or any other manner suitable for modifying the default exposure. In this regard, adaptive exposure modification algorithm210may darken the default exposure of the scene in one exemplary embodiment by a variable size exposure increment that may be a value that is proportional to Bc−Tb. Such a variable size exposure increment may be desirable in one embodiment to provide a more quickly reacting algorithm, i.e., an adaptive exposure modification algorithm that reacts more quickly than the same algorithm would react if it employed a fixed exposure increment. As shown inFIG. 2, adaptive exposure modification algorithm210continues to modify the default exposure to darken the scene viewed by color video camera system100until Inequality (3) is no longer satisfied as may be determined by steps218and220that are described further below.

Still referring toFIG. 2, color video camera system100may continue to operate according to adaptive exposure modification algorithm210until one or more termination criteria are satisfied. Such termination criteria may be selected as needed or desired based on the characteristics of a given application. For example, in one exemplary embodiment, adaptive exposure modification algorithm210may unlock IR block filter114in step222and exit when Inequality 4 is satisfied in step218.
Lc≦LeInequality (4)
where:Lcis the current amount of light energy incident on the camera andLeis the amount of light energy that was incident on the camera before the bright event was detected.

In this exemplary embodiment, Inequality (4) is used to detect when the bright object (e.g., license plate) leaves the field of view of color video camera system100as shown in step218ofFIG. 2, in which case adaptive exposure modification algorithm210unlocks IR block filter114in step222and exits to step202. However, if inequality 4 is not satisfied, an additional exemplary criteria represented by Inequality (5) may be evaluated in step220to determine if some prescribed time interval, treset, has elapsed since bright event detection in step206.
ttimer≧tresetInequality (5)
where:ttimeris a timer measured in seconds that starts when the event is detected andtresetis the amount time before the timer is reset.

When implemented in step220, the criteria described by Inequality (5) enables the adaptive exposure modification algorithm210to unlock IR block filter114in step222and exit to step202in the event that a scene brightens for a long period of time (e.g., sunrise). The value of tresetis in general application dependent, and may be selected as needed or desired based on the characteristics of a given video application. For example, for a video application involving video surveillance of a bank drive-through, a transaction at a drive-through bank teller station might be considered to average about 5 minutes. During such a transaction it is desirable to use color video camera system100to capture license plate information. Thus, for example, a timer reset value tresetmay be selected to be on the order of about 10 minutes, i.e., sufficiently long enough so that adaptive exposure modification algorithm210continues to operate during the anticipated duration of most drive-through transactions, but exits after a time length that is longer than the large majority of drive-through transactions. However, in the event that Inequality (5) is not satisfied (i.e., prescribed time interval, tresethas not elapsed since bright event detection in step206) then adaptive exposure modification algorithm210proceeds to step212where Bcis computed again, and then returns to step214where the new value of Bcis compared to Tb.

FIG. 3shows interrelated states and methodology300of a video camera system that is configured with an adaptive exposure modification algorithm according to one exemplary embodiment of the disclosed methods and systems. Specifically,FIG. 3shows how system exposure state, SEis determined. As shown inFIG. 3, a luma signal is generated in step302and histogram calculated in step304. The current system exposure state, SEis evaluated in step310and the result determines how methodology300proceeds. If the system is not operating in the modified exposure state (i.e., SE=default), then Bcis calculated in step308. Next, in step312, Bcis compared to Teto determine if Bcis greater than or equal to Te, and current IR block filter state, SIRfrom step301is evaluated to determine if it is in unfiltered (i.e., retracted) state. If both the preceding conditions do not concurrently exist, then no action is taken as shown in step316, and the system continues to operate in default exposure state (i.e., SE=default). However, if both conditions are found to concurrently exist in step312, then the IR block filter is locked in the unfiltered (i.e., retracted) position and system exposure state is changed to modified (i.e., SE=modified) as represented by step320ofFIG. 3. At the same time, a timer is started in step318to monitor the amount of time ttimerelapsed since the exposure state modification. This monitored time, ttimer, is used in step314as described further below.

Returning to step310ofFIG. 3, if the system is operating in the modified exposure state (i.e., SE=modified), then a value of Lcis calculated in step306, and then compared to a value of Lein step314. If the value of Lcis found to be greater than the value of Lein step314, and if ttimeris found to be less than tresetin step314, then no action is taken as represented by step316and the system continues to operate in the modified exposure state (i.e., SE=modified). However, if the value of Lcis found to be less than or equal to the value of Lein step314, or if ttimeris found to be greater than or equal to tresetin step314, then the IR block filter is unlocked, and system exposure state is changed to default (i.e., SE=default) as represented in state322ofFIG. 3.

FIG. 4shows various interrelated states and methodology400of a video camera system that is configured with an adaptive exposure modification algorithm according to one exemplary embodiment of the disclosed methods and systems. Specifically,FIG. 4shows how the modified exposure level XPnewis determined. As shown inFIG. 4, a luma signal is generated in step402, histogram calculated in step404, and exposure level of the camera system, XPoutis provided in step406. The current system exposure state, SEis evaluated in step408and the result determines how methodology400proceeds. If the system is not operating in the modified exposure state (i.e., SE=default), then the exposure level of the camera system, XPnewremains equal to the existing and unmodified default exposure level XPoutas represented by step414ofFIG. 4. However, if the system is operating in the modified exposure state (SE=modified), then Bcis compared to Tbin step410. If Bcis greater than or equal to Tbin step410, then the default exposure level XPoutis modified by darkening in a manner as described elsewhere herein (e.g., until Bcis no longer found to be greater than or equal to Tb) to produce an exposure level, XPnewin step416that is darkened relative to XPout. Alternatively, if Bc, is not found to be greater than or equal to Tbin step410, then the exposure level of the camera system, XPnew, remains equal to the existing and unmodified default exposure level XPoutas indicated by step412ofFIG. 4.

It will also be understood that the steps and states ofFIGS. 3,4and5are exemplary, and that fewer or additional steps and/or states may be present or performed, and/or that the indicated steps ofFIGS. 3,4and5may be performed in any alternative sequence that is suitable for either automatically detecting the presence or absence of a bright object in the field of view of a video camera and/or for adaptively modifying video camera exposure level to improve ability to capture information or other details from the bright object.