Patent Description:
Various embodiments described herein are generally directed to methods, systems, and apparatuses that facilitate detecting different types of objects in video frames. In various embodiments, a method, system and apparatus facilitate defining first and second camera parameters optimized for detecting a respective retroreflective and non-retroreflective object. A sequential series of first and second video frames are captured based on the respective first and second camera parameters, and the retroreflective and non-retroreflective object are detected in a camera scene based on the respective first and second video frames of the series.

These and other features and aspects of various embodiments may be understood in view of the following detailed discussion and accompanying drawings.

The discussion below makes reference to the following figures, wherein the same reference number may be used to identify the similar/same component in multiple figures.

The present disclosure relates generally to automated system for identifying objects, such as vehicles, license plates and the like. For example, methods, systems and apparatuses are described below that can identify both general vehicle characteristics (e.g., make, model, color) and vehicle license plates using a single camera, and can capture time resolved images for speed measurement and detection. While embodiments below are described in terms of vehicle imaging and automated license plate detection, it will be appreciated that the concepts may be applied to any imaging application where two or more objects in a scene exhibit widely different reflectivity, such that at least one of the object will tend to be underexposed or overexposed using image capture devices such as a video camera.

The embodiments herein may be usefully applied to any application where large amounts of image data are captured for automatic analysis. Such image capture may use a video camera, which is capable of capturing large numbers of images over time. For example, a traffic monitoring system may utilize a high-resolution camera video suitable for capturing multiple vehicle images simultaneously, such as across multiple lanes of traffic. A high-resolution camera may generally have lower frame rates than cameras sometimes employed in automatic license plate detection system. However, if the high-resolution camera can be adapted to read license plates as well as generally identify vehicles, significant cost savings can be realized compared to devices that are specially tailored for one application or the other. Such an arrangement may be useful for entities engaged in surveillance and detection, and who may also desire high dynamic range imaging for evidentiary purposes, e.g., to obtain quality images of the vehicle as well as the license plate.

For purposes of this disclosure, a high-resolution video camera may be a video camera having a frame resolution of more than <NUM> megapixels. For example, one common high-resolution format is <NUM> x <NUM>, which results in a rendered frame of <NUM> megapixels. The high-resolution camera may be limited to a frame of <NUM>-<NUM> frames per second or less. It will be appreciated that these specifications are a general guide, and may not account for differences such as interlacing, compression, color depth, etc., which can affect the amount of data contained in a particular frame. The camera lens may be a commercially available lens designed for digital cameras with focal lengths ranging from <NUM> to <NUM> and f-numbers ranging from f/<NUM> to f/<NUM>. While the embodiments described below need not be limited to these camera specifications, it has been found that cost-effective yet accurate identification of different types of object can be achieved using cameras with specifications meeting at least those capabilities.

In reference now to <FIG>, example images <NUM>, <NUM> are shown that are produced by an apparatus according to an example embodiment. These figures illustrate how a system that images both license plates and entire vehicles may require different image capture parameters. In <FIG>, an image <NUM> of an automobile is optimized (e.g., camera integration/exposure time, illumination pulse width, camera gain, etc.) so that a license plate of the automobile is readable. This image <NUM> may be suitable for human or machine reading of the characters on the license plate. In <FIG>, another image <NUM> of the same vehicle taken at approximately the same time but using different parameters. This image <NUM> illustrates a result of optimizing the parameters for identifying general features of the vehicle. For example, image <NUM> may facilitate human or machine recognition of make, model, year, etc. The image <NUM> may also make visible other aspects useful for vehicle identification, such as color, dents/damage, bumper stickers, occupants, tire configuration, accessories, etc..

The disclosed embodiments may utilize an imaging and (optional) illumination scheme that enables video or photographic capture of resolvable license plate images contemporaneously with capture of recognizable evidentiary images of moving vehicles. For example, camera parameters (integration times, illumination pulse widths, camera gain, color settings, resolution, etc.) can be alternated between subsequent video frames to both capture resolvable license plate images and capture properly exposed vehicle images. These subsequent frames can be captured over a period of time that is sufficient to produce both types of imagery. Later, the frames can be separated and combined with like frames (e.g., combine all license plate image frames) to form a composite image suitable for identifying objects of interest.

Using this technique, the images can be captured using relatively lower frame rates as compared to cameras sometimes used in specialized license plate imaging systems. This may allow widely-available, commercial, high-resolution cameras to be used for this task. For example, a high resolution camera may allow capture of vehicle data across multiple lanes of traffic. This can result in cost savings due to a reduced number of cameras needed to cover a given traffic area.

One reason that different camera parameters may be needed to capture different types of objects is that different vehicle body parts may have widely different specular and diffuse reflectivity properties. Generally, specular reflection refers to light that reflects at a fairly constant angle over a given area, e.g., such as from a mirrored surface. Specular reflection can be distinguished from diffuse reflection, where light may reflect at widely different angles over a given area. Examples of diffuse reflecting surfaces may include matte surfaces such as sheets of paper.

License plates have specular and diffuse reflectivity properties, and the specular component may be retroreflective as well. Retroreflection refers to reflection of light back to its source with minimal scattering, e.g., the angle that the light approaches a retroreflective surface is substantially parallel to the angle of a substantial amount of the light reflected from the surface. Objects designed for easy night viewing, such as road signs and license plates, use retroreflective surfaces. In night view situations, the light source (e.g., headlights) and receiver (e.g., driver) are in close proximity, so a retroreflective object will appear bright because a significant amount of light is reflected directly back to the light's source. The night-viewable surfaces are generally formed so that the retroreflectivity occurs over a wide range of angles between the surface and the light source/receiver. This allows the surface to be visible over a wide range of light source/receiver locations.

For a camera and illumination system that produces images such as shown in <FIG>, the camera and illuminator light source may be co-located, and the light is specularly retroreflected back to the camera from the license plate. This causes the license plate to appear much brighter than the rest of the car. This phenomenon may occur over a wide variety of view angles, due to the properties of the retroreflective surface of the license plate. Examples are shown in <FIG> of retroreflective structures of objects that may imaged by embodiments described herein.

In <FIG>, a cross sectional view shows features of a beaded specular retroreflective object <NUM>. A plastic outer layer <NUM> forms an outer viewing surface <NUM> of the object <NUM>. The outer layer <NUM> covers glass beads <NUM>, which are covered on the other side by an inner plastic layer <NUM>. The inner plastic layer <NUM> is disposed between the beads <NUM> and a mirror <NUM>. The shape of the beads <NUM> and mirror <NUM>, as well as the optical properties (e.g., index of refraction) of the beads <NUM> and layers <NUM>, <NUM>, cause an incoming beam of light <NUM> to be refracted before being reflected off mirror <NUM> at point <NUM>. The reflected beam <NUM> undergoes similar refractions while passing out through the bead <NUM> and layers <NUM>, <NUM> such that the reflected beam <NUM> exits at an angle substantially parallel to the incoming beam <NUM>. The spherical shape of the balls <NUM> and matching indentations in mirror <NUM> help ensure this parallelism between incident and reflected beams <NUM>, <NUM> over a wide range incidence angle relative to the surface <NUM>.

In <FIG>, a cross sectional view shows features of another retroreflective object <NUM>. The object <NUM> uses a plastic layer <NUM> with angled surfaces <NUM> facing an air gap <NUM>. The surfaces <NUM> are mirrored so that an incoming beam of light <NUM> is reflected by multiple surfaces <NUM>. The surfaces <NUM> are arranged so that a significant number of the reflected beams <NUM> are parallel to the incoming light <NUM>. In order to maintain this parallelism over a number of incident angles of the incoming light <NUM>, the surfaces <NUM> may be oriented at a variety of different angles. Also, as seen in the front view of <FIG>, the angles of surfaces <NUM> may be varied in three-dimensions.

Other than the license plate (and other small objects such as safety reflectors), most car body parts do not specularly retroreflect illuminator light back to the camera. This results in a large brightness difference between the license plate and vehicle as seen from a video camera. In such a case, if the license plate exposure is satisfactory, the vehicle is underexposed (e.g., as seen in <FIG>). If the vehicle exposure is satisfactory, the license plate is overexposed and/or saturated and, therefore, unreadable (e.g., as seen in <FIG>). The present embodiments may alternate different camera integration times, illumination pulse widths, camera gain, aperture settings, etc., to capture both resolvable license plate images and also capture properly exposed vehicle images.

By way of example, <FIG> illustrates a diagram for facilitating the following discussion estimating magnitudes of specular and diffuse reflection. In this simplified diagram, a light source <NUM> emits light that reflects off of a surface <NUM>. The reflected light is received by imaging device <NUM>. An angle θ represents a size of the light source <NUM>, e.g., a range of angles that light from the source <NUM> approaches a point of the surface <NUM>. For purposes of this example, the angle θ is relatively small, e.g., <NUM> degrees, or <NUM> radians, to represent a traffic monitoring scenario. In such a scenario, if the light source <NUM> was <NUM> meters away from the surface <NUM>, θ = <NUM> degrees would correspond to a light source diameter of approximately <NUM>.

If the surface <NUM> is a diffuse scattering surface, then the radiance LDS of the surface seen by the imaging device <NUM> may be estimated as shown in Equation [<NUM>] below. In contrast, if surface <NUM> is a retroreflective scattering surface, then the radiance LRS of the surface seen by the imaging device <NUM> may be estimated as shown in Equation [<NUM>] below. <MAT> <MAT>.

In both of these equations, L is the radiance of the source <NUM>. In Equation [<NUM>], and RDS is reflectance of surface <NUM>, and RDS = <NUM> is assumed for this example. If L = <NUM> W/cm<NUM>-sr, RDS = <NUM>, and θ = <NUM> degrees, the value of LDS is <NUM> mW/cm<NUM>-sr, or <NUM>/<NUM>=<NUM>% of L. In Equation [<NUM>], RRS is the coefficient of retroreflectance, and RRS = <NUM> sr-<NUM> is assumed for purposes of this example. If L = <NUM> W/cm<NUM>-sr, RRS = <NUM> sr -<NUM>, the value of LRS is <NUM> mW/cm<NUM>-sr, or <NUM>% of L.

The results show that, even for an average reflectance of <NUM>%, the effective object brightness of a retroreflective object can be many times higher than for a diffuse reflective object (greater than <NUM> times higher in this case). This effective object brightness difference explains why the license plate image exposure may be much higher than the vehicle image exposure. Referring again to the example images of <FIG>, the image <NUM> of <FIG> was taken with unity camera gain and results in a properly exposed license plate with an underexposed vehicle image. The image <NUM> in <FIG> was taken with a high camera gain and results in a properly exposed vehicle image with a saturated license plate image.

In the examples described herein, a system uses different parameters for adjacent video frames captured by a video camera. Adjacency may be desirable based on the available camera frame rates. It is anticipated that adjacent frames with different parameters, e.g., alternating frames having first and second sets of parameters applied, may provide a good image accuracy with relatively low cost hardware. In such a case, the resulting video stream contains alternating sequences of images of two or more types. The differing types of images are suitable for discerning different types of data, including a first type of image suitable for identifying license plates, and a second type of image suitable for generally vehicle identification. The differing types of images can be separated out from the stream and combined to form enhanced composite images of the first and second types.

In reference now to <FIG>, a series of timing diagrams illustrates how camera and illumination parameters are varied on a per-frame basis according to an example embodiment. In <FIG>, timelines <NUM>, <NUM> are associated with respective settings of camera integration times and illuminator pulse width per each frame. The dashed lines indicated boundaries of video frames, such as frame <NUM> which utilizes a combination of settings from both timelines <NUM>, <NUM>. In this example, the time between camera frames is constant over time. So, if the camera is set up for <NUM> frames per second, the time of each frame (including frame <NUM>) would be <NUM>/<NUM> second.

For time line <NUM>, curves <NUM> represent a first, per-frame, camera integration time (e.g., time over which the camera's light detector gathers received light to form an image) suitable for non-retroreflective objects, such as whole vehicle imaging as seen in <FIG>. Curves <NUM> represent a second, per-frame, camera integration time suitable for retroreflective objects, such as license plates as seen in <FIG>. The shorter integration time <NUM> for the license plate image capture reduces the image smear and yields a resolvable and readable license plate image. The camera integration times <NUM> may vary between <NUM> to <NUM> times larger than the integration times <NUM>. Similarly, curves <NUM> and <NUM> show respective illumination times for the two types of imaging associated with integration times <NUM>, <NUM>. The lower amount of light required from the illuminator in curves <NUM> may reduce the total power required for the illuminator, possibly reducing the cost of the illuminator. The sequential series shown by timelines <NUM>, <NUM> may continue for several frames to provide enough data to enable video tracking and speed measurement of a particular vehicle, in addition to automated license plate recognition (ALPR).

Timing diagrams in <FIG> illustrate an example camera and gain timing scheme that is used to alternately provide a properly exposed vehicle image in first frames of a sequential series, and a properly exposed license plate in second frames of the series. As with <FIG>, curves <NUM>, <NUM> on timeline <NUM> represent first and second camera integration times for the two types of images. On timeline <NUM>, curves <NUM> and <NUM> represent differing values of camera gain associated with the integration times <NUM>, <NUM>, respectively. Unlike the example integration time and illumination time values, the gain values <NUM>, <NUM> vary by magnitude within each frame instead of duration. The gain value <NUM>, <NUM> may represent, for example, an amount of bias voltage applied to a detector to maximize sensitivity over a particular range of radiance. The gain values <NUM> (e.g., for vehicle image capture) may be several times higher than gain values <NUM>, which allows the camera to digitally brighten the image of the vehicle. The lower camera gain <NUM> for the license plate image capture produces an unsaturated, properly exposed image.

It will be appreciated that systems and apparatuses may employ many variations of the implementations shown in <FIG>. For example, all three of the illustrated parameters (camera integration, illumination time, and camera gain) and any other parameter described herein (e.g., color depth, resolution, focus, etc.) may be varied together for each frame. In another variation, the sequence of two frames with different settings may be extended to three or more sequential frame patterns, where each frame in the patter has different combinations of parameters. In yet other variation, some sets of parameters may be applied more often than others, e.g., in the event that certain image types may need more frames to adequately capture desired data. For example, using <FIG> as an example, frames with settings <NUM>, <NUM> could be repeated twice between one frame with settings <NUM>, <NUM> if license plate recognition can benefit from relatively more frames than vehicle identification.

Generally, a system and apparatus can be configured to use alternating video frame camera gain to produce properly exposed resolvable license plate and vehicle video and photographic images for vehicle tracking, speed measurement, and automatic license plate recognition. The system and apparatus can also use alternating video frame integration time, illumination pulse width, camera gain, and any other camera parameter to produce properly exposed resolvable license plate and vehicle video and photographic images for vehicle tracking, speed measurement, and automatic license plate recognition.

In reference now to <FIG>, a block diagram illustrates a system and apparatus according to an example embodiment of the invention. A controller unit <NUM> may include facilities for controlling one or more cameras <NUM> and light sources <NUM>, <NUM>. For purposes of this example, both light sources <NUM>, <NUM> may be dedicated to the camera <NUM>, although at least second light source <NUM> is optional, as indicated by broken lines. The light sources <NUM>, <NUM> may illuminate over any wavelength (e.g., infrared, visible) and each light source <NUM>, <NUM> may have different characteristics (e.g., wavelength, pulse width, power, distance to target, angle from target, etc.) than the other. The controller unit <NUM> may be physically separate from or integrated with the camera <NUM> and/or light sources <NUM>, <NUM>. The controller unit <NUM> may also be coupled to additional cameras (not shown) and respective dedicated light sources (not shown) to provide functions similar to those described below regarding camera <NUM> and light sources <NUM>, <NUM>.

The controller unit <NUM> includes an imagery interface <NUM> that is coupled to a compatible imagery interface <NUM> of the camera <NUM>. Images captured via a detector <NUM> and optics <NUM> of the camera <NUM> can be communicated via the imagery interface <NUM>, e.g., as a digital video stream. A processor <NUM> of the camera <NUM> can facilitate internal processing of the imagery, e.g., capture of images via detector <NUM>, assembly of images into video frames, compression, encoding, transfer of video stream via interface <NUM>, etc. The camera's processor <NUM> is also shown coupled to a frame control interface <NUM> that facilitates selective modification of camera parameters on a frame-by-frame basis. As will be described below, the controller unit <NUM> can access frame control interface <NUM> via its own control interface <NUM> to optimize individual frames for rendering particular types of content in different frames.

The aforementioned imagery interface <NUM> and control interface <NUM> of the controller unit <NUM> may be coupled to general-purpose computer hardware such as a processor <NUM>, memory <NUM>, and a database <NUM>. The database <NUM> may be located within the controller unit <NUM> (e.g., one or more hard disk drives) and/or be remotely located, e.g., via a network. The database <NUM> may be used to store imagery obtained via imagery interface <NUM>. The imagery database <NUM> may be accessed for purposes such as post-processing, analysis, and other end-user needs, as represented by end-user application <NUM>.

Generally, the controller unit <NUM> may coordinate the operation of the camera <NUM> with one or more light sources <NUM>, <NUM> to capture a sequential series of video frames, the frames being captured using two or more parameters that result in at least two subsequent frames being optimized for a different image type. As described above, those types may at least include a specular retroreflective surface such as a license plate, and a non-retroreflective surface such as a vehicle in general. This may be accomplished by instructing the frame control interface <NUM> to alternate between two different settings of gain, integration time, and/or other parameters, between subsequent frames. These could be coordinated with different illumination times of the light sources <NUM>, <NUM>. The controller unit <NUM> may control these illumination times via the control interface <NUM>.

As noted above, if one or more of the light sources <NUM>, <NUM> are co-located with the camera <NUM>, then retroreflective surfaces such as license plates may tend to be overexposed relative to other parts of the image. By reducing illumination time for select frames, either alone or with other camera settings for the same frames, the controller unit <NUM> can ensure the captured imagery has some frames suitable for detecting license plate data, e.g., license numbers/characters. In another variation, one light source <NUM> may be co-located with the camera <NUM>, while the other is physically separated from the camera <NUM>, although still positioned to illuminate a target of interest. In such a case, the controller unit <NUM>, <NUM> could select from between the two light sources <NUM>, <NUM> depending on the type of image optimization desired.

It should be noted that the illustrated system may also be configured to operate without light sources <NUM>, <NUM>, though operating with light sources falls within the scope of the claims. For example, if there is ample daylight illumination, the camera <NUM> may be able to sufficiently resolve various types of desired imagery without other light sources. Even where ambient lighting is used, the camera <NUM> may still vary settings such as gain and integration time between subsequent frames to ensure that those frames are optimal for a particular image type.

The communications between the control interface <NUM> and frame control interface <NUM> may utilize known underlying data transport protocols and media, such as those used by networks, device-to-device communications (e.g., serial lines), fiber optics, etc. The settings may be set/updated infrequently, or in real-time. For example, the camera <NUM> may include a memory register <NUM> accessible via the frame control interface <NUM>. The register <NUM> may include a number of slots <NUM>, <NUM> that can be used to set per-frame parameters. For example, slot <NUM> may have first parameters A1, B1, C1, etc., applied to a first frame. A subsequent, second frame has second parameters A2, B2, C2, of slot <NUM> applied. After a last slot is found (e.g., some parameter is set to a null value), the sequence of slots <NUM>, <NUM> can be started over again and repeated.

In an alternate arrangement, the frame control interface <NUM> may be used to control parameters in real-time or near-real-time. For example, the controller unit <NUM> can send a trigger along with a set of parameters (e.g., slot <NUM>) for each frame. The parameters <NUM> are applied at the same time a frame is captured. This arrangement may require a high bandwidth communication between the control interfaces <NUM>, <NUM> to ensure the camera <NUM> can capture data at a desired frame rate.

The controller unit <NUM> may include features for automatic calibration and adjustment of camera parameters. Calibration/adjustment may be used to account for weather conditions, viewing conditions, equipment variation, etc. For example, two or more indicators (e.g., signs with readable characters) could be mounted in the camera view, one being specular retroreflective and the other not. An application (e.g., user application <NUM>) could over some time period (e.g., at regular intervals) attempt to read the indicators using frames optimized for the appropriate image type, e.g., based on settings values <NUM>, <NUM>. If degradation is detected (e.g., optical character recognition fails to successfully recognize a known string of symbols), then a process of adjusting the parameters <NUM>, <NUM> may be initiated from the controller unit <NUM> and/or camera <NUM>. This adjustment may also involve adjusting parameters that affect light sources <NUM>, <NUM>.

In reference now to <FIG>, a flowchart illustrates a procedure according to an example embodiment. First and second camera parameters optimized for detecting a respective retroreflective and non-retroreflective object are defined <NUM>. A sequential series of first and second video frames is captured <NUM> using the first and second camera parameters. Optionally, first and second composite images may be formed <NUM> based on combinations of the first and second video frames. Whether composite images are formed or not, the retroreflective and non-retroreflective object is detected <NUM> in a camera scene based on the respective first and second video frames of the series.

Claim 1:
A method comprising:
defining first and second parameters optimized for detecting, respectively, a retroreflective and non-retroreflective object, wherein the first and second parameters each comprise an illumination time of a light source, and wherein optimizing the first and second parameters for detecting, respectively, the retroreflective and non-retroreflective object comprises varying the first and second parameters to produce properly exposed images of, respectively, the retroreflective and non-retroreflective object in a camera scene;
capturing a sequential series of first and second adjacent video frames based on, respectively, the first and second parameters, such that a resulting video stream comprises alternating sequences of images of first and second types suitable for discerning, respectively, the retroreflective and non-retroreflective object;
separating the first and second type of images from the video stream;
combining the first type of images to form an enhanced composite image of a first type;
combining the second type of images to form an enhanced composite image of a second type; and
detecting the retroreflective object and non-retroreflective object in the camera scene based on, respectively, the first and second type of enhanced composite image.