Method and apparatus for detecting object movement within an image sequence

Method and apparatus for processing a sequence of images to detect object movement within the sequence. Specifically, the method comprises the steps of: (a) supplying a sequence of image frames; (b) initializing a reference image that contains image information regarding stationary objects within a scene represented by the sequence of images; (c) supplying a next image frame which temporally follows the sequence of image frames; (d) comparing the next image to the reference image to produce a motion image representing motion information regarding movement of objects within the scene; (e) updating the reference image with information within the next image that represents stationary objects within the scene; and (f) repeating steps (c), (d), and (e) for each next image supplied. The method is implemented by image processing apparatus. A specific embodiment of the method and apparatus is a traffic monitoring system that identifies vehicles in varying illumination levels and eliminates erroneous identification of non-physical movement in the scene such as shadows and headlight reflections.

BACKGROUND OF THE INVENTION 
1. Field of the Invention 
The invention relates to a vehicular traffic monitoring system and, more 
particularly, to such a system that digitally processes pixels of 
successive image frames derived from a video camera viewing road traffic. 
2. Description of the Prior Art 
Various types of traffic monitoring systems are known in the prior art and 
examples thereof are respectively disclosed in U.S. Pat. Nos. 4,433,325, 
4,847,772, 5,161,107 and 5,313,295. However, there is a need for a more 
robust traffic monitoring system that is computationally efficient and yet 
is relatively inexpensive to implement. 
Further, the present invention makes use of pyramid teachings disclosed in 
U.S. Pat. No. 4,692,806, which issued to Anderson et al. on September 8, 
and image flow teachings disclosed in the article "Hierarchical 
Model-Based Motion Estimation" by Bergen et al., appearing in the 
Proceedings of the European Conference on Computer Vision, 
Springer-Verlag, 1992. Both of these teachings are incorporated herein by 
reference. 
SUMMARY OF THE INVENTION 
The invention relates to an improvement in digital image processing means 
of a vehicular traffic monitoring system that includes a video camera 
having a given field of view for recording successive image frames of road 
traffic within its field of view. The digital image processing means, is 
responsive to pixel information defined by each of the successive image 
frames. 
Specifically, the digital image processing means comprises first means 
responsive to an initial train of the successive image frames for deriving 
a stored initial reference image defining only stationary objects within 
the field of view and thereafter updating the stored initial reference 
image with a reference image derived from an image frame recorded later 
than the initial train, with each pixel's digital amplitude level of each 
of the reference images being determined by illumination conditions 
existing when the initial train and when the later recorded frame were 
recorded; second means for modifying each pixel's digital amplitude level 
of one of a current image frame and the stored reference image then being 
stored to make their corresponding pixels defining stationary objects 
substantially equal to one another; third means responsive to the digital 
amplitude-level difference between corresponding pixels of each of 
successively occurring ones of the successive image frames and the then 
stored reference image for deriving successive images defining only moving 
objects within the field of view; fourth means for discriminating between 
those moving objects that remain substantially fixed in position with 
respect to one another in each of the successively-occurring ones of the 
successively-occurring images and those moving objects that substantially 
change in position with respect to one another in each of the 
successively-occurring ones of the successively-occurring images; and 
fifth means responsive to the variance of the digital amplitude levels of 
the pixels of those ones of the objects that remain substantially fixed in 
position with respect to one another for distinguishing and then 
eliminating those ones of the moving objects that remain substantially 
fixed in position with respect to one another that define non-physical 
moving objects, such as shadows and headlight reflections cast by physical 
moving objects, from the moving objects that remain substantially fixed in 
position with respect to one another that define the physical moving 
objects.

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The present invention comprises at least one video camera for deriving 
successive image frames of road traffic and a traffic-monitoring image 
processor for digitally processing the pixels of the successive image 
frames. As shown in FIG. 1a, the output of video camera 100 may be 
directly applied as an input to traffic-monitoring image processor 102 for 
digitally processing the pixels of the successive image frames in real 
time. Alternatively, as shown in FIG. 1b, the output of video camera 100 
may be first recorded by VCR 104 and then, at a later time, the pixels of 
the successive image frames may be readout of the VCR and applied as an 
input to traffic-monitoring image processor 102 for digitally processing 
the pixels of the successive image frames. 
Video camera 100, which may be CCD camera or an IR camera which is mounted 
at a given height over a roadway and which has a given field of view of a 
given length segment of the roadway. As shown in FIGS. 2a and 2b, video 
camera 100, by way of example, may be mounted 30 feet above the roadway 
and have a 62.degree. field of view sufficient to view a 60 foot width (5 
lanes) of a length segment of the roadway extending from 50 feet to 300 
feet with respect to the projection of the position of video camera 100 on 
the roadway. FIG. 2c shows that video camera 100 derives a 640.times.480 
pixel image of the portion of the roadway within its field of view. For 
illustrative purposes, vehicular traffic normally present on the length 
segment of the roadway has been omitted from the FIG. 2c image. 
In a designed vehicular traffic monitoring system, video camera 100 was one 
of a group of four time-divided cameras each of which operated at a frame 
rate of 7.5 frames per second. 
A principal purpose of the present invention is to be able to provide a 
computationally-efficient digital traffic-monitoring image processor that 
is capable of more accurately detecting, counting and tracking vehicular 
traffic traveling over the viewed given length segment of the roadway than 
was heretofore possible. For instance, consider the following four factors 
which tend to result in detecting, and tracking errors or in decreasing 
computational efficiency: 
1. Low Contrast 
A vehicle must be detected based on its contrast relative to the background 
road surface. This contrast can be low when the vehicle has a reflected 
light intensity similar to that of the road. Detection errors are most 
likely under low light conditions, and on gray, overcast days. The system 
may then miss some vehicles, or, if the threshold criteria for detection 
are low, the system may mistake some background patterns, such as road 
markings, as vehicles. 
2. Shadows and Headlight Reflections 
At certain times of day vehicles will cast shadows or cause headlight 
reflections that may cross neighboring lanes. Such shadows or headlight 
reflections will often have greater contrast than the vehicles themselves. 
Prior art type traffic monitoring systems may then interpret shadows as 
additional vehicles, resulting in an over count of traffic flow. Shadows 
of large vehicles, such as trucks, may completely overlap smaller cars or 
motor cycles, and result in the overshadowed vehicles not being counted. 
Shadows may also be cast by objects that are not within the roadway, such 
as trees, building, and clouds. And they can be cast by vehicles going the 
other direction on another roadway. Again, such shadows may be mistaken as 
additional vehicles. 
3. Camera Sway 
A camera that is mounted on a utility pole may move as the pole sways in a 
wind. A camera mounted on a highway bridge may vibrate when trucks pass 
over the bridge. In either case camera motion results in image motion and 
that cause detection and tracking errors. For example, camera sway becomes 
a problem if it causes the detection process to confuse one road lane with 
another, or if it causes a stationary vehicle to appear to move. 
4. Computational Efficiency 
Since vehicle travel is confined to lanes and normal travel direction is 
one dimensional along the length of a lane, it is computationally 
inefficient to employ two-dimensional image processing in detecting and 
tracking vehicular traffic. 
The present invention is directed to a digital traffic-monitoring image 
processor that includes means for overcoming one or more of these four 
problems. 
Referring to FIG. 3, there is shown a functional block diagram of a 
preferred embodiment of a preprocessor portion digital traffic-monitoring 
image processor 102. Shown in FIG. 3 are analog-to-digital (A/D) converter 
300, pyramid means 302, stabilization means 304, reference image 
derivation and updating means 306, frame store 308, reference image 
modifying means 310 and subtractor 312. 
The analog video signal input from camera 100 or VCR 104, after being 
digitized by A/D 300, may be decomposed into a specified number of 
Gaussian pyramid levels by pyramid means 302 for reducing pixel density 
and image resolution. Pyramid means 302 is not essential, since the 
vehicular traffic system could be operated at the resolution of the 
640.times.480 pixel density of video camera 100. However, because this 
resolution is higher than is needed downstream for the present vehicular 
traffic system, the use of pyramid means 302 increases the system's 
computational efficiency. Not all levels of the pyramid must be used in 
each computation. Further, not all levels of the pyramid need be stored 
between computations, as higher levels can always be computed from lower 
ones. However, for illustrative purposes it is assumed that all of the 
specified number of Gaussian pyramid levels are available for each of the 
downstream computations discussed below. 
The first of these downstream computations is performed by stabilization 
means 304. Stabilization means 304 employs electronic image stabilization 
to compensate for the problem of camera sway, in which movement may be 
induced by wind or a passing truck. Camera motion causes pixels in the 
image to move. Prior art vehicular traffic systems that do not compensate 
for camera motion will produce false positive detections if the camera 
moves so that the image of a surface marking or a car in an adjacent lane 
overlaps a detection zone. Stabilization means 304 compensates for image 
translation from frame to frame that is due to camera rotation about an 
axis perpendicular to the direction of gaze. The compensation is achieved 
by shifting the current image an integer number of rows and columns so 
that, despite camera sway, it remains fixed in alignment to within one 
pixel with a reference image derived by means 306 and stored within frame 
store 308. The required shift is determine by locating two known landmark 
features in each frame. This is done via a matched filter. 
The problem of low contrast is overcome by the cooperative operation of 
reference image derivation and updating means 306, frame store 308 and 
reference image modifying means 310. Means 306 generates an original 
reference image r.sub.0 simply by blurring the first-occurring image frame 
i.sub.0 applied as an input thereto from means 304 with a large Gaussian 
filter (so that reference image r.sub.0 may comprise a higher pyramid 
level), and then reference image r.sub.0 is stored in frame store 308. 
Following this, the image stored in frame store 308 is updated during a 
first initialization phase by means 306. More specifically, means 306 
performs a recursive temporal filtering operation on each corresponding 
pixel of the first few image frames of successive stabilized image frames 
applied as an input thereto from means 304 with the additional constraint 
that if the difference between the reference image and the current image 
is too large, the reference image is not updated at that pixel. Put 
mathematically, 
##EQU1## 
where r.sub.t represents the reference image after frame t, and i.sub.t 
represents the t'th frame of the input image frame sequence from means 
304. The constant .gamma. determines the "responsiveness" of the 
construction process. 
The "responsiveness" setting of .gamma. must be sufficiently slow to keep 
transitory objects, such as moving vehicles or even vehicles that may be 
temporarily stopped by a traffic jam, out of the reference image, so that, 
at the end of the first few input image frames to means 306 which comprise 
the first initialization phase, the stored reference image in frame store 
308 will comprise only the stationary background objects being viewed by 
camera 100. Such a "responsiveness" setting of .gamma. is incapable of 
adjusting r.sub.t quickly enough to add illumination changes (such as 
those due to a passing cloud or the auto-iris on camera 100) to the 
reference image. This problem is solved at the end of the initialization 
phase by the cooperative updating operation of reference image modifying 
means 310 (which comprises an illumination/AGC compensator) with that of 
means 306 and frame store 308. Specifically, when the initialization phase 
is completed, it is replaced by a second normal operating phase which 
operates in accordance with the following equation 2 (rather than the 
above equation 1): 
##EQU2## 
where k.sub.t and c.sub.t, are the estimated gain and offset between the 
reference image r.sub.t and the current image i.sub.t computed by means 
310. Means 310 computes this gain and offset by plotting a cloud of points 
in a 2D space in which the x-axis represents gray-level intensity in the 
reference image, and the y-axis represents gray-level intensity in the 
current image, and fitting a line to this cloud. The cloud is the set of 
points (r.sub.t-1 (x,y),i.sub.t (x,y)) for all image positions x,y. This 
approach will work using any method for computing the gain and offset 
representing illumination change. For example, the gain might be estimated 
by comparing the histograms of the current image and the reference image. 
Also, the specific update rules need not use an absolute threshold D as 
described above. Instead, the update could be weighted by any function of 
.vertline.i.sub.t (x,y)-r.sub.t-1,(x,y).vertline.. 
The above approach allows fast illumination changes to be added to the 
reference image while preventing transitory objects from being added. It 
does so by giving the cooperative means the flexibility to decide whether 
the new reference image pixel values should be computed as a function of 
pixel values in the current image or whether they should be computed 
simply by applying a gain and offset to the current reference image. By 
applying a gain and offset to the current reference image the illumination 
change can be simulated without running the risk of allowing transitory 
objects to appear in the reference image. 
The result is that the amplitude of the stationary background manifesting 
pixels of the illumination-compensated current image appearing at the 
output of means 310 (which includes both stationary background manifesting 
pixels and moving object (i.e., vehicular traffic)) will always be 
substantially equal to the amplitude of the stationary background 
manifesting pixels of the reference image (which includes solely 
stationary background manifesting pixels) appearing at the output of frame 
store 308. Therefore, subtractor 312, which computes the difference 
between the amplitudes of corresponding pixels applied as inputs thereto 
from means 310 and 304, derives an output made up of significantly-valued 
pixels that manifest solely moving object (i.e., vehicular traffic) in 
each one of successive 2D image frames. The output of subtractor 312 is 
forwarded to the detection and tracking portion of traffic-monitoring 
image processor 102 shown in FIG. 4. 
Referring to FIG. 4, there is shown 2D/1D converter 400, vehicle fragment 
detector 402, image-flow estimator 404, single frame delay 406, 
pixel-amplitude squaring means 408, vehicle hypothesis generator 410 and 
shadow and reflected headlight filter 412. 
2D/1D converter 400 operates to convert 2D image information received from 
FIG. 3 that is applied as a first input thereto into 1D image information 
in accordance with user control information applied as a second input 
thereto. In this regard, reference is made to FIGS. 5 and 5a. FIG. 5 shows 
an image frame 500 derived by camera 100 of straight, 5-lane roadway 502 
with cars 504-1 and 504-2 traveling on the second lane 506 from the left. 
Cars 504-1 and 504-2 are shown situated within an image zone 508 
delineated by the aforesaid user control information applied as a second 
input to converter 400. By integrating horizontally the amplitudes of the 
pixels across image zone and then subsampling the vertically oriented 
integrated pixel amplitudes along the center of zone 508, 1D strip 510 is 
computed by converter 400. The roadway need not be straight. As shown in 
FIG. 5a, curved roadway lane 512 includes zone 514 defined by 
user-delineated lane boundaries 516 which permits the computation of 
medial strip 518 by converter 400. In both FIGS. 5 and 5a, the user may 
employ lane-defining stripes that may be present in the image as landmarks 
for help in defining the user-delineated lane boundaries. 
More specifically, computation by converter 400 involves employing each of 
pixel positions (x, y) to define integration windows. For example, such a 
window might be either (a) all image pixels on row y that are within the 
delineated lane bounds, (b) all image pixels on column x that are within 
the delineated lane bounds, or (c) all image pixels on a line 
perpendicular to the tangent of the medial strip at position (x, y). Other 
types of integration windows not described here may also be used. 
The 1D output from converter 400 is applied as an input to detector 402, 
estimator 404 and single frame delay 406, and through means 408 to filter 
408. While the respective detection, tracking and filtering functions 
performed by these elements are independent of whether they operate on 1D 
or 2D signals, 1D operation is to be preferred because it significantly 
reduces computational requirements. Therefore, the presence of converter 
400, while desirable, is not essential to the performance of these 
detection, tracking and filtering functions. In the following discussion, 
it is assumed that converter 400 is present 
Detector 402 preferably utilizes a multi-level pyramid to provide a 
coarse-to-fine operation to detect the presence and spatial location of 
vehicle fragments in the 1D strip of successive image frames received from 
FIG. 3. A fragment is defined as a group of significantly-valued pixels at 
any pyramid level that are connected to one another. Detector 402 is tuned 
to maximize the chances that each vehicle will give rise to a single 
fragment. However, in practice this is impossible to achieve; each vehicle 
gives rise to multiple fragments (such as separate fragments corresponding 
to the hood, roof and headlights of the same vehicle). Further, pixels of 
more than one vehicle may be connected into a single fragment. 
One technique for object detection at each strip pixel position is to 
compute a histogram of the image intensity values within the integration 
window centered at that pixel position. Based on attributes of this 
histogram (e.g., the number or percentage of pixels over some threshold 
value or values), classify that strip pixel as either "detection" or 
"background". By performing this operation at each strip pixel, one can 
construct a one-dimensional array that contains, for each pixel position, 
the "detection" or "background" label. By performing connected component 
analysis within this array, adjacent "detection" pixels can be grouped 
into "fragments". 
Image-flow estimator 404 in cooperation with delay 406, which employs the 
teachings of the aforesaid Bergen et al. article, to permit objects to be 
tracked over time. Briefly, in this case, this involves, at each pixel 
position, computing and storing the average value contained within the 
integration window. By performing this operation at each strip pixel, a 
one-dimensional array of average brightness values is constructed. Given 
two corresponding arrays for images taken at times t-1 and t, the 
one-dimensional image "flow" that maps pixels in one array to the other is 
computed. This can be computed via one-dimensional least-squares 
minimization or one-dimensional patchwise correlation. This flow 
information can be used to track objects between each pair of successive 
image frames. 
The respective outputs of detector 402 and estimator 404 are applied as 
inputs to vehicle hypothesis generator 410. Nearby fragments are grouped 
together as part of the same object (i.e., vehicle) if they move in 
similar ways or are sufficiently close together. If the positions of 
multiple fragments remain substantially fixed with respect to one another 
in each of a train of successive frames, they are assumed to indicate only 
a single vehicle. However, if the positions of the fragments change from 
frame to frame, they are assumed to indicate separate vehicles. Further, 
if a single fragment of in one frame breaks up into multiple fragments or 
significantly stretches out longitudinally in shape from one frame to 
another, they are also assumed to indicate separate vehicles. 
At night, the presence of a vehicle may be indicated only by its 
headlights. Headlights tend to produce headlight reflections on the road. 
Lighting conditions on the road during both day and night tend to cause 
vehicle shadows on the road. Both such shadows and headlight reflections 
on the road result in producing detected fragments that will appear to 
generator 410 as additional vehicles, thereby creating false positive 
error in the output from generator 410. Shadow and reflected headlight 
filter 412, which discriminates between fragments that produced by valid 
vehicles and those produced by shadows and reflected headlights, 
eliminates such false positive error. 
The output from pixel-amplitude squaring means 408 manifests the relative 
energy in each pyramid-level pixel of the strip output of each of 
successive image frames from converter 400. Filter 412 discriminates 
between fragments that produced by valid vehicles and those produced by 
shadows and reflected headlights based on an analysis of the relative 
amplitudes of these energy-manifesting pixels from means 408. The fact 
that the variance in energy pixel amplitude (pixel brightness) of shadow 
and reflected headlight fragments is significantly less than that of valid 
vehicle fragments can be used as a discriminant. 
Another way of filtering, not shown in FIG. 4, is to employ converter 400 
for discriminating between objects and shadows using the 
background-adjusted reference image. At each pixel position, the following 
information is computed over the integration window: 
(a) the number of pixels with brightness value greater than some threshold 
p, over all image pixels within the integration window 
(b) the maximum absolute value, over all image pixels within the 
integration window 
(c) the number of adjacent pixels (x.sub.1, y.sub.1, and (x.sub.2, y.sub.2) 
within the integration window whose absolute difference, 
.vertline.I(x.sub.1, y.sub.1)-I(x.sub.2, y.sub.2).vertline., exceeded a 
threshold value. 
Fragments that have been extracted as described previously can be 
classified as object or shadow based on these or other properties. For 
example, if the value of measure (a), summed over all strip pixels within 
the fragment, exceeds some threshold, then the fragment cannot be a shadow 
(since shadows would never have positive brightness values in the images 
applied to converter 400 from FIG. 3. A similar summation using measure 
(c) provides another test measuring the amount of texture within the 
fragment, which can also be thresholded to determine whether a fragment is 
an object or a shadow. While the input to filter 412 defines all 
hypothesized vehicle locations, the output therefrom defines only verified 
vehicle locations. The output from filter 412 is forwarded to utilization 
means (not shown) which may perform such functions as counting the number 
of vehicles and computing their velocity and length. 
Vehicle fragment detector 402, image-flow eatimator 404, and vehicle 
hypothesis generator 410 may use pre-determined camera calibration 
information in their operation. Further, each of the various techniques of 
the present invention described above may also be employed to advantage in 
other types of imaging systems from the vehicular traffic monitoring 
system disclosed herein.