Patent ID: 12236617

The foregoing and/or other aspects will become apparent from the following detailed description when considered in conjunction with the accompanying drawing figures.

DETAILED DESCRIPTION

Reference will now be made in detail to features of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. The features are described below to explain the present invention by referring to the figures.

By way of introduction, aspects of the present invention are directed to a system and method for accurately measuring the speeds of vehicles along a stretch of road. A planar road model may be assumed. The present method as disclosed herein may be used to retrofit, with only a software upgrade, an existing camera already in use for license plate recognition, by way of example. Speed measurements, according to embodiments of the present invention may be used to monitor local traffic and/or trigger an issuance of a citation for a speeding violation.

Once setup, the system may be used to measure respective speeds of multiple vehicles within the camera's field of view (FOV), simultaneously including vehicles travelling in opposite directions, to and from the camera. The system and method as disclosed herein includes two stages of operation.

(1) An initial calibration stage during which images are captured, i.e., video, of a sample of vehicles travelling at known speeds.

(2) A measurement stage where the system may operate continuously, measuring and logging vehicle speeds in real-time.

Referring now to the drawings, reference is now made toFIG.1which illustrates camera or pinhole projection which relates a point P(X,Y,Z) in world space Cartesian coordinates to a point p(x,y) image coordinates on image plane8where X is the horizontal Cartesian coordinate in world space, Y is the vertical Cartesian coordinate in world space and Z is the direction along the optical axis of the camera. The origin O of camera projection is at the pinhole, image plane8is in reality behind the origin at focal length f with the image inverted. Image plane8is shown in the projection ofFIG.1in a symmetric position with a non-inverted image in front of origin O at a distance focal length f so the centre of the image plane8is at world space coordinates (0,0,f). The following equations approximate the relation between image coordinates x,y and world space coordinates X,Y,Z assuming camera or pinhole projection:

x⁡(t)=f⁢X⁡(t)Z⁡(t)⁢y⁡(t)=f⁢Y⁡(t)Z⁡(t)

Reference is now also made toFIG.2, which illustrates schematically a system20, according to embodiments of the present invention, a view from above of a road environment. System20illustrates fixed in a road environment, a camera2with field of view (FOV) with the optical axis Z of camera2parallel to the direction of motion of vehicles18. Using the camera coordinate system as shown inFIG.1, the motion of a vehicle18is modelled as starting from (X0, Y0, 0), a point in a plane perpendicular to the camera optical axis, and moving along a straight trajectory in three dimensions, toward a point (X(t), Y(t), Z(t)). The vehicle position is imaged on image plane8as being at (x(t), y(t)). A processor4is shown connected to camera2configured to capture image frames from the road environment.

Reference is now also made toFIG.2Awhich schematically illustrates image plane8imaging the road environment as shown inFIG.2. Vanishing point (x0, y0) is shown, the point on image plane8to which the trajectories of vehicle18converge as Z(t) approaches infinity. Arrows emanating from images of vehicles18moving along the road in the positive Z direction indicate respective image velocity vectors in the direction of the vanishing point. Length of the arrows represents speed v in image space.

Defining image coordinates relative to the vanishing point: Δx=x−x0 and Δy=y−y0, a model relating speed v measured in image space on image plane8and ground speed V of the vehicle is a ratio:

vV=C⁡(a⁢Δ⁢x-Δ⁢y)⁢((Δ⁢x)2+(Δ⁢y)2)(1)where C is a proportionality constant and a is a horizon slope factor related to roll angle of the camera about the optical axis. If the camera horizontal x axis is exactly parallel with the horizon, then a=0. Using equation (1), parameters: C, a, x0 and y0 may be determined during the calibration phase when vehicle speed is known. Subsequently, during the measurement stage, equation (1) may be used to determine respective ground speeds V from speed v measurements in image plane8of image features of vehicles.

Still referring toFIG.2A, a bounding box7is shown with a broken line tightly fitting an image of a vehicle18. When using bounding box7, x′ may be defined as the mid-point of bounding box7along the horizontal image axis x; and measured relative to vanishing point horizontal coordinate x0 \ is:
Δx′=x′−x0

Similarly, y′ may be defined along the vertical image axis y as the bottom of bounding box7; and relative to the vertical coordinate y0 of the vanishing point is:
Δy′=y′−y0

When using bounding box7, we can improve upon equation (1) by assuming that the bottom of bounding box7is on the road to yield:

vV=C⁡(a⁢Δ⁢x′-Δ⁢y′)⁢((Δ⁢x)2+(Δ⁢y)2)(2)
Calibration Stage:

Calibration stage is for initial setup which, in some embodiments of the present invention, may include operator supervision. As long as camera2remains fixed, the initial setup may be performed only once. There are two purposes to calibration mode: (i) to map regions of the road that are straight and planar without dips or turns, and (ii) to find the model parameters, C, a, x0 and y0.

Reference is now made toFIG.3, a flow diagram of a calibration stage30according to embodiments of the present invention. Multiple image frames are captured (step31) of vehicles travelling along a road with known respective speeds. Image features in images of the travelling vehicles are located, (step33). Image features are tracked between image frames. ORB may be used to match features and track between image frames. (Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski: ORB: An efficient alternative to SIFT or SURF. ICCV 2011: 2564-2571).

Optical flow of the image features may be computed between two or more image frames to produce a set of motion vectors. Vehicle images may be stabilised by considering optical flow outside the vehicle images, i.e. the part of the image that is not expected to be moving. Any other known methods for stabilization may be used. Spurious optical flow vectors may be filtered out. Spurious optical flow vectors include: vectors that do not emanate or end in a bounding box. Vectors that are too short and represent noise. Motion vectors with directions very different from most of the optical flow vectors or are too long and may represent an optical flow feature mismatch are outliers and preferably filtered out.

Filtered optical flow vectors are input into a model based on equation (1) to optimally determine (step35) parameters C, a, x0 and y0. Some regions in image plane8may consistently have improved fits to the model, other regions may have worse fits to the model. Image plane8may be mapped so that better fitting images are used for parameter determination. (step35)

Speed Measurement Stage

Reference is now made toFIG.4which is a simplified flow diagram of speed measurement of target vehicles18, according to embodiments of the present invention. Multiple image frames are captured (step41) of vehicles travelling along a road with unknown respective speeds. Image features in images of the travelling vehicles are located, (step43).

As in the calibration stage, similar processing steps may be performed. Target image features are tracked between image frames. Optical flow of the image features may be computed between two or more image frames to produce a set of motion vectors. Vehicle images may be stabilised and spurious optical flow vectors may be filtered out. Filtered optical flow vectors are input into the model based on equation (1) including model parameters C, a, x0 and y0 to optimally determine (step45) the ground speed V of the target vehicle from the image speeds of the target image features.

The term “image feature” as used herein is one or more portion of an image such as a corner or “blob” and may be detected by any method in the art of digital image processing.

The term “optic flow” or “optical flow” as used herein refers to pattern of apparent motion of image features in image space caused by motion of an object relative to the camera.

The term “image speed” as used herein is the magnitude of an optical flow vector in an image plane divided by time.

The term “calibration” as used herein is not “camera calibration” which generally refers to determining internal camera parameters: e.g., focal length, distortion, and external camera parameters: position coordinates and/or angular coordinates of the camera relative to the road. These camera coordinates are not generally required to be known a priori in the present method. The term “calibration” as used herein refers to the initial stage of the method disclosed herein which uses images of a vehicle travelling at known speed to determine parameters of a model relating ground speed to image speed.

The term “vanishing point” as used herein is a point (x0,y0) in image space to which an image of an object emanates as the object approaches infinite distance from the camera along the positive optical axis.

The transitional term “comprising” as used herein is synonymous with “including,” and is inclusive or open-ended and does not exclude additional, unrecited elements or method steps. The articles “a” and “an” are used herein, such as “a vehicle,” “an image feature,” have the meaning of “one or more,” that is “one or more vehicles,” “one or more image features.”

The various aspects, embodiments, implementations or features of the described embodiments can be used separately or in any combination. Various aspects of the described embodiments can be implemented by software, hardware or a combination of hardware and software. The described embodiments can also be embodied as computer readable code on a non-transitory computer readable medium. The non-transitory computer readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of the non-transitory computer readable medium include read-only memory, random-access memory, CD-ROMs, HDDs, DVDs, magnetic tape, and optical data storage devices. The non-transitory computer readable medium can also be distributed over network-coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

All optional and preferred features and modifications of the described embodiments and dependent claims are usable in all aspects of the invention taught herein. Furthermore, the individual features of the dependent claims, as well as all optional and preferred features and modifications of the described embodiments are combinable and interchangeable with one another.

Although selected features of the present invention have been shown and described, it is to be understood the present invention is not limited to the described features.