Patent ID: 12211217

PREFERRED EMBODIMENTS OF THE INVENTION

FIG.1shows a diagram of a preferred embodiment of the method according to the present invention.

The method of detecting moving objects requires a film (or video) of an observation space to be made.

According to the invention, the detection method comprises step1: extracting at least one first image (f1) and at least one second image (f2) from the film.

The images (frames) that form the video are in colour and are made up of pixels. In particular, the resolution is 1280×720 pixels with a variable frame rate of between 24 and 60 frames per second.

Preferably, the second image (f2) is consecutive to the first image (f1).

In step2, corner points are detected in the first image (f1) and in the second image (f2). In particular, in step2a series of edge and corner points are detected in the first image (f1) and in the second image (f2). Corners mean the intersection of two or more edges.

Advantageously, points are detected by applying the FAST (Features from Accelerated Segment Test) algorithm for the detection of corners.

FIG.2shows the FAST algorithm being applied to a frame, in which each circle corresponds to a point of interest.

In step3, feature vectors—that is, n-dimensional vectors describing the features of the object—are extracted using the points identified; in particular, one is extracted for each point identified. Advantageously, the BRISK (Binary Robust Invariant Scalable Keypoints) descriptor is used, which is scale and rotation invariant; this property is suitable for cases where the scene could be subject to scale and rotation variations due to the movement of the recording device, such as a camera installed on a drone.

In step4, corresponding points are identified in the first image (f1) and in the second image (f2). A matching is made, for instance by using the KNN (k-nearest neighbours) algorithm between the points and the descriptors extracted from the two frames (f1), (f2) in order to identify corresponding points in the two scenes.FIG.3shows the match identified between the points of two consecutive frames, in which each point identified in a frame is connected by a segment to the corresponding point in the next frame.

Once the matches have been identified, these are used in step5to estimate the homography, namely the geometric transformation that enables a registration to be made between the first image (f1) and the second image (f2); in other words, aligning/overlapping the two images framing the same scene from two different points of view.

In step6, a warping is applied digitally to the second image (f2), based on the homography, to obtain a warped second image (f2_w).

In step7, the difference in individual pixels is calculated between the warped second image (f2_w) and the first image (f1) to identify the pixels that differ.

A third image (diff) is thus obtained by comparing the second warped image (F2_w) against the first image (f1).

In particular, a third image (diff) is obtained from a comparison between the second warped image (f2_w) and the first image (f1) by calculating the difference in individual pixels between the warped second image (f2_w) and the first image (f1) to identify the pixels that vay.

Preferably, in step8, on the image (diff) obtained from the difference between the second warped image (f2_w) and the first image (f1), a low pass filter (LP) is used to remove the high frequencies present in the image, which could be caused by artifacts due to image compression.

In a preferred embodiment, thresholding is carried out in step9to determine the pixels that differ between the two frames.

Thresholding is an image segmentation process which takes a grayscale image and returns a black-and-white binary image.

Advantageously, Otsu's method of automatic thresholding of image histogram (TH) is used, which graphically shows the tonal distribution of a digital image. This method assumes that only two classes are present in the thresholding image, thus calculating the optimal threshold to separate these two classes by minimising intra-class variance.

FIG.4shows the image following thresholding, in which the pixels that differ most in the scene are shown in black, whereas those that are unchanged are in white.

Steps8and9may be absent, only one of them may be present, or both of them may be present.

In a preferred embodiment, step9is present.

According to the invention, in step10(Finding Contours), all contours (bounding boxes) relating to groups of adjacent pixels are identified in the third image (diff) or in the image obtained by thresholding or by applying the filter.FIG.5shows the image with rectangles (bounding boxes) printed on the contours identified. In particular, a bounding box (bb) is identified in the third image (diff) for each group of contiguous pixels having image difference values above a threshold.

According to the invention, in step11, the optical flow is calculated based on the first image (f1) and the warped second image (f2_w).

In particular, the optical flow is calculated by comparing the second warped image (F2_w) against the first image (f1), as shown inFIG.1.

Step11may be simultaneous to, before or after steps8-10.

In an advantageous embodiment, the optical flow is calculated using the Farnebäck method. This method is described in Gunnar Farnebäck's article “Two-Frame Motion Estimation Based on Polynomial Expansion”.

From the optical flow, a matrix (mag) corresponding to the magnitude of the optical flow is extracted.

The extracted matrix can be viewed as an image in which different colours are used to indicate the magnitude of the optical flow.

An advantageous embodiment includes a step12, in which, for each bounding box (bb) extracted, the area of the rectangle and the ratio between the sides are checked to ensure that the element identified is not too small or too large and that the arrangement of the pixels is not too elongated.

If this condition is not met, the bounding box (bb) is discarded.

According to the invention, in step13, for each bounding box (bb) drawn on the third image (diff), it is checked whether the optical flow magnitude (corresponding to the motion's magnitude) of a detected pixel group (which could be an object of interest) is greater than the average magnitude of the optical flow of the surrounding pixels (i.e. whether they are moving differently relative to background elements, which may also be moving) This allows to determine whether the detected pixel group moves in a different way than background elements. If the optical flow magnitude of the group of adjacent pixels—corresponding to the magnitude of the potential object of interest—is not greater than the average magnitude of the surrounding pixels, the bounding box (bb) is discarded.

In step14, a blob detector is applied to each bounding box (bb) to identify the presence of blobs inside each individual bounding box (bb) forming part of the third image (diff); namely of points and/or regions in the image whose properties such as brightness or colour differ in comparison with the environment.

In step15, if there is no blob, the bounding box (bb) is discarded.

Preferably, the bounding box (bb) is discarded even if more than one blob is present, while a bounding box (bb) with only one blob is considered a valid detection.

An advantageous embodiment includes a step16, in which two profiles are extracted along the reference axes X and Y for each undiscarded bounding box (bb). This is done by adding the value of the pixels along the image's two axes. Then, in step17, it is checked that the two profiles are well aligned in order to verify that the blob is convex. If the profiles are not aligned, the bounding box (bb) is discarded.

FIG.6shows two examples of profile extraction: in the first, on the left, the two profiles are not aligned and therefore the bounding box (bb) is discarded, whereas in the second one, on the right, the profile verification gave a positive result and therefore the bounding box (bb) is considered a valid detection.

A bounding box (bb) that is not discarded is considered a valid detection.

The method according to the present invention therefore allows determined moving objects, even if small, to be detected.

In addition, since it can be implemented by taking a video acquired by a moving camera as its starting point, it allows films to be taken near the moving object and, therefore, enables the detection of elements that are actually of interest.