Patent Description:
A high level of effort and attention is required of the operator in charge of monitoring a certain area when he has to continuously watch video images to look for potential objects of interest, such as targets or threats, which can then be further investigated to make important decisions. This is particularly true if the surveillance system uses a panoramic scanning system to reconstruct a <NUM>° view around the monitored area.

Optoelectronic path surveillance systems are known to highlight potential objects of interest in the images based on some macroscopic characteristics of the objects, for example the colour, shape, size and/or movement of the objects. However, for particular scenarios it is necessary to be able to identify potential objects of interest without making assumptions about the size, shape, colour and/or movements of the object of interest.

Paper titled "<NPL>, discloses a multi-scale seamless image compositing method to minimize the artifacts along the boundaries of multiple images composited with different textures and colours in order to generate visually appealing and natural looking composites. The method automatically detects the visually salient objects from the source image for merging it over a destination image background. The method comprises the use of a salient object detection method for automatic mask generation and the formulation of a <NUM>-D exponentially weighted Savitzky-Golay (<NUM>-D EWS-G) filter, which is an extension of <NUM>-D S-G filter, for multi-scale pyramid based blending with edge preserving blurring property.

Aim of the present invention is to provide a detection system that is able to detect potential objects of interest in an image of a monitored area, without having to make assumptions about characteristics of the object, such as size, shape, colour and/or movements, and that, at the same time, is easy and economical to realize.

In accordance with the present invention, there are provided a detection system to facilitate the detection of a potential object of interest in an original image and a surveillance system, as defined in the appended claims.

The following description is provided to enable a person skilled in the art to realize and use the invention. Various modifications to the embodiments will be readily apparent to those skilled in the art, without departing from the scope of protection of the claimed invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but must be accorded the widest scope of protection consistent with the principles and features described and claimed in the appended claims.

Unless otherwise defined, all technical and scientific terms used in the present disclosure have the same meaning commonly understood by a person with ordinary skill in the art to which the disclosed embodiments belong. In case of conflict, the present invention, including the definitions, will be binding. Furthermore, the examples are illustrative only and are intended to be non-limiting.

In order to promote understanding of the disclosed embodiments, reference will be made to some embodiments and specific language will be used to describe the same. The terminology used in the present disclosure is therefore for the sole purpose of describing particular embodiments, and is not intended to limit the scope of the present disclosure.

In <FIG> generally denotes, as a whole, a surveillance system to detect potential objects of interest, for example target objects or threatening objects, in a certain area to be monitored.

The surveillance system <NUM> comprises at least one optoelectronic device <NUM> for acquiring at least one image, and in particular static images or video image frames, of the area to be monitored, a display device <NUM> for displaying images, for example a normal digital screen, and a detection system <NUM>, which is connected to the optoelectronic device <NUM> to receive the image, is adapted to perform a complex processing of this image and is connected to the display device <NUM> to display the result of this complex processing in order to facilitate the task of an operator in identifying a potential object of interest present in the image. The received image is hereinafter called the original image and is denoted with I(x,y) to underline the representation of the "given image" in the domain of the spatial coordinates x and y in the plane defined by the image.

The original image I(x,y) is either a video image or an infrared image. It is noted that an infrared image is not part of the claimed embodiment of the invention. The video image is encoded using YUV standard. The original image I(x,y) is the luminance component of a video image encoded using the YUV standard.

The detection device <NUM> comprises a programmable logic device <NUM>, which receives the original image I(x,y), at least one processor <NUM>, which is provided with a RAM memory <NUM> and is interfaced with the programmable logic device <NUM>, and at least one non-volatile memory <NUM>, which is adapted to store software that can be loaded into the RAM memory <NUM> to be executed by the processor <NUM>.

Advantageously, the programmable logic device <NUM> is a FPGA (Field Programmable Gate Array). Advantageously, the processor <NUM> is of the ARM type. Advantageously, the programmable logic device <NUM> and the processor <NUM> are part of a single heterogeneous integrated computer device, of the type known as System on Chip (SoC).

<FIG> shows the architecture of a hardware and software implementation of the detection system <NUM>. The hardware part of the detection system is denoted by 4a and is implemented via the programmable logic device <NUM> and the software part of the detection system <NUM> is denoted by 4b and is implemented via the processor <NUM> and the memory <NUM>. In other words, the software stored in the memory <NUM> and executed on the processor <NUM> comprises the software part 4b.

The programmable logic device <NUM> is programmed to implement an image conditioning module <NUM> that performs a padding of the original image I(x,y) in order to neutralize errors near the edge of that image due to discontinuity of the image.

According to an embodiment not shown, in which the errors near the edge of the image I(x,y) have an acceptable extent, the programmable logic device <NUM> is devoid of the image conditioning module <NUM>.

In other words, the image conditioning module <NUM> is optional and has no conceptual impact on downstream processings. Therefore, for simplicity's sake, hereinbelow the original image will be understood indifferently the image I(x,y) or the image at the output of the image conditioning module <NUM>, denoted by f(x,y).

The programmable logic device <NUM> is programmed to implement a plurality of image processing stages <NUM> which are connected in cascade between them. Each image processing stage <NUM> comprises a respective low-pass filter <NUM> and a respective subsampling module <NUM> connected to the output of the low-pass filter <NUM>. The low-pass filter <NUM> of the image processing stage <NUM> further upstream receives the original image f(x,y) and the subsampling modules <NUM> provide respective processed images fi(x,y). Each processed image fi(x,y), except the last one, is supplied to the next image processing stage <NUM>. The example of <FIG> comprises three image processing stages <NUM> for generating three processed images, i.e. fi(x,y) with i = {<NUM>,<NUM>,<NUM>}.

The low-pass filter <NUM> blurs the image at the input. The subsampling module <NUM> reduces the resolution of the blurred image at the input. In particular, the subsampling module <NUM> performs a subsampling of factor <NUM> along each spatial coordinate. Thus, each image processing stage <NUM> provides a processed image fi(x,y) having a greater blur and a resolution fourfold lower than the image at the input. The cascade of the image processing stages <NUM> realizes a so-called pyramid representation of the original image f(x,y). The first level of the pyramid representation is the original image f(x,y) and each subsequent level is a processed image fi(x,y) obtained by blurring and subsampling the previous level so as to obtain a stack of images with progressively increasing blurring and progressively decreasing resolution.

The programmable logic device <NUM> is further programmed to implement a number of saliency map extraction modules <NUM> equal to the number of image processing stages <NUM> plus one to provide respective saliency maps, denoted by S(x,y) and Si(x,y). An image saliency map highlights those portions of the image that deviate from what surrounds them or from the background, regardless of the colour, contrast, and movement of those portions. Each saliency map S(x,y), Si(x,y) is relative to a respective image of a group of images that includes the original image f(x,y) and the processed images fi(x,y). The example of <FIG> therefore comprises four saliency map extraction modules <NUM> for generating four saliency maps, i.e. i = {<NUM>,<NUM>,<NUM>}.

Advantageously, each saliency map extraction module <NUM> obtains the respective saliency map S(x,y), Si(x,y) based on a calculation of the spectral residual of a respective image of the aforementioned group of images. For this reason the saliency map extraction module <NUM> is also called the spectral residue saliency extraction module ("SRSE").

The saliency map extraction modules <NUM> transform the pyramid representation of the original image f(x,y) into a corresponding pyramid representation of saliency maps, as graphically shown in <FIG>. In this way, each saliency map Si(x,y) corresponding to a processed image fi(x,y) allows to highlight small portions with respect to the resolution of that map but which are larger in the original image f(x,y).

The software part 4b of the detection system <NUM>, when loaded into the RAM memory <NUM> and executed by the processor <NUM>, causes the latter to be configured to perform the processing steps described below with reference to <FIG>.

Firstly, a normalization of each of the saliency maps S(x,y) and Si(x,y) is performed as a function of the respective standard deviations, thereby obtaining corresponding normalized saliency maps SN(x,y) and SiN(x,y) (steps <NUM> in <FIG>). In the example of <FIG> the normalized saliency maps are four, i.e. i = {<NUM>,<NUM>,<NUM> }.

Subsequently, the normalized saliency maps SiN(x,y), i.e. those relative to the processed images fi(x,y), are resized so as to bring them to the size of the saliency map SN(x,y) of the original image f(x,y) (steps <NUM>). In particular, the step of resizing the normalized saliency maps SiN(x,y) provides for an interpolation between adjacent pixels. The normalized and resized saliency maps are denoted by Si↑(x,y). In the example of <FIG> the normalized and resized saliency maps are four, i.e. i = {<NUM>,<NUM>,<NUM>}.

Finally, the normalized saliency map SN(x,y) and the normalized and resized saliency maps of the images Si↑(x,y) are merged to obtain a final saliency map SF(x,y) on which to detect the potential object of interest (step <NUM>).

<FIG> graphically shows the effect of resizing the normalized saliency maps SiN(x,y) necessary to be able to perform the subsequent merging of all saliency maps.

Each saliency map extraction module <NUM> comprises a plurality of calculation modules described in detail hereinbelow and shown in <FIG>, with particular reference to the original image f(x,y) and to the corresponding saliency map S(x,y) for ease of treatment.

A first calculation module <NUM> is configured to receive the image f(x,y) and apply to said image f(x,y) a two-dimensional fast Fourier transform (FFT-<NUM>) to provide the spectrum of the image in the domain of the spatial frequencies X and Y corresponding to the spatial coordinates x and y: <MAT> wherein A(X,Y) is the modulus and θ(X,Y) and the phase (argument) of the spectrum represented in exponential form.

A second calculation module <NUM> is configured to receive the modulus A(X,Y) and calculate, based on the latter, a respective spectral residual R(X,Y).

A third calculation module <NUM> is configured to combine the spectral residual R(X,Y) with the spectral phase, using the spectral residual R(X,Y) and the phase θ(X,Y) as the module and argument of a complex number in exponential form, and thus obtain the following signal in the domain of the spatial frequencies: <MAT>.

A fourth calculation module <NUM> is configured to apply a two-dimensional inverse fast Fourier transform (IFFT-<NUM>) to the signal T(X,Y) so as to return to the domain of the spatial coordinates and provide a raw saliency map SR(x,y).

A fifth calculation module <NUM> is configured to apply a two-dimensional Gaussian filter to the squared modulus of the raw saliency map SR(x,y) and thus obtain the saliency map S(x,y) relative to the image f(x,y). In formulas, the saliency map is given by: <MAT>.

In more detail, the calculation module <NUM> comprises a first calculation sub-module <NUM> configured to calculate the logarithm of the modulus A(X,Y) of the spectrum, a second calculation sub-module <NUM> configured to filter the logarithm of the modulus A(X,Y) by a moving average whose result is denoted by Ψ(X,Y) and a third calculation sub-module <NUM> configured to calculate the spectral residual R(X,Y) as the difference between the logarithm of the modulus A(X,Y) and the result of the moving average Ψ(X,Y).

In particular, the calculation sub-module <NUM> implements the moving average by calculating a two-dimensional convolution between the logarithm of the modulus A(X,Y) and a rectangular two-dimensional windowing function W(X,Y) according to the following formula: <MAT>.

The windowing function W(X,Y) is defined by the following formula: <MAT> wherein rect is the rectangular pulse function and the parameter d defines the width of the rectangular pulse.

Thus, the operation of each saliency map extraction module <NUM> is adjustable via the two parameters d and σ. Preferably, d is equal to <NUM> and σ is equal to <NUM>.

The step of normalizing the saliency maps S(x,y) and Si(x,y) comprises a sequence of calculation steps performed for each of the saliency maps and described below, for simplicity's sake, with reference only to the saliency map S(x,y), i.e. the one with the lowest level of the pyramid representation.

In general, the normalized saliency map SN(x,y) is calculated by subtracting the median value of the saliency map S(x,y), denoted below by m, from each pixel of the saliency map S(x,y) and by dividing the result of the subtraction by the standard deviation of the saliency map S(x,y), denoted below by s. In formulas we have: <MAT> <MAT>.

The standard deviation s of the saliency map S(x,y) is calculated as a function of the absolute median deviation around the median value of the saliency map S(x,y), denoted below with MAD: <MAT> wherein the absolute median deviation MAD is given by: <MAT>.

The step of merging the normalized saliency map SN(x,y) with the normalized and resized saliency maps Si↑(x,y) envisages, for each pair of spatial coordinates (x,y) in the plane of the saliency maps, obtaining a corresponding pixel of the final saliency map SF(x,y) by selecting the maximum pixel value between the saliency maps SN(x,y) and Si↑(x,y), i.e. in formula: <MAT>.

The detection system <NUM> described above can adapt to any surveillance system, and in particular to the digital images provided by any optoelectronic device, although it has proved to be particularly effective using as original images the video frames of the luminance component of a video signal encoded using the YUV standard. In particular, the detection system <NUM> proves to be extremely effective in monitoring panoramic images.

The main advantage of the detection system <NUM> described above derives from the combination of using the saliency map of the original image with the pyramid representation of the saliency map. In fact, a saliency map allows very small potential objects of interest to be detected in the original image, whereas larger potential objects of interest would be detected as a cluster of smaller objects, thus losing some of the information on the larger object. With a pyramid representation of the saliency map, each higher level saliency map Si(x,y) contains accurate information on objects of interest that are small in relation to the resolution of that map, but which are larger objects of interest on the saliency map S(x,y) relative to the original image f(x,y). Thus, merging the saliency maps into a final saliency map SF(x,y) allows information to be retrieved about the larger objects of interest that come from the higher-level saliency maps Si(x,y) in the pyramid.

Another advantage of the detection system <NUM> is the operating speed, thanks to the implementation in a heterogeneous environment in which the programmable logic device <NUM> performs in parallel the most onerous processing that involves creating the pyramid representation of the original image f(x,y) and extracting the saliency maps for each level of the pyramid representation.

Finally, the detection system <NUM> makes no assumptions about the characteristics of the potential objects of interest and is therefore also able to detect stationary objects.

Claim 1:
A detection system to facilitate detection of a potential object of interest in an original image (I; f), the detection system (<NUM>) comprising a programmable logic device (<NUM>) and a processor (<NUM>) interfaced with the programmable logic device (<NUM>) and a memory (<NUM>); the programmable logic device (<NUM>) being programmed to implement a cascade of image processing stages (<NUM>), each comprising a respective low-pass filter (<NUM>) and a respective subsampling module (<NUM>) connected to the output of the low-pass filter (<NUM>), the low-pass filter (<NUM>) of a first image processing stage (<NUM>) receiving the original image (I; f) and the subsampling modules (<NUM>) providing respective processed images (fi); the programmable logic device (<NUM>) being programmed to implement a number of saliency map extraction modules (<NUM>) equal to the number of image processing stages (<NUM>) plus one to provide respective saliency maps (S, Si), each of which is relative to a respective image of a group of images including the original image (I; f) and the processed images (fi); the memory (<NUM>) storing a software, which, when executed on the processor (<NUM>), causes the latter to be configured to normalize (<NUM>) the saliency maps (S, Si) according to their respective standard deviations, resize (<NUM>) the normalized saliency maps (SN, SiN) relative to the processed images (fi) so as to bring them to the size of the saliency map (S) of the original image (I; f), and merge (<NUM>) a group of saliency maps including the normalized and resized saliency maps (Si↑) of the processed images (fi) and the normalized saliency map (SN) of the original image (I; f) in order to obtain a final saliency map (SF) on which the potential object of interest can be detected; said original image (I; f) being the luminance component of a video image encoded using the YUV standard.