Patent Description:
In particular, the invention relates to a system of the type indicated above and comprising a plurality of surveillance cameras, each installed in a respective site to be controlled; a plurality of display devices installed in a control area, each of which has a known location in the control area; a network infrastructure adapted to support the transmission of a plurality of videos from the surveillance cameras to the display devices.

The invention also relates to a support method of the control of surveillance cameras.

Several support systems of the control of surveillance cameras are known.

Such systems comprise a plurality of display devices installed in a control area, designed to display surveillance cameras, each installed in a respective site to be controlled, e.g., in a bank or casino. A network infrastructure is adapted to support the transmission of the videos from the surveillance cameras to the display devices.

An operator in the control area is responsible for monitoring the display devices. Monitoring consists in cyclically paying attention to the videos transmitted on the display devices, so as to verify that there are no anomalies or suspicious presences. In order to facilitate the monitoring and increase the security, the display devices are installed to a wall, for example, to be clearly visible and within reach of the operator.

<CIT> discloses an eye tracker that may be used to identify the operator's gaze area and therefore to identify a portion of the display, or one or more displays in a group of displays, at which the operator is not looking. Accordingly, the video monitoring system may allocate more resources to looked image, and reduce resources allocated to other streams corresponding to frames which are not the focus (e.g., in the peripheral vision) of the operator.

<CIT> discloses several displays in a monitoring station receiving images from cameras in a workign area. Line of sight of an operator at the station is detected to determine display or part of display that is been looked at. Corresponding image is enlarged based on this.

<CIT> discloses a line-of-sight detection unit detecting the line of sight of the registration operator who is looking at the captured image which is shown on a display device. If the registration operator's detected line of sight has not moved for an attention time configured in advance, the extraction unit extracts an image of the person or object appearing in the subregion of the captured image where the registration operator's line of sight is directed. The registration unit registers the extracted image in a predetermined storage device.

However, the main difficulties encountered in these monitoring systems relate to maintaining an adequate level of attention. As time passes and in the absence of particular anomalies, it happens that the operator relaxes and inadvertently gets distracted, turning his/her gaze elsewhere, even for a few instants. The lack of attention sometimes appears as an imbalance in the monitoring time of some display devices compared to others. This imbalance is not intentional but dictated by a habit of the operator's movements, which is also mostly unintentional. All these problems reduce the security of the video surveillance system with the risk that some sites to be controlled are not adequately treated.

The technical problem underlying the present invention is to improve the video surveillance system according to the known art, by overcoming the current limitations afflicting it, and in particular by increasing the operator's general attention, balancing his/her attention on all display devices, appropriately drawings his/her attention where required.

A support system of the control of surveillance cameras according to the present invention is defined in claim <NUM>.

A support method of the control of surveillance cameras according to the present invention is defined in claim <NUM>.

Preferred embodiments of the control system according to the present invention are provided in claims <NUM> to <NUM>.

Further features and advantages of the method and system according to the present invention are provided with reference to the accompanying drawings, given only by way of non-limiting example of the present invention.

<FIG> diagrammatically depicts a support system of the control of surveillance cameras, according to the present invention.

With reference to <FIG>, a support system of the control of surveillance cameras <NUM> is diagrammatically depicted and globally indicated by reference numeral <NUM>. The surveillance cameras <NUM> are each designed to be installed in a respective site <NUM> to be monitored, such as in the premises of a bank subject to continuous control, or in the premises of a casino, a jewelry store, or an area requiring a different type of monitoring, not so much for the presence of valuables but rather for the danger of the area for human safety.

The sites <NUM> to be controlled diagrammatically depicted in <FIG> can be part of the same building or different buildings, placed at a distance from one another but all monitored with the system <NUM> of the present invention. Moreover, every site <NUM> to be controlled, e.g., every room <NUM>, can be surveilled by one or more of the surveillance cameras <NUM>.

Surveillance is centralized in a control area <NUM>, and delegated to an operator <NUM>, such as a police officer operating in a control station <NUM>, just to mention an example, and certainly without any limitation on the use of the system. A plurality of display devices <NUM> is installed in the control area <NUM>. The control area <NUM> is typically a room and the display devices <NUM> are LCD or LED displays, for example, placed side by side in rows or in an array or according to other patterns, and fixed to a wall or other load-bearing structure in the control area <NUM>, thus according to a known location in the control area <NUM>.

Every display device <NUM> has its identification code, and an X, Y positioning in the array or a physical location associated with a three-dimensional coordinate system X<NUM>, Y<NUM>, Z<NUM> within the control area <NUM>. The control area <NUM> is at a distance from the sites <NUM> to be controlled, therefore remote therefrom. The videos captured by the surveillance cameras <NUM> are transmitted to the display devices <NUM>, which are controlled by a controller <NUM>.

The video transmission is carried out through a network infrastructure <NUM>, adapted to support the transmission of videos from the surveillance cameras <NUM> to the display devices <NUM>. The network infrastructure can be wired, wireless or partially wired and partially wireless.

In an embodiment, the display devices <NUM> are provided with a communication interface and an IP address, and directly receive the video stream from the surveillance cameras <NUM>. For example, every display device <NUM> receives a video from a respective surveillance camera <NUM>. In another embodiment, the controller <NUM> is associated with an IP address, the video streams are transmitted from the surveillance cameras <NUM> to the controller <NUM>, and directed by the latter to the display devices <NUM>, so that every display device <NUM> receives a video from a respective surveillance camera <NUM> by means of the controller <NUM>. Moreover, in an embodiment, a display device <NUM> can receive a plurality of videos from a plurality of video surveillance cameras, such videos being simultaneously displayed in separate areas of the display device <NUM>.

A plurality of control cameras <NUM> is also installed in the control area <NUM> to film the operator <NUM> in the control area <NUM>. The control cameras <NUM> are synchronized and calibrated.

The control cameras <NUM> are located in predetermined positions in the control area. For example, every control camera <NUM> has its identification code, and a physical location associated with (a system of) three-dimensional coordinates X<NUM>, Y<NUM>, Z<NUM> in the control area <NUM>. The control cameras <NUM> are located at different angles with respect to the operator <NUM> in order that the face of operator <NUM>, and especially his/her eyes, can be filmed, irrespective of the position of operator <NUM>.

The controller <NUM> is configured to receive the plurality of videos from the surveillance cameras <NUM> and carry out some checks repeatedly, which are detailed below.

These checks are primarily intended to determine which display device <NUM> is looked at by operator <NUM> at a certain instant and how long the attention of operator <NUM> remains on such a display device <NUM>.

The time instants are indicated below as time intervals I<NUM>, I<NUM>,.

In each of these intervals I<NUM>, I<NUM>,. , In, the attention of operator <NUM>, and in particular his/her gaze (i.e., the pointing direction of the operator's eyes), can fall on a display device <NUM>. For example, in the first interval I<NUM>, the operator <NUM> could point his/her gaze at a first display device <NUM> for a time t<NUM>I<NUM>.

For every interval Ii considered, the controller <NUM> carries out the following checks.

The controller <NUM> first processes the plurality of videos of the control cameras <NUM> in a time interval Ii for determining a pointing direction <NUM> of the gaze of operator <NUM> within the control area <NUM>. Video processing comprises several steps, in which the controller <NUM> analyzes a plurality of frames of each video, in order to extract specific features from each frame, in particular the face and, inside the face, the eyes. These steps are described more specifically according to an embodiment described later.

Still as part of the checks in the time interval Ii, the controller determines a duration d of the pointing at the display device <NUM> pointed by the operator <NUM>. In the example provided above, the pointing duration in the interval I<NUM> is t<NUM>I<NUM>, and is associated with the display device <NUM>, having a unique identifier, for example <NUM>-id<NUM> (<FIG>).

The checks repeatedly carried out by the controller <NUM> on a plurality of intervals I<NUM>, I<NUM>,. , In are used to determine whether one or more display devices were "ignored" by the operator, i.e., whether they were not sufficiently monitored by the gaze of operator <NUM>.

In particular, for each display device <NUM>, the controller <NUM> counts the overall pointing duration D in the plurality of time intervals I<NUM>, I<NUM>,. , In as the sum of the pointing durations d at the display device <NUM> in each time interval I<NUM>, I<NUM>,. , In; compares the overall pointing duration D of each display device <NUM> with a minimum pointing duration dmin of each display device <NUM>; and outputs an alert if the overall pointing duration D at one or more display devices <NUM> is lower than the minimum pointing duration dmin. The alert activates a signal for drawing the attention of operator <NUM> towards the one or more display devices <NUM> (i.e., those "ignored").

In an embodiment, the interval duration is equal to the minimum pointing duration dmin and the interval number n is equal to the number of display devices <NUM>. Said parameters (interval duration, minimum pointing duration dmin, interval number n, number of display devices <NUM>) are configurable and configured in the controller <NUM>. The advantage of this embodiment is that, on average, in order to avoid alerts from being triggered, the operator's gaze should focus on a display device <NUM> for the duration of an interval I<NUM>, I<NUM>,. , In but no longer (otherwise it would not be possible to monitor every display device <NUM> at least once in the cycle of intervals I<NUM>, I<NUM>,. In other words, the operator's gaze resting on a display device <NUM> for a longer time than the duration of an interval Ii, e.g., for the duration of <NUM> intervals, would already preclude the operator <NUM> from covering the monitoring of all the other display devices <NUM> in the cycle of intervals I<NUM>, I<NUM>,. , In for a sufficient duration. In such a case, according to the present invention, the controller <NUM> attempts to divert immediately the attention of operator <NUM> from the excessively monitored display device <NUM>, for example by altering the video of the device <NUM> (with a coloration of the video, e.g., red, or an image intermittence). Conversely, if the gaze of operator <NUM> is very attentive and dynamic, and thus remains on a display device <NUM> for a shorter time than the duration of an interval Ii, e.g., for half the duration of the interval, it potentially allows the operator <NUM> to cover the monitoring of all the other display devices <NUM> in the cycle of intervals I<NUM>, I<NUM>,. , In and to look back on the device already monitored for half the duration of the interval as well. In this embodiment, correction factors are provided to avoid the too frequent activation of alerts. For example, one of the correction factors consists in considering the interval duration equal to the minimum pointing duration dmin unless a correction factor f (i.e., interval duration = minimum pointing duration dmin * f). The correction factor is configurable in the controller <NUM> and is a lack-of-attention tolerance of operator <NUM>, not only in terms of concentration (focus) on the display devices <NUM>, but also of regularity in the timing of moving the gaze from one device <NUM> to the other.

As mentioned above, some details of the step of determining the pointing direction <NUM> of the operator's gaze, as configured in the controller <NUM>, are shown below according to a possible embodiment.

In particular, in order to determine the pointing direction <NUM> of the gaze of operator <NUM>, the controller <NUM> detects, by means of each control cameras <NUM>, the position of the operator's face in the frames of the videos filmed by the control cameras <NUM>, while the operator is looking at one of the plurality of display devices.

The controller <NUM> then detects the position of each eye of operator <NUM> with respect to the frames filmed by the control cameras <NUM>.

In particular, every control camera <NUM> detects the two-dimensional coordinates X<NUM>, Y<NUM> of the eyes of operator <NUM>. Such two-dimensional coordinates X<NUM>, Y<NUM> can be calculated with an artificial intelligence algorithm which determines, in each frame of the image of operator <NUM>, the position of the face in the frame and then the position (X<NUM>, Y<NUM>) of the eyes in the face. The controller <NUM> then calculates the coordinates Z<NUM> of the eyes of operator <NUM> in the control area <NUM>, by a triangulation. The plurality of two-dimensional coordinates X<NUM>, Y<NUM> in the frame are mapped to two-dimensional coordinates X<NUM>, Y<NUM> in the control area <NUM> and then used to carry out the triangulation and determine the coordinates Z<NUM>.

In a variant, the positioning of the eyes in the three-dimensional space and thus the coordinates X<NUM>, Y<NUM>, Z<NUM> are determined by means of an artificial intelligence algorithm, which is based on volumetric, not two-dimensional, models of the face of operator <NUM>.

At the end of the above-mentioned steps, each eye of the operator is then associated with a three-dimensional coordinate X<NUM>, Y<NUM>, Z<NUM>. The viewpoint of operator <NUM> can be calculated as the equidistant point on a straight eye junction segment (X<NUM>', Y<NUM>', Z<NUM>').

The pointing direction of the gaze of operator <NUM> is then determined. For this purpose, another artificial intelligence algorithm is used, for example. The algorithm determines the shape of some features of the eye in the frame of at least one of the control cameras <NUM>, such as the eyelid or iris. Based on the shape of such features in the frame, the algorithm determines the gaze pointing direction. The determined direction is a straight line R in the three-dimensional space, passing though the viewpoint X<NUM>', Y<NUM>', Z<NUM>' of operator <NUM>.

Considering the pointing direction and thus the straight line R, the display device <NUM> pointed by the gaze of operator <NUM> is determined. For this purpose, a "surface in space" / "straight line R" intersection is preferably calculated as explained below, where the "surface in space" is one of the surfaces occupied by the display devices <NUM>.

As mentioned, every display device <NUM> has known coordinates X<NUM>, Y<NUM>, Z<NUM> in space. In fact, the display device <NUM> involves a plurality of points X<NUM>, Y<NUM>, Z<NUM> in the three-dimensional space of the control area <NUM>, extending over a predetermined surface (that of the LCD or LED monitor).

The straight line R is analytically intersected with the surface X<NUM>, Y<NUM>, Z<NUM> of the display devices <NUM>, in order to determine an intersection point with one of the surfaces. Such an intersection point is the point looked at by the operator <NUM>. The looked-at display device <NUM> is that occupying the surface comprising X<NUM>, Y<NUM>, Z<NUM> of the point intersected by the straight line R.

In essence, the coordinates of the looked-at display device are determined and obtained as intersection points of the straight lines in space R1, R2,. Rn, i.e., the straight lines of the direction of the operator's gaze looking at predetermined points of the display device from different viewpoints.

In the above-mentioned example, the coordinates of the display devices <NUM> are known a priori, i.e., configured in the controller <NUM>.

The configuration step can be dynamic, e.g., it can be determined by means of a calibration. Calibration consists in showing a number of calibration points on the display device <NUM> and asking the operator <NUM> to look at the calibration points in order to construct a model. The model allows associating the calibration point coordinates with the orientation of the operator's face and eyes. In essence, the orientation is an angle of movement of the face and eyes with respect to a centered position. The angle of movement and the looked-at calibration points are used to calculate the distance between the operator <NUM> and the display device <NUM>. For example, given a first calibration point X<NUM>, Y<NUM> and a second calibration point X<NUM>, Y<NUM> having a preset distance d<NUM>-<NUM> on the display device, the greater the angle of movement of the face and eyes of operator <NUM> to move his/her gaze from the calibration point X<NUM>, Y<NUM> to the calibration point X<NUM>, Y<NUM>, the shorter the distance between the display device <NUM> and the operator <NUM>. This determination of distances can also be supported by an artificial intelligence algorithm, based on a model of preset angular movements (of the face and eyes) and respective distances between a viewer and a viewing target.

In essence, the determination of the coordinates of the display devices takes place using different viewpoints of the operator.

In an embodiment, the operator <NUM> / display device <NUM> distance is determined by applying a camera (possibly by temporarily applying the control cameras <NUM>) to the display device and calculating the time of flight (ToF).

According to different embodiments, the support system is expected to output different types of alert. In particular, the step of controlling one or more display devices <NUM> (i.e., the devices "ignored" by the gaze of operator <NUM>) comprises activating an indicator light, such as a LED, for example, associated with the display devices <NUM> or changing a color of the indicator light (which can normally be green, and then switched to red). Alternatively, the step of controlling one or more display devices <NUM> is expected to comprise changing a color of the video of the display device <NUM>. For example, the color of the video can be switched to a blue shade when the display device <NUM> is not sufficiently looked at (which would distinguish it from an excessively looked-at display device <NUM> already described above, where the color of the video would be red). In an embodiment, a plurality of colors of LED indicator lights are taken into account: a green color indicates that the time for looking at the monitor (display devices <NUM>) is in the preset ranges, a yellow color indicates that the viewing time is longer than the preset time, and a red color indicates that the viewing time is insufficient.

According to a further embodiment, the step of controlling the display devices <NUM> comprises displaying a video overlaid on the video received by the display devices <NUM> from the surveillance camera <NUM>. The overlay is of the Picture in Picture (PiP) type, for example. For example, the primary video is the video of the surveillance camera <NUM> associated with the display device <NUM>; the secondary video (that occupying a smaller area in the PiP image) is the video of the surveillance camera <NUM> for which the alert was raised; such a secondary video preferably shows the superimposed identification of the display device <NUM> towards which the gaze is redirected (such a display device <NUM> towards which the gaze is redirected is further brought to the operator's attention as mentioned above, for example by switching the video related to the blue color).

The controlling step comprises simultaneously activating a plurality of signals to draw the attention of operator <NUM> for a plurality of display devices <NUM>, if the overall pointing duration D of the plurality of display devices <NUM> is lower than the minimum pointing duration dmin of said plurality of display devices <NUM>.

In an embodiment, the controller <NUM> is configured to set a plurality of minimum pointing durations dmin, each (or at least one) with a different duration from the other pointing durations. In such a case, it is intended to give some display devices <NUM> viewing priority over others. For example, such a configuration would allow dedicating a significant percentage of the operator's time (<NUM>%) to a specific display device <NUM>, considered of extreme importance (e.g., a bank vault), and a remaining time (<NUM>%) to other devices. The percentages given are only examples, and a plurality of differentiated durations can be set.

The viewing priority can be statistically associated with the physical arrangement of the display devices <NUM> in the control area <NUM>. For example, in an array arrangement of m rows and n columns, the display device <NUM> in position <NUM>*<NUM> has the highest priority (x) and the other devices have a decreasing priority (x-<NUM>, x-<NUM>) moving by rows (<NUM>*<NUM>, <NUM>*<NUM>).

However, according to the present invention, the viewing priority can be established dynamically and not statistically, and in particular increased over time for the display devices <NUM> which were "ignored" (not sufficiently surveilled) in the past. For example, the controller <NUM> is configured to vary the plurality of minimum pointing durations dmin for the various display devices <NUM>, where the variation is an increase in the minimum pointing duration dmin of a display device <NUM> in the time intervals In+<NUM>, In+<NUM>,. following the time intervals I<NUM>, I<NUM>,. , In already considered, if the overall pointing duration D in the time intervals I<NUM>, I<NUM>,. , In already considered is lower than the minimum pointing duration dmin.

Every display device <NUM> preferably displays the viewing priority and/or the minimum viewing duration.

According to an embodiment, the support system <NUM> raises the attention of operator <NUM> when the operator <NUM> repeats the same sequence of viewing operations over time, since a completely predictable monitoring can be less effective and more tiring for the operator <NUM>. In the array arrangement of m rows and n columns, this occurs, for example, if the gaze moves along the display devices of one row, then along those of the next row, up to the last row, and then resumes from the first one.

Therefore, the controller <NUM> is preferably configured to determine sequences of pointing at the plurality of display devices <NUM>, every sequence comprising at least m display devices <NUM>, where m is configurable in the controller <NUM> (and not necessarily corresponding to the number of the devices <NUM> in a row). The controller then determines at least one recurrence in the sequences of pointing at the plurality of display devices <NUM>, and activates the signal for drawing the attention of operator <NUM>, to avoid a further recurrence in the sequences of pointing at the plurality of display devices <NUM>, at least within a preset time period.

For example, considering a preset time period of <NUM> minutes, a total number of <NUM> display devices <NUM> (<NUM><NUM>,. , <NUM><NUM>) and a control sequence of <NUM> in number, the alarm is generated if the operator looks at, in the time period of <NUM> minutes, more than once, the sequence of devices <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> in the same order.

The Applicant carried out experimental tests and observed that the plurality of time intervals I<NUM>, I<NUM>,. , In can preferably be set to a duration Id between <NUM> milliseconds and <NUM> milliseconds, the number In of the plurality of intervals between <NUM> and <NUM>,<NUM>. Given k display devices <NUM>, e.g., k between <NUM> and <NUM>, the minimum pointing duration dmin is preferably between [(Id × In)/k]/w and [(Id × In)/k], where w is the tolerance on the operator's attention.

The example given is not limiting. Preferably, the control system is resettable by the operator. Resetting consists in the possibility to clear the calculations related to the viewing time of the display devices <NUM>. Only by way of example, resetting can be useful in conjunction with a shift change between two operators, when the incoming operator needs to have a cleaned-up situation with respect to the history of the monitoring operations carried out by the previous viewer.

Claim 1:
A support system (<NUM>) of the control of surveillance cameras (<NUM>), comprising:
- a plurality of surveillance cameras (<NUM>), each installed in a respective site (<NUM>) to be controlled;
- a plurality of display devices (<NUM>) installed in a control area (<NUM>), every display device (<NUM>) having a known location in the control area (<NUM>);
- a network infrastructure (<NUM>) adapted to support the transmission of a plurality of videos from the surveillance cameras (<NUM>) to the display devices (<NUM>);
a plurality of control cameras (<NUM>) installed in the control area (<NUM>) to film an operator (<NUM>) in the control area (<NUM>), said control cameras (<NUM>) being located in predetermined positions in the control area, at different angles with respect to the operator (<NUM>);
a controller (<NUM>) configured to receive a plurality of videos from the control cameras (<NUM>) and carry out repeatedly, at time intervals (I<NUM>, I<NUM>, ..., In), the followings steps:
- processing the plurality of videos of the control cameras (<NUM>) in a time interval (Ii) to determine a pointing direction (<NUM>) of the gaze of the operator (<NUM>);
- determining, between said display devices (<NUM>), a display device (<NUM>) pointed, in the time interval (Ii), by the gaze of the operator (<NUM>);
- determining a duration (d) of the pointing at said at least one display device (<NUM>) pointed in the time interval (Ii);
characterized in that said controller (<NUM>) being further configured to:
- count, for each display device (<NUM>), the overall pointing duration (D) in a plurality of time intervals (I<NUM>, I<NUM>, ..., In), said overall pointing duration (D) being the sum of the pointing durations (d) at the display device (<NUM>) in each time interval (I<NUM>, I<NUM>, ..., In);
- compare the overall pointing duration (D) of each display device (<NUM>) with a minimum pointing duration (dmin) of each display device (<NUM>);
- control one or more display devices (<NUM>) by activating a signal for drawing the attention of the operator (<NUM>), if the overall pointing duration (D) at said one or more display devices (<NUM>) is lower than the minimum pointing duration (dmin).