Patent Description:
Surveillance of the public using imaging, in particular video imaging, is common in many areas around the world. Areas that may need monitoring are for example banks, stores, and other areas where security is needed, such as schools and government facilities. Other areas that may need monitoring are processing, manufacturing and logistics applications where video surveillance is primarily used to monitor processes.

In order to protect a video camera from for example dirt and vandalism many surveillance cameras are equipped with a protective housing. However, monitoring with the video camera may still be disturbed by dirt or small objects on the protective cover. The dirt or objects may lower the video quality by distortion of the image or even make it more difficult to detect and/or track persons or objects which should be detected and/or tracked by the video camera.

To overcome this problem the protective cover may be cleaned when necessary. However, there is a problem of detecting if there is dirt or objects on the protective cover which requires cleaning, be that from sabotage or naturally occurring. Document <CIT> discloses a CPU that detects a foreign object adhering to the optical lens by performing shooting under a plurality of shooting conditions in which the exposure value and the focus position are identical and aperture values arc different from each other, and comparing a plurality of the image signals obtained from the imaging element during the shooting.

<CIT> discloses a method and a system for detecting the presence of an impediment on a lens of an image capture device to the passage of light through the lens of the image capture device, see §<NUM>. It is noted that the impediment is detected if the areas of constant luminance substantially coincide.

An object of embodiments herein may thus be to obviate some of the problems mentioned above, or at least reduce the impact of them. Specifically, an object may be to detect a foreign object adhering to a transparent protective cover of a video camera.

In the figures, features that appear in some embodiments are indicated by dashed lines.

The various aspects of embodiments disclosed herein, including particular features and advantages thereof, will be readily understood from the following detailed description and the accompanying drawings, in which:.

As mentioned above, there may be a problem of detecting if there is dirt or objects on the protective cover of a video camera which requires cleaning. Embodiments herein are directed to detecting objects on the protective cover.

Embodiments herein may be implemented in one or more image-processing devices. The one or more image-processing devices may for example be an image-capturing device, such as a video camera, or a video server or a combination of these. The video camera may be a digital video camera. <FIG> depicts an image-processing device in the form of a video camera <NUM>, such as a surveillance camera or a monitoring camera, such as a pan tilt and zoom (PTZ) video camera. Thus, in some embodiments herein the image-processing device is the video camera <NUM>.

<FIG> further depicts an exemplifying video network system <NUM> in which embodiments herein may be implemented. The video network system <NUM> may include the video camera <NUM> which can capture and perform image processing on a digital video image <NUM>. A video server <NUM> in <FIG> may obtain the digital video image <NUM>, for example from the video camera <NUM> over a network or the like, which is indicated in <FIG> with the double-pointing arrows.

A video server is a computer-based device that is dedicated to delivering video. Video servers are used in a number of applications, and often have additional functions and capabilities that address the needs of particular applications. For example, video servers used in security, surveillance and inspection applications typically are designed to capture video from one or more video cameras and deliver the video via a computer network. In video production and broadcast applications, a video server may have the ability to record and play recorded video, and to deliver many video streams simultaneously. Today, many video server functions may be built-in in the video camera <NUM>.

Thus, in <FIG>, the video server <NUM> is connected over the video network system <NUM>, to the video camera <NUM>. The video server <NUM> may further be connected to a video storage <NUM> for storage of video images, and/or connected to a monitor <NUM> for display of video images. In some embodiments the video camera <NUM> is connected directly with the video storage <NUM> and/or the monitor <NUM>, as indicated by the direct arrows between these devices in <FIG>. In some other embodiments the video camera <NUM> is connected to the video storage <NUM> and/or the monitor <NUM> via the video server <NUM>, as indicated by the arrows between the video server <NUM> and the other devices.

<FIG> is a schematic overview illustrating exemplifying embodiments of the video camera <NUM>.

The video camera <NUM> comprises an image sensor <NUM> onto which a lens-based optical imaging system <NUM> of the video camera <NUM> images a scene. The lens-based optical imaging system <NUM> may also be referred to as a lens system. In some embodiments the lens-based optical imaging system <NUM> is a zoom system. That is, the focal length of the optical imaging system <NUM> may be adjustable. The zoom may be used in order to more easily use the captured images for different purposes. For example, a zoomed-out image may be used to discover and/or track objects or persons, while a zoomed-in image may be used to show details and identify objects or persons.

The optical imaging system <NUM> of the video camera <NUM> may comprise an adjustable aperture. The adjustable aperture may control light intensity at the image sensor <NUM> of the video camera <NUM>.

The video camera <NUM> may be arranged on a motorised pan and tilt arrangement <NUM>. The motorised pan and tilt arrangement <NUM> may adjust the orientation of the video camera <NUM> by adjusting pan and tilt angles such that different scenes may be captured. In <FIG> the video camera <NUM> is arranged with an angle alpha against an axis of a protective housing. In <FIG> the video camera <NUM> is arranged with an angle beta against the axis of the protective housing. Adjustment of the orientation of the video camera <NUM> may be used in order to scan different sub-areas of a larger area and to track objects or persons.

The video camera <NUM> is arranged in the protective housing. The protective housing comprises a transparent protective cover <NUM>. The protective housing covers the video camera <NUM> such that no parts of the video camera <NUM> are arranged outside the protective housing. Specifically, the entire optical imaging system <NUM> is covered by the transparent protective cover <NUM>.

<FIG> and <FIG> further illustrate a field of view <NUM> of the optical imaging system <NUM>.

In order to better understand embodiments herein a digital video image processing system will first be described.

Figure 3a is a schematic view of a digital video image processing system <NUM>, in this case of a digital video camera, such as the video camera <NUM>. The digital video image-processing system <NUM> images a scene on an image sensor <NUM> with the lens-based optical imaging system <NUM>. In other words, the digital video image-processing system <NUM> captures the scene with the image sensor <NUM>. The image sensor <NUM> may be provided with a Bayer filter, such that different pixels will receive radiation of a particular wavelength region, in a known pattern. Typically, each pixel of the captured image is represented by one or more values representing the intensity of the captured light within a certain wavelength band. These values are usually referred to as colour components, or colour channels. The term "image" may refer to an image frame or video frame including information originating from an image sensor that has captured the image.

The video camera <NUM> may apply different gain to different exposures. For example, a first exposure with a first light intensity per area at the image sensor <NUM> may have a different gain than a second exposure with a second light intensity per area at the image sensor <NUM>. By changing the gain, it is possible to adjust the apparent brightness of the second exposure to match the brightness of the first exposure even though the image sensor <NUM> received different light intensity per area.

In general, gain may be described as a means of increasing the ISO of the video camera <NUM> and apparent sensitivity to light. In more technical terms, gain in a digital imaging device may be said to represent the relationship between the number of electrons acquired on the image sensor <NUM> and the analog-to-digital units (ADUs) that are generated, representing the image signal. Increasing the gain amplifies the signal by increasing the ratio of ADUs to electrons acquired on the image sensor <NUM>. The result is that increasing gain increases the apparent brightness of an image at a given exposure.

After having read the signal of individual sensor pixels of the image sensor <NUM>, different image processing actions may be performed by an image processing pipeline <NUM>. The image processing pipeline <NUM> may comprise an image processing part 302a and a video post-processing part 302b.

Typically for video processing the images are comprised in a stream of images. Figure 3a illustrates a first video stream <NUM> from the image sensor <NUM>. The first image stream <NUM> may comprise multiple captured image frames, such as a first captured image frame <NUM> and a second captured image frame <NUM>.

Image processing may comprise demosaicing, color correction, noise filtering (for eliminating spatial and/or temporal noise), distortion correction (for eliminating effects of, e.g., barrel distortion), global and/or local tone mapping (e.g., enabling imaging of scenes containing a wide range of intensities), transformation (e.g., rectification and rotation), flat-field correction (e.g., for removal of the effects of vignetting), application of overlays (e.g., privacy masks, explanatory text), etc. The image processing pipeline <NUM> may also be associated with an analytics engine performing object detection, recognition, alarms, etc..

The image processing part 302a may e.g. perform image stabilization, apply noise filtering, distortion correction, global and/or local tone mapping, transformation, and flat-field correction. The video post-processing part 302b may for example crop parts of an image, apply overlays, and comprise the analytics engine.

Following the image processing pipeline <NUM> the image may be forwarded to an encoder <NUM>, wherein the information in the image frames is coded according to an encoding protocol, such as H. <NUM>, and forwarded to for example a receiving client, exemplified here with the monitor <NUM>, to the video server <NUM>, the storage <NUM>, etc..

Exemplifying methods according to embodiments herein will now be described with reference to a flowchart of <FIG>, and with further reference to <FIG>, <FIG> and <FIG>.

The flowchart of <FIG> illustrates methods, performed by an image-processing device, such as the video camera <NUM> and/or the video server <NUM>, for determining whether or not the transparent protective cover <NUM> of the video camera <NUM>, comprising the lens-based optical imaging system <NUM>, is partly covered by a foreign object <NUM> on the protective cover <NUM>.

In other words, the methods are for detecting the foreign object <NUM> adhering to the transparent protective cover <NUM> of the video camera <NUM>. The foreign object <NUM> may be smaller than the protective cover <NUM>.

The methods may be implemented in an image-processing device, in particular the video camera <NUM> of <FIG>. However, the methods may also be implemented in the video server <NUM>. Embodiments herein may also be implemented in several image-processing devices. For example, in some embodiments herein the method for detecting objects on the protective cover of the video camera <NUM> is performed in a distributed manner in both the video camera <NUM>, and one or more further image-processing devices, such as the video server <NUM>. The one or more further image-processing devices may also be comprised in a Video Management System (VMS), or a computer cloud or a VMS implemented in the computer cloud. More specifically, some parts of the method, such as controlling different image-capturing parameters and capturing the image frames may be performed by the video camera <NUM>, while other parts like analysing images may be performed by the one or more further image-processing devices.

One or more of the following actions presented in <FIG> may be performed in the following exemplifying order. In other examples, the order may differ from what is described below.

Embodiments herein comprise analysing a pair of image frames from the video camera <NUM> taken with different depths of field and determining whether or not the transparent protective cover <NUM> of the video camera <NUM> is partly covered by the foreign object <NUM> on the protective cover <NUM>. In other words, the analysis may result in a detection of the foreign object.

Embodiments herein take advantage of the fact that changing the depth of field influences how the object <NUM> is imaged on the image sensor <NUM>, <NUM>. A larger depth of field will sharpen the image of the object <NUM> which will change a distribution of light intensity that hits the image sensor <NUM>, <NUM> on pixels that correspond to the image of the object <NUM> compared to when the object <NUM> is imaged with a shallower depth of field. The light that hits the image sensor <NUM>, <NUM> induces pixel intensity signals which may be stored as the captured image frames <NUM>, <NUM>. For example, the pixel intensity signals may correspond to linear-light quantities, such as linear RGB components. Such linear-light quantities are directly proportional to physical radiance. Another linear-light quantity is relative luminance, sometimes denoted Y. It is directly proportional to physical radiance weighted by the spectral sensitivity of human vision. Luminance involves light having wavelengths in the range of about <NUM> to <NUM>. Luminance may also be computed as a properly weighted sum of linear-light red, green, and blue tristimulus components according to the principles and standards of the Commission Internationale de l'Éclairage (CIE).

Using linear components may be an advantage since the pixel intensity signals may need to be adjusted by for example multiplication in order to correct for differences in exposure values. In embodiments herein.

For RGB color spaces that use the ITU-R BT. <NUM> primaries (or sRGB, which defines the same primaries), relative luminance may be calculated from linear RGB components by first converting gamma-compressed RGB values to linear RGB, and then by the following formula: <MAT>.

In embodiments herein the expression luminance is used to denote pixel intensity signals of a respective image frame. Specifically, luminance may correspond to the relative luminance of the pixels of the image frame. Further, in embodiments herein a luminance pattern denotes a pattern of the pixel intensity signals, such as a pattern of the relative luminance of the respective pixels of the respective image frame. The luminance pattern may also correspond to a pattern of a gradient or a derivative of the relative luminance of the pixels.

Thus, a decrease in luminance due to the object <NUM> may be concentrated to fewer pixels with an increased depth of field. For example, the first captured image frame <NUM> may be captured with a first depth of field which is lower than a second depth of field of the second captured image frame <NUM>. The luminance of the first captured image frame <NUM> may be affected by the object <NUM> in for example 50x50 pixels. For example, the luminance compared to surrounding pixels may be lowered by a few percent. The luminance of the second captured image frame <NUM> may be affected by the object <NUM> in 30x30 pixels and the luminance variation compared to surrounding pixels may be more lowered than for the first captured image frame <NUM>. Thus, by comparing the luminance patterns of the two captured image frames <NUM>, <NUM> it is possible to detect the object <NUM>. However, since the change of depth of field may change the overall luminance on the image sensor <NUM>, <NUM> it may be important to compensate for such overall luminance changes due to the change in depth of field. This may for example be the case if the depth of field is changed by the adjustable aperture.

In some embodiments herein the depth of field is changed by adjusting the focal length of the lens-based optical imaging system <NUM>. Then at least one of the captured images may need to be geometrically transformed before it is possible to compare the images.

When a detection of the foreign object <NUM> occurs the image-processing device <NUM>, <NUM> may trigger a warning indicating the attachment of the foreign object <NUM> on the protective cover <NUM>, which in turn may trigger an appropriate cleaning of the protective cover <NUM>.

The method allows detection of the foreign object <NUM> on the protective cover <NUM> without disturbing the ordinary stream of image frames <NUM> since the method obtains information from image frames in the normal stream of images. Specifically, if the focal length of the lens-based optical imaging system <NUM> is adjustable, e.g., a zoom-lens, then the change in depth of field occurs naturally when changing the focal length. Thus, there is no need to capture images specifically for the purpose of detecting foreign objects on the protective cover <NUM>.

In some embodiments herein the video camera <NUM> may control the optical imaging system <NUM> to obtain a first depth of field. Thus, the image-processing device <NUM>, <NUM> may control the optical imaging system <NUM> to obtain the first depth of field.

The first depth of field may for example be obtained by controlling the focal length of the lens-based optical imaging system <NUM>. Thus, the first depth of field may correspond to a first focal length or a first zoom position of the lens-based optical imaging system <NUM>. A large focal length corresponds to a shallow depth of field compared to a small focal length given that the focus distance is the same.

In some other embodiments herein the first depth of field is obtained by controlling the adjustable aperture by the video camera <NUM>. Thus, the first depth of field may correspond to a first aperture opening of the lens-based optical imaging system <NUM>. Thus, in some embodiments wherein the optical imaging system <NUM> of the video camera <NUM> comprises the adjustable aperture the first depth of field is obtained by the first aperture opening of the optical imaging system <NUM>. For example, the first aperture opening may be a full-aperture opening, i.e., corresponding to the largest aperture opening such as f-number <NUM>. The full-aperture opening may correspond to a shallow depth of field. In another example, the first aperture opening corresponds to a larger depth of field. For example, an f-number of <NUM> may correspond to the larger depth of field. A large aperture opening corresponds to a shallow depth of field. However, since sharpness is reduced at high f-numbers due to diffraction the larger depth of field may be obtained at an f-number that is chosen also based on requirements on sharpness.

In some embodiments the video camera <NUM> controls both the focal length and the aperture opening to obtain the first depth of field. Thus, the first depth of field may correspond to a first setting of the focal length and the aperture opening.

The image-processing device <NUM>, <NUM> obtains the first captured image frame <NUM> captured by the video camera <NUM> with a first depth of field.

For example, the video camera <NUM> may obtain the first captured image frame by capturing it. The video server <NUM> may obtain the first captured image frame by receiving it from the video camera <NUM>.

Thus, in some embodiments herein the focal length of the optical imaging system <NUM> is adjustable and the first captured image frame <NUM> is captured with a first focal length of the optical imaging system <NUM> to obtain the first depth of field.

Since the second captured image frame <NUM> may be captured with a different exposure value than the first captured image frame <NUM> the exposure values or a ratio of the exposure values may be saved in order to compensate for the difference. Thus, the first captured image frame <NUM> may be captured with a first exposure value. The exposure values may be defined by the exposure time, the aperture opening and a gain of the image sensor <NUM>, <NUM>.

The video camera <NUM> may control the optical imaging system <NUM> to obtain the second depth of field which differs from the first depth of field.

The video camera <NUM> may control the optical imaging system <NUM> to obtain the second depth of field by controlling the focal length and/or the aperture opening of the optical imaging system <NUM>.

For example, in case the first aperture opening was a full-aperture opening the second aperture opening may be a smaller aperture opening, such as the smallest aperture opening. In another example, the first aperture opening is the largest aperture opening, then the second aperture opening may be the smallest aperture opening.

The image-processing device <NUM>, <NUM> obtains the second captured image frame <NUM> captured by the video camera <NUM> with a second depth of field which differs from the first depth of field.

At least partly a same part of the protective cover <NUM> may be captured by the first and second captured image frames <NUM>, <NUM> in order to have overlapping images of the foreign object <NUM>. In other words, the second captured image frame <NUM> may comprise an image of at least partly the same part of the protective cover as the first captured image frame. Preferably, a static scene is captured such that the luminance pattern is unaffected by dynamic objects, such as cars or bumble bees in the scene.

When the depth of field is adjusted by adjusting the focal length of the optical imaging system <NUM> then the second captured image frame <NUM> is captured with a second focal length of the optical imaging system <NUM> to obtain the second depth of field.

In some embodiments the second depth of field is obtained by a second aperture opening of the optical imaging system <NUM>.

The second captured image frame <NUM> may be captured with a second exposure value which may differ from the first exposure value. In other words, the first exposure value may differ from the second exposure value.

In embodiments herein the image-processing device <NUM>, <NUM> obtains further image frames <NUM>', <NUM>' based on the first and second captured image frames <NUM>, <NUM>. The further image frames <NUM>', <NUM>' may also be referred to as adjusted or transformed image frames <NUM>', <NUM>'. The further image frames <NUM>', <NUM>' are used for comparing luminance patterns between two images captured with different depths of field since the original captured image frames may be difficult to compare directly due to differences in optical parameters, such as the focal length and/or the aperture opening which in turn may produce differences in exposure values and fields of view. Further, the focal length and the aperture opening may also affect the image frames in other ways which may need to be compensated for, such as vignetting of the image frames. The further image frames <NUM>', <NUM>' may be compensated for vignetting before comparing the image frames with each other.

In order to obtain the further image frames <NUM>', <NUM>' copies of the first and second captured image frames <NUM>, <NUM> may be created, e.g., in the normal image processing pipeline <NUM>. The further image frames <NUM>', <NUM>' may then be created as (local) transforms of the captured image frames <NUM>, <NUM> and may be discarded once the comparison of the luminance patterns is done.

Thus, in some embodiments herein, some sort of adjustment of the captured image frames <NUM>, <NUM> is needed before the image frames may be compared. In an example illustrated by <FIG>, a first image frame <NUM>' is generated as a copy of the first captured image frame <NUM>, while a second image frame <NUM>' is generated as a copy of the second captured image frame <NUM>. As mentioned above, a difference in exposure values may be compensated by adjusting the luminance values of the first image frame <NUM>' with respect to the luminance values of the first captured image frame <NUM>. Each luminance value is associated with a respective pixel of the image frames. In an alternative embodiment, the difference in exposure values is compensated by adjusting the luminance values of the second image frame <NUM>' with respect to the luminance values of the second captured image frame <NUM>. It is also possible to adjust the luminance values of both the first image frame <NUM>' and the second image frame <NUM>'.

For example, a ratio of the second exposure value to the first exposure value may be used to adjust the luminance values of one of the first or second image frames <NUM>', <NUM>'. If a linear pixel intensity signal is used, such as the relative luminance, then the same value, such as the ratio of the second exposure value to the first exposure value, may be used to adjust all pixel intensity values.

Compensating for the difference in exposure value may be performed after compensating for vignetting.

In some embodiments wherein the focal length of the optical imaging system <NUM> is adjustable, at least one of the captured image frames <NUM>, <NUM> may need to be geometrically transformed before the luminance patterns can be compared in order to compensate for a difference in fields of view due to the different focal lengths.

Thus, in some embodiments herein, either the first image frame <NUM>' is obtained by geometrically transforming the first captured image frame <NUM>, or the second image frame <NUM>' is obtained by geometrically transforming the second captured image frame <NUM>. In other words, at least one of the first image frame <NUM>' and the second image frame <NUM>' may be obtained by geometrically transforming the respective first and second captured image frame <NUM>, <NUM>.

For example, if the captured image frames <NUM>, <NUM> have different amounts of optical aberrations, like barrel distortion, these may need to be compensated for by geometrically transforming one or both of the captured image frames <NUM>, <NUM>.

To compensate for different focal lengths a scaling of the captured image frames <NUM>, <NUM> may be performed. The scaling may be performed in combination with a cropping of the captured image frame with the shortest focal length, i.e., the zoomed-out image frame. For example, an area of the zoomed-out image frame may be cropped. What is left of the zoomed-out image frame is then upscaled to the same size as the zoomed-in image frame. Alternatively, the zoomed-in image frame may be down-scaled to the size of the cropped zoomed-out image frame.

The image-processing device <NUM>, <NUM> determines whether or not the protective cover <NUM> is partly covered by the foreign object <NUM> by analysing whether or not the first and second captured image frames <NUM>, <NUM> are affected by presence of the foreign object <NUM> on the protective cover <NUM> such that the difference between the first depth of field and the second depth of field results in a difference in the luminance pattern of corresponding pixels <NUM>', <NUM>' of the first image frame <NUM>' and the second image frame <NUM>'. The first image frame <NUM>' is based on the first captured image frame <NUM> and the second image frame <NUM>' is based on the second captured image frame <NUM>.

<FIG> illustrates a first corresponding pixel <NUM>' of the first image frame <NUM>' and a first corresponding pixel <NUM>' of the second image frame <NUM>'. There may be multiple corresponding pixels in the respective first and second image frames <NUM>', <NUM>'.

How to obtain the first image frame <NUM>' from the first captured image frame <NUM> and the second image frame <NUM>' from the second captured image frame <NUM> was described above in relation to action <NUM>. For example, at least one of the captured image frames <NUM>, <NUM> may be geometrically transformed and compensated for the difference in exposure values before action <NUM>.

Thus, this action may comprise comparing the luminance of pixels of the two derived image frames <NUM>', <NUM>'. Determining whether or not the protective cover <NUM> is partly covered by the foreign object <NUM> may comprise detecting the foreign object at the protective cover <NUM>. Thus, the image-processing device <NUM>, <NUM> may detect that the foreign object adheres to the protective cover <NUM> by detecting a difference in the luminance pattern between one or more pixels of the first image frame <NUM>', which is based on the first captured image frame <NUM> and corresponding one or more pixels of the second image frame <NUM>', which is based on the second captured image frame <NUM>.

As mentioned above when describing action <NUM>, the first image frame <NUM>' may be calculated from the first captured image frame <NUM>. Likewise, the second image frame <NUM>' may be calculated from the second captured image frame <NUM>. Such calculations may for example comprise adjusting luminance values. The luminance values may be adjusted based on the ratio of the second exposure value to the first exposure value. The calculations may further comprise geometrically transforming the captured images <NUM>, <NUM>.

After adjusting at least one of the captured image frames due to differences in exposure and/or focal length a clean protective cover <NUM> should result in a zero difference of the luminance of corresponding pixels. Thus, it may be determined that the protective cover <NUM> is partly covered by the foreign object <NUM> if the difference in the luminance pattern is above a threshold. In other words, the foreign object may be detected if the difference in the luminance pattern is above the threshold. The threshold may be predeterminable. Specifically, the image-processing device <NUM>, <NUM> may determine that the protective cover <NUM> is partly covered by the foreign object <NUM> if a larger depth of field results in a lower luminance of the luminance pattern than a shallower depth of field. The luminance of the luminance pattern may be a total luminance or a mean luminance of the luminance pattern, for example of the corresponding pixels. Combinations of the total luminance and the mean luminance may also be used for different corresponding pixels. In some other embodiments, the luminance of the luminance pattern may correspond to respective luminance values of respective corresponding pixels. For example, in some embodiments each respective pixel luminance value may need to differ by the threshold value in order to determine that the protective cover <NUM> is covered by the foreign object <NUM>.

In other words, the foreign object <NUM> may be detected if the larger depth of field results in the lower luminance of the corresponding pixels <NUM>', <NUM>' than the shallower depth of field. For example, if the luminance of the image frame based on the captured image frame with the larger depth of field is lower by the threshold difference than the luminance of the image frame based on the captured image frame with the shallower depth of field, then the image-processing device <NUM>, <NUM> may determine that the protective cover <NUM> is partly covered by the foreign object <NUM>. This is illustrated in <FIG> where the luminance of some corresponding pixels of first image frame <NUM>' is compared to the luminance of some corresponding pixels of second image frame <NUM>'. In <FIG> the first image frame <NUM>' is based on the shallower depth of field than the second image frame <NUM>'. That is, the first captured image frame <NUM> was captured with the shallower depth of field than the second captured image frame <NUM>.

In some embodiments herein the difference in the luminance pattern of corresponding pixels refers to a threshold number of pixels for which a difference in luminance is greater than a predeterminable threshold. For example, if the low depth of field image frame comprises ten corresponding pixels for which the luminance is greater than the luminance of the ten corresponding pixels of the large depth of field image frame then this may correspond to the difference of the luminance pattern. However, if there are only five corresponding pixels for which the luminance is significantly different then the method may determine that the difference in the luminance pattern is below the threshold and thus non-significant for detecting the foreign object <NUM>.

In some embodiments the difference in the luminance pattern is calculated based on a derivative of the pixel intensity signal, such as the relative luminance. For example, the image-processing device <NUM>, <NUM> may calculate a derivative of the relative luminance for the respective image frames. Regions or blocks of pixels may be compared by summing absolute values of gradients of the luminance of the pixels and then compare the sums between the different image frames.

In some embodiments wherein the first exposure value differs from the second exposure value then luminance of pixels of the first image frame <NUM>' is obtained based on luminance of corresponding pixels of the first captured image frame <NUM> and further based on a ratio of the second exposure value to the first exposure value, and/or luminance of pixels of the second image frame <NUM>' is obtained based on luminance of corresponding pixels of the second captured image frame <NUM> and further based on the ratio of the second exposure value to the first exposure value.

Thus, in order to detect the difference in the luminance pattern the image-processing device <NUM>, <NUM> may take into account the ratio of the second exposure value to the first exposure value.

By analysing the difference in the luminance pattern of corresponding pixels of the first image frame and the second image frame captured with the difference in depths of field determination of whether or not the protective cover is partly covered by the foreign object is improved since the images used for the determination are anyway captured such that no extra images nor image stream is necessary.

Different parts of the protective cover <NUM> will be imaged on different parts of the image sensor <NUM>, <NUM>. Thus, by analyzing differences in the luminance pattern of different parts of the image frames, corresponding to different parts of the image sensor <NUM>, <NUM>, different parts of the protective cover, which are imaged on the image sensor <NUM>, <NUM>, may be scanned for detection of foreign objects. A first pixel <NUM> of the first captured image frame <NUM> may be close to an optical axis. The first pixel <NUM> may then correspond to a part of the protective cover <NUM> close to the middle of the lens-based optical imaging system <NUM>. Pixels further out on the image sensor <NUM>, <NUM>, such as a second pixel <NUM> or a third pixel <NUM>, correspond to parts of the protective cover which are further away from the middle of the lens-based optical imaging system <NUM>.

The video camera <NUM> may adjust a direction in which the video camera <NUM> captures the image frames <NUM>, <NUM>. The direction may for example be adjusted by the motorised pan and tilt arrangement <NUM>. Thus, in some embodiments herein the direction in which the video camera <NUM> captures the image frames <NUM>, <NUM> is adjustable by the motorised pan and tilt arrangement <NUM>. By adjusting the direction of the video camera <NUM> actions <NUM>-<NUM> above may be repeated at multiple orientations of the video camera <NUM>. By adjusting the direction it is possible to analyse and detect multiple areas of the transparent protective cover <NUM>. For example, some parts of the transparent protective cover <NUM> may be extend outside the field of view <NUM> of the video camera <NUM>. Then it is also possible to capture a respective pair of image frames at multiple camera orientations, each camera orientation defined by the pan angle and the tilt angle and compare results of the detection analysis from the different camera orientations. Other parameters describing the camera orientation may also be used. Such parameters may for example be given by an accelerometer and a compass arranged in or at the video camera <NUM>.

In order to be able to compare results from the different camera orientations, the captured images frames capture at least partly a same part of the protective cover <NUM>. For example, in order to compare detection results of the object <NUM> on the protective cover <NUM> the image frames that are used for the comparison capture an image of the object <NUM>.

Such comparisons may corroborate a first result or trigger further analysis. For example, a first analysis at a first camera orientation may indicate that there is an object <NUM> on the protective cover <NUM>. Then a second analysis at a second camera orientation also indicates that there is an object <NUM> on the protective cover <NUM>. Then the two analyses corroborate each other, and appropriate actions may be taken as a result. An example of such an action is given below in action <NUM>. <FIG> illustrates how a further pair of captured images <NUM>, <NUM> is transformed into a further pair of images <NUM>', <NUM>' which may be used for detection of objects and comparison with the first pair of images <NUM>', <NUM>'.

Thus, the first and second captured image frames <NUM>, <NUM> may be captured at a first orientation of the video camera <NUM>. A further first captured image frame <NUM> and a further second captured image frame <NUM> may be captured at a second orientation of the video camera <NUM>.

The further first captured image frame <NUM> is captured with a further first depth of field and the further second captured image frame <NUM> is captured with a further second depth of field which differs from the further first depth of field.

The first orientation corresponds to first values of pan and tilt of the pan and tilt arrangement <NUM> and the second orientation corresponds to second values of pan and tilt of the pan and tilt arrangement <NUM> which at least partly differ from the first values. The image frames captured at the first and the second orientation capture at least partly the same part of the protective cover <NUM>.

In some embodiments herein determining whether or not the protective cover <NUM> is partly covered by the foreign object <NUM> is based on an evaluation of the determining at both the first and the second orientation. In other words, detecting the foreign object <NUM> at the protective cover <NUM> may be based on an evaluation of the detection at both the first and the second orientation.

For example, the image-processing device <NUM>, <NUM> may determine whether or not the protective cover <NUM> is partly covered by the foreign object <NUM> based on analysing whether or not the first and second captured image frames <NUM>, <NUM> are affected by presence of the foreign object <NUM> such that the difference between the first depth of field and the second depth of field results in the difference in the luminance pattern of corresponding pixels of the first image frame <NUM>' and the second image frame <NUM>', and further based on whether or not the further first and further second captured image frames <NUM>, <NUM> are affected by presence of the foreign object <NUM> such that the difference between the further first depth of field and the further second depth of field results in a difference in the luminance pattern of corresponding pixels of the further first image frame <NUM>' and the further second image frame <NUM>'.

In some embodiments the image-processing device <NUM>, <NUM> triggers a warning indication in response to determining that the protective cover <NUM> is partly covered by the foreign object <NUM>. In other words, the image-processing device <NUM>, <NUM> may trigger the warning indication in response to detecting the foreign object <NUM>.

In some embodiments herein the warning indication is sent to the VMS or as an email to an operator. In some other embodiments herein the warning indication is stored in a memory of the video camera <NUM>. The VMS or the operator may then check for the warning, e.g., once every day.

With reference to <FIG>, a schematic block diagram of embodiments of an image-processing device <NUM>, <NUM> is shown. The image-processing device <NUM>, <NUM> is configured to determine whether or not the transparent protective cover <NUM> of the video camera <NUM>, is partly covered by the foreign object <NUM>.

As mentioned above, in some embodiments herein the image-processing device <NUM>, <NUM> is the video camera <NUM>. The video camera <NUM> may be a PTZ video camera. Thus, the image-processing device <NUM>, <NUM> may be a PTZ video camera.

The image-processing device <NUM>, <NUM> may comprise a processing module <NUM>, such as a means for performing the methods described herein. The means may be embodied in the form of one or more hardware modules and/or one or more software modules.

The image-processing device <NUM>, <NUM> may further comprise a memory <NUM>. The memory may comprise, such as contain or store, instructions, e.g. in the form of a computer program <NUM>, which may comprise computer readable code units which when executed on the image-processing device <NUM>, <NUM> causes the image-processing device <NUM>, <NUM> to perform the method of determining whether or not the transparent protective cover <NUM> of the video camera <NUM> is partly covered by the foreign object <NUM>. For example, the computer program <NUM> may be executed on one or more processors, such as a processing circuit <NUM>, of the image-processing device <NUM>, <NUM> to cause the one or more processors to perform the method of determining whether or not the transparent protective cover <NUM> of the video camera <NUM> is partly covered by the foreign object <NUM>.

According to some embodiments herein, the image-processing device <NUM>, <NUM> and/or the processing module <NUM> comprises the processing circuit <NUM> as an exemplifying hardware module, which may comprise one or more processors. Accordingly, the processing module <NUM> may be embodied in the form of, or 'realized by', the processing circuit <NUM>. The instructions may be executable by the processing circuit <NUM>, whereby the image-processing device <NUM>, <NUM> is operative to perform the methods of <FIG>. As another example, the instructions, when executed by the image-processing device <NUM>, <NUM> and/or the processing circuit <NUM>, may cause the image-processing device <NUM>, <NUM> to perform the method according to <FIG>.

In view of the above, in one example, there is provided an image-processing device <NUM>, <NUM> for determining whether or not the transparent protective cover <NUM> of the video camera <NUM> is partly covered by the foreign object <NUM>. Again, the memory <NUM> contains the instructions executable by said processing circuit <NUM> whereby the image-processing device <NUM>, <NUM> is operative for performing the method according to <FIG>:
The image-processing device <NUM>, <NUM> may further be operative to perform the methods according to the detailed embodiments described above in connection to <FIG>.

<FIG> further illustrates a carrier <NUM>, or program carrier, which comprises the computer program <NUM> described above. The carrier <NUM> may be one of an electronic signal, an optical signal, a radio signal and a computer readable medium.

In some embodiments, the image-processing device <NUM>, <NUM> and/or the processing module <NUM> may comprise one or more of an obtaining module <NUM>, a determining module <NUM>, a pan-tilt adjusting module <NUM>, an optical controlling module <NUM>, a warning triggering module <NUM>, as exemplifying hardware modules. In other examples, one or more of the aforementioned exemplifying hardware modules may be implemented as one or more software modules.

Moreover, the processing module <NUM> may comprise an Input/Output unit <NUM>. According to an embodiment, the Input/Output unit <NUM> may comprise the image sensor <NUM>, <NUM> configured for capturing the image frames described above such as the first captured image frame <NUM> and the second captured image frame <NUM>.

Accordingly, the image-processing device <NUM>, <NUM> is configured for determining whether or not the transparent protective cover <NUM> of the video camera <NUM> is partly covered by the foreign object <NUM>.

Therefore, according to the various embodiments described above, the image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the obtaining module <NUM>, <NUM> is configured to obtain the first captured image frame <NUM> captured by the video camera <NUM> with the first depth of field and to obtain the second captured image frame <NUM> captured by the video camera <NUM> with the second depth of field which differs from the first depth of field.

In some embodiments the image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the obtaining module <NUM>, <NUM> is configured to capture at least partly the same part of the protective cover <NUM> by the first and second captured image frames <NUM>, <NUM>.

The image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the determining module <NUM> is configured to determine whether or not the protective cover <NUM> is partly covered by the foreign object <NUM> by analysing whether or not the first and second captured image frames <NUM>, <NUM> are affected by presence of the foreign object <NUM> on the protective cover <NUM> such that the difference between the first depth of field and the second depth of field results in the difference in the luminance pattern of corresponding pixels <NUM>', <NUM>' of the first image frame <NUM>' and the second image frame <NUM>'. The first image frame <NUM>' is based on the first captured image frame <NUM> and the second image frame <NUM>' is based on the second captured image frame <NUM>.

In some embodiments herein the focal length of the optical imaging system <NUM> is adjustable. Then the image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the obtaining module <NUM> may be configured to capture the first captured image frame <NUM> with the first focal length of the optical imaging system <NUM> to obtain the first depth of field and capture the second captured image frame <NUM> with the second focal length of the optical imaging system <NUM> to obtain the second depth of field.

When focal length of the optical imaging system <NUM> is adjustable and the second focal length differs from the first focal length, then the image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the obtaining module <NUM> may be configured to obtain either the first image frame <NUM>' by geometrically transforming the first captured image frame <NUM>, or the second image frame <NUM>' by geometrically transforming the second captured image frame <NUM>.

In some embodiments the optical imaging system <NUM> of the video camera <NUM> comprises the adjustable aperture. Then the image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the obtaining module <NUM> may be configured to obtain the first depth of field by the first aperture opening of the optical imaging system <NUM>, and the second depth of field by the second aperture opening of the optical imaging system <NUM>.

The image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the determining module <NUM> may be configured to determine that the protective cover <NUM> is partly covered by the foreign object <NUM>, i.e., the foreign object is detected, if the difference in the luminance pattern is above the threshold.

The image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the determining module <NUM> may be configured to determine that the protective cover <NUM> is partly covered by the foreign object <NUM> if the larger depth of field results in the lower luminance of the luminance pattern than the shallower depth of field.

In some embodiments wherein the first captured image frame <NUM> is captured with the first exposure value, the second captured image frame <NUM> is captured with the second exposure value, which differs from the first exposure value. Then the image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the obtaining module <NUM> may be configured to obtain luminance of pixels of the first image frame <NUM>' based on luminance of corresponding pixels of the first captured image frame <NUM> and further based on the ratio of the second exposure value to the first exposure value, and/or luminance of pixels of the second image frame <NUM>' based on luminance of corresponding pixels of the second captured image frame <NUM> and further based on the ratio of the second exposure value to the first exposure value.

In some embodiments the further first captured image frame <NUM> and the further second captured image frame <NUM> are captured at the second orientation of the video camera <NUM> and the further first captured image frame <NUM> is captured with the further first depth of field and the further second captured image frame <NUM> is captured with the further second depth of field which differs from the further first depth of field. Then the image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the determining module <NUM> is configured to determine whether or not the protective cover <NUM> is partly covered by the foreign object <NUM> further based on whether or not the further first and further second captured image frames <NUM>, <NUM> are affected by presence of the foreign object <NUM> such that the difference between the further first depth of field and the further second depth of field results in the difference in the luminance pattern of corresponding pixels of the further first image frame <NUM>' and the further second image frame <NUM>'.

The image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the pan-tilt adjusting module <NUM> may be configured to adjust the motorised pan and tilt arrangement to adjust the direction in which the video camera <NUM> captures the image frames <NUM>, <NUM>.

The image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the warning triggering module <NUM> may be configured to trigger the warning indication in response to determining that the protective cover <NUM> is partly covered by the foreign object <NUM>.

The image-processing device <NUM>, <NUM> and/or the processing module <NUM> and/or the optical controlling module <NUM> may be configured to control the optical imaging system <NUM> to obtain the first depth of field and control the optical imaging system <NUM> to obtain the second depth of field which differs from the first depth of field.

As used herein, the term "module" may refer to one or more functional modules, each of which may be implemented as one or more hardware modules and/or one or more software modules and/or a combined software/hardware module. In some examples, the module may represent a functional unit realized as software and/or hardware.

As used herein, the term "computer program carrier", "program carrier", or "carrier", may refer to one of an electronic signal, an optical signal, a radio signal, and a computer readable medium. In some examples, the computer program carrier may exclude transitory, propagating signals, such as the electronic, optical and/or radio signal. Thus, in these examples, the computer program carrier may be a non-transitory carrier, such as a non-transitory computer readable medium.

As used herein, the term "processing module" may include one or more hardware modules, one or more software modules or a combination thereof. Any such module, be it a hardware, software or a combined hardware-software module, may be an obtaining means, determining means, pan-tilt adjusting means, optical controlling means, warning triggering means or the like as disclosed herein. As an example, the expression "means" may be a module corresponding to the modules listed above in conjunction with the figures.

As used herein, the term "software module" may refer to a software application, a Dynamic Link Library (DLL), a software component, a software object, an object according to Component Object Model (COM), a software component, a software function, a software engine, an executable binary software file or the like.

The terms "processing module" or "processing circuit" may herein encompass a processing unit, comprising e.g. one or more processors, an Application Specific integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or the like. The processing circuit or the like may comprise one or more processor kernels.

As used herein, the expression "configured to/for" may mean that a processing circuit is configured to, such as adapted to or operative to, by means of software configuration and/or hardware configuration, perform one or more of the actions described herein.

As used herein, the term "action" may refer to an action, a step, an operation, a response, a reaction, an activity or the like. It shall be noted that an action herein may be split into two or more sub-actions as applicable. Moreover, also as applicable, it shall be noted that two or more of the actions described herein may be merged into a single action.

As used herein, the term "memory" may refer to a hard disk, a magnetic storage medium, a portable computer diskette or disc, flash memory, Random Access Memory (RAM) or the like. Furthermore, the term "memory" may refer to an internal register memory of a processor or the like.

As used herein, the term "computer readable medium" may be a Universal Serial Bus (USB) memory, a DVD-disc, a Blu-ray disc, a software module that is received as a stream of data, a Flash memory, a hard drive, a memory card, such as a MemoryStick, a Multimedia Card (MMC), Secure Digital (SD) card, etc. One or more of the aforementioned examples of computer readable medium may be provided as one or more computer program products.

As used herein, the term "computer readable code units" may be text of a computer program, parts of or an entire binary file representing a computer program in a compiled format or anything there between.

As used herein, the terms "number" and/or "value" may be any kind of number, such as binary, real, imaginary or rational number or the like. Moreover, "number" and/or "value" may be one or more characters, such as a letter or a string of letters. "Number" and/or "value" may also be represented by a string of bits, i.e. zeros and/or ones.

As used herein, the expression "in some embodiments" has been used to indicate that the features of the embodiment described may be combined with any other embodiment disclosed herein.

Claim 1:
A method, performed by an image-processing device (<NUM>, <NUM>), for determining whether or not a transparent protective cover (<NUM>) of a video camera (<NUM>) comprising a lens-based optical imaging system (<NUM>) is partly covered by a foreign object (<NUM>) on the protective cover (<NUM>), wherein a focal length of the optical imaging system (<NUM>) is adjustable, the method comprising:
obtaining (<NUM>) a first captured image frame (<NUM>) captured by the video camera (<NUM>) with a first depth of field, wherein the first captured image frame (<NUM>) is captured with a first focal length of the optical imaging system (<NUM>) to obtain the first depth of field;
obtaining (<NUM>) a second captured image frame (<NUM>) captured by the video camera (<NUM>) with a second depth of field which differs from the first depth of field, wherein the second captured image frame (<NUM>) is captured with a second focal length of the optical imaging system (<NUM>) to obtain the second depth of field; and
determining (<NUM>) whether or not the protective cover (<NUM>) is partly covered by the foreign object (<NUM>) by analysing whether or not the first and second captured image frames (<NUM>, <NUM>) are affected by presence of the foreign object (<NUM>) on the protective cover (<NUM>) such that the difference between the first depth of field and the second depth of field results in a difference in a luminance pattern of corresponding pixels of a first image frame (<NUM>') and a second image frame (<NUM>'), wherein the first image frame (<NUM>') is based on the first captured image frame (<NUM>) and the second image frame (<NUM>') is based on the second captured image frame (<NUM>);
wherein the luminance pattern corresponds to a pattern of a gradient of a relative luminance of the corresponding pixels,
wherein the first image frame (<NUM>') is obtained by geometrically transforming the first captured image frame (<NUM>), and/or the second image frame (<NUM>') is obtained by geometrically transforming the second captured image frame (<NUM>), and
wherein it is determined that the protective cover (<NUM>) is partly covered by the foreign object (<NUM>) if the difference in the luminance pattern is above a threshold.