Patent Description:
The present invention relates to cameras, and more specifically to detecting objects that are located in close proximity to a lens of a camera.

Camera health monitoring is becoming popular, especially in large monitoring systems that may contain hundreds or thousands of where it may not be practically feasible to manually monitor the status of each camera. One common problem that occurs with monitoring cameras, both in indoor and outdoor environments, is that spiders and insects get attracted to the infrared light that is often built into the cameras.

The insects (e.g., butterflies, moths, etc.) and spiders may either sit on the lens itself or be located in close proximity to the lens, thereby blocking the view of the camera and making the camera less useful for detecting events that occur in the scene that is monitored by the camera. When the insect or spider moves across the lens or across the view, it may also trigger false alarms, as the camera analytics software may interpret its movements as being movements in the scene rather than movements in close proximity to or on the lens itself.

Further, spiders in particular (but also some insects) may leave "belongings" in front of the lens. One example of such a belonging is a spiderweb. As smaller insects and spiders get attracted by the infrared light in the camera, insects and spiders tend to leave their belongings in front of the lens. When belongings, such as a spiderweb, are illuminated by the lights of the camera, especially when there are water droplets in the belongings, a significant portion of the light gets reflected back into the camera sensor by the belongings, rendering the camera more or less useless for monitoring purposes until the belongings have been removed. There may also be situations when a camera is tampered with by a human for illegal purposes, for example, by obscuring the lens by placing some kind of object in close proximity to the lens. <CIT> disclosing a photographing device and method used for monitoring and recognizing the appearing of objects. The intensity comparison is made with information stored within a database, under different illumination conditions.

For at least these reasons, it is desirable to have ways of automatically detecting objects that are close to the lens, such that false alarms can be avoided and/or to get an indication that some action on the camera (e.g., removing the obstacle) is required.

According to a first aspect, the invention relates to a method, in a computer system, for determining a presence of an object located in close proximity to a lens through which an image capturing device monitors a scene as claimed in the appended independent claim <NUM>, system claim <NUM> and computer program product claim <NUM>.

This provides a way of improving techniques for detecting objects that are close to the lens, and often without having to make any modifications to the hardware setup of the camera, as conventional cameras typically have illuminators that can be controlled to illuminate scenes from differing angles. Intensity comparisons are also computationally low-cost comparisons to make, and by first compensating for overall intensity differences, any local intensity differences will show up more clearly in the image comparison, thus allowing for an easy determination of the presence of an object in close proximity to the lens. As a consequence of being able to automatically determine whether an object is present close to the lens, the number of false alarms can be decreased, and appropriate alerts can be generated in the event anything needs to be attended to with respect to the camera. It also reduces the need for manual intervention and monitoring of individual camera feeds, and can thus also contribute to cost savings with respect to maintaining the camera system.

According to one embodiment the first infrared illumination source and the second infrared illumination source are arranged to illuminate the object with essential equal intensities and wavelengths. Having equal intensities and wavelengths for the illumination from both sides creates conditions that are as similar as possible in terms of illumination and take away any discrepancies that might occur from the illuminated object having different reflecting properties for different wavelengths. It also facilitates comparison of images and makes it easier to more clearly identify the differences that arise purely from the illumination of the object from two different angles.

According to one embodiment, the first and second infrared illumination sources each comprise one or more infrared light emitting diodes. Light emitting diodes (LEDs) are standard elements in most cameras, and by using already existing components, no major modifications need to be made to the camera hardware. The number of LEDs in each illumination source may also vary. In some cases, a single LED in each source may be sufficient and in other cases multiple LEDs in each illumination source may be required.

According to one embodiment, the method further comprises deactivating the second infrared illumination source, and activating a third infrared illumination source arranged to illuminate the scene from a third angle; acquiring a third image; and comparing intensity information of the first image, the second image, and the third image to determine the presence of an object located in close proximity to the lens. Being able to use more than two infrared illumination sources can allow a better determination as to the presence of an object, both because more images can be obtained, and because it allows pairwise comparison among the three images. This creates greater certainty in determining the presence of an object close to the lens.

According to one embodiment, the method further comprises activating and deactivating a third infrared illumination source along with activating and deactivating the first infrared illumination source; and activating and deactivating a fourth infrared illumination source along with activating and deactivating the second infrared illumination source, wherein the first and third infrared illumination sources are arranged essentially orthogonally from the second and fourth infrared illumination sources. Using two groups of illumination sources that are arranged essentially orthogonally with respect to each other provides not only good illumination, but also enhances any intensity differences that may arise as a result of an object located close to the lens, thereby improving the accuracy of the detection of such an object.

According to one embodiment, the method further comprises: comparing a local intensity difference pattern to a reference intensity difference pattern; and providing an indication of the presence of an object located in close proximity to the lens when there is a match between the local intensity difference pattern and the reference intensity difference pattern. Having a different reference intensity patterns may allow not only detection of an object close to the lens, but also - at least to some degree - identification of what the object might be. For example, a butterfly may provide a different intensity pattern compared to a spider web, or compared to someone having tampered with the camera, and may require different types of actions to be taken. Thus, different types of alerts could be generated based on what the most likely object is.

According to one embodiment, the method can include using a neural network to evaluate the local intensity differences and make a determination as to the presence or absence of an object located in close proximity to the lens. Similar to what was described above, a neural network can be taught to automatically identify specific objects based on their intensity profile.

According to one embodiment, the method can include: in response to determining that there is an object located in close proximity to the lens, sending a notification to a camera operator. As was mentioned above, the notification can be automatically generated based on the result of the detection. There may also be various types of notifications, or different types of personnel may be notified based on what the object possibly might be. For example, it may be more urgent to address a situation where a camera has been tampered with, compared to a situation where a spiderweb has been detected.

According to one embodiment, the object can be a spider, an insect, a belonging of a spider, or a belonging of an insect. These are some examples of various insects and "belongings" that are common causes of problems in environments where monitoring cameras have been set up.

According to one embodiment, acquiring the first image by the image capturing device and acquiring the second image by the image capturing device includes acquiring several short exposure images and combining the short exposure images into the first image and the second image, respectively. This can be useful in situations where the scene monitored by the camera includes a lot of movement, such as when the camera looks out over water with waves or where is a tree in the background with branches and leaves moving in the wind. By adding or averaging several short exposure pictures of this kind, such movement can be evened out and more accurate identification of an object close to the lens can be obtained.

According to one embodiment, acquiring the initial image by the image capturing device and acquiring the final image by the image capturing device includes acquiring several short exposure images and combining the short exposure images into the initial image and the final image, respectively. Similar advantages to those described in the previous paragraph can be obtained for the initial and final image.

The computer program involves advantages corresponding to those of the method and may be varied similarly.

Other features and advantages of the invention will be apparent from the description and drawings, and from the claims.

As was described above, one goal with the various embodiments of the invention is to be able to automatically detect objects that are close to the lens, such that false alarms can be avoided and/or to get an indication that some action on the camera (e.g., removing the obstacle) is required. The invention stems from the general principle that objects that are close to the lens of the camera will look different, that is, have different intensities in an image, when they are illuminated from different angles. Therefore, when two images are recorded by a camera, and the images are captured using IR illumination from different angles, objects in the image will look different (i.e., have different intensities in the acquired images). Thus, by comparing the acquired images, and in particular looking for local intensity differences between the acquired images, it can be concluded whether an object is present close to the lens. What is considered "close" may vary, depending on the particular situation at hand, but as a general guideline, "close" to the camera in this context refers to objects that are located within approximately <NUM> meters of the camera, and preferably within <NUM> meters from the lens of the camera. In response to detecting an object, an appropriate alert can be generated to address the issue. Various embodiments will now be described in further detail by way of example and with reference to the figures.

<FIG> is a flowchart showing a method <NUM> for determining whether an object is present close to a lens, in accordance with one embodiment. This method <NUM> can be performed automatically at various intervals, as needed, to efficiently detect objects that are located close to the lens. As can be seen in <FIG>, the method <NUM> begins by activating a first infrared illumination source, step <NUM>. This is schematically illustrated in <FIG>, which schematically shows a monitoring camera <NUM> with a first infrared illumination source <NUM> activated. Typically, the infrared illumination source includes several IR LEDs, which are conventional in most cameras that are used for purposes of monitoring scenes. However, IR LEDs are not the only type of illuminators that can be used in conjunction with the invention and other types of illuminators such as laser diodes could also be used.

Next, a first image is captured, step <NUM>. This image is shown schematically as image #<NUM>, <NUM>, in <FIG>. The image is typically captured using the same type of image capture as is used by the camera <NUM> during normal operation, for example, using a CMOS sensor in combination with rolling shutter technique.

When the first image has been captured, the first infrared illumination source <NUM> is turned off and a second infrared illumination source <NUM> is turned on, step <NUM>. Typically, the first and second infrared illumination sources <NUM>, <NUM> are located at the opposite side of the lens <NUM> to be able to illuminate the object from two opposite angles (e.g., top and bottom, or left and right, etc.). However, it should be noted that this is not a requirement, and that there could be a smaller angular distance between the first infrared illumination source <NUM> and the second infrared illumination source <NUM>. One or more IR LEDs could also be part of both the first group <NUM> and the second group <NUM>. Further, it should be noted that while the first and second infrared illumination sources <NUM>, <NUM> are shown in <FIG> as being integrated in the camera <NUM>, they could also be separate units that are placed outside (e.g. on either side) of the camera <NUM> itself. With the second infrared illumination source turned on, a second image <NUM> is acquired, step <NUM>, in the same way as the first image <NUM> was acquired.

Finally, image analysis <NUM> is performed to determine whether an object is present close to the lens, step <NUM>. It should be noted that this image analysis <NUM> is not concerned with the actual depths of the objects themselves - it merely pertains to determining whether an object is present or not close to the lens. This is a significant advantage, as depth determination typically requires considerable data processing. The comparably low computing resources needed for the various embodiments of the present invention makes it possible to implement it in environments where there may not be significant computing resources available, such as in the camera itself, for example. In addition, having lower computing resource requirements allows overall power consumption for the camera to be reduced. This is important, not only to reduce heat, but also when the LEDs are used nighttime, since the camera power supply is limited by the power available in accordance with the Ethernet standard.

In one embodiment, the image analysis <NUM> works as follows. First, any global intensity differences between the two images are calculated and compensated for. Such global intensity differences can be, for example, due to exposure differences or IR LED strength differences. This global intensity difference is the same for all pixels in the image and can be compensated for by adding the global intensity difference to one of the images to even out the global intensity difference.

Next, it is determined whether there are any local intensity difference patterns between the first image <NUM> and the second image <NUM>. Expressed differently, it is determined whether the intensities differ more in some areas than other areas between the images. If such a local pattern exists, this is an indication of the presence of a close object. In some embodiments, this analysis alone may result in a positive determination of a close object. In other embodiments, it can serve as an indication that a more advanced analysis may be needed, for example, where the determined local pattern is compared to reference intensity difference patterns to identify the type of object. Based on this image analysis, an alert can be generated to a user of the camera system or to some maintenance crew. Alternatively, the alert can serve as a trigger to an automated system, such as wipers, vibration, noise, etc., for removing, for example, spiderweb from the area close to the lens. Many such variations can be envisioned by those having ordinary skill in the art.

In many cameras, a portion of their housing protrudes above and/or on the sides of the lens a certain distance to act as a weather protection (similar to how awnings on houses often protrude above the windows). Generally this camera housing protrusion is not shown in the images captured by the lens. However, when the first and second infrared illumination sources are lit, this could sometimes cause light to bounce off the protrusion, which creates a weak, but stable intensity difference between the two images that are recorded with the first and second illumination sources are activated. This discrepancy can be addressed by recording two calibration images, in which each illumination source is activated in a completely dark room and without any object close to the lens, and then storing these calibration images in the camera. When the camera later on is in use and registers the first and second images, each of these two calibration pictures can first be subtracted from its respective image prior to performing the image processing described above. In this manner any intensity differences arising as a result of the configuration of the camera housing can be taken into account.

<FIG> is a flowchart showing a method <NUM> for determining whether an object is present close to a lens, in accordance with a different embodiment. In <FIG>, an initial image is first captured in step <NUM>. The initial image is captured with both IR illumination sources turned on when the scene is dark (e.g., at night), and with both IR illumination sources turned off when the scene is light (e.g. during daytime). Steps <NUM>-<NUM> are then performed in a corresponding way to steps <NUM>-<NUM> described above with reference to <FIG>. After acquiring the second image in step <NUM>, a final image is acquired in step <NUM>. This final image is, just like the initial image, also captured with both IR illumination sources turned on or turned off, depending on whether the scene is dark or light.

The image analysis in step <NUM> is similar to the image analysis in step <NUM>, but with one significant difference. In addition to the image analysis described in step <NUM>, the initial image captured in step <NUM> is compared to the final image captured in step <NUM>. If these images, which were both captured with both IR illumination sources turned on or turned off, differ from each other, this is an indication that there likely was movement in the scene during the capture of the first and second image. That is, any differences detected when comparing the first and second images may not actually be due to an object being close to the lens, but could instead be due to movement in the scene captured by the camera. Therefore, capturing and comparing the initial image and the final image on either side of the capturing of the first and second images, may serve as a good additional "verification" that the result obtained when comparing the first and second images is indeed valid.

It should be noted that while the description has focused on two images and two IR illumination sources, the same concepts can be extended to any number of images and any number of IR illumination sources. The exact choice of how many images or IR illumination sources should be used in a given situation fall well within the skills of those having ordinary skill in the art.

Further, as was mentioned above, instead of capturing a single first image, a single second image, and so on, in some implementations multiple short exposure images are captured and then added together to form the first and second images, respectively. This is beneficial in that it allows intensity differences that are due to movement in the image to be taken into account. For example, a camera may be overlooking the ocean, or there might be a tree moving in the wind within the camera's field of view. Effects from these types of movement can be mitigated with this type of setup where several short exposure images are taken rather than a single one.

A computer readable signal medium may be any computer medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

Aspects of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. Each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions.

The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Thus, many other variations that fall within the scope of the claims can be envisioned by those having ordinary skill in the art.

Claim 1:
A method for determining a presence of an object located in close proximity to a lens (<NUM>) through which an image capturing device (<NUM>) monitors a scene, the method comprising:
activating a first infrared illumination source (<NUM>) arranged to illuminate the scene from a first angle;
acquiring, by the image capturing device (<NUM>), a first image (<NUM>) through the lens;
deactivating the first infrared illumination source (<NUM>), and activating a second infrared illumination source (<NUM>) arranged to illuminate the scene from a second angle;
acquiring, by the image capturing device (<NUM>), a second image (<NUM>) through the lens; and
comparing intensity information of the first image (<NUM>) and the second image (<NUM>) to determine the presence of an object located in close proximity to the lens (<NUM>) wherein comparing intensity information includes:
compensating for overall intensity differences between the first image (<NUM>) and the second image (<NUM>);
determining whether any local intensity differences remain; and
in response to a positive determination that there are local intensity differences, providing an indication of the presence of an object located in close proximity to the lens.