Abstract:
A camera ( 1 ) arranged to generate an IR image ( 21 ) and a visible light image ( 23 ) is proposed, said camera comprising a thermal imaging part ( 3 ) for registering IR radiation from a first field of view, a visible light imaging part ( 9 ) for registering visible light from a second field of view at least partially overlapping with the first field of view, and a processing device ( 15 ) for processing the information related to the registered IR radiation and the registered visible light information. The processing device ( 15 ) is arranged to identify any moving objects that are detectable by the thermal imaging part ( 3 ) and not by the visible light imaging part ( 9 ), and presenting said moving objects, if any, in such a way that they will be easily recognizable by an operator.

Description:
TECHNICAL FIELD 
       [0001]    The present invention relates to a camera for providing IR images and visible light images as defined in the preamble of claim  1 , a method using such a camera, as defined in the preamble of claim  10  and a computer program product as defined in the preamble of claim  18 . 
       BACKGROUND AND RELATED ART 
       [0002]    In many applications there is a desire to detect the presence of gas. For example, there is often a need to monitor gas pipes for leakages. Many gases are not visible with visible-light cameras, since they do not absorb light in the visible range. 
         [0003]    Cameras exist that are specially adapted for gas detection. Such cameras operate in a wave length range where the gas is easily detectable. Such cameras are specially designed only for gas detection, and will not necessarily work for different types of gas. It would be possible to use an IR camera for imaging gas since gas generally fluctuates, but it will not appear very clearly in the thermal image especially in the case of small amounts of gas. Therefore, there is a need for an IR camera that will assist the operator in identifying the presence gas in the thermal image. 
       OBJECT OF THE INVENTION 
       [0004]    It is an object of the invention to facilitate the detection of gas in thermal images captured by an IR camera. 
       SUMMARY OF THE INVENTION 
       [0005]    This object is achieved according to the present invention by a camera arranged to generate an IR image and a visible light image, said camera comprising a thermal imaging part for registering IR radiation from a first field of view, a visible light imaging part for registering visible light from a second field of view at least partially overlapping with the first field of view, and a processing device for processing the information related to the registered IR radiation and the registered visible light information, said camera being characterized in that the processing device is arranged to identify any moving objects that are detectable by the thermal imaging part and not by the visible light imaging part, and presenting said moving objects, if any, in such a way that they will be easily recognizable by an operator. 
         [0006]    The object is also achieved by a method of presenting information related to IR radiation comprising the following steps:
       registering IR radiation from a first field of view,   registering visible light from a second field of view at least partially overlapping with the first field of view,   processing the information related to the registered IR radiation and the registered visible light information, characterized in that it comprises the steps of   identifying any moving objects that are detectable by the thermal imaging part and not by the visible light imaging part, and   presenting said moving objects, if any, in such a way that they will be easily recognizable by an operator.       
 
         [0012]    The object is also achieved by a computer program product for use in a processing device arranged to control a camera arranged to generate IR images and visible light images, the camera comprising a thermal imaging part for registering IR radiation from a first field of view, a visible light imaging part for registering visible light from a second field of view at least partially overlapping with the first field of view, said computer program product comprising computer readable code means which, when run in the processing device, will cause the processing device to identify any moving objects that are detectable by the thermal imaging part and not by the visible light imaging part, and presenting said moving objects, if any, in such a way that they will be easily recognizable by an operator. 
         [0013]    Hence, the inventive idea based on the recognition of the fact that most moving objects will be detectable both in thermography and in visible light imaging, such as machines or human beings. Other moving objects, such as gas, will be detectable only by thermography. Many IR cameras today also comprise a visible light camera and are arranged to present images that are in some way based on both IR and visible light information. 
         [0014]    The object is therefore achieved according to the invention by capturing at least two IR images and at least two visible light images, to identify any moving objects that are detectable with visible light photography and IR photography, respectively. The functions of the processing device are typically controlled by a computer program product stored in the processing device. Preferably, the processing device is arranged to compare a moving object identified in the visible light image to a moving object identified in the thermal image and determine if they represent the same object. 
         [0015]    Moving objects in the IR image are potentially gas, which will generally not be detectable in the visible light image. Other moving objects will be detectable both in the IR image and in the visible light image. By identifying the moving objects that appear in both the visible light image and the IR image and eliminating them from the IR image, the remaining moving objects in the IR image are potentially caused by gas. The remaining moving objects, that is, the ones that are only visible in the IR image and not in the visible light image are marked in the IR image, preferably by applying a suitable colour. 
         [0016]    The processing device is preferably arranged to determine the presence of a moving object in the visible light image and/or the thermal image, respectively, by obtaining a first image at a first point in time, obtaining a second image at a second point in time, comparing the first and the second image and determining the presence of a moving object based on the difference between the first and the second image. The presence of a moving object may be determined in different ways. For example a difference image between the first and the second image may be established and the presence of any moving object may be determined based on the difference image. Alternatively, the presence of a moving object may be determined based on the “optical flow” method 
         [0017]    In a preferred embodiment processing device is arranged to filter out small changes between the images to eliminate the effects of camera movements. 
         [0018]    Preferably, the processing device is arranged to align and stabilize the visible light image and the thermal image before identifying any moving objects in the images. This will also serve to eliminate the effects of camera movements, which will occur, for example if the camera is hand-held. 
         [0019]    The processing device is preferably arranged to compare the moving object in the visible light image and the thermal image on the basis of their position in the respective image, their size and/or their shape. While the size and shape of an object will never coincide completely between an IR image and a visible light image, they should be approximately the same, within certain limits to ensure that what is treated as the same object really is the same object. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0020]    The invention will be described in more detail in the following, by way of example and with reference to the appended drawings in which: 
           [0021]      FIG. 1  illustrates a camera that may be used according to the invention. 
           [0022]      FIG. 2  illustrates schematically an IR image and a visible light image of the same scene and the resulting image after applying the inventive method. 
           [0023]      FIG. 3  is a flow chart of a method according to an embodiment of the invention. 
           [0024]      FIG. 4  is a flow chart of a possible method for identifying moving objects. 
       
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
       [0025]      FIG. 1  shows, schematically, a camera  1  that may be used according to the invention. The camera comprises a first imaging part  3  for capturing thermal images based on IR radiation, The first imaging part comprises optics  5  for focusing IR radiation onto a first focal plane array  7  The camera also comprises a second imaging part  9  for capturing visible light images. The second imaging part  9  comprises optics  11  for focusing visible light onto a second focal plane array  13 . The camera  1  also comprises a processing device  15  for receiving and processing IR image data from the first focal plane array  7  and visible light image data from the second focal plane array  13 . Each imaging part is devised as is well known in the art. As the skilled person will understand one or more processing units may be arranged between the respective focal plane array  7 ,  13  and the processing device  15 , for processing of image data in ways known in the art. For clarity, however, no such units are shown here. The processing device  15  preferably comprises software having functions for processing the image data received from the imaging parts, to produce image files that can be stored in a memory  17  and/or presented to an operator on a display  19 . As will be understood, the processing device  15  could be implemented as two (or more) separate processing devices. The processing device  15  could also be used to perform other functions. 
         [0026]    According to the invention, the processing device  15  also comprises software for processing the image data in order to identify objects that change their position between a first and a second visible light image taken at a first and a second point in time t 1 , t 2 , respectively, and a first and a second IR image taken at a third and a fourth point in time, t 3 , t 4 , respectively, which should be as close as possible to the first and the second point in time t 1 , t 2 , respectively. How to achieve this, and how to use the information about the moving objects, will be discussed in more detail in the following. 
         [0027]      FIG. 2  illustrates in a) an IR image  21 , and in b) a visible light image  23  taken of the same scene at the same time. As will be understood the two focal plane arrays  7 ,  13  have different numbers of pixels, and the pixels have different sizes. Also, the two imaging parts have different fields of view and different optical axes, so that the IR image and the visible light image overlap to a certain extent but not completely. 
         [0028]    Further, there is a parallax error between the imaging parts. Therefore, the processing device  15  has functions for compensating for the parallax error and adjust the image sizes so that the two images will overlap as much as possible and that a pixel in one image will correspond to a pixel in the other image. Such functions are known from prior art cameras enabling the display of pictures that are based on both the IR and the visible light information, so called blended, or fused images. It is assumed that the images  21 ,  23  shown in  FIG. 2  a) and b) have been adjusted to be fully overlapping images. 
         [0029]    Preferably, as discussed above, at least two IR images are taken at two different points in time and at least two visible light images are taken, at substantially the same two different points in time, to enable the identification of moving objects in each image. This can be achieved in a number of different ways, as will be explained in more detail below. 
         [0030]    In the IR image  21  a first  25  and a second  27  moving object have been identified. In the visible light image  23  a third  29  and a fourth  31  moving object have been identified. As can be seen, the first object  25  and the third object  29  are located in essentially the same position in the respective image. Generally, they will not exhibit exactly the same shape, but they should have approximately the same size and shape, within certain limits. If so, it may be determined that they are probably an IR image and a visible light image, respectively, of the same object. Hence, since this moving object is also visible in a visible light image, and by the bare eye, it is not gas, or if it is gas, it can still be detected without using the inventive camera or method. The fourth moving object  31  appears only in the visible light image and can be neglected for the purposes of the invention. The second moving object  27 , on the other hand, appears only in the thermal image and may be assumed to represent a flow of gas. 
         [0031]    The second moving object  27 , being now identified as gas, can be presented in any appropriate way to make it easily identifiable to the operator. It may be displayed as a field of particular colour, shade or intensity in the IR image, in the visible light image, or in any type of blended image. The image  33  denoted c) in  FIG. 2  shows the resulting image having only the second moving object, since this is the part of the image that is of interest in the context of the invention. That is, the objects that are visible in both images, or only in the visible light image, have been eliminated. Of course, other objects, or the complete screen, may be displayed as well, but with the second object  27  clearly distinguishable from the rest of the image. 
         [0032]    The detection of any moving objects in each of the images may be performed in any way known in the art. For example, a method known as “optical flow” may be used. In this method the velocity vector for each pixel is determined along the rows and the columns, respectively, of the focal plane array  7 ,  13 . This may be difficult to achieve in IR images, because of noise. The optical flow method is described, for example, in Barron, J L, et. al. “Performance of Optical Flow Techniques”, CVPR, 1992. 
         [0033]    In a preferred embodiment, therefore, an alternative method is used, which comprises determining the difference between two images taken at different points in time and which may be applied to IR images or visible light images. If the difference in a given part of the image is larger than a given threshold, there has very likely been a movement. Hence, there is no need to define each object; instead, the number of moving pixels, and the distance moved can be used to detect and quantify movement. 
         [0034]    Since the method is based on detecting moving objects, it is very important that the images themselves have exactly the same field of view, that is, that any movement of the camera is compensated for. Often, especially when the camera is hand-held by an operator, the camera is moved or shaken during use. Therefore, an image stabilization method is preferably applied. Several such methods exist, based on optical or mechanical means, or on pure image processing, and any one of these methods can be used with the invention, as long as it produces a stable thermal image and a stable visible light image. 
         [0035]      FIG. 3  is a flow chart of the method according to a preferred embodiment of the invention. In step S 31  a first IR image and a first visible light image are obtained at a first point in time t 1 . The IR image and the visible light image do not have to be taken at exactly the same point in time, but they should be close enough in time to enable correct identification of moving objects. In step S 32  the IR image and the visible light image are aligned in the same way as would be done to produce a blended, or fused image. 
         [0036]    In step S 33  a second IR image and a second visible light image are obtained at a second point in time t 2 . In step S 34  these images are also aligned in the same way as the first images and stabilized based on comparison with the first images. As discussed above, any image stabilization method may be used, or different methods may be used for the thermal image and the visible light image. In step S 35  any moving objects in the visible light image and the thermal image are identified using the first and second visible light image and the first and second thermal image, respectively. As explained above, in a preferred embodiment, a thermal differential image and a visible light differential image are obtained. In step S 36  any changes above a certain threshold in each of the differential images are identified as movements. Thresholds may be applied to filter out small changes as well as small areas. 
         [0037]    In step S 37  the areas in which movements have been identified in each of the differential images are compared and associated with each other if possible. This means determining, for each moving object in the thermal image, whether there is a corresponding object in the visible light image. In step S 38  the moving objects that appear in the visible light image are discarded from the image to produce the image shown in  FIG. 2   c . Any moving object in the thermal image that does not appear in the visible image is identified as a potential field of gas and is marked in the resulting image, for example, by using a particular colour. In step S 39 , if desired, further investigation is performed to determine if the moving object identified as gas is really gas. 
         [0038]    As will be understood, the order of steps S 32  and S 33  may be changed. In step S 35  alternative methods of identifying movement in the respective image could be applied, for example, the “optical flow” method. 
         [0039]    An object will normally not be imaged as having exactly the same size and shape in the thermal image and the visible light image. Therefore, the association of the objects performed in step S 37  cannot be based on the requirement that the objects should appear in identical ways in the visible light image and the IR image. Instead it should be based on one or more of the following parameters: The positions in the images taken at the first t 1  and second t 2  point in time will be essentially the same in both the IR and the visible light images. The position may, for example, be determined based on the centre of the object in the relevant image. The objects should have the same size within a certain margin, for example, the sizes between the objects in the images may not differ by more than a certain percentage. The objects should have the same shape within certain limits, that is, even or uneven edges, extremities, etc. Further, the contours of the object should coincide in both images, within certain limits. As will be understood, once the movement has been identified, the size, shape and position of the moving object could be determined from the images taken at the first point in time t 1 , the images taken at the second point in time t 2  or a combination of them. 
         [0040]      FIG. 4  is a flow chart of one possible method for identifying moving objects, that is, a more detailed description of one possible embodiment of step S 35  of  FIG. 3 . According to this embodiment, moving objects are identified by finding areas in which the intensity varies strongly between frames. As the skilled person will understand, various modifications to the steps of this method are possible. 
         [0041]    In step S 41 , the image I t  is divided into B×B pixel distinct squares. In step S 42 , the mean of each square is calculated resulting in an image Ic t  with coarser scale. 
         [0042]    For the frame Ic t  the difference from the preceding frame is calculated in step S 43 . 
         [0000]        Id   t   =Ic   t   −Ic   t-1   (1) 
         [0043]    In step S 44 , the resulting image Id t  from step S 43  is median filtered using the pixels in a window of size M around each pixel. This is done to reduce the extreme differences likely to be caused by noise and not by diffusing gas. 
         [0044]    Depending on the frame rate of the currently used camera the difference between Ic t  and IC t-1  could be one or more frames. Ic t  and Ic t-1  can also be estimated in step S 44  as the mean value of a number of frames. 
         [0045]    Now the image obtained from step S 44 , Idm t , contains values indicating the amount of intensity change. Some of the intensity change is caused by moving objects. But there will also be intensity change if the camera has changed its position in some way, which may be falsely identified as moving objects. This has been compensated for in the stabilization step but only to a certain degree. Basically if one part of the image is perfectly stabilized it does not guarantee that other parts are, due to the depth of the scene. Parts of the scene that are not perfectly stabilized will then contribute to the intensity change. An area with large gradient magnitude will if it is not perfectly stabilized contribute to the intensity change more than an area with smaller gradient magnitude. 
         [0046]    Therefore, in step S 45 , the horizontal and vertical gradient Ifx t  and Ify t  is calculated and then the gradient magnitude Ig t  is determined. 
         [0000]        Ig   t =√{square root over ( Ifx   t   2   +Ify   t   2 )} 
         [0047]    The gradient can be estimated in various ways. The most basic method is to approximate the gradient with the central difference of the adjacent pixels. Another method is to use the Sobel operator by convolve the image with two gradient estimating convolution masks. 
         [0000]    This step results in the image Ing t . 
         [0000]    
       
      
       Ing 
       t 
       =Idm 
       t 
       −Ig 
       t  
      
     
         [0000]    Areas with strong edges are less likely to contain gas. Therefore, preferably in step S 46 , areas having strong edges are identified. Finding strong edges can be done by using one of the common edge detector algorithms, see for example Gonzalez, R. C. and Woods, R. E.  Digital Image Processing  (3 rd Edition ). s.1.: Prentice-Hall Inc., 2006. The result is a binary image with edges stronger than a certain threshold set to one. By setting the edge threshold to a relatively high value only the strongest edges will survive. This is repeated N times and the resulting images are added together. Consequently the strongest edges will have the intensity N. This intensity image is then filtered with an averaging filter. The resulting image Ie t  is then subtracted from the result from the step S 45  (or S 44  if S 45  is not performed) and a gas probability image is obtained. 
         [0000]    
       
      
       Igp 
       t 
       =Ing 
       t 
       −Ie 
       t  
      
     
         [0048]    From the procedure above, the image with values corresponding to probable moving gas is obtained. In step S 47  a threshold determining a minimum acceptable motion value is applied to the image, to obtain a binary mask identifying areas of movement. In the simplest form, a fixed threshold is used, although it would be possible to use a moving threshold. The threshold is depending on the camera type and frame rate. These are known parameters; therefore a fixed threshold T can be used and the possible benefits of having a moving threshold would be very small. When deciding which threshold to use it should be taken into account that setting a too high threshold will leave the gas undetected whereas setting a too low threshold will be sensitive to noise and small uninteresting image changes. 
         [0000]      Im t =Igp t &gt;T 
         [0049]    The binary mask that is the result of thresholding contains larger moving regions as well as smaller unconnected regions. Therefore, in step S 48  some filtering steps are performed. First the mask is dilated to smooth regions and connect neighboring regions that probably represent the same object. Then small regions are discarded as noise and unlikely to be gas. Small regions are preferably discarded based on the number of pixels in the region. 
         [0050]    After the steps performed according to  FIG. 4 , additional steps may be performed to reduce further the risk of false identification of movement. One problem with motion detection that is related to stabilization is that when the stabilization fails objects that where not supposed to be detected show up as regions after the thresholding step. This occurs quite often in singular frames. Thus a new binary mask based on two consecutive frames may be applied, to reduce the problem. 
         [0000]      Ims t =Im t &amp;Ims t-1    
         [0051]    When moving the camera too swiftly, the stabilization algorithm might have troubles to stabilize the images properly. Consequently the image will be full of regions with moving objects. To solve this problem a threshold Tm may be applied. If the quotient between the number of masked pixels in Ims t  and the total number of pixels in is above Tm no gas detection will be made.