Patent Publication Number: US-11640654-B2

Title: Image processing method and apparatus

Description:
This application claims priority to GB Patent Application No. 1915640.5 filed Oct. 29, 2019, the entire contents of which are hereby incorporated by reference. 
     FIELD 
     The invention relates to a method of image processing and an apparatus using the method. 
     BACKGROUND 
     Digital images of the same scene but with different brightness may have blocks or local areas, which contain movement that causes blur. The blur may be reduced by combining the images by taking pixels of the movement from the darker image which leaves noise in the pixels of the movement. Alternatively, the blur may be reduced by combining the images by taking pixels of the movement from the brighter image, but in this case the pixels of the movement may be saturated. 
     Hence, there is a need to improve the image processing. 
     BRIEF DESCRIPTION 
     The present invention seeks to provide an improvement in the image processing. 
     The invention is defined by the independent claims. Embodiments are defined in the dependent claims. 
    
    
     
       LIST OF DRAWINGS 
       Example embodiments of the present invention are described below, by way of example only, with reference to the accompanying drawings, in which 
         FIG.  1    illustrates an example of two images, which are used to form a composed image; 
         FIG.  2    illustrates an example of an image processing apparatus; 
         FIG.  3    illustrates an example where a local block and a corresponding block are in the same image; and 
         FIG.  4    illustrates of an example of a flow chart of an image processing method. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     The following embodiments are only examples. Although the specification may refer to “an” embodiment in several locations, this does not necessarily mean that each such reference is to the same embodiment(s), or that the feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments. Furthermore, words “comprising” and “including” should be understood as not limiting the described embodiments to consist of only those features that have been mentioned and such embodiments may contain also features/structures that have not been specifically mentioned. All combinations of the embodiments are considered possible if their combination does not lead to structural or logical contradiction. 
     It should be noted that while Figures illustrate various embodiments, they are simplified diagrams that only show some structures and/or functional entities. The connections shown in the Figures may refer to logical or physical connections. It is apparent to a person skilled in the art that the described apparatus may also comprise other functions and structures than those described in Figures and text. It should be appreciated that details of some functions, structures, and the signalling used for measurement and/or controlling are irrelevant to the actual invention. Therefore, they need not be discussed in more detail here. 
     When a digital image is captured, each pixel gathers light, the intensity of which is converted into a digital number. The number is within a dynamical range of the imaging sensor, the dynamical range being expressed using a certain number of bits. At one end of the dynamical range there are the darkest pixels and at the other end of the dynamical range there are the brightest pixels. By lengthening the duration of the exposure or having a larger aperture it is possible to increase the intensity received by each pixel. 
     However, when the intensity is too strong, pixels saturate i.e. they cannot carry information on the intensity they receive. On the other hand, when the intensity is too low, pixels become quantized which is observed as noise. Pixels in a digital image can be considered to count incoming photons. Because radiation of photons itself is a random process, a signal-to-noise ratio can be improved by collecting more photons. Thus, an underexposed image contains less information compared to a well exposed image of the same target or scene. 
     In addition to the saturation and noise, images may be blurred because of movement. All these features which decrease the image quality should be compensated. 
       FIG.  1    illustrates an example of two images  100 ,  102 , where the image  100  is darker, i.e. the image  100  has a lower brightness than the image  102 . In general, the number of the images may be two or more. The lower brightness is illustrated with a hatch pattern. The brightness is based on a visual perception and refers, in a certain manner, to lightness or darkness of an image as a whole. On the other hand, the brightness can be considered a mean or average of values of the red, green, and blue in the RGB (Red-Green-Blue) color coordinate system. An image having a low brightness may be captured when using a short exposure or a small aperture. Additionally or alternatively, an image having a low brightness may be captured in a low illumination of a scene or target of the image. In the example of  FIG.  1   , the image  102  may have a longer exposure than an exposure of the image  100 , for example. 
     The image processing apparatus  200  comprises an image processing electric circuit system, which converts the images  100 ,  102  to a common brightness range. At least one image  102  has initially a higher brightness than at least one other image  100  as shown in the example of  FIG.  1   . The conversion to the common brightness may be performed by converting intensity values of images  100 ,  102  to a common brightness range. That is, the brightness does not need to be exactly the same as long as the brightness is about the same, i.e. within a predetermined range. The predetermined range may be decided by a person skilled in the art on the basis of his/her experience, tests, simulation or his/her knowledge of theory. The predetermined range may be a general range, or it may be decided for each application or each group of images separately. 
     The images  100 ,  102  represent at least partly the same scene at different moments, although they may or may not be shifted with respect to each other. The images  100 ,  102  illustrate a shift S between the images. The shift S is shown with an arrow in the image  102 . In addition to a global shift, motion between images can be rotation, scaling, shearing, perspective projection etc., which, per se, are known to a person skilled in the art. 
     The image processing apparatus  200  selects one of images  100 ,  102 , which includes a local block  104 A with movement, for forming a composed image  10  from the images  100 ,  102 . The selected image is thus used as a base from which the composition may start. In the example of  FIG.  1   , the selected image is the image  102 . In general, more than one of the images  100 ,  102  may include local movement. In the example of  FIG.  1   , the movement may be clear in the image  102  because of a longer exposure than an exposure of the image  100 , for example. The movement causes blur. In the image  102 , the leaves and the outer contour of the tree may be blurred. Additionally, there is a higher probability that pixels are saturated in the brighter image  102  than in the darker image  100 . A pixel of an image is saturated at a threshold above which an increasing intensity of optical radiation hitting a corresponding pixel of a camera, which records said image, causes a constant output. The constant output is typically the maximum output. Saturation may result in undesirable artifacts in the images. 
     The image processing electric circuit system determines one or more corresponding blocks  104  of the images  100 ,  102 , the corresponding blocks  104  corresponding to the local block  104 A of said one  102  of the images  100 ,  102  on the basis of their visual information. The corresponding blocks  104  may be in the one or more images  100  other than said one  102  of the images  100 ,  102 . Alternatively or additionally, the one or more corresponding blocks  104  may also be found in the same image  102  as the local block  104 A (see  FIG.  3   ). A block of an image, such as either one of the one or more corresponding blocks  104  or the local block  104 A, comprises one or more pixels. 
     The image processing electric circuit system forms the corresponding block  104  (or refines the local block  104 A) by taking weighted average of the corresponding blocks  104  from different images. The image processing electric circuit system then weights each of said corresponding blocks  104  with at least one of the following: similarity with the local block  104 A, a distance from a location of the local block  104 A, saturation, and noise of the corresponding blocks  104 . In an embodiment, the noise of the corresponding blocks  104  may be compared with noise of the local block  104 A. The location of the local block  104 A may be determined in a coordinate system of an image  100  different from the image  102  including the local block  104 A. 
     The image processing electric circuit system combines the local block  104 A and at least one of the corresponding blocks  104 , or replaces the local block  104 A by at least one of the corresponding blocks  104  based on the weighting for forming the composed image  10  from the images  100 ,  102 . In this manner, the new image  10  is a combination of the images  100 ,  102  such that disturbance, such as blur caused by the movement and the saturation, is minimized, while trying to maximize signal-to-noise ratio and/or to minimize noise. 
     In an embodiment, the composed image  10  may be formed by combining the plurality of images  100 ,  102  when combining the local block  104 A and the at least one of the one or more corresponding blocks  104 , or replacing the local block  104 A by the at least one of the one or more corresponding blocks  104 . 
     This kind of composition on the basis of weighting of the blocks having movement can be considered a trilateral filtering which is edge-preserving and noise reducing. 
     The local block  104 A and the corresponding blocks  104  may have noise, which may be divided in several types. Gaussian noise, which may also be called thermal noise, comes from the camera that captured the image. Salt-and-pepper noise has dark pixels in bright regions and bright pixels in dark regions. Salt-and-pepper noise may also be called or spike noise. Shot noise, in turn, is caused by quantum fluctuations and it can be included the Gaussian noise because of it statistical properties. The shot noise and thus the Gaussian noise may include dark current in the image sensor. Because the images are in a digital form, the conversion into the digital form causes quantization noise. Additionally, the image  100 ,  102  may have periodic noise, row noise, and/or column noise, for example. 
       FIG.  2    illustrates an example of the image processing apparatus  200 , which may comprise one or more processors  202  and one or more memories  204  including at least one computer program code. The one or more processors  202 , the one or more memories  204  and the at least one computer program code cause the image processing apparatus  200  at least to perform image processing method steps, which include the conversion of the images  100 ,  102  to the common brightness range, the determination of the corresponding blocks  104 , the weighting of each of said corresponding blocks  104 , and the combination or replacement relating to the local block  104 A and the corresponding blocks  104 . The image processing apparatus  200  may comprise also a user interface  206 , which may comprise a screen and a keyboard, and/or a touch screen, for example. Additionally, the image processing apparatus  200  may comprise an imaging device  208 . The imaging device  208  may be a camera or the like, which captures or forms the images. 
       FIG.  3    illustrates an example where the corresponding block  104  is in the same image  102  as the local block  104 A. If an image has a shape  300  that is larger than the local block  104 A (the corresponding block has the same size as the local block), the same shape can also be found outside the local block  104 A. The shape  300  may continue in an unchanged manner or it may appear repeatedly in the image  102 . The saturation or movement of the shape  300  may differ in different locations of the image  102 , which may sometimes be utilized. 
     In an embodiment, the image processing apparatus  200  may use the coordinate system of the image  100  having a lowest brightness to determine locations in the images. Thus, a location for the local block  104 A and a location of the one or more corresponding blocks  104  may be determined in the coordinate system of the image  100 . 
     In an embodiment, the image processing apparatus  200  may convert the brightness of the images to the common brightness range by increasing brightness of dark images. If a dynamical range of the image  100  is 10 bits and it is brightened eight times, for example, which corresponds to an exposure with eight times more light, the values of pixel intensities are multiplied by eight. The dynamical range of the pixels of the initially darker image  100  is now 14 bits. In this case, it may be so that the pixels of the initially darker image  100  are quantized and the pixels of the initially brighter image  102  may be saturated. However, if the quantized pixels and the saturated pixels are not completely the same, it is possible to have an image without or with limited saturation and quantization by combining the images. 
     In an embodiment, the image processing apparatus  200  may form the similarity with the local block  104 A by computing the similarity between brightness of the local block  104 A and each of the one or more corresponding blocks  104 . Additionally or alternatively, the image processing apparatus  200  may form the similarity with the local block  104 A by computing the similarity between colors of the local block  104  and each of the one or more corresponding blocks  104 . Additionally or alternatively, the image processing apparatus  200  may form the similarity with the local block  104 A by computing the similarity of a distribution of brightness between the local block and one of the corresponding blocks. 
     In an embodiment, the image processing apparatus  200  may compute histograms of values of pixels of each of the one or more corresponding blocks  104  and the local block  104 A, and form the similarity based on a comparison of the histograms. A histogram can be used to show a distribution of numerical data of an image or a block of an image. The similarity may also be determined on the basis of dissimilarity. 
     The similarity may also be based on scale invariant feature transform (SIFT) algorithm. The similarity may further be determined using a plurality of method such as Pearson correlation, Tanimoto measure, Spearman&#39;s ρ, Kendall&#39;s τ, correlation, and/or Shannon/Rényi/Tsallis mutual information, for example. The dissimilarity may be measured using L 1  or L 2  norm, incremental sign distance, intensity-ratio variance, intensity-mapping-ratio variance, rank distance, joint entropy, and/or exclusive F-information. These methods may also measure noise and blurring as well as to intensity and geometric changes. The similarity may estimate the probability that the blocks represent the same target, when taking known or estimated amount of noise into an account. 
     In an embodiment, the image processing apparatus  200  may weight said each of the one or more corresponding blocks  104  by values, which decrease with an increasing distance from the location of the local block  104 A. 
     In an embodiment, the image processing apparatus  200  may weight said one or more corresponding blocks  104  by values, which decrease with increasing noise. 
     In an embodiment, the image processing apparatus  200  may estimate noise of the one or more corresponding blocks  104  and the local block  104 A based on a gain of a sensor, which captured the images. The gain of the sensor refers to amplification of the intensity values of the pixels, for example. Gain may also be taken into account in an embodiment, when different images are adjusted to be at a same intensity level. When the values of pixels of an image are multiplied by a value n, also the noise will increase with the value n. In this multiplication, the signal-to-noise ratio will remain constant. On the other hand, if the number of photons is increased by the value n, the photon noise will increase by a square root of the value n, which increases the signal-to-noise ratio by the square root of the value n. That is, the same brightness achieved with the gain as with an increase of a corresponding optical radiation power will result in an increase of noise by the square root of the increase/gain n. 
     Additionally or alternatively, the image processing apparatus  200  may estimate noise of the one or more corresponding blocks  104  and the local block  104 A based on one or more image transformations performed to the image. The image transforms may be geometric image transforms, for example. The image transformation may refer to a geometrical distortion and/or a combination of images, for example. 
     In an embodiment, the image processing apparatus  200  may estimate noise of the one or more corresponding blocks  104  and the local block  104 A based on a pre-gain performed to the images  100 ,  102  prior to entering the method of this application. 
     In an embodiment, the image processing apparatus  200  may estimate noise of the one or more corresponding blocks  104  and the local block  104 A based on noise, which depends on a location in the images. 
     In an embodiment, the image processing apparatus  200  may estimate noise of the one or more corresponding blocks  104  and the local block  104 A based on image fusion. 
     Noise of an image may be reduced, when at least two images have been fused. When combining the images, at least one of the images may have been adjusted (contrast may have been changed, for example), which may have increased a gain of a pixel i.e. a pixel-specific gain may have been altered. Alternatively or additionally, a possible vignetting caused by a lens and/or any other physical disturbance or deformation caused by optical components of an image capturing device or environment (turbulence of atmosphere) may have been adjusted by a multiplication of pixel values which changes the pixel-specific gain. 
     In an embodiment, the image processing apparatus  200  may select an image  102 , which has the best information, where the best information may mean that the image  102  has the greatest brightness without saturation or with least saturation. In an embodiment, the image processing apparatus  200  may select an image  102 , which has the greatest brightness among the images  100 ,  102 , as said one of images  100 ,  102 . In this embodiment, the local block  104 A is selected to be in an image, which is the brightest image among the images and includes the movement. Here the brightest image may be a non-saturated image. 
     In an embodiment an example of which is illustrated in  FIG.  1   , the image processing apparatus  200  may form a bright image  102  by combining images having a higher brightness than an image, which has a lowest brightness, before converting the images  100 ,  102  to the common brightness range. In the example of  FIG.  1   , the image  102  may have been formed as a combination of bright images  102 ′,  102 ″, which are brighter than the dark image  100 . 
     In an embodiment an example of which is illustrated in  FIG.  1   , the image processing apparatus  200  may combine a plurality of images of a lower brightness than an image, which has a highest brightness, before converting the images  100 ,  102  to the common brightness range. In the example of  FIG.  1   , the image  100  may have been formed as a combination of dark images  100 ′,  100 ″, which are darker than the image  102 . 
     In an embodiment, a plurality of dark images  100 ″ and a plurality of bright images  102 ″ may be used separately. The formation of the corresponding blocks  104  may automatically combine information from multiple dark images. The formation of corresponding blocks  104  may automatically combine information from multiple bright images. 
     In an embodiment, the image processing apparatus  200  may register the images  100 ,  102  before converting the images to the common brightness range. In this manner, the images  100 ,  102  are aligned to each other. The shift S may be utilized in the alignment. The coordinate system of the images is unified, and a location of an area of the local block  104 A becomes the same in each of the images  100 ,  102 . 
     In an embodiment, the corresponding blocks  104  may originate from several different input images, which may have been captured with different settings. Thus, in addition to spatial distance from the local block  104 A and similarity of blocks, information content (noise and saturation) of each the corresponding blocks  104  may affect the weighting. In general, input images may be captured with the same or different settings (brightness). 
     At extreme, non-moving and non-saturated areas of images are obtained merely from at least one bright image  102 ″, which produces a better signal-to-noise ratio for these areas than a darker image. 
     If any of the corresponding blocks  104  contains a lot of saturated pixels, several dark images  100 ″ may be combined. 
     In an embodiment, a dark image  100  may be used as a reference. As an effect of this kind of filtering, if target in the block  104  is moving, but is non-saturated, the at least one corresponding block  104  may be taken from at least one bright image  102  where it is at a slightly different location in addition to shift S of a local motion. The result of a combination of the images  100 ,  102  is that a signal-to-noise level comes from the bright image  102 , but a location of a moving object is based to the dark image  100 . If the corresponding block  104  is attached to pixels that are saturated in the bright image  102 , but non-saturated in dark image  100 , this will result in a continuous object. Its location corresponds to that in the dark image  100 , without saturated pixels, where pixels are taken partly from a bright image  102  (less noisy), and partly from a dark image  100 . Also, these pixels that have been taken from the dark image  100  are rather bright (as these are saturated in the bright image  102 ), so noise is not as high in these pixels as it would be in the corresponding block  104  of the dark image  100 . 
     There can be cases where a corresponding block  104  of a dark image  100  is not visible in a bright image  102  (with good enough similarity). Then, the bright image  102  cannot be utilized even if area of the corresponding block  104  would be non-saturated in an intensity level of the bright image  102 . In an embodiment, a location of each single moving object may vary, either coming from a dark or bright image  100 ,  102 , where the object may move image-specifically. An algorithm may prefer an image that contains a best or optimum information, which may mean that the image is bright but the pixels are non-saturated, for example. Irrespective of a location, areas of these larger moving objects in the corresponding blocks  104  of a plurality of images may be partly combined (see  FIG.  3    although it refers to one image only). 
     In images, there may be a whole target/object that moves. Then, when the shift S has been compensated between a plurality of images, an outline of the target/object may correspond well to each other in different images. However, in an area surrounded by the outline of the target/object there may be pixel values that differ in different images  100 ,  102  because of movement. The target/object may be divided in blocks like blocks  104 A,  104 , which may contain one pixel, 2×2 pixels, 8×8 pixels, 5×9 pixels or the like. Each of the blocks may be filtered separately in order to have a high quality composed image  10  despite the movement. 
     In an embodiment, different base frames may be selected for each block  104 . In an embodiment, different base locations for each of the corresponding blocks  104  may be selected. That means that some areas of the composed image  10  may come from the image  100  and some areas of the composed image  10  may come from the image  102 . This can be used to avoid ghost edges of an object in the composed image  10  and to increase total quality of the composed image  10 . 
     In an embodiment, the image processing apparatus  200  may determine noise pixel by pixel of the composed image  10 , and equalize the noise of the composed image  10  by filtering. This kind of filtering may be performed by any single image based spatial filtering (like Gaussian filtering, bilateral filtering, non-local means BM3D). Alternatively, this information of pixel-by-pixel noise may be utilized in a previous step in order to adjust a weight to merge a corresponding block  104  with a local block  104 A, based on a noise level of the corresponding block  104 . This filtering may take known noise level into an account. As a result, the image formed as a combination of images filtered in this manner contains an equal noise level over the whole image, irrespective if the corresponding block  104  has originated from a dark or bright image, or if the corresponding block  104  is moving (when combined with one or several bright frames, for example). 
     In an embodiment, the image processing apparatus  200  may process images  100 ,  102  which are raw Bayer-pattern data. In other words, no preprocessing steps or less than usual amount preprocessing steps have been made for inputs. This facilitates the processing and estimation of the movement, helps in finding the local block  104 A and the corresponding blocks  104 , and helps estimating amount of noise remaining per image pixel. 
     In an embodiment an example of which is illustrated in  FIG.  2   , the image processing apparatus comprises one or more processors  202 ; and one or more memories  204  including computer program code. The one or more memories  204  and the computer program code are configured to, with the one or more processors  202 , cause image processing apparatus at least to: convert the intensity values of the images to the common brightness range; select one of images  100 ,  102  including a local block  104 A with movement for forming a composed image  10  from the images  100 ,  102 ; determine the corresponding blocks  104 ; weight each of said corresponding blocks  104  with respect to the local block  104 A; and perform the combination or replacement. 
       FIG.  4    presents an example of a flow chart of the image processing method. In step  400 , images  100 ,  102  of different brightness are converted to a common brightness range, the images  100 ,  102  representing at least partly the same scene. In step  402 , one of images  100 ,  102  including a local block  104 A with movement is selected for forming a composed image  10  from the images  100 ,  102 . In step  404 , one or more corresponding blocks  104  to the local block  104 A determined in the images  100 ,  102 . In step  406 , each of said one or more corresponding blocks  104  are weighted with at least one of the following: similarity with respect to the local block  104 A, a distance from a location of the local block  104 A, saturation of the one or more corresponding blocks  104 , and noise of the one or more corresponding blocks  104 . In step  408 , the local block  104 A and at least one of the one or more corresponding blocks  104  are combined, or the local block  104 A is replaced by at least one of the one or more corresponding blocks  104  based on the weighting for forming the composed image  10  from the images  100 ,  102 . 
     The method shown in  FIG.  4    may be implemented as a logic circuit solution or computer program. The computer program may be placed on a computer program distribution means for the distribution thereof. The computer program distribution means is readable by a data processing device, and it encodes the computer program commands, and carries out the image processing. 
     The computer program may be distributed using a distribution medium, which may be any medium readable by the controller. The medium may be a program storage medium, a memory, a software distribution package, or a compressed software package. In some cases, the distribution may be performed using at least one of the following: a near field communication signal, a short distance signal, and a telecommunications signal. 
     It will be obvious to a person skilled in the art that, as technology advances, the inventive concept can be implemented in various ways. The invention and its embodiments are not limited to the example embodiments described above but may vary within the scope of the claims.