Image processing method and apparatus

A method of image processing converts images of different brightness to a common brightness range, the images representing at least partly the same scene. One of images including a local block with movement is selected for forming a composed image from the images. One or more corresponding blocks corresponding to the local block are determined in the images. Each of said one or more corresponding blocks is weighted with at least one of the following: similarity with respect to the local block, a distance from a location of the local block, saturation of the one or more corresponding blocks, and noise of the one or more corresponding blocks. The local block and at least one of the one or more corresponding blocks are combined, or the local block is replaced by at least one of the one or more corresponding blocks based on the weighting for forming the composed image from the images.

This application claims priority to GB Patent Application No. 1915640.5 filed Oct. 29, 2019, the entire contents of which are hereby incorporated by reference.

FIELD

The invention relates to a method of image processing and an apparatus using the method.

BACKGROUND

Digital images of the same scene but with different brightness may have blocks or local areas, which contain movement that causes blur. The blur may be reduced by combining the images by taking pixels of the movement from the darker image which leaves noise in the pixels of the movement. Alternatively, the blur may be reduced by combining the images by taking pixels of the movement from the brighter image, but in this case the pixels of the movement may be saturated.

Hence, there is a need to improve the image processing.

BRIEF DESCRIPTION

The present invention seeks to provide an improvement in the image processing.

The invention is defined by the independent claims. Embodiments are defined in the dependent claims.

DESCRIPTION OF EMBODIMENTS

It should be noted that while Figures illustrate various embodiments, they are simplified diagrams that only show some structures and/or functional entities. The connections shown in the Figures may refer to logical or physical connections. It is apparent to a person skilled in the art that the described apparatus may also comprise other functions and structures than those described in Figures and text. It should be appreciated that details of some functions, structures, and the signalling used for measurement and/or controlling are irrelevant to the actual invention. Therefore, they need not be discussed in more detail here.

When a digital image is captured, each pixel gathers light, the intensity of which is converted into a digital number. The number is within a dynamical range of the imaging sensor, the dynamical range being expressed using a certain number of bits. At one end of the dynamical range there are the darkest pixels and at the other end of the dynamical range there are the brightest pixels. By lengthening the duration of the exposure or having a larger aperture it is possible to increase the intensity received by each pixel.

However, when the intensity is too strong, pixels saturate i.e. they cannot carry information on the intensity they receive. On the other hand, when the intensity is too low, pixels become quantized which is observed as noise. Pixels in a digital image can be considered to count incoming photons. Because radiation of photons itself is a random process, a signal-to-noise ratio can be improved by collecting more photons. Thus, an underexposed image contains less information compared to a well exposed image of the same target or scene.

In addition to the saturation and noise, images may be blurred because of movement. All these features which decrease the image quality should be compensated.

FIG.1illustrates an example of two images100,102, where the image100is darker, i.e. the image100has a lower brightness than the image102. In general, the number of the images may be two or more. The lower brightness is illustrated with a hatch pattern. The brightness is based on a visual perception and refers, in a certain manner, to lightness or darkness of an image as a whole. On the other hand, the brightness can be considered a mean or average of values of the red, green, and blue in the RGB (Red-Green-Blue) color coordinate system. An image having a low brightness may be captured when using a short exposure or a small aperture. Additionally or alternatively, an image having a low brightness may be captured in a low illumination of a scene or target of the image. In the example ofFIG.1, the image102may have a longer exposure than an exposure of the image100, for example.

The image processing apparatus200comprises an image processing electric circuit system, which converts the images100,102to a common brightness range. At least one image102has initially a higher brightness than at least one other image100as shown in the example ofFIG.1. The conversion to the common brightness may be performed by converting intensity values of images100,102to a common brightness range. That is, the brightness does not need to be exactly the same as long as the brightness is about the same, i.e. within a predetermined range. The predetermined range may be decided by a person skilled in the art on the basis of his/her experience, tests, simulation or his/her knowledge of theory. The predetermined range may be a general range, or it may be decided for each application or each group of images separately.

The images100,102represent at least partly the same scene at different moments, although they may or may not be shifted with respect to each other. The images100,102illustrate a shift S between the images. The shift S is shown with an arrow in the image102. In addition to a global shift, motion between images can be rotation, scaling, shearing, perspective projection etc., which, per se, are known to a person skilled in the art.

The image processing apparatus200selects one of images100,102, which includes a local block104A with movement, for forming a composed image10from the images100,102. The selected image is thus used as a base from which the composition may start. In the example ofFIG.1, the selected image is the image102. In general, more than one of the images100,102may include local movement. In the example ofFIG.1, the movement may be clear in the image102because of a longer exposure than an exposure of the image100, for example. The movement causes blur. In the image102, the leaves and the outer contour of the tree may be blurred. Additionally, there is a higher probability that pixels are saturated in the brighter image102than in the darker image100. A pixel of an image is saturated at a threshold above which an increasing intensity of optical radiation hitting a corresponding pixel of a camera, which records said image, causes a constant output. The constant output is typically the maximum output. Saturation may result in undesirable artifacts in the images.

The image processing electric circuit system determines one or more corresponding blocks104of the images100,102, the corresponding blocks104corresponding to the local block104A of said one102of the images100,102on the basis of their visual information. The corresponding blocks104may be in the one or more images100other than said one102of the images100,102. Alternatively or additionally, the one or more corresponding blocks104may also be found in the same image102as the local block104A (seeFIG.3). A block of an image, such as either one of the one or more corresponding blocks104or the local block104A, comprises one or more pixels.

The image processing electric circuit system forms the corresponding block104(or refines the local block104A) by taking weighted average of the corresponding blocks104from different images. The image processing electric circuit system then weights each of said corresponding blocks104with at least one of the following: similarity with the local block104A, a distance from a location of the local block104A, saturation, and noise of the corresponding blocks104. In an embodiment, the noise of the corresponding blocks104may be compared with noise of the local block104A. The location of the local block104A may be determined in a coordinate system of an image100different from the image102including the local block104A.

The image processing electric circuit system combines the local block104A and at least one of the corresponding blocks104, or replaces the local block104A by at least one of the corresponding blocks104based on the weighting for forming the composed image10from the images100,102. In this manner, the new image10is a combination of the images100,102such that disturbance, such as blur caused by the movement and the saturation, is minimized, while trying to maximize signal-to-noise ratio and/or to minimize noise.

In an embodiment, the composed image10may be formed by combining the plurality of images100,102when combining the local block104A and the at least one of the one or more corresponding blocks104, or replacing the local block104A by the at least one of the one or more corresponding blocks104.

This kind of composition on the basis of weighting of the blocks having movement can be considered a trilateral filtering which is edge-preserving and noise reducing.

The local block104A and the corresponding blocks104may have noise, which may be divided in several types. Gaussian noise, which may also be called thermal noise, comes from the camera that captured the image. Salt-and-pepper noise has dark pixels in bright regions and bright pixels in dark regions. Salt-and-pepper noise may also be called or spike noise. Shot noise, in turn, is caused by quantum fluctuations and it can be included the Gaussian noise because of it statistical properties. The shot noise and thus the Gaussian noise may include dark current in the image sensor. Because the images are in a digital form, the conversion into the digital form causes quantization noise. Additionally, the image100,102may have periodic noise, row noise, and/or column noise, for example.

FIG.2illustrates an example of the image processing apparatus200, which may comprise one or more processors202and one or more memories204including at least one computer program code. The one or more processors202, the one or more memories204and the at least one computer program code cause the image processing apparatus200at least to perform image processing method steps, which include the conversion of the images100,102to the common brightness range, the determination of the corresponding blocks104, the weighting of each of said corresponding blocks104, and the combination or replacement relating to the local block104A and the corresponding blocks104. The image processing apparatus200may comprise also a user interface206, which may comprise a screen and a keyboard, and/or a touch screen, for example. Additionally, the image processing apparatus200may comprise an imaging device208. The imaging device208may be a camera or the like, which captures or forms the images.

FIG.3illustrates an example where the corresponding block104is in the same image102as the local block104A. If an image has a shape300that is larger than the local block104A (the corresponding block has the same size as the local block), the same shape can also be found outside the local block104A. The shape300may continue in an unchanged manner or it may appear repeatedly in the image102. The saturation or movement of the shape300may differ in different locations of the image102, which may sometimes be utilized.

In an embodiment, the image processing apparatus200may use the coordinate system of the image100having a lowest brightness to determine locations in the images. Thus, a location for the local block104A and a location of the one or more corresponding blocks104may be determined in the coordinate system of the image100.

In an embodiment, the image processing apparatus200may convert the brightness of the images to the common brightness range by increasing brightness of dark images. If a dynamical range of the image100is 10 bits and it is brightened eight times, for example, which corresponds to an exposure with eight times more light, the values of pixel intensities are multiplied by eight. The dynamical range of the pixels of the initially darker image100is now 14 bits. In this case, it may be so that the pixels of the initially darker image100are quantized and the pixels of the initially brighter image102may be saturated. However, if the quantized pixels and the saturated pixels are not completely the same, it is possible to have an image without or with limited saturation and quantization by combining the images.

In an embodiment, the image processing apparatus200may form the similarity with the local block104A by computing the similarity between brightness of the local block104A and each of the one or more corresponding blocks104. Additionally or alternatively, the image processing apparatus200may form the similarity with the local block104A by computing the similarity between colors of the local block104and each of the one or more corresponding blocks104. Additionally or alternatively, the image processing apparatus200may form the similarity with the local block104A by computing the similarity of a distribution of brightness between the local block and one of the corresponding blocks.

In an embodiment, the image processing apparatus200may compute histograms of values of pixels of each of the one or more corresponding blocks104and the local block104A, and form the similarity based on a comparison of the histograms. A histogram can be used to show a distribution of numerical data of an image or a block of an image. The similarity may also be determined on the basis of dissimilarity.

The similarity may also be based on scale invariant feature transform (SIFT) algorithm. The similarity may further be determined using a plurality of method such as Pearson correlation, Tanimoto measure, Spearman's ρ, Kendall's τ, correlation, and/or Shannon/Rényi/Tsallis mutual information, for example. The dissimilarity may be measured using L1or L2norm, incremental sign distance, intensity-ratio variance, intensity-mapping-ratio variance, rank distance, joint entropy, and/or exclusive F-information. These methods may also measure noise and blurring as well as to intensity and geometric changes. The similarity may estimate the probability that the blocks represent the same target, when taking known or estimated amount of noise into an account.

In an embodiment, the image processing apparatus200may weight said each of the one or more corresponding blocks104by values, which decrease with an increasing distance from the location of the local block104A.

In an embodiment, the image processing apparatus200may weight said one or more corresponding blocks104by values, which decrease with increasing noise.

In an embodiment, the image processing apparatus200may estimate noise of the one or more corresponding blocks104and the local block104A based on a gain of a sensor, which captured the images. The gain of the sensor refers to amplification of the intensity values of the pixels, for example. Gain may also be taken into account in an embodiment, when different images are adjusted to be at a same intensity level. When the values of pixels of an image are multiplied by a value n, also the noise will increase with the value n. In this multiplication, the signal-to-noise ratio will remain constant. On the other hand, if the number of photons is increased by the value n, the photon noise will increase by a square root of the value n, which increases the signal-to-noise ratio by the square root of the value n. That is, the same brightness achieved with the gain as with an increase of a corresponding optical radiation power will result in an increase of noise by the square root of the increase/gain n.

Additionally or alternatively, the image processing apparatus200may estimate noise of the one or more corresponding blocks104and the local block104A based on one or more image transformations performed to the image. The image transforms may be geometric image transforms, for example. The image transformation may refer to a geometrical distortion and/or a combination of images, for example.

In an embodiment, the image processing apparatus200may estimate noise of the one or more corresponding blocks104and the local block104A based on a pre-gain performed to the images100,102prior to entering the method of this application.

In an embodiment, the image processing apparatus200may estimate noise of the one or more corresponding blocks104and the local block104A based on noise, which depends on a location in the images.

In an embodiment, the image processing apparatus200may estimate noise of the one or more corresponding blocks104and the local block104A based on image fusion.

Noise of an image may be reduced, when at least two images have been fused. When combining the images, at least one of the images may have been adjusted (contrast may have been changed, for example), which may have increased a gain of a pixel i.e. a pixel-specific gain may have been altered. Alternatively or additionally, a possible vignetting caused by a lens and/or any other physical disturbance or deformation caused by optical components of an image capturing device or environment (turbulence of atmosphere) may have been adjusted by a multiplication of pixel values which changes the pixel-specific gain.

In an embodiment, the image processing apparatus200may select an image102, which has the best information, where the best information may mean that the image102has the greatest brightness without saturation or with least saturation. In an embodiment, the image processing apparatus200may select an image102, which has the greatest brightness among the images100,102, as said one of images100,102. In this embodiment, the local block104A is selected to be in an image, which is the brightest image among the images and includes the movement. Here the brightest image may be a non-saturated image.

In an embodiment an example of which is illustrated inFIG.1, the image processing apparatus200may form a bright image102by combining images having a higher brightness than an image, which has a lowest brightness, before converting the images100,102to the common brightness range. In the example ofFIG.1, the image102may have been formed as a combination of bright images102′,102″, which are brighter than the dark image100.

In an embodiment an example of which is illustrated inFIG.1, the image processing apparatus200may combine a plurality of images of a lower brightness than an image, which has a highest brightness, before converting the images100,102to the common brightness range. In the example ofFIG.1, the image100may have been formed as a combination of dark images100′,100″, which are darker than the image102.

In an embodiment, a plurality of dark images100″ and a plurality of bright images102″ may be used separately. The formation of the corresponding blocks104may automatically combine information from multiple dark images. The formation of corresponding blocks104may automatically combine information from multiple bright images.

In an embodiment, the image processing apparatus200may register the images100,102before converting the images to the common brightness range. In this manner, the images100,102are aligned to each other. The shift S may be utilized in the alignment. The coordinate system of the images is unified, and a location of an area of the local block104A becomes the same in each of the images100,102.

In an embodiment, the corresponding blocks104may originate from several different input images, which may have been captured with different settings. Thus, in addition to spatial distance from the local block104A and similarity of blocks, information content (noise and saturation) of each the corresponding blocks104may affect the weighting. In general, input images may be captured with the same or different settings (brightness).

At extreme, non-moving and non-saturated areas of images are obtained merely from at least one bright image102″, which produces a better signal-to-noise ratio for these areas than a darker image.

If any of the corresponding blocks104contains a lot of saturated pixels, several dark images100″ may be combined.

In an embodiment, a dark image100may be used as a reference. As an effect of this kind of filtering, if target in the block104is moving, but is non-saturated, the at least one corresponding block104may be taken from at least one bright image102where it is at a slightly different location in addition to shift S of a local motion. The result of a combination of the images100,102is that a signal-to-noise level comes from the bright image102, but a location of a moving object is based to the dark image100. If the corresponding block104is attached to pixels that are saturated in the bright image102, but non-saturated in dark image100, this will result in a continuous object. Its location corresponds to that in the dark image100, without saturated pixels, where pixels are taken partly from a bright image102(less noisy), and partly from a dark image100. Also, these pixels that have been taken from the dark image100are rather bright (as these are saturated in the bright image102), so noise is not as high in these pixels as it would be in the corresponding block104of the dark image100.

There can be cases where a corresponding block104of a dark image100is not visible in a bright image102(with good enough similarity). Then, the bright image102cannot be utilized even if area of the corresponding block104would be non-saturated in an intensity level of the bright image102. In an embodiment, a location of each single moving object may vary, either coming from a dark or bright image100,102, where the object may move image-specifically. An algorithm may prefer an image that contains a best or optimum information, which may mean that the image is bright but the pixels are non-saturated, for example. Irrespective of a location, areas of these larger moving objects in the corresponding blocks104of a plurality of images may be partly combined (seeFIG.3although it refers to one image only).

In images, there may be a whole target/object that moves. Then, when the shift S has been compensated between a plurality of images, an outline of the target/object may correspond well to each other in different images. However, in an area surrounded by the outline of the target/object there may be pixel values that differ in different images100,102because of movement. The target/object may be divided in blocks like blocks104A,104, which may contain one pixel, 2×2 pixels, 8×8 pixels, 5×9 pixels or the like. Each of the blocks may be filtered separately in order to have a high quality composed image10despite the movement.

In an embodiment, different base frames may be selected for each block104. In an embodiment, different base locations for each of the corresponding blocks104may be selected. That means that some areas of the composed image10may come from the image100and some areas of the composed image10may come from the image102. This can be used to avoid ghost edges of an object in the composed image10and to increase total quality of the composed image10.

In an embodiment, the image processing apparatus200may determine noise pixel by pixel of the composed image10, and equalize the noise of the composed image10by filtering. This kind of filtering may be performed by any single image based spatial filtering (like Gaussian filtering, bilateral filtering, non-local means BM3D). Alternatively, this information of pixel-by-pixel noise may be utilized in a previous step in order to adjust a weight to merge a corresponding block104with a local block104A, based on a noise level of the corresponding block104. This filtering may take known noise level into an account. As a result, the image formed as a combination of images filtered in this manner contains an equal noise level over the whole image, irrespective if the corresponding block104has originated from a dark or bright image, or if the corresponding block104is moving (when combined with one or several bright frames, for example).

In an embodiment, the image processing apparatus200may process images100,102which are raw Bayer-pattern data. In other words, no preprocessing steps or less than usual amount preprocessing steps have been made for inputs. This facilitates the processing and estimation of the movement, helps in finding the local block104A and the corresponding blocks104, and helps estimating amount of noise remaining per image pixel.

In an embodiment an example of which is illustrated inFIG.2, the image processing apparatus comprises one or more processors202; and one or more memories204including computer program code. The one or more memories204and the computer program code are configured to, with the one or more processors202, cause image processing apparatus at least to: convert the intensity values of the images to the common brightness range; select one of images100,102including a local block104A with movement for forming a composed image10from the images100,102; determine the corresponding blocks104; weight each of said corresponding blocks104with respect to the local block104A; and perform the combination or replacement.

FIG.4presents an example of a flow chart of the image processing method. In step400, images100,102of different brightness are converted to a common brightness range, the images100,102representing at least partly the same scene. In step402, one of images100,102including a local block104A with movement is selected for forming a composed image10from the images100,102. In step404, one or more corresponding blocks104to the local block104A determined in the images100,102. In step406, each of said one or more corresponding blocks104are weighted with at least one of the following: similarity with respect to the local block104A, a distance from a location of the local block104A, saturation of the one or more corresponding blocks104, and noise of the one or more corresponding blocks104. In step408, the local block104A and at least one of the one or more corresponding blocks104are combined, or the local block104A is replaced by at least one of the one or more corresponding blocks104based on the weighting for forming the composed image10from the images100,102.

The method shown inFIG.4may be implemented as a logic circuit solution or computer program. The computer program may be placed on a computer program distribution means for the distribution thereof. The computer program distribution means is readable by a data processing device, and it encodes the computer program commands, and carries out the image processing.

The computer program may be distributed using a distribution medium, which may be any medium readable by the controller. The medium may be a program storage medium, a memory, a software distribution package, or a compressed software package. In some cases, the distribution may be performed using at least one of the following: a near field communication signal, a short distance signal, and a telecommunications signal.

It will be obvious to a person skilled in the art that, as technology advances, the inventive concept can be implemented in various ways. The invention and its embodiments are not limited to the example embodiments described above but may vary within the scope of the claims.