Abstract:
The present invention relates to a method for sharpness enhancing digital image data (I, ID). A basic idea is to choose the degree of enhancing the sharpness in a local manner to thereby enhance only local parts of the underlying image (I) and to thereby avoid or reduce enhancing noise components contained in the underlying image (I).

Description:
FIELD OF THE INVENTION  
       [0001]     The present invention relates to a method for sharpness enhancing digital images and digital image data. More particular, the present invention relates to a method for controlling image sharpness enhancement, in particular on noisy images.  
       BACKGROUND OF THE INVENTION  
       [0002]     In many applications and customer devices, before displaying images the underlying images and image data have to be pre-processed in order to better adapt the image properties. One aspect is the enhancement of sharpness of the image. The problem when enhancing the sharpness of an image is that the process in general is applied to all image details represented by the image data. However, some of the image details indeed stem from noise components which are also contained in the image. Under the process of sharpness enhancement also the noise components contained in the image data are enhanced and therefore amplified.  
       SUMMARY OF THE INVENTION  
       [0003]     It is therefore an object of the present invention to provide a method for sharpness enhancing of digital image data which is capable of enhancing the sharpness of useful image structures and to not enhance noise components contained in the images.  
         [0004]     The object is achieved by a method for sharpness enhancing digital image data according to independent claims  1  and in addition by a system or device, by a computer program product, and by a computer readable storage medium according to claims  29 ,  30 , and  31 , respectively.  
         [0005]     The present invention relates to a method for sharpness enhancing digital image data. A basic idea is to choose the degree of enhancing the sharpness in a local manner to thereby enhance only local parts of the underlying image and to thereby avoid or reduce enhancing noise components contained in the underlying image.  
         [0006]     According to a most general view of the present invention the inventive method for enhancing sharpness of image data is based on the aspect of locality of the sharpness enhancing process. Locality in the sense of the present invention means that local areas of the underlying image are treated in general differently from each other, namely in dependence of the local properties of the local areas. Therefore, the process of enhancing sharpness is according to the present invention locally designed in order to better take care and take into account the local aspects of the different areas contained in the underlying image. For instance, but not restricted thereto is the local adjustment of the degree of sharpness enhancement in order to treat areas with different noise and/or texture in a different way. Thereby the enhancement of noise can be avoided, whereas useful structures to be enhanced may be for instance enhanced in the usual way.  
         [0007]     According to a first general solution for the object underlying the present invention the invention provides a method for sharpness enhancing digital image data wherein in a process of enhancing sharpness of image data ID representing an image I noise or noise components contained in the underlying image I to be sharpness enhanced and/or in the underlying digital image data ID or a part thereof are taken into account and/or are treated in a local manner with respect to the underlying image I and/or in the underlying digital image data ID or a part thereof, wherein thereby enhancing said noise or said noise components in said image I and/or in the underlying digital image data ID or the respective part thereof is one of reduced or avoided and sharpness enhancement is achieved with received or no side effects.  
         [0008]     According to a second general solution for the object underlying the present invention the invention provides a method sharpness enhancing digital image data wherein a process of enhancing sharpness of image data ID representing an image I is locally applied to the underlying image I to be sharpness enhanced or in the underlying digital image data ID or a part thereof, wherein the degree of enhancing sharpness is controlled by respective local properties in said underlying image I, the underlying digital image data ID or the respective part thereof, and wherein thereby enhancing said noise or said noise components in said image I and/or in the underlying digital image data ID or the respective part thereof is one of reduced or avoided and sharpness enhancement is achieved with received or no side effects. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0009]     The invention will now be explained based on preferred embodiments thereof and by taking reference to the accompanying and schematical figures.  
         [0010]      FIG. 1  is a schematic diagram for elucidating the structure of a sequence of images, which is used in the present invention.  
         [0011]      FIG. 2  is a schematic diagram for elucidating the structure of an image used in the context of the present invention.  
         [0012]      FIG. 3  is a schematic diagram, which elucidates the structure of a block within a given image used in the context of the present invention.  
         [0013]      FIG. 4  is a schematic block diagram for elucidating some basic aspects of a preferred embodiment of the inventive method for estimating noise in an image sequence.  
         [0014]      FIG. 5  is a schematic block diagram for elucidating the process of sharpness enhancement according to a first preferred embodiment of the present invention.  
         [0015]      FIG. 6  is a schematic block diagram for elucidating the process of sharpness enhancement according to a second preferred embodiment of the present invention.  
         [0016]      FIG. 7  is a schematic diagram for elucidating the process of noise cancellation according to a further preferred embodiment of the present invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0017]     In the following functional and structural similar or equivalent element structures will be denoted with the same reference symbols. Not in each case of their occurrence a detailed description will be repeated.  
         [0018]     According to a first solution for the object underlying the present invention the invention provides a method for sharpness enhancing digital image data wherein in a process of enhancing sharpness of image data ID representing an image I noise or noise components contained in the underlying image I to be sharpness enhanced and/or in the underlying digital image data ID or a part thereof are taken into account and/or are treated in a local manner with respect to the underlying image I and/or in the underlying digital image data ID or a part thereof, wherein thereby enhancing said noise or said noise components in said image I and/or in the underlying digital image data ID or the respective part thereof is one of reduced or avoided and sharpness enhancement is achieved with received or no side effects.  
         [0019]     According to a second solution for the object underlying the present invention the invention provides a method sharpness enhancing digital image data wherein a process of enhancing sharpness of image data ID representing an image I is locally applied to the underlying image I to be sharpness enhanced or in the underlying digital image data ID or a part thereof, wherein the degree of enhancing sharpness is controlled by respective local properties in said underlying image I, the underlying digital image data ID or the respective part thereof, and wherein thereby enhancing said noise or said noise components in said image I and/or in the underlying digital image data ID or the respective part thereof is one of reduced or avoided and sharpness enhancement is achieved with received or no side effects.  
         [0020]     Of course both solutions can be combined.  
         [0021]     The inventive method may comprise (a) a step of providing S 1  said digital image ID as input data InD which are representative for said underlying image I to be sharpness enhanced, (b) a step of sharpness enhancing S 2  said underlying image I by appropriately processing said digital image data ID or a derivative thereof to thereby generate processed digital image data PID being representative for the processed image PI, I′, and (c) a step of providing said processed digital image data PID as output data outD which are representative for the processed and therefore sharpness enhanced image PI, I′, (d) wherein the locality of said taking into account of said noise or noise components and/or said treatment of said noise or noise components is performed and realised at least in said step (b) of sharpness enhancing S 2  said image I.  
         [0022]     The locality of said taking into account said noise or noise components and/or said treatment of said noise or noise components is realised by (A) a step of defining and selecting T 1  in said underlying image I, said respective digital image data ID or said derivative thereof a plurality of blocks Bjk and (B) a step of classifying T 2  each of said blocks Bjk by deriving block classification data BCD which are locally descriptive for at least one of the group consisting of noise level, flatness level, and texture level of said underlying image I, said respective digital image data ID or said derivative thereof in the respective block Bjk, (C) wherein said step (b) of sharpness enhancing S 2  said underlying image I is based on said blocks Bjk and controlled based on said block classification data BCD.  
         [0023]     Within said step (B) of classifying T 2  said blocks Bjk a local analysis of the structure of the underlying image I, of the respective digital image data ID or of the respective derivative thereof may be performed, in particular based on said blocks Bjk.  
         [0024]     Sections may be detected in said underlying image I, in said respective digital image data ID or in said respective derivative thereof as uniform sections USjk, in particular based on said blocks Bjk, which fulfill a predefined uniformity criterion.  
         [0025]     Said uniformity criterion may be descriptive for at least one of the group consisting of uniformity, flatness, structure, and noise content.  
         [0026]     No or only a reduced degree of sharpness enhancement may be applied to said uniformity sections USjk.  
         [0027]     Said uniformity criterion may be described based on an uniformity function U.  
         [0028]     At least one of the group consisting of statistical properties, statistical standard deviation σ and statistical variance may be used for said uniformity criterion and in particular as a uniformity function U, each of which being applied.  
         [0029]     A given block Bjk may be classified as a uniform section USjk if the corresponding uniformity function value Ujk of the respective block Bjk based on the uniformity function U is below a given threshold value Th.  
         [0030]     A certain uniformity value Uglob may be used as said threshold value Th which is obtained by applying said uniformity function U to the entire underlying image I, the respective digital image data ID or the respective derivative thereof, in particular according to the following formula I: 
 
 Th≈Uglob≈U ( I ),  (I) 
 
 with I denoting the image, U denoting the uniformity function, Uglob denoting the value of the uniformity function for the entire image I, and Th denoting the threshold value. 
 
         [0031]     Then said uniform sections USjk may be cleaned from noise with preserving at least one of the group consisting of contours, edges, wedges, structures, and/or texture in said underlying image I, said respective digital image data ID or said respective derivative thereof.  
         [0032]     Said process of cleaning uniform sections USjk from noise or noise components may be based on a filter process.  
         [0033]     A Kuwahara filter process may be used as said filter process.  
         [0034]     Said filter process is applied to each picture element or pixel PBjkab of said image I of a respective uniform section USjk.  
         [0035]     For each picture element or pixel PBjkab of a uniform section USjk to be processed respective four rectangular neighbourhood regions N 1 , . . . , N 4  having said picture element or pixel PBjkab as a corner may be defined and evaluated with respect to respective statistical properties, in particular with respect to a mean value, a standard deviation and/or a statistical variance.  
         [0036]     For cleaning noise or noise components the value of the given pixel PBjkab is replaced by a lowest value of the four values of the statistical property Var of the respective four neighbourhood regions N 1 , . . . , N 4 , in particular according to the following formula (II): 
 
 PBjkab (original)           PBjkab (cleaned)≈min{Var( N 1), Var( N 2), Var( N 3), Var( N 4)},  (II) 
 
 with N 1 , . . . , N 4  denoting the neighbourhood regions, Var(.) denoting the function for obtaining the statistical property of the respective neighbourhood region Nj with j=1, . . . , 4—in particular its variance over all picture elements or pixel PBjkab within said neighbourhood region Nj—and min(.) denoting the minimum function. 
 
         [0037]     Thereby noise or noise components may be removed and said contours, edges, wedges, structures, and/or textures essentially remain unchanged.  
         [0038]     Then said contours, edges, wedges, structures and/or textures are extracted and refined in order to obtain an intermediate processed image IPI and respective intermediate digital image data IDI in which uniform sections USjk may therefore be cleaned from noise and noise components and in which said respective contours, edges, wedges, structures, and/or textures are enhanced.  
         [0039]     An edge image may be derived as said intermediate processed image IPI.  
         [0040]     Said edge image IPI is derived by detecting said contours, edges, wedges, structures, and/or textures using the absolute value of first derivatives in said underlying image I, said digital image data ID or said respective derivative thereof.  
         [0041]     The respective first derivates may be obtained in a diagonal manner from top/left to bottom/right and from top/right to bottom/left.  
         [0042]     Then an erosion algorithm may be applied in order to extent said contours, edges, wedges, structures, and/or textures, in particular in order to close holes and/or gaps and to finally obtain the intermediate processed image IPI.  
         [0043]     Said processed image PI and said respective processed digital image data PID may be obtained by blending said intermediate processed image IPI with that original underlying image I or the respective digital image data PID, ID or respective derivatives thereof with the original image I being subjected to a two-dimensional sharpness enhancement process.  
         [0044]     The process of blending may be performed on the basis of pixels PBjkab using a so-called blend value b, in particular ranging between 0 and 1 and further in particular according to the following formula (II): 
 
 PBjkab ( PI )≈ b·PBjkab ( IPI )+(1 −b )· PBjkab ( I )  (III) 
 
 with I denoting the image, IPI denoting the intermediate processed image, PI denoting the processed image, PBjkab(.) denoting the respective picture element or pixel, and b denoting the blending or mixing value. 
 
         [0045]     A plurality of L blocks Bjk may be defined in said image, frame, or picture I, with L being an integer.  
         [0046]     Said plurality of blocks Bjk or a sub-set thereof may cover the respective image, frame, or picture I completely.  
         [0047]     Each of said blocks Bjk may be given by a number M×N of image elements or pixels PBjkab, with M, N being integers.  
         [0048]     Each of said blocks Bjk have the same—in particular rectangular—shape and/or extension, in particular with respect to the number of M×N pixels PBjkab.  
         [0049]     Blocks Bjk may be non-overlapping.  
         [0050]     Blocks Bjk may overlap.  
         [0051]     According to the present invention a system or an apparatus for sharpness enhancing digital image data is provided which is adapted and which comprises means for realising and performing a method for sharpness enhancing digital image data according to the present invention.  
         [0052]     In addition, a computer program product is provided comprising computer program means which is adapted and arranged in order to perform and realise a method for sharpness enhancing digital image data according to the present invention and the steps thereof when it is executed on a computer or on a digital signal processing means.  
         [0053]     According to a still further aspect of the present invention a computer readable storage medium is provided comprising a computer program product according to the present invention.  
         [0054]     These and further aspects of the present invention will be further discussed in the following:  
         [0055]     The present invention inter alia also relates to a method for controlling image sharpness enhancement on noisy images.  
         [0056]     A new method for controlling the sharpness enhancement of noisy sequences is proposed. The method starts with local the analysis of the structure of the images. A pre-processing step cleans the uniform area from noise, leaving all contours in the image intact. In this image, edges and textures are extracted and refined. The resulting data is used to blend the unprocessed image with the sharpness enhanced image so that sharpness enhancement is only applied in textured areas and around edges but not in uniform areas.  
         [0057]     Current state of the art for improving the image sharpness is based on a global sharpness control depending on the noise level of the image. If an image is very noisy, sharpness enhancement is typically switched off totally, as it enhances noise. If an image is not noisy, sharpness enhancement is applied. Intermediate steps are possible, but always the level of sharpness for the total image is changed.  
         [0058]     One basic problem solved by the invention is the local change of the level of sharpness enhancement applied on an image depending on the content and the content&#39;s sensitivity to noise. For example in uniform area, noise is very visible, especially noise enhanced by sharpness enhancement algorithms. So no sharpness enhancement is applied in this area.  
         [0059]     The method is described also in  FIGS. 5 and 6 .  
         [0060]     First, a filter is applied on the image to remove noise in the flat area. The filter in use calculates mean and variance of local areas and flattens the image, leaving edges and textures intact.  
         [0061]     From this image, an edge image is generated using the absolute value of the first derivatives. The first derivative is calculated diagonally top/left-bottom/right and top/right-bottom/left.  
         [0062]     Then, an erosion algorithm is applied to extend the area with textures and details, i.e. to close holes and gaps.  
         [0063]     The result, which has the same size as the original image, can be seen as a map of the flat area of the image. A small value of the map indicates a uniform area like a sky or a flat background. A big value indicates an area close to an edge (like a character on white paper) or within a textured area (like grass or leaves on a tree).  
         [0064]     Then, two-dimensional sharpness enhancement is applied to the original image. This map is used to blend the sharpness enhanced image with the original image. In flat areas (map value is small), mostly the unprocessed signal is sent to the output. It textured areas (map value is big), mostly the processed signal is sent to the output.  
         [0065]     With this improvement, a sharpness enhancement algorithm can be applied in full strength even on noisy sequences with minimized visual disturbance.  
         [0066]     The optimum and robust solution according to the present invention works well on noisy and on noise-free sequences and adaptable to software and/or hardware solutions.  
         [0067]     In the following reference is again taken to the figures:  
         [0068]      FIG. 1  is a schematic diagram for elucidating a structure of a sequence SI of images I or of a image sequence SI which can be processed according to the present invention.  
         [0069]     In  FIG. 1  the image sequence SI is composed of a sequence of images I 1 , I 2 , . . . , I 1 , I 1 +1, I 1 +2, . . . For each but the first image the image I 1  is a directly preceding image, also denoted by I and I 1 +1 is referred to as a directly succeeding image. This is demonstrated by the linearly arranged sequence of squares I 1 , I 1 +1, I 1 +2 in  FIG. 1 .  
         [0070]     In  FIG. 2 a  schematic demonstration of the structure of each of said images I 1  is given. Each of the images I 1 —being a preceding image I or a succeeding image—is formed by a certain number of lines or rows and by a certain number of columns of picture elements which are also called pixels. According to the present invention the images I are subdivided into blocks Bjk. Therefore, each image I, I 1  is subdivided into m lines and into n columns of blocks Bjk; j=1, . . . , m, k=1, . . . n.  
         [0071]     As the images I, I 1  are formed by lines and rows of picture elements or pixels, the blocks Bjk, Bjk′ are also formed by certain rows and columns of pixels PBjkab. One can assume—but it is not necessary—that each of the blocks has an identical extension, e.g. of M×N pixels. Then one has for each image I, I 1  L=m×n blocks Bjk per image, picture or frame with each of said blocks Bjk having M×N pixels PBjkab; j=1, . . . , m; k=1, . . . , n; a=1, . . . , M; b=1, . . . , N.  
         [0072]     Based on such a subdivision of the images I into blocks Bjk and into pixels PBjkab the inventive processing will be explained in the following based on the further FIGS.  4  to  7 .  
         [0073]      FIG. 4  elucidates in a schematic flow chart an embodiment of the inventive method for estimating noise in an image sequence, in particular in a video stream or the like.  
         [0074]     After a starting or initialisation phase S 0  digital image data ID for an image I to be sharpness enhanced are provided in a first step S 1  as input data inD. The image I has a certain format, which might be in particular fixed for all images I of the sequence SI.  
         [0075]     In a following step S 2  the image I underlying the provided digital image data ID is sharpness enhanced by appropriately processing the image data ID. It is also possible to first of all derive some kind of data from the image data ID, e.g. the luminance, the chrominance or the like and to then perform the sharpness enhancement on such a derivative. By the enhancement process of step S 2  processed image data PID derived which are representative for a processed, i.e. sharpness enhanced image PI or I′.  
         [0076]     In a following fourth step S 3  said processed image data PID and/or said processed image PI are provided and output as output data outD.  
         [0077]      FIG. 5  is a schematic block diagram for elucidating a process of sharpness enhancement according to a first preferred embodiment of the present invention.  
         [0078]     Again the whole process can be understood as a concatenation of a step S 1  of providing digital image data ID, a step of sharpness enhancement S 2 , and a step S 3  of providing the processed digital image data PID and the processed image I′.  
         [0079]     In a first step U 1  input data InD are received, provided or generated which represent as digital image data ID an underlying image I which might be one of a sequence SI of images I.  
         [0080]     Said image data ID are fed into two independent processing streams. The first processing stream comprises a step U 2  for pre-processing the image data ID in order to clean flat or uniform areas or sections USjk which are formed by or constituted by the respective image blocks Bjk from noise or noise components, a step U 3  for two-dimensional edge detection and refinement, and a step U 4  of post-processing the obtained cleaned and etch-refined image data in order to compensate erosion processes by filling holes or gaps.  
         [0081]     In the parallel processing stream a single processing step U 5  of two-dimensional sharpness enhancement on the original input data ID for the original image I is performed.  
         [0082]     According to a following blending step U 6  the two-dimensionally sharpness enhanced original image data ID and the intermediate image data IID which are cleaned from noise in the flat areas, etch-refined and erosion compensated are blended, i.e. they are mixed in order to provide processed image data PID corresponding to a processed image PI, I′ in which only areas with contours, edges, wedges, structures or the like are enhanced, flat areas or uniform areas are cleaned from noise, wherein side effects of amplifying or enhancing the noise are reduced or avoided.  
         [0083]     In a final step U 7  the processed image PI, I′ and the processed image data PID are provided and output as output data OutD.  
         [0084]      FIG. 6  is a schematical block diagram for elucidating the structure of the enhancement process according to another preferred embodiment of the present invention. This embodiment is realised inter alia by three independent processing streams which are again considered by a blending or mixing process.  
         [0085]     In a first step U 1  input data InD are received, provided or generated which represent as digital image data ID an underlying image I which might be one of a sequence SI of images.  
         [0086]     Said image data ID are fed into three independent processing streams. The first processing stream comprises a step U 2  for pre-processing the image data ID in order to clean flat or uniform areas or sections USjk which are formed by or constituted by the respective image blocks Bjk from noise or noise components, a step U 3  for two-dimensional edge detection and refinement, and a step U 4  of post-processing the obtained cleaned and etch-refined image data in order to compensate erosion processes by filling holes or gaps.  
         [0087]     In the parallel processing second stream a single processing step U 5  of two-dimensional sharpness enhancement on the original input data ID for the original image I is performed.  
         [0088]     In the parallel third processing stream non operation is performed on the original image data ID, they are just conveyed in an unchanged manner.  
         [0089]     According to a following blending step U 6  the original image, the two-dimensionally sharpness enhanced original image data ID and the processed intermediate image data PID which are cleaned from noise in the flat areas, etch-refined and erosion compensated are blended, i.e. they are mixed in order to provide processed image data PI corresponding to a processed image I′ in which only areas with contours, edges, wedges, structures or the like are enhanced, flat areas or uniform areas are cleaned from noise, wherein side effects of amplifying or enhancing the noise are reduced or avoided.  
         [0090]     In a final step U 7  the processed image I′ and the processed image data PID are provided and output as output data OutD.  
         [0091]     The processing of the embodiment shown in  FIG. 6  is in so far very similar to the processing of the embodiment shown in  FIG. 5 . However, in the embodiment of  FIG. 6  in step U 6  three signal components or data components are blended and mixed. First of all—comparable to the embodiment of  FIG. 5 —the noise cleaned, etch-refined, and erosion compensated intermediate processed image data IPD as well as the two-dimensional sharpness enhanced original image data ID are taken. However, in addition also the original image data ID are included into the blending or mixing process in order to obtain a processed image I′ via respective processed image data PID.  
         [0092]      FIG. 6  schematically elucidates the process of noise cleaning.  
         [0093]     In  FIG. 7  an image I is given as a matrix of pixels PBjkab. The complete image I is again subdivided into blocks Bjk.  FIG. 7  in detail demonstrates a certain block Bjk. Each of the pixels PBjkab is processed in order to clean the pixels from noise content or noise components. To do this, each pixel PBjkab is taken and respective sections N 1 , N 2 , N 3 , and N 4  are defined. The relationship between the pixel PBjkab to be processed and the sections N 1  to N 4  is, that the respective pixel PBjkab in each case forms an edge element or edge pixel of the rectangular sections N 1 , N 2 , N 3 , N 4 . Then the variance values Var (Nl) for l=1, . . . , 4 are calculated for the cleaned image data a respective cleaned pixel PBjkab′ is defined as the minimum value of the variance values Var (Nl) for l=1, . . . , 4.  
       REFERENCE SYMBOLS  
       [0000]    
       
          B block, preceding block  
          Bjk block, preceding block, block of a preceding image/frame, picture  
          I image/frame /picture  
          I′ processed image/frame/picture  
          I 1  image/frame/picture in the sequence SI  
          ID digital image data  
          inD input data  
          IPI intermediate processed image/frame/picture  
          L number of selected blocks per image/frame/picture  
          L′ number of blocks in an image/frame/picture  
          m number of blocks in a column in an image/frame/picture  
          min(.) minimum function  
          M number of pixels in a column in a block  
          n number of blocks in a line/row in an image/frame/picture  
          N number of pixels in a line/row in a block  
          N 1 , N 2 , N 3 , N 4  neighbourhood regions or sections  
          outD output data  
          PI processed image/frame/picture  
          PID processed image data  
          Pjkab image element, pixel in block Bjk of image/frame/picture I  
          Pjkab′ cleaned image element, cleaned pixel in block Bjk of  
          image/frame/picture I  
          SI sequence of images/frames/pictures  
          σ(.), sigma(.) uniformity function  
          U(.) uniformity function  
          Ujk uniformity function value for block Bjk  
          USjk uniform section  
          V, V(I) variance vector, vector of variance values for image I  
          Var(.) variance function