Multi-modal tone-mapping of images

A system for multi-modal mapping of images is described. Embodiments are described where the image mapping system is used for visualizing high dynamic range images such as medical images, satellite images, high dynamic range photographs and the like and also for compressing such images. In examples, high bit-depth images are tone-mapped for display on equipment of lower bit-depth without loss of detail. In embodiments, the image mapping system computes statistics describing an input image and fits a multi-modal model to those statistics efficiently. In embodiments, the multi-modal model is a Gaussian mixture model and a plurality of sigmoid functions corresponding to the multi-modal model are obtained. In an embodiment the sigmoid functions are added to form a tone-mapping function which is used to transform a high bit-depth image such as 16 or 12 bits per pixel to a low bit-depth image such as 8 bits per pixel.

BACKGROUND

Image mapping is carried out for many purposes such as image compression, contrast enhancement, and for enabling images captured with capture devices of particular types to be displayed using display devices of different capabilities. For example, in the field of medical imaging (or in other fields such as professional photography, robotic imaging systems, high dynamic range photography, depth cameras, capture devices often produce 16-bit images where for example, each pixel may be one of 65,536 levels of grey (in the case of a greyscale image). Other image capture devices may produce 12-bit images or 32-bit images depending on the image capture device. The term “bit-depth” is used to refer to the number of bits available per pixel at an image capture or display device.

Where images have been captured with high bit-depth devices it is often required to reduce the bit-depth to enable the captured images to be displayed on a display device with lower bit-depth. This is difficult to achieve whilst preserving as much information as possible, so as not to lose the original dynamic range captured in the high bit-depth device. This is important for many types of images and particularly so in the field of medical imaging, where images often have particularly high dynamic range and where it is required to visualize fine details in images and remove noise as far as possible in order to make accurate medical diagnoses.

Dynamic range of an image may be thought of as the ratio between the intensities of the brightest and darkest recordable parts of that image. Tone-mapping functions are typically used to compress the dynamic range of an image to allow more detail in the original image to be visualized on a display whilst preferably preserving the “natural look” of the image. Improved tone-mapping systems are required which may produce more realistic, useful results in a computationally inexpensive, fast and robust manner.

Where images are captured at devices with relatively high bit-depth, it is often required to compress those captured images to reduce their size for storage and/or transmission. Image compression is difficult to achieve in a manner which is computationally inexpensive, fast, which does not produce visible artefacts and which is reversible (that is, the original image can be obtained from the compressed image without loss of quality).

Previous approaches for mapping images of one bit-depth to another bit-depth have included histogram equalization, linear mappings and gamma mappings. Linear mappings and gamma mappings are straightforward techniques but which are also very limited in the quality of results they give.

Histogram equalization tone-mapping processes typically involve taking the cumulative histogram of an image to be tone-mapped. The cumulative histogram is then normalized to 255 (in the case that the output bit-depth is 8 bits) and the normalized cumulative histogram is then used as a mapping function to transform the original image to the required bit-depth. However, histogram equalization processes are often found to be very aggressive and as a result fine details in images are lost. Artefacts may also be introduced such as gradient reversal and quantization or banding artefacts.

Local histogram equalization tone-mapping processes are also known. These are sometimes referred to as adaptive histogram equalizing techniques. They involve applying different transforms to equalize the histograms of sub-regions of an image. These approaches are typically highly computationally intensive and difficult to implement in real-time applications. Noise and artefacts may also be introduced for example, because a rectangular window is typically used around each pixel for histogram equalization in this window. The resulting transform is then not spatially smooth.

Previously, a single sigmoid function has been used as a tone-mapping function, with the sigmoid function determined from original image statistics and taking perceptual preference guidelines into account. Taking subjective preferences into account allows the image to look as pleasing as possible to the viewer. This is desirable in consumer imaging and commercial photography. However, this approach is not suitable for medical imaging applications, satellite imaging, archiving and the like where the goal is information preservation. In such cases it is required to preserve or enhance details at all regions and luminance levels of an image, not just those suited to the human visual system.

The embodiments described below are not limited to implementations which solve any or all of the noted disadvantages of known image mapping systems.

SUMMARY

A system for multi-modal mapping of images is described. Embodiments are described where the image mapping system is used for visualizing high dynamic range images such as medical images, satellite images, high dynamic range photographs and the like and also for compressing such images. In examples, high bit-depth images are tone-mapped for display on equipment of lower bit-depth without loss of detail. In embodiments, the image mapping system computes statistics describing an input image and fits a multi-modal model to those statistics efficiently. In embodiments, the multi-modal model is a Gaussian mixture model and a plurality of sigmoid functions corresponding to the multi-modal model are obtained. In an embodiment the sigmoid functions are added to form a tone-mapping function which is used to transform a high bit-depth image such as 16 or 12 bits per pixel to a low bit-depth image such as 8 bits per pixel.

DETAILED DESCRIPTION

Although the present examples are described and illustrated herein as being implemented in a medical imaging system, the system described is provided as an example and not a limitation. As those skilled in the art will appreciate, the present examples are suitable for application in a variety of different types of imaging systems.

FIG. 1ais a schematic diagram of a cumulative histogram100of a high dynamic range image.FIG. 1bis a schematic diagram of a single sigmoid function101which may be used as a tone-mapping function for the image which produced the histogram ofFIG. 1a. For a given pixel in the high dynamic range input image, the intensity value is used to look-up the corresponding low dynamic range pixel value using the sigmoid function. This process is satisfactory in the case that the cumulative histogram of the input image is uni-modal as in the example ofFIG. 1a. However, this is not the case for many types of high dynamic range images such as medical images, satellite images, dental images, high dynamic range photographs and other images of scenes or items where there are many distinct ranges of pixel intensities. This is explained in more detail with reference toFIG. 2.

FIG. 2shows a schematic diagram of a CT scan and an associated cumulative image histogram. The CT scan is of a human body comprising muscle201, bone200and lungs202. The parts of the image representing bone tend to have pixel intensities in a high range203as shown in the histogram. The parts of the image representing muscle tend to produce pixel intensities in a different range204and the parts of the image representing lungs205produce pixel intensities in yet another range205. These three different ranges in the histogram are not adequately taken into account by a single sigmoid tone-mapping function such as that ofFIG. 1b. If such a tone-mapping function is used the results are poor because much of the detail of the lungs and bone regions are lost. This is problematic for applications where information preservation is key such as for medical diagnosis.

To address such problems the embodiments described herein use a multi-modal model which is able to take into account more than one mode in the cumulative histogram of the input image. Multi-modal models are more complex than single mode models (such as that ofFIG. 1b) and yet the embodiments described herein provide an image mapping system which may be used in real time for practical applications such as in hospitals, dental surgeries, weather forecasting systems and the like.

FIG. 2is a flow diagram of a method of image mapping. An input digital image is received300which is a high dynamic range image of any type. A cumulative histogram is computed of the input image301. For example, for a grey scale image a histogram generator determines a bin size (e.g. a range of values such as 0-10, 11-20, 21-30, etc.) and counts the number of occurrences of pixel grey scale values in each bin. A graph of the bins against the frequency of occurrence in each bin gives a graphical representation of the histogram such as that illustrated inFIG. 2.

A processor is arranged to fit302a multi-modal model to the histogram. Any suitable type of multi-modal model may be used such as a Gaussian mixture model (GMM). The number of modes (n) that are required may be a configurable parameter303or may be determined automatically by the image mapping system. For many practical applications, including CT scans and MRI (magnetic resonance imaging) scans it is found that using only two modes provides greatly improved results as compared with previous tone-mapping processes and that increasing the number of modes to three or more is advantageous in some cases.

In an embodiment, an n-component GMM model is fitted to the histogram. Various different methods of computing the GMM model are possible as discussed below with reference toFIG. 5. GMM models are described in detail in Bishop. C. M Pattern Recognition and Machine Learning. 2006 Springer which is incorporated herein by reference in its entirety.

Using the Gaussian mixture model n corresponding mapping sigmoid functions are computed304and summed306to obtain a multi-modal tone-mapping function (also called transformation function). The bit-depth of the output display305may be taken into account. This is described in more detail below with reference toFIG. 4.

Once the tone-mapping function is obtained it is used to transform307the high dynamic range input image into an output image with the required bit-depth. The output image is stored or displayed on a display device such as a computer screen or printer308.

FIG. 4ais a schematic diagram of a cumulative histogram of a high dynamic range image. The histogram is multi-modal and a GMM model is fitted with 3 components represented by Gaussian distributions400,401and402. Each Gaussian distribution has an associated mean p and standard deviation sigma. The parameters of the Gaussian distributions are used as known in the art to form a sigmoid function corresponding to each Gaussian distribution. These sigmoid functions403,404,405are added to form a tone-mapping function as illustrated inFIG. 4b. The resulting tone-mapping function may be adjusted if necessary to ensure that it begins at the origin in order that black pixels from the HDR image map to black pixels in the output image. This ensures that the full range of available bit-depth in the output is made use of.

The step of fitting the multi-modal model is relatively complex compared with other stages of the process and compared with previous approaches which have used uni-modal models. Various different methods of fitting the multi-modal model may be used. For example, an n-component GMM model 500 may be fitted using an expectation maximization logic provided in the image mapping system. Expectation maximization is explained in detail in Bishop. C. M Pattern Recognition and Machine Learning. 2006 Springer reference above. Another approach is to use a K-means clustering logic provided in the image mapping system. K-means clustering is also explained in detail in Bishop. C. M Pattern Recognition and Machine Learning. 2006 Springer which is referenced above. In order to provide a particularly fast implementation, an embodiment uses 1D K-means clustering in an efficient manner. In this embodiment image data to be processed is received in the form of integer values. For non-integer values these are first quantized and then treated as integers. The minimum and maximum values in the integer data set are determined and a histogram of the integer values is created. Cluster center positions are initialized to be equally spaced throughout the range of values. The following steps are then performed iteratively until convergence:

For each of K clusters the 1D Voronoi interval is determined around the current center position. For each histogram entry in the current interval, the number of values in that cluster are accumulated and the sum of those values is determined. After going through the whole interval the cluster's center of mass is re-computed as the sum divided by the count. If the procedure produces de minimis change in the cluster center positions then the procedure is terminated.

In some embodiments the image mapping process is dynamic in that the tone-mapping function is re-computed in real time for sub-regions of the input image, for example, as a result of user input “zooming in” on a particular part of the image. This is useful in many data visualization applications such as medical imaging where a medical doctor may need to see more detail in a particular region of an image in order to make a diagnosis. The sub-region of the image may be specified by user input, such as via a graphical user interface, speech interface or other interface. Alternatively, the sub-region may be selected automatically by an image processing system.

For example as described with reference toFIG. 6, a tone-mapped image is obtained using a first mapping function (which is multi-modal)600and displayed. User input is received specifying a sub-region of the image601and a second mapping function is computed in real-time602. For example, the n-component GMM model is fitted to a cumulative histogram obtained from the image sub-region. The display is then dynamically updated using the second mapping function603and optionally using a smoothing process to avoid an abrupt change in the display. For example, the second mapping function is applied to the image sub-region and only the transformed sub-region displayed.

The processes described herein may also be used to enhance image compression. For example, previously a high bit-depth image700has been compressed with a codec701suitable for high bit-depth images. Known compression algorithms for high bit-depth images are complex and time consuming. Often these introduce artifacts to the image. Once compressed the image is transmitted702for example, over the Internet or other communications network before being uncompressed at a DE-CODEC703. The resulting high bit-depth image704typically contains artifacts.

In an embodiment a high bit depth image700is tone mapped to a lower bit depth (such as 8 bits per pixel (bpp))705using any of the embodiments described herein. The tone-mapped image is then compressed706using any suitable known 8-bpp image compression algorithm. This results in a higher quality compressed image because the 8 bpp range is utilized optimally thanks to the non-linear tone-mapping. Furthermore, existing 8-bpp compression algorithms are considerably faster than corresponding 16-bpp ones. The compressed image is transmitted702and decompressed707using the known compression process and may then be displayed708. For example, in the case of a medical image, a doctor may receive an email with the attached compressed image, decompress that image and display it at his or her PC. The doctor is then able to visualize the tone-mapped image with the same quality as the tone-mapped image before sending. It is also possible to reverse the tone-mapping process709to obtain a high-dynamic range image back. In this case, the transmission comprises both the compressed image and the parameters of the multi-modal model. These are used to reverse the tone-mapping process709and provide the high bit-depth image without artifacts as would have previously been introduced. That high bit-depth image may then be used for further processing or other purposes as required.

FIG. 8is a schematic diagram of an image mapping apparatus805comprising a tone mapping processor801arranged to use a multi-modal model such as those described herein to map a source image800of high bit-depth to a display device802of lower bit-depth. The display device may be a printer, display screen or other display apparatus. The image mapping apparatus805also comprises an image statistics processor803which determined statistics describing the source image800for use in creating a multi-modal model. The image statistics processor is arranged to fit a multi-modal model to the image statistics and to store the resulting model804in memory.

FIG. 9illustrates various components of an exemplary computing-based device900which may be implemented as any form of a computing and/or electronic device, and in which embodiments of an image mapping system may be implemented.

The computing-based device900comprises one or more inputs906which are of any suitable type for receiving media content, Internet Protocol (IP) input, high dynamic range images, high bit-depth images and other input. The device also comprises communication interface907for example to connect the device to the Internet or other communications network for transmission of images and/or other data.

Computing-based device900also comprises one or more processors901which may be microprocessors, controllers or any other suitable type of processors for processing computing executable instructions to control the operation of the device in order to map high bit-depth images to lower bit-depth images whilst retaining dynamic range information. Platform software comprising an operating system904or any other suitable platform software may be provided at the computing-based device to enable application software903to be executed on the device.

The computer executable instructions may be provided using any computer-readable media, such as memory902. The memory is of any suitable type such as random access memory (RAM), a disk storage device of any type such as a magnetic or optical storage device, a hard disk drive, or a CD, DVD or other disc drive. Flash memory, EPROM or EEPROM may also be used.

An output is also provided such as an audio and/or video output to a display system integral with or in communication with the computing-based device. A display interface905may provide a graphical user interface, or other user interface of any suitable type although this is not essential.

The methods described herein may be performed by software in machine readable form on a tangible storage medium. The software can be suitable for execution on a parallel processor or a serial processor such that the method steps may be carried out in any suitable order, or substantially simultaneously.