Patent Description:
Dynamic Range in images is defined as the ratio between the maximum and minimum measurable light intensities (white and black, respectively). The human visual system has a very broad dynamic range. Humans can perceive details in very bright and very dark scenes. However, cameras struggle to capture such richness of details due to the nature of their photoreceptors.

The sensor of a camera is composed by photosites, which act as light cavities which produce an electric signal when photons "fall" into them, as illustrated in <FIG>. Due to the inherent noise level in the signal, there is a minimum amount of photons each cavity can detect, and thus this will set the "minimum" light measurable by the camera. <FIG> illustrates the minimum measurable light due to the sensor noise level. At the other end of the luminosity, when there are too many electrons falling into the photosites, they can overflow and saturate, setting effectively the maximum measurable light, i.e. white colour. <FIG> illustrates the maximum measurable light due to an overflowing photosite. All of the luminosity present in the scene outside of this range, would be under- or over-exposed and thus parts of the scene luminosity would be lost.

If the luminosity of the scene is higher, several frames with different exposure values can help reveal all the details of the scene, as illustrated by <FIG>. In <FIG>, bright details around the windows and doors are revealed, but the back of the room is under-exposed. In <FIG>, the region around the windows and doors is over-exposed, but details in the back of the room are well exposed.

The current dynamic range of cameras is far from being able to capture all the details present in high contrast scenes. For that reason, and given a static camera and scene, several frames are captured with different exposure values, i.e. Automatic Exposure Bracketing, and merged via a weighted fusion. This enables the obtained image to retain details present in all input images thus effectively increasing the dynamic range, which has been named a HDR image.

Prior approaches can generally be separated into two broad areas: multi-frame fusion, where multiple images exposed differently are combined through a linear weighted combination or a neural network, and single-image neural networks.

In multi-frame approaches, through bracketed exposure images, different ranges can be captured and fused together, leveraging a lot of information of the scene luminosity beyond the dynamic range of a single capture. However, multi-frame HDR poses several challenges. It requires accurate alignment and pixel registration between different frames, which is very difficult, particularly for scenes with high motion.

Additionally, for certain types of photography, shooting several frames might not be possible, as the element of interest to be captured might only be available for a short span of time (for example, in wildlife photography, or sports).

In contrast, single-image HDR processing does not require any alignment between frames and thus is effective in reconstructing images with any amount of motion. However, the single-image HDR methods rely on hallucinating details that are under- or over-exposed in the low-dynamic-range input image. Thus, scenes that have a very broad luminosity range are not very well reconstructed, and such techniques are mostly suited to reconstructing HDR images whose luminosity range is only moderately higher than that of a single image.

It is desirable to develop a method that overcomes these problems.

'<NPL>' discusses a framework for jointly enhancing the resolution and the dynamic range of an image, i.e., simultaneous super-resolution (SR) and high dynamic range imaging (HDRI), based on a convolutional neural network (CNN). <CIT> discloses techniques for convolutional neural network-based multi-exposure fusion of multiple image frames and for deblurring multiple image frames.

Embodiments not covered by the claims are illustrative.

According to one aspect there is provided an image processing apparatus for forming an enhanced image as in claim <NUM>, the apparatus comprising one or more processors configured to: receive one or more input images; form from each of the one or more input images a respective feature representation, each feature representation representing features of the respective input image; and subject the one or more feature representations to a symmetric pooling operation to form an enhanced image from at least some of the features of the one or more feature representations identified by the symmetric pooling operation.

The apparatus may process any arbitrary number of input frames, process frames in any order, and process frames with any exposure value. Content which is over-or under-exposed can be hallucinated when not available in any of the input frames. The apparatus and method can handle frames from different camera sensors/ISPs and handle different levels of noise per frame. The apparatus may result in images with increased photoreceptive dynamic range (i.e. capacity to capture shadows and highlights within the same scene), increased bit depth and signal-to-noise ratio, with less quantization error and richer colour representation.

The symmetric pooling operation may be such as to subject the or each feature representation to a common processing logic so as to identify the components of the respective feature representation. This may allow the components of each feature representation to be identified and used to form the enhanced image.

The symmetric pooling operation is such as to operate independently of an order in which feature representations are provided to the symmetric pooling operation. The apparatus may therefore process input frames in any order and be permutation invariant.

The symmetric pooling operation may comprise a max pooling operation and/or an average pooling operation. These may be convenient implementations for performing the symmetric pooling operation.

The one or more processors may be configured to: subject each input image or data derived from it to a first convolution operation to form the respective feature representation(s); and form the enhanced image by subjecting an output of the symmetric pooling operation to a second convolution operation. A number of convolutions may therefore be applied to the input image to form the feature representation(s), followed by a symmetric pooling operation that creates a "single feature representation" of all the input images (or single image) and apply one or more convolutions to obtain the estimated HDR image.

The enhanced image has a higher dynamic range than each of the one or more input images. This may allow a HDR image to be formed from one or more LDR input images.

The one or more processors may be configured to: subject an output of the symmetric pooling operation and the respective feature representation to a concatenation operation to form a respective intermediate feature representation. The second convolution operation may be performed on the intermediate feature representations. In the preferred implementation, this may be a single convolution after and a single convolution before the symmetric pooling operation. However, any number of convolutions before and after the symmetric pooling operation may be used.

The one or more processors may be configured to form the or each feature representation by performing one or more of the following refinement tasks: denoising, dequantization, linearisation, camera response calibration and over and under-exposure correction on the respective input image. The refinement tasks may be intermediate refinement tasks. This may allow the deep learning model to comprise multiple neural networks that act sequentially to perform dynamic range enhancement of low dynamic range images, in both the single image and multi-image settings.

The symmetric pooling operation is performed in conjunction with a trained neural network. For example, the convolutions around the symmetric pooling blocks may be learnable and may be implemented as a neural network.

The image processing apparatus may comprise a camera (or other imaging device). The, or each, processor may be configured to receive the input images from the camera. The camera or other imaging device may be configured to capture the input images at different times and with different exposure settings. This may allow the apparatus to process a series of LDR images captured at different exposures.

The at least one input image may be subject to exposure normalization. This may result in improved image quality.

The apparatus may be configured to receive a single input image. Alternatively, the one or more processors may be configured to: receive multiple input images; form from each of the multiple input images a respective feature representation, each feature representation representing features of the respective input image; and subject the feature representations to a symmetric pooling operation to form an enhanced image by fusing at least some of the features of the one or more feature representations identified by the symmetric pooling operation. Thus, the apparatus may process one or multiple LDR images to form the enhanced image.

The enhanced image is an estimated high-dynamic-range image and the one or more input images are of lower dynamic range than the high-dynamic-range image. This may allow a HDR image to be produced from one or more LDR images.

According to a second aspect there is provided a computer-implemented method for forming an enhanced image as in claim <NUM>, the method comprising: receiving one or more input images; forming from each of the one or more input images a respective feature representation, each feature representation representing features of the respective input image; and subjecting the one or more feature representations to a symmetric pooling operation to form an enhanced image from of at least some of the features of the feature representations identified by the symmetric pooling operation.

This method may process any arbitrary number of input frames, process frames in any order, and process frames with any exposure value. Content which is over-or under-exposed can be hallucinated when not available in any of the input frames. The method can handle frames from different camera sensors/ISPs and handle different levels of noise per frame.

The step of receiving may comprise receiving multiple input images and the fields of view of each input image are the same. The input images may therefore be of identical fields of view (for example, derived from a common photo).

The step of receiving may comprise receiving multiple input images and the fields of view of at least two of the input images are different. For example, the input images may be sequentially captured photos.

Embodiments of the present invention relate to high-dynamic-range image estimation from one or multiple low-dynamic-range images.

The present invention solves a problem of obtaining a HDR image from either a single LDR image or a set of them (i.e. multiframe) by means of a permutation invariant modular deep learning model that can, in some embodiments. jointly denoise, dequantize and reconstruct under- and over-exposed regions.

In summary, the image processing apparatus forms an enhanced image, which is an estimated HDR image. The apparatus is configured to receive one or more input images. From each of the one or more input images, the apparatus is configured to form a respective feature representation, each feature representation representing features of the respective input image. The apparatus is configured to subject the one or more feature representations to a symmetric pooling operation to form the enhanced image from at least some of the features of the one or more feature representations identified by the symmetric pooling operation.

The symmetric pooling operation is such as to subject the feature representation(s) to a common processing logic so as to identify the components of the respective feature representation. A feature representation is formed by applying a transformation to the input image. The feature representation outlines characteristics of the input image that are useful for the final task. In the present case, the feature representations identify characteristics that help the HDR task and are compatible with the symmeric pooling operation.

The apparatus and method described herein can handle any number of input frames (single or multiple, with any configuration of exposure values and input in any order), and can handle both fusion of image details as well as hallucination of lacking details if some regions of the input image or images are not available.

The apparatus achieves this flexibility by a permutation invariant-aware design that takes advantage of using symmetric operations (such as max pooling) across different copies of the neural network. Therefore, the symmetric pooling operation is such as to operate independently of an order in which feature representations are provided to the symmetric pooling operation.

The algorithm used by apparatus can adapt any deep single image HDR model to a deep permutation invariant single and multi-image HDR model through inclusion of specific intermediate and global symmetric pooling operations across multiple identical copies of the model. This may allow for the use of an arbitrary number of inputs in any permutation with any exposure value.

The deep learning model can, in some embodiments which will be described herein, jointly perform denoising or joint denoising and demosaicing (JDD), dequantization, camera response calibration, over- and under-saturation correction as sequential building blocks for dynamic range enhancement. The method allows for use of images from different camera sensors/ISPs and can handle different levels of noise per frame (i.e. each input image may have a different level of noise).

Embodiments of the present invention can provide a training strategy where subtasks are trained separately in a way such that no fully annotated data is needed. This may make it easier to collate training data.

The network takes as input any number of low dynamic range images, from multiple different sensors and ISPs if necessary, in any permutation. Where the network receives multiple input images, the inputs images can have different noise levels and different exposure values. In summary, the input is an arbitrary number of LDR images and the output is an estimated HDR image.

The symmetric pooling operations is a means to provide a scalable fusion mechanism that will work with any number of images. The symmetric pooling resorts to a single image model when only feeding one image as input and provides an effective mechanism for sharing information via pooling when multiple frames are available.

The symmetric pooling operations are defined in such a way that if applied to a single input they do not modify it: the output will be the same as the input (i.e. identity transformation). A max pooling of a single image feature representation will be that same image feature representation.

In the multi-image case, the apparatus is configured to receive multiple input images, form from each of the multiple input images a respective feature representation, and subject the feature representations to a symmetric pooling operation to form an enhanced image by fusing at least some of the features of the one or more feature representations identified by the symmetric pooling operation. The fused features may correspond to the same features of a respective image. Therefore, one or more features of each of the feature representations may at least partly (in some cases entirely) refer to the same image characteristic.

For the single image case, the input is Im_1. The feature extractor is applied to the image, i.e. F(Im_1). The set of features are passed through the max pooling operation max(F(lm_1)). The max pooling of a single image is just the identity, so max(Im_1)=Im_1 and thus the model will work without having to change the architecture.

This global max pooling can happen in any of the intermediate tasks as described later in the examples (for example, denoising, dequantization, over- and under-exposure correction). Preferably, the global max pooling is applied towards the end of the pipeline, for example in the over- and under-exposure correction stages.

For permutation invariance, the global symmetric pooling preferably happens at least once, and after the global symmetric pooling all of the heads are then collapsed into a single one. In contrast, local symmetric pooling can happen as often as desired and does not collapse the heads into one single head.

The mechanism of the local symmetric pooling operation is illustrated in <FIG> for two input images <NUM> and <NUM>.

Each input image <NUM>, <NUM> is subjected to a first convolution operation <NUM>, <NUM> to form the respective feature representations. The symmetric pooling operation, which in this example is a max pooling operation, is shown at <NUM>. The outputs are concatenated at <NUM>, <NUM> and the enhanced image is formed by subjecting the output to a second convolution operation <NUM>, <NUM>. The concatenation operation <NUM>, <NUM> forms a respective intermediate feature representation and the second convolution operation <NUM>, <NUM> is performed on the intermediate feature representations.

<FIG> show the approach with only a global symmetric pooling operation. <FIG> shows the single image backbone with input image <NUM> input to network <NUM> with skip connections, resulting in output <NUM>. In <FIG>, multiple input images <NUM>, <NUM>, <NUM> are each input to such a network. The multiple encoders have shared weights, one for each input image. The symmetric pooling layers (in this example, max pooling) are applied across the multiple copies of the network on the resulting feature representations, as shown at <NUM>. The enhanced output image is shown at <NUM>.

<FIG> shows the approach using local and global symmetric pooling across the encoders <NUM>. Each input image <NUM>, <NUM>, <NUM> is subjected to a first convolution operation to form the respective feature representations. The outputs of the local symmetric pooling operation are concatenated and the HDR image is formed by subjecting the output to a second convolution operation. The concatenation operation forms a respective intermediate feature representation and the second convolution operation is performed on the intermediate feature representations. Local symmetric pooling enables more channels of information sharing across different copies of the network or heads, while each head retains distinctive features thanks to the concatenation operation. The HDR output image is shown at <NUM>.

<FIG> illustrates an exemplary summary of the stages of the single and multi-image HDR deep learning model. In <FIG>, the input images are shown at <NUM>, <NUM> and <NUM>. In this example, the deep learning model comprises four neural networks that act sequentially to perform dynamic range enhancement of low dynamic range images, in both the single image and multi-image settings. In <FIG>, the stages of the architecture are denoising/JDD and dequantization <NUM>, linearization <NUM>, over exposure correction <NUM> and refinement <NUM>. The resulting estimated HDR image is shown at <NUM>.

The denoising architecture is illustrated in <FIG>. This network jointly performs denoising/JDD and dequantization. It has a U-Net structure, with skip connections between the encoder and decoder, indicated by the arrows. The noisy input image is shown at <NUM>. The feature representations formed from the input images are put through convolutional layers <NUM>. Symmetric pooling layers are applied across multiple copies of the network with shared weights, one for each input image. The downsampling and upsampling are shown at <NUM> and <NUM> respectively. The upsampled feature maps are then put through convolutional layers <NUM> to give the denoised output at <NUM>.

The network architecture shown in <FIG> performs linearization by predicting the weights needed to generate an inverse camera response function, which is applied to the nonlinear input image <NUM>. The network has a ResNet structure comprising residual blocks. The input <NUM>, along with sobel filter features <NUM> and histogram maps <NUM>, is put through convolutional layers and residual blocks, as shown at <NUM>. Pooling is performed at <NUM>, an inverse Camera Response Function (CRF) step at <NUM>, and CRF at <NUM> to give the linearized output at <NUM>.

The network architecture shown in <FIG> performs exposure correction by hallucination of details in the over-exposed regions of the image, and also fusion of details from multiple sources when there are multiple input images. It has a U-net structure with skip connections between the encoder and decoder, as shown by the arrows. The over-exposed input is shown at <NUM>. The input is downsampled as shown at <NUM>, before going through a convolutional layer at <NUM>. At <NUM>, upsampling is performed with symmetric pooling layers applied across multiple copies of the network with shared weights, one for each input image. After going through a convolutional layer at <NUM> and concatenation at <NUM>, the corrected output is shown at <NUM>.

The network architecture shown in <FIG> performs refinement of the image to improve visual quality. It has a U-Net structure, with skip connections between the encoder and decoder. This network takes the denoised output <NUM>, the exposure corrected output <NUM> and the linearized output <NUM> and puts them through convolutional layers <NUM> before downsampling <NUM>, upsampling <NUM> and a further convolutional layer <NUM> before concatenating at <NUM> to give the refined output <NUM>.

In the preferred implementation, as shown in <FIG>, the deep learning model comprises the four networks described above with reference to <FIG> in sequence. For simplification, in <FIG>, a single copy of each model is shown, although multiple copies of the same model, with shared weights, are preferably used for denoising and exposure correction, as shown in <FIG>.

To train the network, each subtask is first trained separately. To train the denoising/JDD network, the mean of the squared difference between pixels in the predicted image DPred and ground truth image DGT is minimized. If N is the number of pixels in the image, the loss function is: <MAT>.

To train the linearization network, the mean of the squared difference between pixels in the predicted image LPred and ground truth image LGT is minimized. The squared difference between the predicted inverse CRF CPred and ground truth inverse CRF CGT is also minimized. If N is the number of pixels in the image and M is the number of elements in the predicted inverse CRF, then the loss is given by: <MAT>.

To train the exposure correction network, the predicted image HPred is tonemapped using the µ-Law tonemapping function with y = <NUM> to obtain HPred_tonemapped. The ground truth image HGT is also tonemapped in the same way to obtain HGT_tonemapped. A mask of the overexposed regions of the input image Hmask_overexposed and also a mask of non-overexposed regions <NUM> - Hmaskoverexposed are computed. These masks are applied to the tonemapped prediction and ground truth images to get HPred_overexposed, HPred_normal, HGT_overexposed, HGT_normal.

The loss function is then given by the pixel-wise reconstruction loss (L1 loss) on the masked images as follows: <MAT>.

After training the three subtasks separately, the three networks can be jointly trained together to improve performance: <MAT>.

To train the refinement network, this network is jointly finetuned with the other three subnetworks. The pixelwise reconstruction loss (L1 loss) between the tonemapped predicted image RPred_tonemapped and the tonemapped ground truth RGT_tonemapped is minimized: <MAT>.

For a multi-image HDR dataset, the single image HDR reconstruction dataset can be adapted to train the model. A series of transformations are applied to the ground truth HDR images to generate LDR-HDR pairs for training.

In one example, the longer side of the HDR image is resized to size <NUM> and the shorter side to size <NUM>. This image is then split into two 512x512 ground truth HDR images. This set of images is the ground truth for the exposure correction and refinement networks. A burst size, b, between <NUM> and n is then randomly chosen. b exposures in the range [-<NUM>,<NUM>] are randomly sampled and these exposures are applied to the ground truth image to get b exposed images. The dynamic range of these images is clipped to be between <NUM> and <NUM>. This is the ground truth of the linearization network. CRFs are randomly sampled from the Database of Response Function (DoRF) and applied to the clipped images. This is the ground truth of the denoising network. Noise is added with two components, a fixed element and a signal-dependent element. Using the noise model, the variance of the ground truth is approximated by using the noisy image instead and this estimated variance is concatenated to each image. All images within a burst are aligned by exposure value.

Therefore, embodiments of the present invention may provide apparatus configured to implement an algorithm to adapt any single image HDR model to become a permutation invariant single and multi-image HDR model through the inclusion of specific intermediate and global symmetric pooling operations (for example average or max pooling) as described. The deep learning model described herein may jointly perform denoising/JDD, dequantization, camera response calibration, over- and under-saturation correction as sequential building blocks for both single and multiframe dynamic range enhancement. Embodiments of the present invention may provide a training strategy whereby subtasks are trained separately in such a way that no fully annotated data is required.

<FIG> shows an example of a computer-implemented image processing method. At step <NUM>, the method comprises receiving one or more input images. At step <NUM>, the method comprises forming from each of the one or more input images a respective feature representation, each feature representation representing features of the respective input image. At step <NUM>, the method comprises subjecting the one or more feature representations to a symmetric pooling operation to form an enhanced image from of at least some of the features of the feature representations identified by the symmetric pooling operation.

The apparatus may comprise an imaging device, such as a camera. Where multiple images are used as input, the apparatus may be configured to, in response to an input from a user of the apparatus, cause the camera, or other imaging device, to capture each of the input images with a different exposure from the other input images. The captured images may also have different noise levels.

<FIG> shows an example of apparatus <NUM> comprising a imaging device <NUM>, for example a camera, configured to use the method describe herein to process image data captured by at least one image sensor in the device. The device <NUM> comprises image sensors <NUM>, <NUM>. Such a device <NUM> typically includes some onboard processing capability. This could be provided by the processor <NUM>. The processor <NUM> could also be used for the essential functions of the device.

The transceiver <NUM> is capable of communicating over a network with other entities <NUM>, <NUM>. Those entities may be physically remote from the device <NUM>. The network may be a publicly accessible network such as the internet. The entities <NUM>, <NUM> may be based in the cloud. Entity <NUM> is a computing entity. Entity <NUM> is a command and control entity. These entities are logical entities. In practice they may each be provided by one or more physical devices such as servers and data stores, and the functions of two or more of the entities may be provided by a single physical device. Each physical device implementing an entity comprises a processor and a memory. The devices may also comprise a transceiver for transmitting and receiving data to and from the transceiver <NUM> of device <NUM>. The memory stores in a non-transient way code that is executable by the processor to implement the respective entity in the manner described herein.

The command and control entity <NUM> may train the model(s) used in the device. This is typically a computationally intensive task, even though the resulting model(s) may be efficiently described, so it may be efficient for the development of the algorithm to be performed in the cloud, where it can be anticipated that significant energy and computing resource is available. It can be anticipated that this is more efficient than forming such a model at a typical imaging device.

In one implementation, once the algorithms have been developed in the cloud, the command and control entity can automatically form a corresponding model and cause it to be transmitted to the relevant imaging device. In this example, the model is implemented at the device <NUM> by processor <NUM>.

In another possible implementation, an image may be captured by one or both of the sensors <NUM>, <NUM> and the image data may be sent by the transceiver <NUM> to the cloud for processing. The resulting image could then be sent back to the device <NUM>, as shown at <NUM> in <FIG>.

Therefore, the method may be deployed in multiple ways, for example in the cloud, on the device, or alternatively in dedicated hardware. As indicated above, the cloud facility could perform training to develop new algorithms or refine existing ones. Depending on the compute capability near to the data corpus, the training could either be undertaken close to the source data, or could be undertaken in the cloud, e.g. using an inference engine. The method may also be implemented at the device, in a dedicated piece of hardware, or in the cloud.

<FIG> show examples of results produced using the method described herein (using the Single Image HDR test set). <FIG> shows the result using a single image and an example of the result for multiple images (two images in this example) is shown in <FIG>. An improvement in image quality compared to the input(s) is observed in both cases.

The apparatus and method described herein may process any arbitrary number of input frames, process frames in any order, and process frames with any exposure value. Content which is over-or under-exposed can be hallucinated when not available in any of the input frames. The apparatus and method can handle frames from different camera sensors/ISPs and handle different levels of noise per frame. The deep learning model may train without data that is fully annotated (only pairs of tasks are needed).

This deep learning pipeline retains the benefits of both single- and multi-frame HDR models.

The apparatus and method described herein may result in images with increased photoreceptive dynamic range (i.e. capacity to capture shadows and highlights within the same scene), increased bit depth and signal-to-noise ratio, with less quantization error and richer colour representation (for example, <NUM> bit per channel RGB images.

Claim 1:
An image processing apparatus (<NUM>) for forming an enhanced image, the apparatus comprising one or more processors (<NUM>) configured to:
receive (<NUM>) one or more input images (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>);
form (<NUM>) from each of the one or more input images a respective feature representation, each feature representation representing features of the respective input image; and
subject (<NUM>) the one or more feature representations to a symmetric pooling operation (<NUM>) to form an enhanced image (<NUM>, <NUM>, <NUM>) from at least some of the features of the one or more feature representations identified by the symmetric pooling operation;
wherein the symmetric pooling operation is performed across different copies of a trained neural network;
wherein the enhanced image is an estimated high-dynamic-range image and the one or more input images are of lower dynamic range than the high-dynamic range image; and
wherein the symmetric pooling operation is such as to operate independently of an order in which feature representations are provided to the symmetric pooling operation.