Patent Description:
High-dynamic-range imaging (HDRI) is a high dynamic range (HDR) technique used in imaging and photography to reproduce a greater dynamic range of luminosity than that possible with standard digital imaging or photographic techniques. The aim is to present a similar range of intensity to what is experienced through the human visual system.

The human eye, through adaptation of the iris and other methods, adjusts constantly to adapt to a broad range of intensities present in the environment. The brain continuously interprets this information so that a viewer can see in a wide range of light conditions.

Due to the limitations of printing and display contrast, the extended luminosity range of an HDR image has to be compressed to improve its visibility on Low Dynamic Range (LDR). The method of rendering an HDR image to a standard monitor or printing device is called tone mapping. This method reduces the overall contrast of an HDR image to facilitate display on devices or printouts with lower dynamic range and can be applied to produce images with preserved local contrast or exaggerated for artistic effect.

In prior art, document <CIT> discloses an image tone adjustment to a video stream using local tone curve computation, wherein the images of the video stream are divided into multiple logical blocks to compute the local tone curves and wherein the local tone curves computed for one image in the video stream are applied to adjust the luminance range of a later image.

The invention is defined in the appended claims and addresses all or some of the drawbacks of known tone mapping processes.

One embodiment provides a tone mapping method for a succession of images implemented by an image processing device, the method including: a) the division of the images of the succession of images in a plurality of sub-blocks of first pixels; b) for a first image of the succession of images, the creation of a first mini-image comprising pixels of the first mini-image, each pixel of the first mini-image representing a corresponding sub-block of the first image, the intensity of each pixel of the first mini-image being representative of the intensity of the first pixels of the corresponding sub-block; c) the storage of the first mini-image in a memory; d) for a second image of the succession of images, the modification of the second image according to the first mini-image in order to generate an output image.

According to an embodiment, the operation d) comprises for each first pixel of the second image: the computing of a context value, the context value depending on: the spatial proximity between the first pixel and the centers of the sub-blocks adjacent to the sub-block comprising the first pixel; and the affinity in intensity between the first pixel and the pixels of the first mini-image of the second image; the modification of the intensity of each first pixel according to the context value in order to generate the value of a pixel of the output image.

According to an embodiment, each pixel OUTPUT_Pq of the output image is equal to OUTPUT_Pq = GMPq * Pq, where Pq represents the value of any of the channel pf the pixel, OUTPUT_Pq represents the value of the corresponding channel in the output image, and the value GMPq is a gain value dependent on the context value.

According to an embodiment, the gain value GMPq is equal to GMPq = (LOCAL_GAINq - <NUM>)*GLOBAL_POWER + <NUM>, where GLOBAL_POWER is a value representing the power, or influence, of the modification by the process on the whole image and the value LOCAL_GAINq is the local contrast gain, computed based on the image LOGY_IMAGEf and the mini-image MPICf-<NUM>.

According to an embodiment, the method further comprises storing the output image in the memory or another memory or displaying the output image.

According to an embodiment, each sub-block comprises at least two first pixels.

According to an embodiment, in operation b), the intensity of each pixel of the first mini-image is computed based on an average of the intensities of the first pixels of the corresponding sub-block.

According to an embodiment, in operation b), the intensity of each pixel of the first mini-image is computed based on: the average of the intensities of the first pixels in a square centered on the corresponding sub-block, the size of the square being equal to k times the size of the sub-block, k being a number between <NUM> and <NUM>, not including <NUM>; or the average of the intensities of the first pixels in a circle centered on the corresponding sub-block, the radium of the circle being equal to k times the size of the sub-block, k being a number between <NUM> and <NUM>, not including <NUM>.

According to an embodiment, the first mini-image used for the generation of the output image is erased from the memory after the generation of the output image.

According to an embodiment, operation b) is applied to all the images of the succession of images.

According to an embodiment, the modification of the first pixels is independent of the context value if: the intensity of the pixel is above a first threshold, or below a second threshold; and/or the first pixel is a color pixel and the intensity of one of the color of the first pixel is above a third threshold; and/or the accuracy of the context value is below a fourth threshold.

According to an embodiment, the intensity of the pixels of the first mini-image is computed, in operation b), based on the logarithmic value of the intensity of the first pixels of the corresponding sub-block.

Another embodiment provides a computer-readable non-transitory storage medium storing instructions that, when executed by a processing device, cause the previous method to be implemented.

Another embodiment provides an image processing device for tone mapping a succession of images, each image being divided into a plurality of sub-blocks of first pixels, comprising: a circuit configured to: for a first image, create a first mini-image comprising pixels of the first mini-image, each pixel of the first mini-image representing a corresponding sub-block of the first image, the intensity of each pixel of the first mini-image being representative of the intensity of the first pixels of the corresponding sub-block; and for a second image of the succession of images, the modification of the second image according to the first mini-image in order to generate an output image, at least one memory configured to store the second image created by the circuit.

According to an embodiment, the circuit is configured to: compute a context value, the context value depending on: the distances between the first pixel and the centers of the sub-blocks adjacent to the sub-block comprising the first pixel; and the differences in intensity between the pixel of the first mini-image corresponding to the first pixel and the adjacent pixels of the first mini-image of the second image created during the reading of the first image, and modify the intensity of each first pixel according to the context value in order to generate the intensity of a pixel of the output image.

For the sake of clarity, only the operations and elements that are useful for an understanding of the embodiments described herein have been illustrated and described in detail.

Unless indicated otherwise, when reference is made to two elements connected together, this signifies a direct connection without any intermediate elements other than conductors, and when reference is made to two elements linked or coupled together, this signifies that these two elements can be connected or they can be linked or coupled via one or more other elements.

Unless specified otherwise, the expressions "around", "approximately", "substantially" and "in the order of" signify within <NUM>%, and preferably within <NUM>%.

For ease of reading, the following table summarizes the various references and variables used, in particular in the equations.

<FIG> is a flow diagram illustrating an embodiment of a method, or process, of tone mapping. More specifically, <FIG> illustrates operations of a process of local tone mapping, meaning a process in which each pixel of an image can be modified depending on the pixels adjacent to or surrounding said pixel. Local tone mapping is to be distinguished from global tone mapping, in which the modification of each pixel is defined only by global statistics from the whole image.

The embodiment of <FIG> is implemented by a circuit of an electronic device, such as an image processing device. An example of a circuit implementing the process is described below with reference to <FIG>. This process is, for example, part of a larger process of image processing implemented in the electronic device. The larger process can for example include processes of global tone mapping and other processes of local tone mapping. The larger process can also include operations of pre- and post-processing, which are for example standard processes and will not be described in detail herein. Alternatively, the process can be implemented within a sensor.

The electronic device for example comprises an image sensor configured to capture a sequence of successive images, for example the electronic device is an image sensor capable of capturing video. The image sensor is for example in a video camera.

The process is used on a group of successive images. The group of images comprises at least two successive images. Each couple of two successive images is comprised of images of substantially the same scene at two different instants. Preferably, the two different instants are two close instants, for example less than <NUM> apart. For example, several of the successive images are taken every second. Preferably, the group of images corresponds to the various frames of a video. The frames are for example less than <NUM> apart, for example <NUM> or <NUM> apart.

The process can be used with greyscale images, color images, or hyperspectral images. In the case of color images or hyperspectral images, the process is applied to an intensity image computed from the color or hyperspectral image. The intensity is a weighted combination of the various channels of the image. The result of the process is then applied to the initial color or hyperspectral image. Alternatively, instead of the intensity image, the process can be applied to an image computed by extracting the information of one of the channels of the color or hyperspectral image.

The process of <FIG> is used on a plurality of successive input images INPUT_IMAGEf, where f is an integer going from <NUM> to n-<NUM>, and where n is the total number of successive images. The integer f corresponds to the position of the image in the succession of images. Thus, the image INPUT_IMAGEf directly precedes the image INPUT_IMAGEf+<NUM>. For example, the image INPUT_IMAGE<NUM> precedes the image INPUT_IMAGE<NUM>.

Each input image INPUT_IMAGEf comprises an array of pixels Pq, the value q being the number of the pixel in the order in which the pixels are read out from an image sensor. For example, the pixels are read in each row from left to right, and from the top row to the bottom row. Thus, the value q is equal to: q=COL*y+x, COL being the number of columns in the array, y being the index of row of the pixel Pq and x being the index of column of the pixel Pq in the array. The index x goes from <NUM> to COL-<NUM>. The index y goes from <NUM> to ROW-<NUM>, ROW being the number of rows in the array.

The images INPUT_IMAGEf are for example images created by a Bayer sensor. The images are for example usual camera images, preferably high dynamic range images (HDR images), having a bit depth for example superior or equal to <NUM>, preferably greater than <NUM>, for example superior or equal to <NUM>, for example superior or equal to <NUM>, for example superior or equal to <NUM>. In other words, the intensity Yq of each pixel Pq is coded on at least <NUM> bits, for example on at least <NUM> bits, for example on at least <NUM> bits, for example on at least <NUM> bit. Thus, the intensity of each pixel can for example be one of at least <NUM><NUM> values, for example one of at least <NUM><NUM> values, for example one of at least <NUM><NUM> values, for example one of at least <NUM><NUM> values.

The successive images have for example substantially the same dimensions, preferably exactly the same dimensions. Preferably, the dimensions of the images are such that each image can be divided in an array of squares.

The process is described once for an image INPUT_IMAGEf, f being a number between <NUM> and n. At the start of the process, the pixel of the image INPUT_IMAGEf starts being read, and therefore the circuit implementing the process starts to receive the values of the pixels Pq of the image INPUT_IMAGEf (block <NUM>, START INPUT_IMAGEf). The pixels Pq have for example already been through other processes, for example through a preprocessing.

The circuit implementing the image processing receives successively the intensity Yq corresponding to each pixel Pq of the image, pixel by pixel, in an order corresponding to the incrementation of the number q. In other words, the pixel Pq is read before the pixel Pq+<NUM>. The brightness intensity Yq of the pixel Pq is computed for each pixel is read. The brightness intensity Yq of the pixel Pq is for example computed as a linear combination of the available channels of INPUT_IMAGEf. For example, in the case of a greyscale image, the intensity is equal to the value of the pixel. For example, in the case of a color image, the intensity is a linear combination of the values of the red channel, the blue channel and the green channel.

A group <NUM> of steps of the process is performed on each pixel Pq as they are read. Thus, the various steps of the group <NUM> are all carried out on the pixel Pq, then on the pixel Pq+<NUM>, etc. The steps of the group <NUM> are described only once, in regard with a pixel Pq, the steps being reiterated for each pixel of the image INPUT_IMAGEf.

During a step represented by the block <NUM> (APPLY LOG TO INPUT_IMAGEf) of the group <NUM>, a logarithmic function, preferably with base two, is applied to the intensity of the current pixel. This operation results in a value LYq. The value LYq corresponds to the logarithmic value of the intensity of the pixel Pq. The value LYq corresponds to the intensity of a pixel LPq of an image LOGY_IMAGEf. The image LOGY_IMAGEf correspond to the image INPUT_IMAGEf after computation of the intensity and application of the logarithmic function to each pixel. The image LOGY_IMAGEf comprises the exact same number of pixels as the image INPUT_IMAGEf, each pixel Pq corresponding to a pixel LPq.

This logarithmic value is used to create a mini-image MPICf and to generate a gain map GMf, which will be used to obtain an output image. The creation of the mini-image is carried out in parallel with the generation of the gain map.

Using the logarithmic value has the advantage of simplifying the computations. Indeed, operations of multiplication and division applied to the value Yq are equivalent to operations of addition and subtraction on the value LYq, the operations of addition and subtraction being easier and less costly to implement. Furthermore, the use of the logarithmic value increases the stability of the video where highly contrasted object of the screen appear close to each other.

More specifically, the group <NUM> comprises an operation represented by the block <NUM> (CREATE MINI-IMAGE MPICf).

During operation <NUM>, a mini-image MPICf, based on the image LOGY_IMAGEf, is created. The mini-image MPICf comprises an array of pixels PMb, where b is in the range from <NUM> to sb-<NUM>, sb being the number of pixels in the mini-image, the value b being the number of the pixel of the mini-image in the order in which the pixels are processed. For example, the pixels are read in each row from left to right, and from top row to bottom row. The mini-image MPICf is a reduced, or subsampled, image having a number of pixels PMb inferior to the number of pixels of the image LOGY_IMAGEf. Each pixel of the mini-image is representative of the intensity of a plurality of pixels of the corresponding image LOGY_IMAGEf. For example, each pixel of the mini-image represents the average of the plurality of pixels located in the same region of the image LOGY_IMAGEf.

The mini-image is for example a greyscale image. Each pixel PMb of the mini-image MPICf for example corresponds to an intensity LYMb. The value of the intensity LYMb of each pixel of the mini-image is for example representative of the intensity of a sub-block Rb of pixels of the image LOGY_IMAGEf. The mini-image is a subsampled version of the image LOGY_IMAGEf.

The mini-image MPICf is of finite size and far smaller than the image INPUT_IMAGEf. The mini-image MPICf comprises for example at least <NUM> pixels, for example at least <NUM> pixels. The mini-image MPICf comprises for example between <NUM> and <NUM> times less pixels than the image LOGY_IMAGEf. Each pixel of the image LOGY_IMAGEf is associated with at least a pixel of the mini-image MPICf. For example, each pixel of the image LOGY_IMAGEf can be associated with only one pixel of the mini-image MPICf.

In the case that the operations are carried out pixel by pixel, the operation represented by the block <NUM> corresponds, for each pixel, to the updating of the mini-image. This step will be described in more detail in relation with <FIG> and <FIG>.

The step represented by the block <NUM> is followed by an operation represented by a block <NUM> (STORE MPICf), in which the mini-image MPICf is stored in a memory. In other words, the value(s) of the pixel, or pixels, of the mini-image updated during the operation represented by block <NUM> is/are stored, preferably in the same memory location as of the same pixel before having been updated. Preferably, the values of the pixels of the image INPUT_IMAGEf and of the image LOGY_IMAGEf are not stored.

In parallel with operations <NUM> and <NUM>, the process for example comprises an operation <NUM> in which the gain map is created (block <NUM>, GENERATE GAIN MAP GMf USING MPICf-<NUM>) and an operation <NUM> in which the gain map is applied on the image INPUT_IMAGEf (block <NUM>, APPLY GMf TO INPUT_IMAGEf).

The gain map GMf is an array of gain values GMPq, each gain value GMPq corresponding to a pixel Pq of the image INPUT_IMAGEf. The values of the gain map are generated based on the image LOGY_IMAGEf and on the mini-image MPICf-<NUM>. The generation of the gain map will be described in more detail below.

The operation represented by block <NUM> generates an output image OUTPUT_IMAGEf.

More specifically, during the operation represented by block <NUM>, the gain value GMPq, corresponding to the current pixel Pq, is generated based on the image LOGY_IMAGEf and more specifically, based on the value of the pixel Pq, and the mini-image MPICf-<NUM>. Only one gain value GMPq is generated for each iteration of the operations of the group <NUM>.

During the operation represented by block <NUM>, the value of each pixel Pq of the input image INPUT_IMAGEf is for example multiplied with the corresponding gain value GMPq of the gain map OUTPUT_IMAGEf. The values of the pixels OUTPUT_Pq of the output image are equal to: OUTPUT_Pq = GMPq * Pq, where Pq represents the value of the pixel, and OUTPUT_Pq represents the value of the pixel in the output image. For example, if the image is a color image or a hyperspectral image, the gain value is multiplied with the values of all the channels. The value of the gain GMPq determines the gain or attenuation to be applied on the pixel.

The gain value GMPq is preferably erased after the generation of the output pixel OUTPUT_Pq. Preferably, the values of the gain map are not stored.

Each value of the gain map is preferably computed as follows: <MAT> where GLOBAL_POWER is a value representing the power, or degree of influence, of the modification by the process on the whole image and the value LOCAL_GAINq is the local contrast gain, computed based on the image LOGY_IMAGEf and the mini-image MPICf-<NUM>.

The value GLOBAL_POWER is for example the same for all the pixels of an image and preferably the same for all the images of the sequence of images. The value GLOBAL_POWER is for example chosen by the user or is preprogrammed during the programing of the process. For example, the value GLOBAL_POWER is greater than <NUM>, for example in the range from <NUM> to <NUM>, where the value <NUM> is preferably not included. If the value GLOBAL_POWER was equal to <NUM>, the gains GMPq would all be equal to <NUM> and the output image would be identical to the input image. In other words, if the value GLOBAL_POWER was equal to <NUM>, the method of tone mapping would have no impact on the images.

The value LOCAL_GAINq is preferably computed as follows: <MAT> where:.

The values LOCAL_POWER, CLIP_GAIN, and fbw are for example the same for all the pixels of an image and for example the same for all the images of the sequence of images. The values LOCAL_POWER, CLIP_GAIN, and fbw are for example chosen by the user or are preprogrammed during the programing of the process. For example, the value LOCAL_POWER, is in the range from -<NUM> to <NUM>. If the value LOCAL _POWER is negative, the local contrast will be diminished and if the value LOCAL _POWER is positive, the local contrast will be increased. For example, the value LOCAL_POWER, is equal to <NUM>. For example, the value CLIP_GAIN, is higher than <NUM>, preferably substantially equal to <NUM>. For example, the value fbw is equal to or higher than <NUM>.

The computation of the value CTXq will be described in more detail in relation with <FIG>, <FIG>. The value CTXq is dependent on the spatial proximity and the affinity of the pixel with the other pixels.

As each input image is taken by the sensor at an instant sufficiently close to the instant of the previous image, the mini-images of two successive instants are similar enough that the mini-image MPICf-<NUM> can be used in the operation of tone mapping the following image INPUT_IMAGEf taken at an instant directly after the instant in which the mini-image MPICf-<NUM> was created. The mini-image MPICf-<NUM> is the result of a heavy subsampling of the image LOGY_IMAGEf-<NUM>. The successive images are for example taken quickly one after another. Therefore, changes in the scene usually induce very small, almost unnoticeable, changes in the mini-images from one image to the other.

After the mini-image MPICf-<NUM> is used for the tone mapping of image INPUT_IMAGEf, the mini-image MPICf-<NUM> is preferably erased from the memory. Thus, there are preferably at most two mini-images stored at any one time during the process, one being the mini-image being created and the other being the mini-image being used for the gain map.

Preferably, other processes are applied to the output image OUTPUT_IMAGEf, for example other processes of tone mapping and post processing. The output image OUTPUT_IMAGEf is then for example displayed and/or stored by the device.

In the case that the operations of the group <NUM> are applied to the image INPUT_IMAGEf pixel by pixel, the operation of modification (bloc <NUM>) of the pixels of the image INPUT_IMAGEf is for example applied dynamically to the pixels as they are read.

If the image INPUT_IMAGEf processed is not the last image of the sequence of successive images (branch NO, operation <NUM>, f = n), the value of the index f is incremented by <NUM> (block <NUM>, f = f + <NUM>). The process, starting with operation <NUM> is applied on the new image INPUT_IMAGEf, corresponding to the next image of the succession of images.

In other words, if the integer f is not equal to the integer n, n being the number of successive images INPUT_IMAGEf, the process of tone mapping is applied to the new image INPUT_IMAGEf.

If the image processed was the last image (branch YES, operation <NUM>, f = n), the process is finished (block <NUM>, END).

In order to modify an image INPUT_IMAGEf, a mini-image MPICf-<NUM> generated based on the previous image LOGY_IMAGEf-<NUM> is used. As such, at least one of the first images of the sequence of images is not modified, but is used to generate a mini-image.

For example, the first images are not displayed or stored. For example, one of the first images is used to create a mini-image but is not displayed or used to generate an output image.

The method of <FIG> can be implemented in software, for example by instructions stored on a computer-readable non-transitory storage medium.

<FIG> is a timing diagram illustrating the operation of the embodiment of <FIG> during three cycles of the method, in other words during processing of three successive images. <FIG> represents the image (LOGY_IMAGE) being read (blocks <NUM> and <NUM> of <FIG>), the mini-image (MPIC) being created (block <NUM> of <FIG>) and the gain map (TONE MAPPING) being applied to an image, as a function of time (t).

At an instant t0, an image IMAGEf-<NUM> is captured by the image sensor and the values of the pixels are used to generate the values of the pixels of the image LOGY_IMAGEf-<NUM> (block <NUM> of <FIG>).

Between the instant t0 and an instant t1, the image INPUT_IMAGEf-<NUM> is read and processed and the image LOGY_IMAGEf-<NUM> is generated. In other words, the voltage generated by each pixel is received by the circuit and thus the value of the various pixels is provided to the circuit implementing the method and a logarithmic function is applied.

Between the instant t0 and an instant t1, the mini-image MPICf-<NUM> is also created based on the image LOGY_IMAGEf-<NUM> (block <NUM> of <FIG>). Furthermore, the gain map GMf-<NUM> is generated (block <NUM> of <FIG>), in parallel with the generation of the mini-image MPICf-<NUM>, based on the image LOGY_IMAGEf-<NUM> and the mini-image MPICf-<NUM> (not represented) generated previously.

At the instant t1, an image INPUT_IMAGEf is captured by the image sensor.

Between the instant t1 and an instant t2, the image INPUT_IMAGEf is read and processed and the image LOGY_IMAGEf is generated. In other words, the voltage generated by each pixel is received by the circuit and thus the value of the various pixels are provided to the circuit implementing the method and a logarithmic function is applied.

Between the instant t1 and an instant t2, the mini-image MPICf is also created based on the image LOGY_IMAGEf (block <NUM> of <FIG>). Furthermore, the gain map GMf is generated (block <NUM> of <FIG>), in parallel with the generation of the mini-image MPICf, based on the image LOGY_IMAGEf and the mini-image MPICf-<NUM> generated previously.

At the instant t2, an image INPUT_IMAGEf+<NUM> is captured by the image sensor.

Between the instant t2 and an instant t3, the image INPUT_IMAGEF+<NUM> is read and processed and the image LOGY_IMAGEf+<NUM> is generated. In other words, the voltage generated by each pixel is received by the circuit and thus the value of the various pixels are provided to the circuit implementing the method and a logarithmic function is applied.

Between the instant t2 and an instant t3, the mini-image MPICf+<NUM> is also created based on the image LOGY_IMAGEf+<NUM> (block <NUM> of <FIG>). Furthermore, the gain map GMf+<NUM> is generated (block <NUM> of <FIG>), in parallel with the generation of the mini-image MPICf+<NUM>, based on the image LOGY_IMAGEf+<NUM> and the mini-image MPICf generated previously.

At the instant t3, a following image INPUT_IMAGEf+<NUM> is captured by the image sensor and the process continues.

The operation of creating a mini-image based on an image LOGY_IMAGEf is preferably applied to all the images of the succession of images.

<FIG> is a flow diagram illustrating in more detail the operation <NUM> of the embodiment of <FIG>, and <FIG> represents an image during this operation <NUM>. <FIG> and <FIG> thus illustrate the creation of a mini-image. The value of each pixel of the mini-image is preferably an average intensity of a corresponding region in the image LOGY_IMAGEf. To compute the average intensity, the values of all the pixels of the corresponding region are summed as the pixels go through the process, and the total sum is divided by the size of the corresponding region.

We consider that operation <NUM> is applied to an image LOGY_IMAGEf in order to create a mini-image MPICf, f being an integer in the range from <NUM> to n-<NUM>. More specifically, the operation described is applied to the pixel LPq. The operation described in relation with <FIG> is repeated for each pixel, for example as they are provided to the circuit.

As illustrated in <FIG>, the image LOGY_IMAGEf is divided into an array of sub-blocks Rb, b being the index of the pixel PMb of the mini-image MPICf and the index of the corresponding sub-block Rb of the image LOGY_IMAGEf. The integer l goes from <NUM> to sb-<NUM>. The integer sb corresponds to the number of sub-blocks in the image LOGY_IMAGEf. The integer sb is preferably higher than <NUM>. In <FIG>, the image LOGY_IMAGEf is divided into twenty-five sub-blocks arranged in five rows and five columns. The index b is therefore in the range from <NUM> to <NUM> in the example of <FIG>. More generally, the number of rows can be different from the number of columns, as dictated by the aspect ratio of the image LOGY_IMAGEf.

Each sub-block comprises an array of pixels <NUM> of the image LOGY_IMAGEf. Preferably, all the images are formed of the same number of pixels <NUM>. Alternatively, some of the sub-blocks, preferably at the bottom and right-hand side, can have a different number of pixels, for example comprising less pixels. Preferably, the arrays of pixels <NUM> are square arrays, meaning they comprises the same number of rows as of columns of pixels.

The pixels of the mini-image are in the same order as the corresponding sub-blocks. As such, the pixel corresponding to a sub-block Rb is surrounded by the pixels corresponding to the sub-blocks surrounding the sub-block Rb.

The division is preferably the same for all images LOGY_IMAGEf of the sequence of successive images. All images LOGY_IMAGEf are for example divided into an array of sub-blocks comprising the same number of sub-blocks, having the same number of rows of sub-blocks and the same number of columns of sub-blocks. If the pixels LPq of the image LOGY_IMAGEf correspond to the sub-block Rb of LOGY_IMAGEf, the pixel LPq of the image LOGY_IMAGEf+<NUM> correspond to the sub-block Rb of LOGY_IMAGEf+<NUM>.

The dimensions of the images INPUT_IMAGEf and thus of the images LOGY_IMAGEf depend on the image sensor that is used. The dimensions of the sub-blocks are preferably determined image by image, before the start of processing of each image.

Alternatively, the dimensions of the sub-blocks are for example determined before the start of the application of the method of the first image INPUT_IMAGE<NUM>. When the image INPUT_IMAGE<NUM> is read, pixel by pixel, meaning that the value of each pixel is provided to the circuit implementing the method, each pixel has for example already been associated with a sub-block.

The operation <NUM> described herein updates the value of some of the pixels PMb of the mini-image MPICf based on the values of a group <NUM> of pixels LPq of the image LOGY_IMAGEf. Let Rb' be the group <NUM> corresponding to the pixel PMb.

The value PMb is for example based on, and in some cases equal to, an average of the values LYq of the group <NUM> of pixels of the image LOGY_IMAGEf. Alternatively, the value PMb is a median value of the values LYq of the group <NUM> of pixels, the minimum value of the values LYq of the group <NUM> of pixels, the maximum value of the values LYq of the group <NUM> of pixels or any function of the values LYq of the group <NUM> of pixels.

According to an embodiment, the group <NUM> of pixels includes all the pixels of the sub-block Rb and only the pixels of the sub-block Rb.

According to another embodiment, the group of pixels <NUM> includes all the pixels of the sub-block Rb and some pixels of the adjacent sub-blocks. For example, the group of pixels <NUM> includes all the pixels of the sub-block Rb and pixels of the adjacent sub-blocks forming a ring around the sub-block Rb.

More generally, the group of pixels <NUM> may include all the pixels inside a square region centered on the sub-block.

Let m represent the size of each sub-block. In some embodiments, the sub-blocks are square, and m is the width or length of the sub-block, expressed as a number of pixels. Each sub-block is for example a square of m pixels by m pixels. The group of pixels is for example a square whose size h is given by the formula: h = floor (k * m), where floor represent a function giving the highest integer inferior to the variable of the function, and where k is a number strictly greater than <NUM>. The number k is, preferably, equal to or greater than <NUM>. Indeed, if k is lesser than <NUM>, some pixels of the image LOGY_IMAGEf are not considered in the computation of the mini-image. The number k can be chosen by a user, or can be preprogrammed. For example, the values k, m and h are identical for all images of the sequence.

At the beginning of the computation of the mini-image (<FIG>, block <NUM>, START), in other words, when the first pixel of one of the images INPUT_IMAGEf is provided to the circuit implementing the method, the index q is equal to <NUM>, corresponding to the first pixel. Furthermore, the index <NUM> corresponding to the current pixel PMb is equal to <NUM>, and the value ΣRb', corresponding to the cumulative sum of the values of the pixels of the group Rb', is equal to <NUM>.

During the operation of block <NUM>, the value LYq of the pixel LPq of the group Rb' is added to the value ΣRb'. In other words, the values representative of the pixels of the mini-image are updated if the pixel on which the operation is carried out is part of the group <NUM> corresponding to these pixels of the mini-image. The process is described for a single pixel LPq. The process is reiterated for each pixel during the process of the group <NUM> of <FIG>.

More precisely, the circuit determines (block <NUM>, LPq ∈ Rb'), whether the pixel LPq is comprised in the group Rb' taken into account for the computation of the value of the pixel PMb. If it is not the case (Branch NO of block <NUM>), the index b is incremented (block <NUM>, b=b+<NUM>) and the operation of block <NUM> is repeated until b has a value such that the pixel LPq is comprised in the group Rb'. If it is the case (Branch YES of block <NUM>), the value ΣRb' is updated (block <NUM>, ΣRb' = ΣRb' + LYq). More precisely, the value LYq of the pixel LPq is added to the current value of ΣRb'.

The circuit then determines (block <NUM>, b = sb-<NUM>) if this group Rb' is the last group of the image, and thus if the circuit has determined, for all the groups Rb', whether or not the pixel LPq belongs to the group. If it is not the case (Branch NO, block <NUM>), the value b is incremented (block <NUM>) and the process goes back to the step of block <NUM>. If it is the case (Branch YES, block <NUM>), the operation is finished (block <NUM>, END). The value ΣRb' is for example stored. The remaining operations of the method of <FIG> are then for example carried out.

Alternatively, one or several additional operations can be carried out on the value ΣRb' in order to obtain the value of the pixel PMb. For example, the value ΣRb' can be divided by a value, for example by h<NUM>, in order to generate an average value.

Alternatively, the operation of block <NUM>, corresponding to the updating of a value used to determine the value of the pixel PMb can be a different operation. For example, the updated value corresponds to a minimum or maximum value between the value LYq and the current value of the pixel of the mini-image MPICf.

According to an embodiment, the process comprises a step during which the circuit compares the value LYq of the current pixel with a stored value, the stored value being the minimum value of the previous pixel of the image. If the stored value is higher than the value of the pixel, the value of the pixel is stored in place of the previously stored value. This value is for example used in the computation of a context value. A similar value representing the maximum value is for example generated.

The operation <NUM> represented in <FIG> is carried out for each image LOGY_IMAGEf. Before the reiteration of the process for another image, the values of q, b and ΣRb' are reset for example to <NUM>.

The operations of <FIG> are for example carried out in parallel with other operations of the process, in particular with the operations <NUM> and <NUM> of <FIG>.

As the number k, which is the ratio between each sub-block and the corresponding group <NUM>, can be greater than <NUM> some pixels of the image IMAGEf can be used for the computing of several of the values of the mini-image MPICf.

An advantage of using a number k greater than <NUM>, is that it decreases the risk of artefacts, especially edge artefacts, between the various regions. It also smoothens the passage from one region of the image LOGY_IMAGEf to the other.

<FIG> is a flow diagram illustrating in more detail a step of generating the context value CTXq.

This operation is described for a pixel LPq of a sub-block Rb of an image LOGY_IMAGEf.

During an operation <NUM> (DETERMINING WEIGHT LINKED TO PROXIMITY), the circuit determines weights di for each sub-block Ri of the image LOGY_IMAGEf, the integer i being in the range from <NUM> to sb-<NUM>, sb being the number of sub-blocks in the image. Each weight di is linked to the spatial proximity of the various sub-blocks to the pixel LPq. For example, the circuit computes each weight di based on the spatial proximity of the pixel LPq to the center of the sub-block Ri.

For example, the weight di associated to a sub-block Ri is computed as follows: <MAT> where Dmax is an arbitrary constant distance, for example preprogrammed in a memory, and δi is the distance between the pixel LPq and the center of the sub-block Ri.

The value δi is for example computed as follows: <MAT> where ydiff is the difference between the index of the row of the pixel LPq and the index of the row of the center of the sub_block Ri and xdiff is the difference between the index of the column of the pixel LPq in the array and the index of the column of the center of the sub_block Ri. The value δi may, for example, be an approximated value, in order to avoid the computation of a square root operation.

The value of di is in this example in the range from <NUM> to <NUM>. If a value of di is equal to <NUM>, which correspond to a value δi equal to, or greater than, Dmax, the associated sub-block Ri is not used in the computation, as it is not considered close enough to pixel LPq.

The values of the weights di are for example computed for each image of the succession of images. Alternatively, as the value of each weight di is independent of the content of the images, and only depends on the size of the image and of the sub-block, the values of the weights di are for example computed once and stored in order to be used for all the images.

<FIG> represents part of an operation of determining a weight value. <FIG> illustrates the determination of the weights di, linked to the spatial proximity of the sub-block Ri to the pixel LPq.

<FIG> represents an image LOGY_IMAGEf <NUM>, divided into <NUM> sub-blocks <NUM>. The pixel LPq is included in one of the sub-blocks.

The spatial proximity di between the pixel LPq and the centers of the sub-blocks Ri <NUM> has been computed. A border <NUM> delimits in <FIG> the sub-blocks that will not be used to compute the context value CTXq (outside the border <NUM>), in other words, the pixels with a spatial proximity equal to <NUM>, from the sub-blocks that can be used to compute the context value (inside the border <NUM>), in other words, the pixels with a spatial proximity higher than <NUM>.

Referring again to <FIG>, during an operation <NUM> (DETERMINING WEIGHT LINKED TO AFFINITY), the circuit determines weights aj, the integer j being in the range from <NUM> to sb-<NUM>. The weight aj is linked to the affinity of the pixel PMj with the pixel LPq. For example, the closer the value of the pixel PMj to the value LYq, the higher the affinity. The affinity represents the difference in intensity between the pixel LPq and the pixels PMj of the mini-image.

For example, the weight aj can be computed as follows: <MAT> where r is the difference between the highest and the lowest value of the intensity LYq of the pixels of the image LOGY_IMAGEf-<NUM>, fbw is a value representative of the number of bits used to code logarithmic values and aff _factor is a percentage. If the difference between the value of the intensity LYq and the value LYMj of the pixel PMj is greater than aff_factor*r, then the pixel LPq and the pixel PMj have no affinity. The value of aj is for example in the range from <NUM> to <NUM>.

The values aff_factor and fbw are preferably the same for all the pixels of an image and preferably the same for all the images of the sequence of images. The values aff _factor and fbw are for example chosen by the user or are preprogrammed during the programing of the process. Preferably, the value aff _factor is in the range from <NUM> to <NUM>, and is preferably not equal to <NUM>.

<FIG> represents the operation of determining a weight value aj. <FIG> illustrates the determination of the weight aj, linked to the affinity of the pixel LPq with the pixel PMj.

<FIG> represents a mini-image MPICf <NUM>, divided into <NUM> pixels, corresponding to an image <NUM> LOGY_IMAGEf comprising <NUM> regions Rb. The affinity aj between the pixel LPq and the pixel PMj has been computed.

Preferably, for a given pixel LPq of the image LOGY_IMAGEf, the sensor computes the values of the spatial proximity for all the sub-blocks of the image LOGY_IMAGEf. Afterwards, the device determines which sub-block have a spatial proximity different from <NUM> and computes the values of the affinity of the pixel with the pixels of the mini-image corresponding to those sub-blocks. Therefore, for a given pixel LPq, the device does not compute the affinity between the pixel LPq of the image LOGY_IMAGEf and the pixels of the mini-image which corresponds to sub-block having a spatial proximity equal to <NUM>.

Referring again to <FIG>, during an operation <NUM> (COMPUTE CTXq), the circuit computes the context value CTXq. The context value is for example a weighted average. For example, the context value is computed as follows: <MAT>.

The value ECTXq corresponds for example to the estimated context value and is for example computed as follows: <MAT>.

The value Likelihoodq corresponds to the likelihood that the estimated context value is an accurate representation of the context of the pixel or that it is neither too extreme in intensity or colorfulness. It is for example computed as follows: <MAT>.

The value Confidenceq corresponds to the confidence in the estimated context and is computed as follows: <MAT>.

The confidence is for example in the range from <NUM> to <NUM>, including <NUM> and <NUM>, as the weights ai and di are in the same range.

The confidence Confidenceq is substantially equal to <NUM> when the circuit considers that there is not enough information for the estimated context to be considered accurate. For example, the confidence is substantially equal to <NUM> for the pixels situated at the border between two different objects of the scene, at which point there is generally an abrupt variation in brightness and the sum of di *ai will be low.

The value Biasq corresponds to a variable referred to as the bias, and is for example computed as follows: <MAT> where normPq is the normalized value of the intensity LYq of the pixel Pq. In other words, the value normPq is equal to the value
of the pixel LPq minus the minimum value LYqmin of the previous image LOGY_IMAGEf. The number r is the range of the values of the previous image LOGY_IMAGEf, in other words, the number r is equal to the maximum value LYqmax of the intensity LYq of the image minus the minimum value LYqmin. The maximum value of the number r is equal to the bit depth minus one. The value bq is representative of the parts at the extremity of the dynamic range in which the bias is lower than <NUM>.

The value bq is for example equal to one of several values depending on the intensity value of the pixel LPq. The possible values of the value bq are for example chosen by the user or preprogrammed during the programing of the process.

<FIG> illustrates the bias Biasq associated with a pixel Pq as a function of the intensity LYq. The value of the bias increases from <NUM> to <NUM> between the minimum intensity value LYqmin and a value Y1 of the intensity equal to Y1 = LYqmin + bq x r. The value of the bias is constant between the value Y1 and a value Y2 equal to Y2 = LYqmax - bq x r. The value of the bias decreases from <NUM> to <NUM> between the value Y2 and the maximum value of the intensity LYqmax.

The bias is used to avoid saturation of the brightest and darkest pixels. Indeed, if the intensity of the brightest pixels is increased too much, it will be difficult to distinguish detail. Similarly, if the intensity of the darkest pixels is lowered too much, it will also be difficult to distinguish detail.

When the value of bias is equal to <NUM>, the likelihood is equal to <NUM> and therefore, no change is made to the pixel. For example, the gain value corresponding to the pixel is then equal to <NUM>.

The value Greynessq is a variable referred to as the greyness and depends on a variable Gq, which is computed as follows: <MAT> where the number s is a sensitivity threshold, for example in the range from <NUM> to <NUM>. The maximum or minimum values over R,Gr,Gb,B correspond to the minimum or maximum values among the red, green, and blue channels of the pixel. Alternatively, the red, green, and blue channels can be replaced by other channels, for example in the case of hyperspectral images.

The value s is for example chosen by the user or is preprogrammed during the programing of the process.

If the variable Gq is higher than a first threshold chosen by the user or preprogrammed, the value Greynessq is equal to <NUM>. If the variable Gq is less than a second threshold, chosen by the user or preprogrammed, the value Greynessq is equal to <NUM> minus the value s. The second threshold is lower than or equal to, preferably lower than, the first threshold. If the variable G is in the range from the second threshold to the first threshold, preferably including the first and second thresholds, the value Greynessq is equal to the value Gq.

Similarly to the bias, the greyness is equal to <NUM> if, for a pixel, the value of a color is above a threshold, in which case the likelihood is also <NUM> and no change is made to the corresponding pixel.

If the images of the succession of images are in monochrome, the greyness is not computed.

The value of the likelihood Likelihoodq is substantially equal to <NUM>, which means that the image will not be modified or, will be modified in a negligible way, when:.

<FIG> illustrates a circuit providing an example implementation of the embodiment of <FIG>. More precisely, <FIG> represents the circuit <NUM> of the device, which implements the process of <FIG>.

The circuit <NUM> for example comprises an image sensor <NUM> (SENSOR). The image sensor <NUM> for example comprises a Bayer sensor. The image sensor <NUM> is configured to capture successive images. For example, the image sensor <NUM> is able to capture images at video rates. In some embodiments, the image sensor <NUM> is configured to capture stills. The image sensor <NUM> is preferably a video camera.

The circuit <NUM> for example comprises a preprocessing circuit <NUM> (PREPROCESSING). The preprocessing circuit <NUM> receives, as an input, the images captured by the image sensor <NUM>. The output of the preprocessing unit corresponds, in this example, to the pixels of one of the images INPUT_IMAGE.

The circuit <NUM> for example comprises a circuit <NUM> (INTENSITY). The circuit <NUM> computes the value Yq corresponding to the pixel Pq being sent by the circuit <NUM>.

The circuit <NUM> for example comprises a circuit <NUM> (LOG) configured to implement a logarithmic function, preferably with base <NUM>, on the value of the intensity Yq and generate the corresponding value LYq.

The circuit <NUM> for example comprises a circuit <NUM> (MINI-IMAGE) configured to create the mini-image MPICf from the values LYq generated by the circuit <NUM>. The circuit <NUM> for example comprises a memory configured to store the mini-images created. Preferably, the memory comprises a region 311a (MEM1) in which the mini-image MPICf that is created is stored and a region 311b (MEM2) in which the mini-image MPICf-<NUM> created based on the previous image is stored. Preferably, the mini-image MPICf-<NUM> stored in the region 311b is erased when the modification of the image INPUT_IMAGEf is completed, and the mini-image MPICf having just been created is moved from region 311a to region 311b.

The circuit <NUM> for example comprises a circuit <NUM> (COMPUTE) configured to compute the gain map to be applied to the image INPUT_IMAGEf in order to obtain the image OUTPUT_IMAGEf. More specifically, the circuit <NUM> computes the gain value GMPq corresponding to the pixel Pq as described above.

The circuit <NUM> for example comprises a circuit <NUM> (MODIFICATION) configured to apply the gain map to the image INPUT_IMAGEf and more specifically, to multiply the gain value GMPq and the value of the pixel Pq.

The circuit <NUM> for example comprises a circuit <NUM> (POSTPROCESSING) configured to implement other functions. The other functions may include other tone mapping processes and other processes of image processing.

The circuit <NUM> for example comprises a circuit <NUM> (MEM), for example a memory, configured to store the output image. Alternatively, the output image is not stored. The output image is for example only displayed.

In this example, the circuit is at least partially implemented in hardware. The circuit may, for example, comprise a software part.

Preferably, some or all of the circuits of the circuit <NUM>, except for example for the circuits <NUM> and <NUM>, are shared by at least two processes of image processing. As such, the addition of the process of local tone mapping described herein only involves the addition of circuits <NUM> and <NUM>, which is advantageous in terms of silicon area and cost, as the number of hardware blocks is limited.

<FIG> illustrates a processing device providing another example of implementation of the embodiment of <FIG>. This example represents an implementation entirely by way of software.

The circuit <NUM> implementing the process of tone mapping described in relation with <FIG> comprises a microprocessor, or a microcontroller <NUM> (µ). The microprocessor implements the various operations of computation and of modification of the pixels comprised in the embodiment of <FIG>.

The circuit <NUM> also comprises at least one memory (MEM) <NUM>. The memory, or memories, <NUM> are for example non-transitory memories configured to store at least the mini-images and the output image. The memory <NUM> comprises a region 405a (MEM1) in which the mini-image MPICf being created is stored and a region 405b (MEM2), in which the mini-image MPICf-<NUM>, created based on the previous image is stored. Preferably, the mini-image MPICf-<NUM> stored in the region 405b is erased when the modification of the image INPUT_IMAGEf is completed, and the mini-image MPICf having just been created is moved from region 405a to region 405b.

The circuit <NUM> comprises an input/output interface <NUM> (I/O). For example, the interface <NUM> is coupled with an image sensor in a similar fashion to the image sensor described in relation with <FIG>.

The circuit <NUM> preferably comprises at least one other function <NUM> (FUNC).

The various elements of the circuit <NUM>, in particular the microprocessor <NUM>, the memory or memories <NUM>, the interface <NUM> and the functions <NUM>, are coupled together via a bus <NUM>. Various information, such as the information concerning the images IMAGEf and the output image, is for example transmitted over the bus <NUM>.

An advantage of the embodiments described herein is that, as the image IMAGEf generated by the image sensor is not stored, the memory requirement is relatively small. Indeed, the storage of two mini-images is significantly less costly than the storage of an image IMAGEf. In a traditional regional tone mapping operation, a number of rows equal.

Another advantage is that, as each pixel of a mini-image corresponds to an average of a bigger area, there are few differences between a mini-image and the following one. Therefore, it is possible to use a mini-image, generated using a first image, to process the following image of the successive images. Furthermore, it is possible to process the pixels with the previous mini-image, as they are caught and read by the sensor. Indeed, the mini-image is already fully available when we process the pixel. There is, therefore, no delay in this part of the processing.

Another advantage is that the use of a logarithmic value for the creation of the mini-image allows the process not to be limited by the bit depth. It is therefore possible to use this process on images having a high dynamic and to obtain an output image with the same a high dynamic.

Another advantage of the embodiments described herein is that the size of the mini-image can be chosen for a given sequence of successive images. The bigger the mini-image, the more precise the tone mapping. However, increasing the size of the mini-image increases the size of the memory needed to store the mini-images. Being able to choose the size of the mini-image in regard with the input image allows to determine a compromise between memory space and precision depending on the application.

Another advantage of the embodiments described herein is that, in the computations, the pixels of the images are compared with pixels of the mini-image. Therefore, there are less comparisons and less computations than if it was needed to compare with as many pixels as there are in the images of the succession of images.

Various embodiments and variants have been described. Those skilled in the art will understand that certain features of these embodiments can be combined and other variants will readily occur to those skilled in the art. In particular, the succession of images on which the method is applied may be part of a larger group of images. For example, the succession of images comprises an image out of Z images of the larger group of images, Z being an integer higher than or equal to <NUM>.

Alternatively, the method can be applied to a sequence of successive images stored in a memory. The method is then identical at the exception of the fact that the input images come from a storage element instead of a sensor.

Claim 1:
A tone mapping method for a succession of images (INPUT_IMAGEf) implemented by an image processing device, the method including:
a) the division of the images (INPUT_IMAGEf) of the succession of images in a plurality of sub-blocks (Rb) of first pixels (Pq);
b) for a first image (INPUT_IMAGEf) of the succession of images, the application of a logarithmic function to the intensity of each pixel of the first image, and the creation of a first mini-image (MPICf) comprising pixels of the first mini-image, each pixel of the first mini-image representing a corresponding sub-block (Rb) of the first image, the intensity (PMb) of each pixel of the first mini-image being representative of the intensity of the first pixels of the corresponding sub-block after application of the logarithmic function to each pixel;
c) the storage of the first mini-image (MPICf) in a memory;
d) for a second image (IMAGEf+<NUM>) of the succession of images, the creation of a gain map of the second image according to the first mini-image and the second image, the modification of the second image according to the gain map in order to generate an output image, the gain map being erased after the generation of the output image.