Patent Description:
Images captured by a camera or image capturing device of an object are often illuminated by ambient light as well as a separate source of illumination. In order to be able to accurately compare images of the same object, or determine colours of an object, the ambient light must be corrected for to produce an ambient light corrected image. However, such images are often generated by comparing pixels of captured images, which introduces errors when there is any movement of the object between frames.

<CIT> discloses a device and method of capturing a video frame using an electronic device to provide enhanced video. A first sub-frame is captured using ambient light, and a second sub-frame is captured using ambient light and a light source under control of the electronic device. A primary frame is generated based on a combination of the first sub-frame and the second sub-frame. It discloses detecting motion of an object captured and modifying the second sub-frame to compensate for the detected motion. <CIT> discloses systems and methods for minimizing motion clutter in image-generation devices. Temporally-interleaved image-subtraction reduces the magnitude of motion clutter and has no adverse effect on the desired ambient-light cancellation of static images. Embodiments of image-interleaved generation devices employing temporally-interleaved image-subtraction include single, double, triple, and series accumulator configurations. All four embodiments allow synchronization with scene illuminators and may be implemented on a single electronic chip. Temporally-interleaved image-subtraction is particularly well suited for use in video eyetracking applications where ambient light and scene motion can cause significant problems. <CIT> discloses a method for generating an ambient light rejected output image which includes providing a sensor array including a two-dimensional array of digital pixels where the digital pixels output digital signals as digital pixel data representing the image of the scene, capturing a pair of images of a scene within the time period of a video frame using the sensor array where the pair of images includes a first image being illuminated by ambient light and a second image being illuminated by the ambient light and a light source, storing the digital pixel data associated with the first and second images in a data memory, and subtracting the first image from the second image to obtain the ambient light rejected output image.

According to a first specific aspect, there is provided the method of claim <NUM> for obtaining an ambient light correction (ALC) image having a minimised error due to motion.

Additional preferred embodiments of the method are described in dependent claims <NUM>-<NUM>.

According to a second aspect, there is provided the non-transitory computer readable storage medium comprising computer-readable instructions of claim <NUM> that, when executed by a processor, causes the performance of a method in accordance with the first aspect.

According to a third aspect, there is provided the computer program of claim <NUM> that, when read by a computer, causes performance of a method in accordance with the first aspect.

According to a fourth aspect, there is provided the image capturing apparatus of claim <NUM> comprising: at least one processor; at least one memory comprising computer-readable instructions; the at least one processor being configured to read the computer readable instructions and cause performance of a method in accordance with the first aspect.

These and other aspects will be apparent from and elucidated with reference to the embodiments described hereinafter.

<FIG> shows an image capturing apparatus <NUM> comprising an image capturing device <NUM>, a light <NUM>, a processor <NUM> and a memory <NUM> comprising computer readable instructions. The image capturing device <NUM> is connected to the processor <NUM> and is configured to capture images of an object <NUM>, which in this example is a human, which can be read and processed by the processor <NUM>.

The light <NUM> is configured to illuminate the object <NUM> and is also connected to the processor <NUM>. The light <NUM> is configured to emit variable intensity of illumination which is controlled by the processor <NUM>. In other examples, the light <NUM> may be connected to, and controlled by a separate controller.

The memory <NUM> comprises computer readable instructions and the processor <NUM> is configured to read the computer readable instructions to perform a method for minimising error in ambient light corrected (ALC) images.

<FIG> is a flow chart showing steps of a first example method <NUM> of obtaining an ALC image having a minimised error due to motion. <FIG> shows the steps of the method in terms of the outputs of captured and processed images.

In block <NUM>, the method comprises capturing a plurality of primary images <NUM> of the object <NUM> over a time window <NUM> illuminated by the light <NUM>, which illumination is controlled to vary over time. In <FIG>, the passage of time is represented by the arrow <NUM>. Each primary image <NUM> comprises a plurality of pixels <NUM> which each have a pixel index corresponding to the location of the pixel <NUM> in the image, and a pixel value defining the intensity of the light captured at that pixel <NUM>.

In block <NUM>, following block <NUM>, the method <NUM> comprises performing ambient light correction on at least a portion of the plurality of primary images <NUM> to generate a plurality of ALC images <NUM>. Ambient light correction of primary images <NUM> involves adjusting the intensity of pixels <NUM> in the captured primary images <NUM> to correct for ambient light which illuminates the object <NUM>. A sub-set of primary images <NUM>, which are captured with differing controlled illumination, can be demodulated to generate an ALC image <NUM>. In a simplified example of ambient light correction by demodulation, a first primary image may be captured with the light switched on to illuminate the object, and a second primary image may be captured with the light switched off such that only ambient light illuminates the object. For each pixel of the first primary image, the intensity of the second primary image at that pixel can be subtracted to generate an ALC pixel, and combining all of the ALC pixels generates the ALC image. A more complex example of ambient light correction by demodulation can be found in<NPL>, or a further example in which a time varying signal for each pixel in the image sequence is multiplied with a modulation function relating to the periodically varying controlled, active illumation, followed by a low-pass filter operation which results in the amplitude of the periodically varying incoming light. This amplitude is proportional to the amount of reflection of light by the object, and is used to extract an ALC image in which the ambient light intensity or amplitude has been removed.

In this example, each ALC image <NUM> is generated based on demodulating a pair of temporally adjacent primary images <NUM>. In other words, the sub-set of primary images <NUM> which are demodulated to generate a single ALC image <NUM> comprises two primary images <NUM>. Therefore, each primary image <NUM> is used in the generation of two temporally adjacent ALC images <NUM>. In other examples, each ALC image may be generated based on demodulating more than two primary images.

In this example, each ALC image <NUM> is generated from a sub-set of primary images <NUM> which is offset by one primary image <NUM> from a preceding sub-set of primary images <NUM> used to generate another ALC image <NUM>. In other examples, there may be an offset of more than one primary image for each sub-set of primary images.

In the example shown in <FIG>, there are six primary images <NUM> captured, and each pair of temporally adjacent primary images <NUM> is demodulated to generate five ALC images <NUM>.

ALC images <NUM> generated by demodulating two or more primary images <NUM> can have errors introduced due to movement of the object <NUM> while the two or more primary images <NUM> are captured. The following steps of the method <NUM> are intended to select an ALC image <NUM> from a plurality of ALC images <NUM> which has the lowest error.

In block <NUM>, the method <NUM> comprises calculating an error value <NUM> for pairs of ALC images <NUM>. The calculation of the error value <NUM> for a pair of ALC images <NUM> will be described below with reference to a single pair of ALC images <NUM> comprising a first ALC image 36a and a second ALC image 36b, to calculate error value E<NUM>, as shown in <FIG>. It will be appreciated that these steps can be repeated for every pair of ALC images <NUM> for which an error value <NUM> is to be calculated.

In this example, the first ALC image 36a is temporally adjacent to the second ALC image 36b. In other examples, error values may be calculated for pairs of non-temporally adjacent ALC images.

Block <NUM> of the method <NUM> comprises block <NUM> and block <NUM>. In block <NUM>, each pixel <NUM> of the first ALC image 36a is compared with a corresponding pixel <NUM>, having the same pixel index, of the second ALC image 36b, and a change in intensity is determined for each pixel <NUM> between the first ALC image 36a and the second ALC image 36b.

In some examples, if each pixel comprises a plurality of colour channels, then determining the change in intensity for each pixel may comprise determining a change in intensity for each colour channel in the respective pixel and aggregating the changes in intensity over the colour channels.

In block <NUM>, the error value E<NUM> is calculated between the first ALC image 36a and the second ALC image 36b based on the change in intensity over a plurality of pixels <NUM>. In this example, the change in intensity for each pixel <NUM> between the first ALC image 36a and the second ALC image 36b is aggregated by determining a sum of absolute changes in intensity of the plurality of pixels <NUM>. This sum is used to determine the error value E<NUM>. In other examples, the error value may be determined by aggregating the change in intensity values over the plurality of pixels by other means, such as determining a mean square deviation in changes in intensity over the plurality of pixels, or determining an absolute mean of the change in intensity of each of the plurality of pixels. In yet further examples, the error value may be determined simply based on selecting a maximum absolute change in intensity over the plurality of pixels <NUM>. Selecting a maximum absolute change in intensity may be useful to assign larger errors to pairs of ALC images in which there is a low error throughout the images, but a large distortion in a small area of the ALC images. The plurality of pixels <NUM> from which the error value <NUM> is calculated comprises all of the pixels <NUM> in the whole ALC image in this example. However, in other examples, the plurality of pixels may only comprise a sub-set of all of the pixels in the ALC images such as a central portion of the ALC images, or a sub-set of pixels corresponding to a feature of interest (described in more detail in <FIG> below).

In block <NUM>, the method <NUM> comprises comparing the error values <NUM> for each pair of ALC images <NUM>, and selecting an error minimised pair of ALC images <NUM> having the lowest error value Ex. An error minimised image <NUM> may then be selected from the error minimised pair of ALC images <NUM>. This may be one of the ALC images <NUM> of the error-minimised pair of ALC images <NUM> selected arbitrarily or following a condition, or it may be both of the ALC images <NUM> of the error-minimised pair of ALC images <NUM>.

Referring back to <FIG>, the image capturing apparatus <NUM> comprises a user interface <NUM> which is connected to the processor <NUM>. The connection may be wired or wireless. The user interface <NUM> may display the error minimised pair of ALC images <NUM> or the selected error minimised image <NUM>. The user interface <NUM> may additionally or alternatively display the error value Ex of the selected error minimised image <NUM>.

The first example method <NUM> may further comprise block <NUM>, which comprises outputting the selected error minimised image <NUM> to the user interface <NUM> and/or outputting the error value of the selected error minimised image <NUM> to the user interface <NUM>. In other examples, block <NUM> may be omitted.

In the first example method <NUM>, the plurality of primary images <NUM> may be captured over a predetermined time window <NUM> and analysed after all of the primary images <NUM> are captured. In some examples, the primary images <NUM> may be captured continuously over the time window and the ALC images <NUM> may be generated dynamically during capturing of subsequent primary images <NUM>. The error value <NUM> may also be calculated dynamically in real time while subsequent ALC images <NUM> are being generated and primary images <NUM> are being captured.

Although it is described that an error value is calculated for a pair of whole ALC images <NUM>, in some examples, the primary images <NUM> and/or the ALC images <NUM> may be divided into sub-images to generate ALC sub-images each corresponding to a portion of the primary images. Error values may then be calculated for pairs of ALC sub-images to select an error-minimised ALC sub-image, for example as described with reference to <FIG>, or sub-images corresponding to portions of the primary images may be selected in any other suitable manner such as by dividing each image into predetermined blocks of pixels, with each block of pixels being a single ALC sub-image. In such examples, a full error-minimised ALC image may be generated or reconstructed by splicing together a plurality of error-minimised ALC sub-images, where each of the error-minimised ALC sub-images corresponds to a different portion of the primary images. This may be particularly useful if different portions of the object images by primary images are subject to different amounts of motion or movement.

<FIG> is a flow chart showing steps of a second example method <NUM> of selecting a single error minimised image <NUM> from the error minimised pair of ALC images <NUM>.

<FIG> shows a visual representation of the steps of the method <NUM> of selecting the error minimised image <NUM> from the error minimised pair of ALC images <NUM> in the form of an example plot <NUM>. The plot <NUM> comprises an x-axis of time, t, and a y-axis of error value, E.

Each error value <NUM> of the pairs of ALC images <NUM> is plotted against a time, t, corresponding to a time that each primary image <NUM> was captured from which each of the respective pair of ALC images <NUM> was generated. For example, each ALC image <NUM> is generated, in this example, from two temporally adjacent primary images <NUM>, and therefore each pair of ALC images <NUM> is generated from three temporally adjacent primary images <NUM>. The time, t, against which each error value <NUM> is plotted may be any time within the time that the three temporally adjacent primary images <NUM> are captured, such as the time that the middle primary image <NUM> is captured, or a time at a midpoint between the temporal ends of the three temporally adjacent primary images <NUM>. It will be appreciated that in other examples, there may be more than three primary images associated with each pair of ALC images, and that the time against which a respective error value is plotted can relate in any suitable way to the times that the associated primary images were captured.

In block <NUM>, the method <NUM> comprises interpolating discrete error values <NUM> of a plurality of pairs of temporally adjacent ALC images <NUM> calculated within the time window <NUM> to generate a continuous function <NUM> of error values <NUM> over time.

The lowest discrete error value <NUM> having a point at (tx, Ex) in <FIG> is the corresponding error value <NUM> of the error minimised pair of ALC images <NUM>. The continuous function <NUM> defines a minimum error Em at a minimum error time tm. In this example, there is only one minimum error Em at one minimum error time tm. However, it will be appreciated that there may be more than one minimum error defined by the continuous function having a gradient of zero between a preceding negative gradient and an ensuing positive gradient.

In block <NUM>, the method <NUM> comprises finding the minimum error time tm. In some examples, where there is more than one minimum error time tm, block <NUM> may comprise finding the minimum error time tm which his closest in time to the time, tx, of the error minimised pair of ALC images <NUM>.

In block <NUM>, the method <NUM> comprises selecting the error minimised image <NUM> that is closest to the minimum error time tm from the error minimised pair of ALC images <NUM>. In other words, selecting the error minimised image <NUM>, from the error minimised pair of ALC images <NUM>, that is generated from primary images <NUM> which are temporally closest to the minimum error time tm. In short, if tm is larger than tx, then the error minimised image <NUM> is selected as the later of the pair of ALC images <NUM>. If tm is smaller than tx, then the error minimised image <NUM> is selected as the earlier of the pair of ALC images <NUM>.

<FIG> is a flow chart showing steps of a third example method <NUM> for minimising the error in ALC images <NUM> due to motion. Referring back to <FIG>, the third example method <NUM> produces the same basic outputs at various steps as the first example method <NUM>, and therefore the description below will refer to the reference numerals of <FIG>.

In block <NUM>, the third example method <NUM> comprises capturing a plurality of primary images <NUM> of the object <NUM> with controlled varying illumination of the object <NUM> from the light <NUM> over time. The method <NUM> comprises continuously capturing primary images <NUM> and analysing them in real time in the subsequent steps of the method <NUM>.

In block <NUM>, following block <NUM>, the method <NUM> comprises performing ambient light correction to generate an ALC image <NUM> by demodulating at least a sub-set of the plurality of primary images <NUM> captured.

In block <NUM>, following from block <NUM>, the method <NUM> comprises determining whether there have been at least two ALC images <NUM> generated. If there have not been at least two ALC images <NUM> generated, the method <NUM> returns to block <NUM> to capture further primary images <NUM> and to generate a further ALC image <NUM> from a later sub-set of primary images <NUM>, offset by one temporally adjacent primary image <NUM> from which the preceding ALC image <NUM> was generated. In other examples, the offset may be more than one temporally adjacent primary image.

If it is determined in block <NUM> that there are at least two ALC images <NUM>, the method moves to block <NUM>, in which the error value <NUM> of a pair of ALC images <NUM> is calculated, where the pair of ALC images <NUM> includes the two most recently generated ALC images <NUM>. Block <NUM> includes the same steps as block <NUM> in the first example method <NUM> to calculate the error value <NUM> of the pair of ALC images <NUM>. The calculated error value <NUM> is stored in the memory <NUM> together with the pair of ALC images <NUM> to which it refers.

In block <NUM>, following block <NUM>, the method <NUM> comprises comparing stored error values <NUM> for pairs of ALC images <NUM> and determining the lowest error value <NUM>. In block <NUM>, following block <NUM>, the method <NUM> comprises determining whether the lowest error value <NUM> is below a first threshold. If the lowest error value <NUM> does not fall below the first threshold, the method returns to block <NUM> to capture further primary images <NUM> and to generate another ALC image <NUM> from a sub-set of primary images <NUM> temporally adjacent to the sub-set of primary images from which the preceding ALC image <NUM> was generated.

If the lowest error value <NUM> is below the first threshold, the method <NUM> moves to block <NUM>, in which it is determined whether the latest error value <NUM> associated with the latest pair of ALC images <NUM> generated is above a second threshold. The second threshold is higher than the first threshold. If the latest error value <NUM> is not above the second threshold, the method <NUM> returns to block <NUM> to capture more primary images <NUM> and to generate another ALC image <NUM> from a sub-set of primary images <NUM> temporally adjacent to the sub-set of primary images from which the preceding ALC image <NUM> was generated. If the latest error value <NUM> is above the second threshold, the method continues to block <NUM> in which the time window <NUM> ends and an error minimised pair of ALC images <NUM> having the lowest error value Ex is selected. An error minimised image <NUM> may then be selected from the error minimised pair of ALC images <NUM> in a similar manner to block <NUM> described with reference to the first example method <NUM>.

In other examples, the method may move to block <NUM> after block <NUM>, when it is determined that the lowest error value <NUM> falls below the first threshold. Using the additional step in block <NUM> can ensure that the process is not stopped as soon as an error value <NUM> falls below the first threshold. This is particularly helpful if motion of the object <NUM> slows down over an extended time period, in which case, there may be pairs of ALC images <NUM> which can be generated with lower error values if the method is not immediately stopped when the first threshold is met. Waiting until the error values exceed a second threshold, higher than the first threshold ensures that the object <NUM> is no longer slowing down.

In this example, it has been described that a single ALC image <NUM> is generated in block <NUM>. It will be appreciated that any suitable number of ALC images <NUM> may be generated in one go at block <NUM> and that the second example method may continue by analysing any number of ALC images generated in block <NUM>.

<FIG> are flow charts showing first example sub-method <NUM> and second example sub-method <NUM> respectively for replacing blocks <NUM>-<NUM> in the first example method <NUM> and blocks <NUM>-<NUM> in the second example method <NUM>.

The first example sub-method <NUM> and the second example sub-method <NUM> are intended to define areas of interest in the ALC images <NUM> which can be analysed for an error value <NUM> instead of the whole ALC image <NUM>.

In the first example sub-method <NUM>, block <NUM> comprises capturing a plurality of primary images <NUM> in a similar manner to block <NUM> in the first example method <NUM> or block <NUM> in the second example method <NUM>.

In block <NUM>, following from block <NUM>, the sub-method <NUM> comprises performing feature detection of the object <NUM> to define a plurality of primary sub-images 30a (shown best in <FIG>). In this example, the object <NUM> is a human, and the feature detection may include identifying particular features of interest, such as an area around the eyes of the human, the cheeks of the human or the forehead of the human. Each primary sub-image 30a may therefore comprise a feature of interest.

In block <NUM>, following from block <NUM>, the sub-method <NUM> comprises performing ambient light correction on the plurality of primary sub-images 30a from the plurality of primary images <NUM> to generate a plurality of ALC sub-images. The first example sub-method <NUM> may then continue to block <NUM> of the first example method <NUM> or may continue to block <NUM> or <NUM> of the second example method <NUM>, in which the comparison of ALC-images is instead a comparison of the ALC sub-images.

In the second example sub-method <NUM>, block <NUM> comprises capturing a plurality of primary images <NUM> in a similar manner to block <NUM> in the first example method <NUM> or block <NUM> in the second example method <NUM>.

In block <NUM>, the sub-method <NUM> comprises performing ambient light correction to generate a plurality of ALC images <NUM>. The sub-method <NUM> then moves on to block <NUM>.

In block <NUM>, the sub-method <NUM> comprises performing feature detection of the object <NUM> in the ALC images <NUM> to define a plurality of ALC sub-images. In this example, the object <NUM> is a human, and the feature detection may include identifying particular features of interest, such as an area around the eyes of the human, the cheeks of the human or the forehead of the human. Each ALC sub-image may therefore comprise a feature of interest. The second example sub-method <NUM> may then continue to block <NUM> of the first example method <NUM> or may continue to block <NUM> or <NUM> of the second example method <NUM>, in which the comparison of ALC-images is instead a comparison of the ALC sub-images.

The methods described herein are based on the assumption that, the more a generated image differs in two ALC images <NUM>, the larger the error will be in each ALC image <NUM>. Therefore, rather than calculating motion of an object <NUM> and trying to correct the actual error introduced into an ALC image by that motion, the methods produce the ALC images, and select the ALC image or pair of ALC images with the lowest estimated error.

Claim 1:
A method for obtaining an ambient light correction (ALC) image having a minimised error due to motion, the method comprising:
capturing a plurality of primary images (<NUM>) of an object (<NUM>) over a time window (<NUM>), with controlled illumination of the object (<NUM>) varying in intensity over time;
performing ambient light (<NUM>) correction on at least a portion of the plurality of primary images (<NUM>), which have been captured with different illumination intensity, to generate a plurality of ALC images (<NUM>), each ALC image (<NUM>) being generated based on demodulating a sub-set of the plurality of primary images (<NUM>) which are temporally adjacent by determining, from the sub-set of temporally adjacent primary images with differing controlled illumination, an intensity of ambient light, and generating the ALC image (<NUM>) by removing the ambient light intensity from at least one image in the sub-set of temporally adjacent primary images (<NUM>);
and characterized in that the method further comprises:
calculating an error value (<NUM>) for pairs of ALC images (<NUM>), each pair of ALC images (<NUM>) including a first ALC image (36a) and a second ALC image (36b), wherein the first ALC image (36a) and the second ALC image (36b) are temporally adjacent;
comparing the error values (<NUM>) for each pair of ALC images (<NUM>) and determining a pair of ALC images (<NUM>) which has the lowest error value (<NUM>) within the time window (<NUM>); and
selecting an ALC image (<NUM>) as an error-minimised image (<NUM>) from the determined pair of ALC images (<NUM>) having the lowest error value (<NUM>) within the time window (<NUM>),
wherein calculating the error value (<NUM>) comprises:
comparing each pixel (<NUM>) of the first ALC image (36a) with a corresponding pixel (<NUM>) of the second ALC image (36b), and determining a change in intensity for each pixel (<NUM>) between the first ALC image (36a) and the second ALC image (36b), and
calculating an error value (<NUM>) for the pair of ALC images (<NUM>) based on the change in intensity over a plurality of pixels (<NUM>).