Patent Description:
Camera flashes provide illumination in low-light conditions. For optimal color rendering, the color of the flash should match that of the subject and/or the full image. This can be accomplished by using a collection of light sources of different colors so that the combined color point produced by the collection matches that of the subject and/or the full image. Due to their compact size and low power requirements, light-emitting diodes (LEDs) are attractive candidates for light sources used in camera flashes for hand-held, battery-powered devices, such as cameras and cell phones.

<CIT> describes a device including a flash unit and a controller. The controller is configured to generate a third image based on a first image and a second image.

<CIT> relates to systems and techniques for color temperature analysis and matching. For example, three or more camera flash LEDs of different output colors can be used to match any of a range of ambient color temperatures in a non-linear space on the black body curve. The scene color temperature can be analyzed in a preliminary image by determining actual sensor R/G and B/G ratios, enabling more accurate matching of foreground flash lighting to background lighting by the brightness emitted from, each LED can be individually controlled based on the determined sensor response to provide a dynamic and adaptive mix of the output colors of the LEDs.

The present invention provides technology for improved post-post-processing of multi color flash images. The present invention in its first aspect provides a camera device as defined in claims <NUM> to <NUM>.

The present invention in its second aspect provides a method for operating a camera device as defined in claims <NUM> and <NUM>.

The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

For purposes of illustrating multi-color flash with image post-processing described herein, it might be useful to understand phenomena that may come into play in multi-color flash cameras. The following foundational information may be viewed as a basis from which the present disclosure may be properly explained. Such information is offered for purposes of explanation only and, accordingly, should not be construed in any way to limit the broad scope of the present disclosure and its potential applications.

LED flash in cameras (or, more generally, any imaging devices), e.g., in cell phone imaging, provides illumination for when images are acquired in low-light conditions. As described above, for optimal color rendering, the color of the flash should match that of the subject and/or the full image, which can be accomplished by using a collection of light sources of different colors to set the optimal color point. One conventional method to set the right color point is to, first, measure the color point with a camera or other device and, next, tune the collection of LEDs to adjusting the current used to drive each of the LEDs. One challenge with such an approach is that the measurement and tuning should all be done in between the time when the user pushes the button to acquire an image and the time when the camera actually acquires the image. This is a time-critical step and typically requires a specialized processing unit, which can be quite costly in terms of processing power and required components. Executing this kind of pre-processing may be particularly cost-prohibitive for mobile applications, such as multi-flash cameras used in cell phones.

Embodiments of the present disclosure provide an approach that may be referred to as "multi-color flash with image post-processing" in that it uses a camera device with a multi-color flash (i.e., flashes of at least two different colors or color points) and implements post-processing (i.e., processing after the images have been acquired by a camera) to generate images. In one aspect, the multi-color flash with image post-processing is implemented by a controller configured to control a camera and flashes of at least two different colors (or associated with at least two different color points), referred to as a "first flash" and a "second flash. " Generally, as used herein, the term "first flash" refers to a collection of light sources, e.g., a collection of LEDs, which, when the first flash is used when an image is acquired, cause the image to be associate with a first color point. Similarly, the term "second flash" refers to a collection of light sources, e.g., a collection of LEDs, which, when the second flash is used when an image is acquired, cause the image to be associate with a second color point, different from the color point. The controller is configured to cause the camera to acquire a first image of a scene while the scene is being illuminated with the first flash but not the second flash (which may cause the first image to be associated with a first color point), then cause the camera to acquire a second image of the scene while the scene is being illuminated with the second flash but not the first flash (which may cause the second image to be associated with a second color point, different from the first color point), and generate a final image of the scene in post-processing based on a combination of the first image and the second image. For example, the final image may be generated as a weighted average, a squared weighted average, a weighted median, or any other suitable combination of the first and second images to represent the desired balance between the first and second flashes in the final image. In this manner, selection of the optimal/desired color point for the multi-color flash may be performed in post-processing, which may alleviate some of the demands or requirements on the pre-processing steps that have to be performed, advantageously allowing to, e.g., use a less complicated processing unit, which may lead to significant cost savings. For example, in some embodiments, the post-processing of generating the final image may be performed by an application-layer processor. Other features and advantages of the disclosure will be apparent from the following description and the claims.

In the following, the multi-color flash is described with reference to flashes of two different colors (or color points). For example, the first flash could be such that the first color point is a color point of a cold white (CW) color and the second flash could be such that the second color point is a color point of a warm white (WW) color. However, these descriptions can be easily extended to other embodiments which do not fall into the scope of the appended claims where other colors of the first and second flashes may be used, and/or where flashes of more than two different colors may be used. Furthermore, while some descriptions may refer to LEDs as the light sources for the flashes of different colors, in other embodiments, any suitable light sources, not limited to the LEDs, may be used. Still further, descriptions provided herein that refer to a flash of one color providing illumination while a flash of another color is off are equally applicable and cover embodiments which do not fall into the scope of appended claims, where the second flash is not completely off but is dimmed, e.g., is dimmed to less than about <NUM>% of its nominal output, to less than about <NUM>%, or to less than about <NUM>%, including all values and ranges therein. Since each flash may be implemented as a collection of light sources, e.g., a collection of LEDs, embodiments which do not fall into the scope of appended claims, could be envisioned where at no point of acquiring the images with different color flashes as described herein all of the LEDs are off. Instead, it could be that different sets of LEDs emit light with different intensities at different moments in time, which result in different color points of the images which are acquired to generate the final image as described herein. Thus, generally, descriptions referring to acquiring a given image while a given flash is illuminating the scene and while other flashes are not illuminating the scene include any embodiments where a given set of light sources emit light, each at a certain intensity (which could be different from one light source to another), resulting in illumination of the scene associated with a certain color point, different from the color points of other one or more flashes. Therefore, multi-color flash with image post-processing described herein refers, generally, to any embodiments where multiple sequential pictures are taken with various flash color points that are combined afterwards, where the flash color point does not necessarily need to be the same as the color point of a given light source or a given set of light sources as defined by the appended claims.

As will be appreciated by one skilled in the art, aspects of the present disclosure, in particular aspects of multi-color flash with image post-processing, described herein, may be embodied in various manners - e.g. as aa camera device defined in claims <NUM> to <NUM>, a method as defined in claims <NUM> and <NUM>, a computer program which does not fall into the scope of the appended claims, or a computer-readable storage medium which does not fall into the scope of the appended claims. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system. " Functions described in this disclosure may be implemented as an algorithm executed by one or more hardware processing units, e.g. one or more microprocessors, of one or more computers. In various embodiments, different steps and portions of the steps of each of the methods described herein may be performed by different processing units. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s), preferably non-transitory, having computer readable program code embodied, e.g., stored, thereon. In various embodiments, such a computer program may, for example, be downloaded (updated) to the existing devices and systems (e.g. to the existing camera devices and/or their controllers, etc.) or be stored upon manufacturing of these devices and systems.

In the following detailed description, various aspects of the illustrative implementations may be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. For example, if used, the term "connected" means a direct electrical or magnetic connection between the things that are connected, without any intermediary devices, while the term "coupled" means either a direct electrical or magnetic connection between the things that are connected, or an indirect connection through one or more passive or active intermediary devices. The term "circuit" means one or more passive and/or active components that are arranged to cooperate with one another to provide a desired function. The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within +/- <NUM>%, preferably within +/- <NUM>%, of a target value based on the context of a particular value as described herein or as known in the art.

For the purposes of the present disclosure, the phrase "A, B, and/or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges. As used herein, the notation "A/B/C" means (A), (B), and/or (C).

The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

In the following detailed description, reference is made to the accompanying drawings that form a part hereof, showing, by way of illustration, some of the embodiments that may be practiced. In the drawings, same reference numerals refer to the same or analogous elements/materials so that, unless stated otherwise, explanations of an element/material with a given reference numeral provided in context of one of the drawings are applicable to other drawings where elements/materials with the same reference numerals may be illustrated. The accompanying drawings are not necessarily drawn to scale. Moreover, it will be understood that certain embodiments can include more elements than illustrated in a drawing, certain embodiments can include a subset of the elements illustrated in a drawing, and certain embodiments can incorporate any suitable combination of features from two or more drawings.

Various operations may be described as multiple discrete actions or operations in turn in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed, and/or described operations may be omitted in additional embodiments.

In some examples provided herein, interaction may be described in terms of two, three, four, or more electrical components. However, this has been done for purposes of clarity and example only. It should be appreciated that the devices and systems described herein can be consolidated in any suitable manner. Along similar design alternatives, any of the illustrated components, modules, and elements of the accompanying drawings may be combined in various possible configurations, all of which are clearly within the broad scope of the present disclosure. In certain cases, it may be easier to describe one or more of the functionalities of a given set of flows by only referencing a limited number of electrical elements.

The following detailed description presents various descriptions of specific certain embodiments. However, is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. In general, the innovations described herein can be embodied in a multitude of different ways, as long as they fall into the scope of appended claims.

<FIG> provides a block diagram illustrating an example camera device <NUM> in which multi-color flash with image post-processing may be implemented, according to some embodiments of the present disclosure. As shown in <FIG>, the camera device <NUM> may include a first flash <NUM> (i.e., a flash associated with a first color point), a second flash <NUM> (i.e., a flash associated with a second color point), a camera <NUM>, a controller <NUM>, and, optionally, a measuring unit <NUM>.

Each of the first flash <NUM> and the second flash <NUM> may include a collection of light sources, e.g., a collection of LEDs, which, when the camera <NUM> acquires an image with one of these flashes illuminating the scene, causes the acquired image to be associate with a different color point. For example, the first flash <NUM> could be a CW color flash and the second flash <NUM> could be a WW color flash. In some embodiments, a scene being illuminated with the first flash <NUM> includes the scene being illuminated by a first plurality of light sources of one or more of a plurality of colors, and a scene being illuminated with the second flash <NUM> includes the scene being illuminated by a second plurality of light sources of one or more of the plurality of colors, illumination provided by the second plurality of light sources having a different color than illumination provided by the first plurality of light sources. The color of a given flash (or a given light source or a collection of light sources) may originate from different emission spectra of the light source and is characterized by different location (color point) in the RGB color space of the camera <NUM>.

The collection of LEDs of the first flash <NUM> and the second flash <NUM> may be arranged in a matrix or a vector. Thus, LEDs of a specific color point may be in a specific matrix. In one implementation, the camera device <NUM> includes two matrices for two color points, e.g., one matrix for a CW color, another matrix for a WW color. Thus, all of the LEDS of the matrices can be activated dependent on a desired color point. In an implementation, the matrices share the same LEDs and are controlled to emit different intensities of colored light at different times.

The LEDs in the collection of LEDs may be microLEDs. Thus, the first flash <NUM> and the second flash <NUM> may be microLED (µLED) pixel arrays with dozens, hundreds, thousands, or millions of LEDs positioned together on centimeter-scale-area substrates or smaller. In some implementations, the microLEDs are sized between <NUM> microns and <NUM> microns. In various implementations, the light emitting pixels are positioned less than <NUM> millimeter apart and are typically spaced apart by distances ranging from <NUM> microns to <NUM> microns. In many instances, the micro LED pixels are individually addressable.

The camera <NUM> may include any suitable imaging device configured to acquire images of a scene. The camera <NUM> may be communicatively coupled to the each of the first flash <NUM> and the second flash <NUM> so that these flashes may be synchronized to provide illumination of the scene when the camera <NUM> is acquiring images.

The timing of acquiring images by the camera <NUM> and the selective engagement of the flashes <NUM> and <NUM> in providing illumination when the images are acquired is controlled by the controller <NUM>, e.g., as described below. The controller <NUM> is also configured to generate the final image of a scene based on the images acquired by the camera <NUM>. The controller <NUM> may be implemented as, or include at least portions of, a data processing system <NUM> shown in <FIG>.

If used, the measuring unit <NUM> may be configured to perform additional measurements that may be used by the controller <NUM> in generating the final images. For example, in various embodiments, the measuring unit <NUM> may be configured to measure one or more of ambient (white) color point, spectrum, ambient (white) RGB response, of ambient correlated color temperature (CCT), and other parameters related to acquired images, such as auto focus, auto exposure, etc. In some embodiments, the measuring unit <NUM> may also be configured to perform measurements of, or measurements that enable calculation of, the values of Ra, Ba, Ga, described below, e.g., the values used in equations (<NUM>)-(<NUM>) provided below.

Although the camera device <NUM> illustrates only two flashes of different colors, namely, the flashes <NUM> and <NUM>, in other embodiments, additional flashes of different colors may be included in the camera device <NUM>. In addition, in further embodiments the camera device <NUM> may include other components. For example, in some embodiments, the camera device <NUM> may further include an output device, configured to display, e.g., one or more of the first image, the second image, and the final image as described herein. The camera device <NUM> may further include an input device, configured to receive user input to be used by the camera device to perform one of more of <NUM> cause the camera to acquire the first image, cause the camera <NUM> to acquire the second image, and the controller <NUM> generate the final image. The camera device <NUM> may further include one or more communication chips and an antenna, configured to wirelessly transmit one or more of the first image, the second image, and the final image, and/or wirelessly receive input to be used by the camera device <NUM> to perform one of more of cause the camera to acquire the first image, cause the camera to acquire the second image, and generate the final image.

Furthermore, while various components are shown in <FIG> as included within the camera device <NUM>, in various embodiments, the camera device <NUM> may refer to a device that includes to any combination of one or more of these components, in which case the other components shown in <FIG> may be implemented externally (i.e., in a separate device) and may be communicatively coupled to components of the camera device <NUM> as needed, via any appropriate communication channel, to implement the multi-flash with image post-processing as described herein.

In various embodiments, the camera device <NUM> may be, e.g., a wearable camera device (e.g., a smart watch), a hand-held camera device (e.g., a mobile phone), or a stationary camera (e.g., a security/surveillance camera).

<FIG> illustrates a flow chart of a method <NUM> for performing multi-flash with image post-processing, according to some embodiments of the present disclosure. Although method <NUM> is now described with reference to elements illustrated in <FIG>, any system or apparatus configured to perform various processes of this method, in any order, is within the scope of the present disclosure as long as they fall into the scope of the appended claims.

As shown in <FIG>, the method <NUM> begins with a process <NUM> which includes the controller <NUM> causing the camera <NUM> to acquire a first image of a scene while the scene is being illuminated with the first flash <NUM> but not with the second flash <NUM>. Similarly, in a process <NUM> of the method <NUM>, the controller <NUM> causes the camera <NUM> to acquire a second image of the scene while the scene is being illuminated with the second flash <NUM> but not with the first flash <NUM>. In general, the processes <NUM> and <NUM> represent that the method <NUM> includes the camera <NUM> acquiring separate images with flashes of different colors, where for each of these images, one or more of the flashes are illuminating the scene during image acquisition, while one or more other flashes are off. Typically, ambient light would also be present and also provide illumination for the scene when images with the external illumination from the flashes <NUM> or <NUM> are acquired. Thus, more generally, the process <NUM> includes the controller <NUM> causing the camera <NUM> to acquire the first image while the scene is being illuminated with the first flash <NUM> and the ambient light, while the process <NUM> includes the controller <NUM> causing the camera <NUM> to acquire the second image while the scene is being illuminated with the second flash <NUM> and the ambient light.

As shown in <FIG>, the method <NUM> may also include an optional process <NUM> which includes the controller <NUM> causing the camera <NUM> to acquire a third image of the scene while the scene is not being illuminated with the first or second flashes <NUM>, <NUM> (or, more generally, while the scene is not being illuminated with any flash and only ambient light may be present).

While the images acquired in the processes <NUM>, <NUM>, and <NUM> are referred to as first, second, and third images, in general, these processes may be performed in order other than what is shown in <FIG>. The order may be as follows: first, the process <NUM> is performed, then the process <NUM> is performed after the process <NUM>, then the process <NUM> performed after the process <NUM>.

Furthermore, the method <NUM> may include performing any of the processes <NUM>, <NUM>, and <NUM> multiple times, e.g., to compensate for motion-related artefacts. Any of the processes <NUM>, <NUM>, and <NUM> may be repeated multiple times consecutively. The method <NUM> may include performing the processes <NUM>, <NUM>, and <NUM> in the following order: <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. In general, any combination of these processes may be included in the method <NUM>. In this context, for all descriptions provided herein, the first, second, and third images may refer to not necessarily the pixel values of the images as they were acquired by the camera <NUM>, but to any combination of the pixel values from multiple instances of acquiring each of the images. For example, if the method <NUM> includes repeating the process <NUM> multiple times, the "first image" may refer to a matrix of pixel values where each pixel value is a combination, e.g., an average or any other statistical representation, of the corresponding pixel values in multiple instances of the images originally acquired in each of the processes <NUM>. In this context, a pixel value of one image may be referred to as being "corresponding to a pixel value of another image" when these pixel values are of pixels in the same position/location within a matrix of pixels of the images. For example, a pixel value of the pixel (<NUM>,<NUM>) of the matrix of pixels of one image (i.e., the pixel in the first row and the first column of the array of pixels of that image) corresponds to a pixel value of the pixel (<NUM>,<NUM>) of the matrix of pixels of another image.

Furthermore, what is considered to be a pixel value may be different in the method <NUM>. Different pixels may be monochromatic pixels. In other embodiments, different pixels may be (R, G, B)-pixels. In still other embodiments, different pixels may be other combination of colors, e.g., RGBW or RGB/Cyan. Unless specified otherwise, referring to "pixel values" in any of the explanations provided herein, values of pixels according to any of these embodiments are within the scope of the present disclosure as long as they fall into the scope of the appended claims.

Acquisition of the images with different color flashes providing illumination, and with no external flashes but only with ambient light providing illumination, causes these images to be associated with different color points which are used later by the controller <NUM> to combine the images in post-processing to generate the final image. To that end, in some embodiments, a color point of an image may be illustrated in a plot, some examples of which being shown in <FIG>, described in greater detail below, each of which illustrates a white dot representing a color point of the first image (i.e., the image acquired in the process <NUM>), a black dot representing a color point of the second image (i.e., the image acquired in the process <NUM>), and a grey dot representing a color point of the third image (i.e., the image acquired in the process <NUM>).

Turning back to <FIG>, the method <NUM> includes a process <NUM> which includes the controller <NUM> generating a final image of the scene based on a combination of at least the first image and the second image, possibly in combination with the third image (since the acquisition the third image is optional). Some examples of the combination are described below. It is to be understood that, there could be additional processing steps involved in performing the process <NUM>, which are not specifically described here but are known in the art of image processing, such as normalization of brightness of the acquired images (e.g., relative to one or more of image sensor sensitivity, exposure time, and possible other settings of the camera <NUM> during acquisition of the images), depending on the type of input images (e.g., jpeg or raw or. ), a correction including the Gamma curve, application of the same color correction matrix on all pictures (if not fixed before acquiring the images), etc..

In some embodiments, the process <NUM> may include the controller <NUM> applying respective weights to the pixel values of the first and second images to generate what may be referred to as "modified" first and second images (e.g., where each pixel value of the originally acquired image is multiplied by a certain weight), and then combining the pixel values of the modified first and second images to generate the final image. For example, in some embodiments, the process <NUM> may include the controller <NUM> generating a modified first image by multiplying each of pixel values of the first image by a first weight (the first weight being a value equal to or greater than zero and equal to or less than <NUM>) and generating a modified second image by multiplying each of pixel values of the second image by a second weight (the second weight being a value so that the sum of the first and second weights is equal to <NUM>, for the embodiments where only two different flashes are used). The process <NUM> may also include the controller <NUM> generating the final image by adding, e.g., on a pixel-by-pixel basis, a pixel value of the modified first image with a corresponding pixel value of the modified second image. In this manner, the final image may be generated as a weighted average of the first and second images. However, the weights may be used differently to generate the final images. The final image may be generated as a squared weighted average, a weighted median, or any other suitable combination of the first and second images (possibly with weights applied thereto) to represent the desired balance between the first and second flashes in the final image. The weights are used, the weights may be such that a sum of all weights applied to images with different color of flashes add up to <NUM>. Some examples of how weights may be used by the controller <NUM> to implement the process <NUM> of the method <NUM> are described with reference to <FIG>, where the controller <NUM> may be configured to compute a correction factor γ to apply to the first image to generate the modified first image (i.e., the weight applied to the first image is equal to, or is based on, the correction factor γ), and apply a correction factor equal to <NUM>-γ to the second image to generate the modified second image (i.e., the weight applied to the second image is equal to, or is based on, the correction factor <NUM>-y).

Besides various embodiments being possible for generating and applying weights to the first and second images, how the modified first and second images are combined in the process <NUM> may also be carried out in various manners as long as they fall into the range of the appended claims. The controller <NUM> may be configured to combine the modified first and second images to generate the final image on a pixel-by-pixel basis by somehow combining pixel values of the modified first image with corresponding pixel values of the modified second image. In this context, as used herein, describing an action being performed "on pixel-by-pixel basis" refers to performing the action for each one of the pixels individually. For example, the controller <NUM> adding, on a pixel-by-pixel basis, a pixel value of the modified first image with a corresponding pixel value of the modified second image refers to the controller <NUM> adding a pixel value of the pixel (<NUM>,<NUM>) of the modified first image with a pixel value of the pixel (<NUM>,<NUM>) (i.e., the corresponding pixel value) of the modified second image, adding a pixel value of the pixel (<NUM>,<NUM>) of the modified first image with a pixel value of the pixel (<NUM>,<NUM>) (i.e., the corresponding pixel value) of the modified second image, and so on.

In some embodiments of the method <NUM>, the method may further include receiving a user input indicative of a factor representing a balance between the first image and the second image in the final image, and then generating the final image in the process <NUM> based on the received factor. For example, in some such embodiments, the controller <NUM> may be configured to generate a modified first image by multiplying each of pixel values of the first image by a first weight indicative of the received factor, generating a modified second image by multiplying each of pixel values of the second image by a second weight, indicative of the received factor, and generating the final image by adding, on a pixel-by-pixel basis, a pixel value of the modified first image with a corresponding pixel value of the modified second image. Discussions provided above with respect to generating the final image in a way other than the simple weighted combination of the first and second images are applicable to such embodiments as well.

While discussions of the method <NUM> and other discussions provided herein refer to processing of images, these discussions are equally applicable to processing of video signals that may be acquired if the camera <NUM> is a video recording device that generates a video that includes a plurality of consecutive frames. In this context, each of the frames may be generated as the final image described herein.

As briefly described above, in some embodiments, the process <NUM> may include the controller <NUM> computing what is referred to herein as a "correction factor" (denoted as γ) based on the color points of the first, second, and third images, and then applying the correction factor to the first and second images to compute pixels values of the final image. <FIG> provide different examples of how the correction factor γ could be computed.

Each of <FIG> illustrates a white dot <NUM>, representing a color point of the first flash <NUM> (e.g., computed by the controller <NUM> based on the first image acquired in the process <NUM>), a black dot <NUM>, representing a color point of the second flash <NUM> (e.g., computed by the controller <NUM> based on the image acquired in the process <NUM>), and a grey dot <NUM>, representing a color point of the ambient light (e.g., computed by the controller <NUM> based on the third image acquired in the process <NUM> or obtained based on the measurements of the measurement unit <NUM>). The horizontal axis of the examples of each of <FIG> illustrates a value indicative of a sum of all or a sub-set of all red pixels of a given image (R) divided by a value indicative of a sum of all or a sub-set of all green pixels of a given image (G), i.e., R/G, while the vertical axis of each of <FIG> illustrates a value indicative of a sum of all or a sub-set of all blue pixels of a given image (B) divided by a value indicative of a sum of all or a sub-set of all green pixels of a given image (G), i.e., B/G. However, this is just one example of how a plot with color points of different flashes and the color point of ambient light may be represented in order to compute a correction factor to apply to the first and second images to generate the final image in the process <NUM>, the example being one particular example given for an RGB camera. In other embodiments, measures indicative of various pixel values of the flash-illuminated images and ambient light color points could be different.

For the specific example shown in <FIG>, the coordinates of the first flash color point <NUM> within the plot <NUM> may be (Rcw/Gcw;Bcw/Gcw), the coordinates of the second flash color point <NUM> within the plot <NUM> may be (Rww/Gww;Bww/Gww), while the coordinates of the ambient color point <NUM> within the plot <NUM> may be (Ra/Ga;Ba/Ga). In this notation, "CW" refers to "cold white" indicating that the first flash <NUM> could be of a first color that is CW, "WW" refers to "warm white" indicating that the second flash <NUM> could be of a second color that is WW, while "a" refers to "ambient.

In particular, <FIG> illustrates that, in some embodiments, the correction factor γ may be computed with reference to the horizontal axis of the plot <NUM>. In particular, <FIG> illustrates that the correction factor may be computed based on where the ambient color point <NUM> is located, along the horizontal axis of the plot <NUM>, with respect to the location of the first flash color point <NUM> and the location of the second flash color point <NUM> along the horizontal axis. To that end, <FIG> illustrates the coordinates of the color points <NUM>, <NUM>, and <NUM> along the horizontal axis, and these values may be used to compute the correction factor γ as follows: <MAT>.

<FIG> illustrates that, in some embodiments, the correction factor γ may be computed with reference to the vertical axis of the plot <NUM>. In particular, <FIG> illustrates that the correction factor may be computed based on where the ambient color point <NUM> is located, along the vertical axis of the plot <NUM>, with respect to the location of the first flash color point <NUM> and the location of the second flash color point <NUM> along the vertical axis. To that end, <FIG> illustrates the coordinates of the color points <NUM>, <NUM>, and <NUM> along the vertical axis, and these values may be used to compute the correction factor γ as follows: <MAT>.

Finally, <FIG> illustrates that, in some embodiments, the correction factor γ may be computed with reference to the straight line connecting the first flash color point <NUM> and the second flash color point <NUM> of the plot <NUM>. In particular, <FIG> illustrates that the correction factor may be computed based on where the ambient color point <NUM> is located, along the straight line connecting the first flash color point <NUM> and the second flash color point <NUM> in the plot <NUM>, with respect to the location of the first flash color point <NUM> and the location of the second flash color point <NUM> along that line. To that end, <FIG> illustrates the coordinates of the color points <NUM> and <NUM>, and the coordinates (R'a/G'a;B'a/G'a) of the projection (point <NUM>, labeled in <FIG>) of the ambient color point <NUM> onto the line connecting the color points <NUM> and <NUM>. In such embodiments, the correction factor γ may be computed as follows: <MAT>.

In each of these examples, the correction factor γ may be a value between <NUM> and <NUM>, and it may be applied to the first and second images in any of the ways described above. Furthermore, while <FIG> illustrates identifying the position of the ambient color point with respect to the straight line connecting the color points <NUM> and <NUM>, in other embodiments, the ambient color point <NUM> could be referenced to a curved line, or some other line that is not straight, between the color points <NUM> and <NUM>. Still further, when flashes of more than two colors are used, the ambient color point could be determined with respect to correspondingly more than two color points.

<FIG> provides a block diagram illustrating a data processing system <NUM> that may be configured to implement at least portions of camera devices with multi-flash with image post-processing as described herein, e.g., of the camera devices as described with reference to <FIG>.

As shown in <FIG>, the data processing system <NUM> may include at least one processor <NUM>, e.g. a hardware processor <NUM>, coupled to memory elements <NUM> through a system bus <NUM>. It should be appreciated, however, that the data processing system <NUM> may be implemented in the form of any system including a processor and a memory that is capable of performing the functions described within this disclosure.

The processor <NUM> can execute software or an algorithm to perform the activities as discussed in this specification, in particular activities related to multi-flash with image post-processing described herein. The processor <NUM> may include any combination of hardware, software, or firmware providing programmable logic, including by way of non-limiting example a microprocessor, a DSP, a field-programmable gate array (FPGA), a programmable logic array (PLA), an integrated circuit (IC), an application specific IC (ASIC), or a virtual machine processor. The processor <NUM> may be communicatively coupled to the memory element <NUM>, for example in a direct-memory access (DMA) configuration, so that the processor <NUM> may read from or write to the memory elements <NUM>.

In general, the memory elements <NUM> may include any suitable volatile or nonvolatile memory technology, including double data rate (DDR) random access memory (RAM), synchronous RAM (SRAM), dynamic RAM (DRAM), flash, read-only memory (ROM), optical media, virtual memory regions, magnetic or tape memory, or any other suitable technology. Unless specified otherwise, any of the memory elements discussed herein should be construed as being encompassed within the broad term "memory. " The information being measured, processed, tracked or sent to or from any of the components of the data processing system <NUM> could be provided in any database, register, control list, cache, or storage structure, all of which can be referenced at any suitable timeframe. Any such storage options may be included within the broad term "memory" as used herein. Similarly, any of the potential processing elements, modules, and machines described herein should be construed as being encompassed within the broad term "processor. " Each of the elements shown in the present figures, e.g., any of the circuits/components shown in <FIG>, can also include suitable interfaces for receiving, transmitting, and/or otherwise communicating data or information in a network environment so that they can communicate with, e.g., the data processing system <NUM> of another one of these elements.

In certain implementations, mechanisms for implementing multi-flash with image post-processing in camera devices as outlined herein may be implemented by logic encoded in one or more tangible media, which may be inclusive of non-transitory media, e.g., embedded logic provided in an ASIC, in DSP instructions, software (potentially inclusive of object code and source code) to be executed by a processor, or other similar machine, etc. In some of these instances, memory elements, such as e.g. the memory elements <NUM> shown in <FIG>, can store data or information used for the operations described herein. This includes the memory elements being able to store software, logic, code, or processor instructions that are executed to carry out the activities described herein. A processor can execute any type of instructions associated with the data or information to achieve the operations detailed herein. The processors, such as e.g. the processor <NUM> shown in <FIG>, could transform an element or an article (e.g., data) from one state or thing to another state or thing. In another example, the activities outlined herein may be implemented with fixed logic or programmable logic (e.g., software/computer instructions executed by a processor) and the elements identified herein could be some type of a programmable processor, programmable digital logic (e.g., an FPGA, a DSP, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM)) or an ASIC that includes digital logic, software, code, electronic instructions, or any suitable combination thereof.

The local memory may refer to RAM or other non-persistent memory device(s) generally used during actual execution of the program code.

As shown in <FIG>, the memory elements <NUM> may store an application <NUM>. The application <NUM> may be stored in the local memory <NUM>, the one or more bulk storage devices <NUM>, or apart from the local memory and the bulk storage devices.

Input/output (I/O) devices depicted as an input device <NUM> and an output device <NUM>, optionally, can be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, or the like. The output device <NUM> may be any type of screen display, such as plasma display, liquid crystal display (LCD), organic light emitting diode (OLED) display, electroluminescent (EL) display, or any other indicator, such as a dial, barometer, or light emitting diode (LED). In some implementations, the system may include a driver (not shown) for the output device <NUM>. Input and/or output devices <NUM>, <NUM> may be coupled to the data processing system either directly or through intervening I/O controllers.

The input and the output devices may be implemented as a combined input/output device (illustrated in <FIG> with a dashed line surrounding the input device <NUM> and the output device <NUM>). The input to the device may be provided by a movement of a physical object, such as e.g. a stylus or a finger of a user, on or near the touch screen display.

A network adapter <NUM> may also, optionally, be coupled to the data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and/or remote storage devices through intervening private or public networks.

It is to be understood that not necessarily all objects or advantages may be achieved in accordance with any particular embodiment described herein. Thus, for example, those skilled in the art will recognize that certain embodiments may be configured to operate in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other objects or advantages as may be taught or suggested herein.

It should be appreciated that the electrical circuits of the accompanying drawings and its teachings are readily scalable and can accommodate a large number of components, as well as more complicated/sophisticated arrangements and configurations. Accordingly, the examples provided should not limit the scope or inhibit the broad teachings of the electrical circuits as potentially applied to a myriad of other architectures.

Any number of electrical circuits of the accompanying drawings may be implemented on a board of an associated electronic device. The board can be a general circuit board that can hold various components of the internal electronic system of the electronic device and, further, provide connectors for other peripherals. More specifically, the board can provide the electrical connections by which the other components of the system can communicate electrically. Any suitable processors (inclusive of digital signal processors, microprocessors, supporting chipsets, etc.), computer-readable non-transitory memory elements, etc. can be suitably coupled to the board based on particular configuration needs, processing demands, computer designs, etc. Other components such as external storage, additional sensors, controllers for audio/video display, and peripheral devices may be attached to the board as plug-in cards, via cables, or integrated into the board itself. The functionalities described herein may be implemented in emulation form as software or firmware running within one or more configurable (e.g., programmable) elements arranged in a structure that supports these functions. The software or firmware providing the emulation may be provided on non-transitory computer-readable storage medium comprising instructions to allow a processor to carry out those functionalities.

The electrical circuits of, or associated with, the accompanying drawings may be implemented as stand-alone modules (e.g., a device with associated components and circuitry configured to perform a specific application or function) or implemented as plug-in modules into application specific hardware of electronic devices. Note that some embodiments of the present disclosure may be readily included in a system on chip (SOC) package, either in part, or in whole. An SOC represents an integrated circuit (IC) that integrates components of a computer or other electronic system into a single chip. It may contain digital, analog, mixed-signal, and often radio frequency functions: all of which may be provided on a single chip substrate. Other embodiments may include a multi-chip-module (MCM), with a plurality of separate ICs located within a single electronic package and configured to interact closely with each other through the electronic package. In various other embodiments, at least some aspects of the multi-flash with image post-processing may be implemented in one or more silicon cores in Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), and other semiconductor chips.

Claim 1:
A camera device, comprising:
a controller, configured to
cause a camera to acquire a first image of a scene while the scene is being illuminated with a first set of light sources, the first set of light sources for causing the first image to be associated with a first color point;
cause the camera to acquire a second image of the scene while the scene is being illuminated with a second set of light sources, the second set of light sources for causing the second image to be associated with a second color point; illumination provided by the second set of light sources having a different color than illumination provided by the first set of light sources, wherein the first set of light sources and the second set of light sources provide flash illumination of the scene;obtain a third color point, the third color point associated with the scene not being illuminated with the first set of light sources and not being illuminated with the second set of light sources, wherein each of the first color point, the second color point, and the third color point has a respective location in a plot having a first axis and a second axis, the first axis indicating values of a ratio of a sum of at least a sub-set of pixel values of a first primary color to a sum of at least a sub-set of pixel values of a third primary color, and the second axis indicating values of a ratio of a sum of at least a sub-set of pixel values of a second primary color to the sum of at least the sub-set of pixel values of the third primary color; and
generate a final image of the scene based on a combination of the first image and the second image characterised by
computing a correction factor based on at least a value indicative of the location of the first color point, at least on a value indicative of the location of the second color point, and at least on a value indicative of the location of the third color point,
generating a modified first image by modifying each of pixel values of the first image by a first value based on the correction factor,
generating a modified second image by modifying each of pixel values of the second image by a second value based on the correction factor, and
generating the final image based on pixel values of the modified first image and pixel values of the modified second image.