Patent Description:
Many mobile electronic devices, such as smartphones and tablet computers, include cameras that can be used to capture still and video images. While convenient, cameras on mobile electronic devices typically suffer from a number of shortcomings, including poor performance in low-light situations. For example, some mobile electronic devices simply use a flash when capturing low-light images. However, the flashes used in mobile electronic devices typically act as point sources of bright light (not diffuse sources of light), so the use of a flash typically causes over-exposure or "blow out" for nearby people or objects and under-exposure of the background. In other words, the use of a flash creates non-uniform radiance in the images, resulting in low aesthetic quality. The captured images also tend to have a bluish cast, which is not constant across the images and therefore not easily removable. Other mobile electronic devices attempt to combine multiple images together to produce more aesthetically-pleasing images. However, these approaches often suffer from unnatural saturation artifacts, ghosting artifacts, color twisting, bluish color casts, or noise. <CIT> relates to a method in which a plurality of images are taken in order to achieve an image with improved pixel exposure.

This disclosure provides an apparatus and method for capturing and blending multiple images for high-quality flash photography using a mobile electronic device.

In a first embodiment, a method according to claim <NUM> includes capturing multiple ambient images of a scene using at least one camera of an electronic device and without using a flash of the electronic device. The method also includes capturing multiple flash images of the scene using the at least one camera of the electronic device and during firing of a pilot flash sequence using the flash. The method further includes analyzing multiple pairs of images to estimate exposure differences obtained using the flash, where each pair of images includes one of the ambient images and one of the flash images that are both captured using a common camera exposure and where different pairs of images are captured using different camera exposures. In addition, the method includes determining a flash strength for the scene based on the estimate of the exposure differences and firing the flash based on the determined flash strength.

In a second embodiment, an electronic device according to claim <NUM> includes at least one camera, a flash, and at least one processing device. The at least one processing device is configured to capture multiple ambient images of a scene using the at least one camera and without using the flash. The at least one processing device is also configured to capture multiple flash images of the scene using the at least one camera and during firing of a pilot flash sequence using the flash. The at least one processing device is further configured to analyze multiple pairs of images to estimate exposure differences obtained using the flash, where each pair of images includes one of the ambient images and one of the flash images that are both captured using a common camera exposure and where different pairs of images are captured using different camera exposures. In addition, the at least one processing device is configured to determine a flash strength for the scene based on the estimate of the exposure differences and fire the flash based on the determined flash strength.

In a third embodiment, a non-transitory machine-readable medium according to claim <NUM> contains instructions that when executed cause at least one processor of an electronic device to capture multiple ambient images of a scene using at least one camera of the electronic device and without using a flash of the electronic device. The medium also contains instructions that when executed cause the at least one processor of the electronic device to capture multiple flash images of the scene using the at least one camera of the electronic device and during firing of a pilot flash sequence using the flash. The medium further contains instructions that when executed cause the at least one processor of the electronic device to analyze multiple pairs of images to estimate exposure differences obtained using the flash, where each pair of images includes one of the ambient images and one of the flash images that are both captured using a common camera exposure and where different pairs of images are captured using different camera exposures. In addition, the medium contains instructions that when executed cause the at least one processor of the electronic device to determine a flash strength for the scene based on the estimate of the exposure differences and fire the flash based on the determined flash strength.

For a more complete understanding of this disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts:.

As used here, terms and phrases such as "have," "may have," "include," or "may include" a feature (like a number, function, operation, or component such as a part) indicate the existence of the feature and do not exclude the existence of other features. Also, as used here, the phrases "A or B," "at least one of A and/or B," or "one or more of A and/or B" may include all possible combinations of A and B. For example, "A or B," "at least one of A and B," and "at least one of A or B" may indicate all of (<NUM>) including at least one A, (<NUM>) including at least one B, or (<NUM>) including at least one A and at least one B. Further, as used here, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate different user devices from each other, regardless of the order or importance of the devices. A first component may be denoted a second component and vice versa without departing from the scope of this disclosure.

It will be understood that, when an element (such as a first element) is referred to as being (operatively or communicatively) "coupled with/to" or "connected with/to" another element (such as a second element), it can be coupled or connected with/to the other element directly or via a third element. In contrast, it will be understood that, when an element (such as a first element) is referred to as being "directly coupled with/to" or "directly connected with/to" another element (such as a second element), no other element (such as a third element) intervenes between the element and the other element.

As used here, the phrase "configured (or set) to" may be interchangeably used with the phrases "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of" depending on the circumstances. The phrase "configured (or set) to" does not essentially mean "specifically designed in hardware to. " Rather, the phrase "configured to" may mean that a device can perform an operation together with another device or parts. For example, the phrase "processor configured (or set) to perform A, B, and C" may mean a generic-purpose processor (such as a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (such as an embedded processor) for performing the operations.

The terms and phrases as used here are provided merely to describe some embodiments thereof, but not to limit the scope of other embodiments of this disclosure. All terms and phrases, including technical and scientific terms and phrases, used here have the same meanings as commonly understood by one of ordinary skill in the art to which the embodiments of this disclosure belong. It will be further understood that terms and phrases, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined here. In some cases, the terms and phrases defined here may be interpreted to exclude embodiments of this disclosure.

Examples of an "electronic device" according to embodiments of this disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic appcessory, an electronic tattoo, a smart mirror, or a smart watch). Definitions for other certain words and phrases may be provided throughout this patent document.

None of the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claim scope. The scope of patented subject matter is defined only by the claims.

<FIG>, discussed below, and the various embodiments of this disclosure are described with reference to the accompanying drawings. However, it should be appreciated that this disclosure is not limited to these embodiments, and all changes and/or equivalents or replacements thereto also belong to the scope of this disclosure. The same or similar reference denotations may be used to refer to the same or similar elements throughout the specification and the drawings.

As noted above, many mobile electronic devices suffer from a number of shortcomings, including poor performance in low-light situations. Some mobile electronic devices simply use a flash for capturing low-light images, which typically results in non-uniform radiance and low aesthetic quality. Other mobile electronic devices attempt to combine multiple images together to produce more aesthetically-pleasing images but suffer from problems such as unnatural saturation artifacts, ghosting artifacts, color twisting, bluish color casts, or noise.

This disclosure provides techniques for using multiple images captured using a flash by combining principles of multi-frame high dynamic range (HDR) imaging, where camera exposure settings are adjusted to capture multiple images in the presence of the flash. This is accomplished by analyzing pairs of images captured by an electronic device to determine how to control a flash of the electronic device. Multiple images are then captured by the electronic device based on the flash control, and those images are processed and blended to produce a final image having a more uniform radiance. This may allow, for example, more aesthetically-pleasing images having more natural colors in low-light situations to be produced. These images may suffer from little or no blow-out and may have backgrounds and foregrounds that are more evenly illuminated. These images may also suffer from less saturation artifacts, ghosting artifacts, color twisting, bluish color casts, or noise.

<FIG> illustrates an example network configuration <NUM> including an electronic device in accordance with this disclosure. The embodiment of the network configuration <NUM> shown in <FIG> is for illustration only. Other embodiments of the network configuration <NUM> could be used without departing from the scope of this disclosure.

According to embodiments of this disclosure, an electronic device <NUM> is included in the network environment <NUM>. The electronic device <NUM> can include at least one of a bus <NUM>, a processor <NUM>, a memory <NUM>, an input/output (I/O) interface <NUM>, a display <NUM>, a communication interface <NUM>, or a sensor <NUM>. In some embodiments, the electronic device <NUM> may exclude at least one of these components or may add at least one other component. The bus <NUM> includes a circuit for connecting the components <NUM>-<NUM> with one another and for transferring communications (such as control messages and/or data) between the components.

The processor <NUM> includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor <NUM> is able to perform control on at least one of the other components of the electronic device <NUM> and/or perform an operation or data processing relating to communication. In some embodiments, the processor <NUM> can be a graphics processor unit (GPU). For example, the processor <NUM> can receive image data captured by at least one camera during a capture event. The processor <NUM> can process the image data (as discussed in more detail below) to perform multi-pair image analysis and multi-scale blending.

The memory <NUM> can include a volatile and/or non-volatile memory. For example, the memory <NUM> can store commands or data related to at least one other component of the electronic device <NUM>. According to embodiments of this disclosure, the memory <NUM> can store software and/or a program <NUM>. The program <NUM> includes, for example, a kernel <NUM>, middleware <NUM>, an application programming interface (API) <NUM>, and/or an application program (or "application") <NUM>. At least a portion of the kernel <NUM>, middleware <NUM>, or API <NUM> may be denoted an operating system (OS).

The kernel <NUM> can control or manage system resources (such as the bus <NUM>, processor <NUM>, or memory <NUM>) used to perform operations or functions implemented in other programs (such as the middleware <NUM>, API <NUM>, or application program <NUM>). The kernel <NUM> provides an interface that allows the middleware <NUM>, the API <NUM>, or the application <NUM> to access the individual components of the electronic device <NUM> to control or manage the system resources. The application <NUM> includes one or more applications for image capture as discussed below. These functions can be performed by a single application or by multiple applications that each carries out one or more of these functions. The middleware <NUM> can function as a relay to allow the API <NUM> or the application <NUM> to communicate data with the kernel <NUM>, for instance. A plurality of applications <NUM> can be provided. The middleware <NUM> is able to control work requests received from the applications <NUM>, such as by allocating the priority of using the system resources of the electronic device <NUM> (like the bus <NUM>, the processor <NUM>, or the memory <NUM>) to at least one of the plurality of applications <NUM>. The API <NUM> is an interface allowing the application <NUM> to control functions provided from the kernel <NUM> or the middleware <NUM>. For example, the API <NUM> includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.

The I/O interface <NUM> serves as an interface that can, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic device <NUM>. The I/O interface <NUM> can also output commands or data received from other component(s) of the electronic device <NUM> to the user or the other external device.

The display <NUM> includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display <NUM> can also be a depth-aware display, such as a multi-focal display. The display <NUM> is able to display, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display <NUM> can include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.

The communication interface <NUM>, for example, is able to set up communication between the electronic device <NUM> and an external electronic device (such as a first electronic device <NUM>, a second electronic device <NUM>, or a server <NUM>). For example, the communication interface <NUM> can be connected with a network <NUM> or <NUM> through wireless or wired communication to communicate with the external electronic device. The communication interface <NUM> can be a wired or wireless transceiver or any other component for transmitting and receiving signals, such as images.

The electronic device <NUM> further includes one or more sensors <NUM> that can meter a physical quantity or detect an activation state of the electronic device <NUM> and convert metered or detected information into an electrical signal. For example, one or more sensors <NUM> can include one or more buttons for touch input, one or more cameras, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red green blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasound sensor, an iris sensor, or a fingerprint sensor. The sensor(s) <NUM> can also include an inertial measurement unit, which can include one or more accelerometers, gyroscopes, and other components. The sensor(s) <NUM> can further include a control circuit for controlling at least one of the sensors included here. Any of these sensor(s) <NUM> can be located within the electronic device <NUM>. The one or more cameras can capture images as discussed below and are used in conjunction with at least one flash <NUM>. The flash <NUM> represents a device configured to generate illumination for use in image capture by the electronic device <NUM>, such as one or more LEDs.

The first external electronic device <NUM> or the second external electronic device <NUM> can be a wearable device or an electronic device-mountable wearable device (such as an HMD). When the electronic device <NUM> is mounted in the electronic device <NUM> (such as the HMD), the electronic device <NUM> can communicate with the electronic device <NUM> through the communication interface <NUM>. The electronic device <NUM> can be directly connected with the electronic device <NUM> to communicate with the electronic device <NUM> without involving with a separate network. The electronic device <NUM> can also be an augmented reality wearable device, such as eyeglasses, that include one or more cameras.

The wireless communication is able to use at least one of, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (<NUM>), millimeter-wave or <NUM> wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection can include, for example, at least one of a universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard <NUM> (RS-<NUM>), or plain old telephone service (POTS). The network <NUM> includes at least one communication network, such as a computer network (like a local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The first and second external electronic devices <NUM> and <NUM> and server <NUM> each can be a device of the same or a different type from the electronic device <NUM>. According to certain embodiments of this disclosure, the server <NUM> includes a group of one or more servers. Also, according to certain embodiments of this disclosure, all or some of the operations executed on the electronic device <NUM> can be executed on another or multiple other electronic devices (such as the electronic devices <NUM> and <NUM> or server <NUM>). Further, according to certain embodiments of this disclosure, when the electronic device <NUM> should perform some function or service automatically or at a request, the electronic device <NUM>, instead of executing the function or service on its own or additionally, can request another device (such as electronic devices <NUM> and <NUM> or server <NUM>) to perform at least some functions associated therewith. The other electronic device (such as electronic devices <NUM> and <NUM> or server <NUM>) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device <NUM>. The electronic device <NUM> can provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example. While <FIG> shows that the electronic device <NUM> includes the communication interface <NUM> to communicate with the external electronic device <NUM> or server <NUM> via the network <NUM>, the electronic device <NUM> may be independently operated without a separate communication function according to some embodiments of this disclosure.

The server <NUM> can optionally support the electronic device <NUM> by performing or supporting at least one of the operations (or functions) implemented on the electronic device <NUM>. For example, the server <NUM> can include a processing module or processor that may support the processor <NUM> implemented in the electronic device <NUM>.

Although <FIG> illustrates one example of a network configuration <NUM> including an electronic device <NUM>, various changes may be made to <FIG>. For example, the network configuration <NUM> could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. Also, while <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

<FIG> illustrates an example process <NUM> for multi-pair image analysis for flash control in a mobile electronic device in accordance with this disclosure. For ease of explanation, the process <NUM> shown in <FIG> is described as being performed using the electronic device <NUM> of <FIG>. However, the process <NUM> shown in <FIG> could be used with any other suitable electronic device and in any suitable system.

The process <NUM> is generally used to identify and make intelligent adjustments to the flash strength of the flash <NUM> and possibly other parameters such as camera exposure and number of frames. The flash strength and optionally the other parameters that are identified using the process <NUM> can then be used as described below to support multi-scale blending of images (which is described with respect to <FIG>). The flash strength is identified here using pilot flash scene analysis/subject detection. A pilot flash sequence generally refers to one or more flashes that occur prior to the main flash used to produce a final image of a scene. A pilot flash sequence is often used in electronic devices for other purposes as well, such as red-eye reduction, and may include a single flash or multiple flashes. The scene analysis/subject detection can be used to identify object types in a scene, a distance to a closest object, a scene type (such as indoor or outdoor, night or day, or macro or wide-angle), or other characteristics of the scene. The determined flash strength and optionally the other parameters are determined using this information.

As shown in <FIG>, a capture request <NUM> is received by the electronic device <NUM>. The capture request <NUM> represents any suitable command or input indicating a need or desire to capture an image of a scene using the electronic device <NUM>. For example, the capture request <NUM> could be initiated in response to a user's pressing of a "soft" button presented on the display <NUM> or the user's pressing of a "hard" button. In response to the capture request <NUM>, the processor <NUM> performs a capture operation <NUM> using the camera of the electronic device <NUM> to capture multiple ambient images <NUM> of the scene. An ambient image generally refers to an image of a scene in which little or no light from the electronic device <NUM> is illuminating the scene, so the flash <NUM> may not be used in the capture of the ambient images <NUM>. In some instances, during the capture operation <NUM>, the processor <NUM> can control the camera of the electronic device <NUM> so that the ambient images <NUM> are captured rapidly in a burst mode. Different ambient images <NUM> can be captured using different camera exposures.

In response to the capture request <NUM>, the processor <NUM> also performs a capture operation <NUM> using the camera of the electronic device <NUM> to capture multiple flash images <NUM> of the scene. A flash image generally refers to an image of a scene in which light from the electronic device <NUM> is illuminating the scene, so the flash <NUM> is used in the capture of the flash images <NUM>. In some instances, during the capture operation <NUM>, the processor <NUM> can control the camera of the electronic device <NUM> so that the flash images <NUM> are captured rapidly in a burst mode. The flash <NUM> is used here to generate the pilot flash sequence, and the flash images <NUM> may be captured using a common flash strength. The flash strength used here may denote a default flash strength or other flash strength used by the camera or the electronic device <NUM>. Different flash images <NUM> can be captured using different camera exposures.

In this example, the ambient images <NUM> and the flash images <NUM> form multiple ambient-flash image pairs. That is, the processor <NUM> can control the camera so that multiple pairs of images are obtained, where each pair includes one ambient image <NUM> captured without using the flash <NUM> and one flash image <NUM> captured using the flash <NUM>. Each image pair can be captured using a common camera exposure and camera sensitivity (ISO setting), and different image pairs can be captured using different camera exposures or camera sensitivities. It should be noted, however, that there is no need to capture the images in each image pair consecutively. The ambient images <NUM> and the flash images <NUM> can be captured in any suitable order, as long as the processor <NUM> obtains multiple ambient-flash image pairs.

The images <NUM> and <NUM> are used by the processor <NUM> in an analysis operation <NUM> to identify the exposure differences that are obtained in the scene using the flash <NUM>. The analysis operation <NUM> occurs in order to quantify the exposure differences that are obtained using the flash <NUM> and the different camera exposures/camera sensitivities. In this way, the analysis operation <NUM> can identify the exposure differences between ambient lighting and flash lighting in a scene, which could occur in any suitable manner (such as at the pixel level in the images or for the foreground or one or more objects in the images). This information can then be used to identify the ideal or desired flash strength for capturing an image of the scene. This information can also be used to perform other functions, such as color correction. The analysis operation <NUM> includes any suitable operations to identify exposure differences between images. Two example implementations of the analysis operation <NUM> are described below, although other implementations of the analysis operation <NUM> could also be used. One benefit of using multiple pairs of ambient/flash images is that the resulting analysis is more robust to over-exposed and under-exposed regions of the images, yielding a more accurate estimate of the exposure differences from the use of the flash <NUM>.

The exposure differences identified by the analysis operation <NUM> are used by the processor <NUM> during a mapping operation <NUM> to map the exposure differences to a suitable flash strength. The mapping here essentially translates the exposure differences into a suitable strength for the flash <NUM> to be used when capturing subsequent images of the scene. Here, the mapping can consider various aspects of the exposure differences, such as sizes of foreground regions/objects in the scene and the sizes of background regions in the scene. The mapping can also consider the types of objects in the scene, such as whether the scene appears to include at least one person or one or more inanimate objects. The mapping can further be based on an estimated distance to the closest object in the scene. In addition, the mapping can be based on whether the image is being captured indoors or outdoors, at night or during the day, or using a macro lens or a wide-angle lens. The specific mappings used can vary based on a number of circumstances, such as the design of the camera being used in the electronic device <NUM>.

The identified flash strength can optionally be used by the processor <NUM> during a color cast determination operation <NUM>. During this operation <NUM>, the processor <NUM> attempts to estimate the regions of any subsequent images where blue casting or other color casting may form as a result of the use of the flash <NUM> at the identified flash strength. This information can be useful in later processing of the subsequent images to remove the casting from the subsequent images. The identification of the regions performed here can be based on the exposure differences identified by the analysis operation <NUM> and can identify the likely areas where casting may occur based on the exposure differences.

A modulated flash firing <NUM> occurs using the identified flash strength. For example, when the processor <NUM> is ready to capture additional images of the scene in order to produce a final image of the scene, the processor <NUM> can trigger the flash <NUM>. The additional images of the scene are then captured by the camera of the electronic device <NUM> while the scene is being illuminated using the flash <NUM>, which operates at the identified flash strength. Ideally, the use of the identified flash strength allows the additional images to then be blended or otherwise processed to provide a more uniform illumination in the final image of the scene.

<FIG> and <FIG> illustrate example analysis operations <NUM> for analyzing exposure differences in the process <NUM> of <FIG> in accordance with this disclosure. In particular, <FIG> illustrates an example implementation of the analysis operation <NUM> in which the analysis is performed using a prior model on a per-pixel basis, and <FIG> illustrates an example implementation of the analysis operation <NUM> in which the analysis is performed using artificial intelligence. Of course, other implementations of the analysis operation <NUM> are also possible and fall within the scope of this disclosure.

As shown in <FIG>, two images <NUM> and <NUM> in an image pair are being analyzed. The images here include one ambient image <NUM> (representing one of the images <NUM>) and one flash image <NUM> (representing one of the images <NUM>). The images <NUM> and <NUM> are subject to a division operation <NUM>, which divides the value of each pixel in one image <NUM> or <NUM> by the value of the corresponding pixel in the other image <NUM> or <NUM>. The quotient values resulting from the division are subjected to a logarithmic operation <NUM> (a log2 operation in this example) to convert the quotient values into the logarithmic domain. A rectifier linear unit <NUM> operates to prevent the values in the logarithmic domain from being negative, such as by selecting (for each value in the logarithmic domain) the greater of that value or zero. The operations <NUM>, <NUM>, and <NUM> here can be performed for each pair of ambient/flash images captured by the electronic device <NUM> during the capture operations <NUM> and <NUM>.

Because the ambient/flash images can be captured by the electronic device <NUM> using different camera exposures and/or camera sensitivities, different images may often have resulting data that is reliable in some areas and not reliable in other areas. The data resulting from the operations <NUM>, <NUM>, and <NUM> for the different pairs of ambient/ flash images can therefore be averaged in an averaging operation <NUM>, which averages the values obtained for the different camera exposures/camera sensitivities. The averaged values are passed through an edge-preserving filter <NUM>, which smooths out the averaged data and reduces noise while preserving edges within the averaged data. The edges could denote the edges of one or more people or objects in the foreground of the images or in the background of the images. Various types of edge-preserving filters are known in the art. In some embodiments, the edge-preserving filter <NUM> could represent a bilateral filter, which operates to replace the intensity of each average pixel with a weighted average of intensity values from nearby average pixels. Note, however, that other implementations of the edge-preserving filter <NUM> could be used.

The outputs of the edge-preserving filter <NUM> are the exposure differences <NUM> obtained through the use of the flash <NUM>. The exposure differences could be expressed in any suitable manner. In some embodiments, for example, the exposure differences can be expressed as a grayscale image, where darker pixels in the grayscale image identify areas where the exposure differences were smaller and brighter pixels in the grayscale image identify areas where the exposure differences were larger. For instance, if the original ambient and flash images <NUM> and <NUM> included a person in the foreground and a dark background, the grayscale image would likely include many white pixels in the area of the images where the person was located, since the illumination from the flash <NUM> would greatly improve the brightness of the person in the flash images. In contrast, the grayscale image would likely include many dark pixels in the area of the images where the background was located, since the illumination from the flash <NUM> may not improve (or would only slightly improve) the brightness of the background in the flash images.

As shown in <FIG>, once again, two images <NUM> and <NUM> of each image pair are analyzed. In this example, however, the pairs of images <NUM> and <NUM> are passed through a convolutional neural network (CNN) <NUM>. A convolutional neural network <NUM> generally represents a type of deep artificial neural network, and convolutional neural networks are often applied to analyzing images. In the convolutional neural network <NUM>, layers of convolutional neurons apply a convolution operation that emulates the response of individual neurons to visual stimuli. Each neuron typically applies some function to its input values (often by weighting different input values differently) to generate output values. Pooling layers can be used to combine the output values of neuron clusters from one layer into input values for single neurons in another layer.

The convolutional neural network <NUM> here can be used to process the image pairs and generate exposure differences (such as in the form of a grayscale image). This can be accomplished by training the convolutional neural network <NUM> so that the weights of the neurons have appropriate values. The convolutional neural network <NUM> can also be trained to perform other functions, such as specularity removal (the removal of small bright spots where distant specular surfaces in the background might still yield a strong response to a flash) and de-ghosting (the removal of movement from one image to another).

Although <FIG> illustrates one example of a process <NUM> for multi-pair image analysis for flash control in a mobile electronic device and <FIG> and <FIG> illustrate examples of analysis operations <NUM> for analyzing exposure differences in the process <NUM> of <FIG>, various changes may be made to <FIG>, <FIG>, and <FIG>. For example, while shown as sequences of steps, various operations shown in <FIG>, <FIG>, and <FIG> could overlap, occur in parallel, occur in a different order, or occur any number of times. As a particular example, the operations <NUM> and <NUM> in <FIG> could be reversed or interleaved so that the ambient images <NUM> and the flash images <NUM> are captured in a different order than that shown here. Also, the specific analyses shown in <FIG> and <FIG> are examples only, and other techniques could be used to identify exposure differences involving any number of images.

<FIG> illustrates an example process <NUM> for multi-scale blending of images in a mobile electronic device in accordance with this disclosure. For ease of explanation, the process <NUM> shown in <FIG> is described as being performed using the electronic device <NUM> of <FIG>. However, the process <NUM> shown in <FIG> could be used with any other suitable electronic device and in any suitable system.

The process <NUM> is generally used to capture multiple images of a scene using different camera exposures at the same flash strength, namely the flash strength determined using the process <NUM> described above. In some embodiments, the different camera exposures can be achieved by varying the camera's sensor gain and exposure time. Generally, the electronic device <NUM> can capture one or more images having shorter exposures, one or more images having longer exposures, and optionally one or more images having mid-range exposures between the shorter and longer exposures. The images are then aligned geometrically and photometrically, and one of the images (often a mid- or longer-exposure image) is selected as a reference. The images are blended to, among other things, replace one or more blown-out regions in the reference image with one or more regions based on or extracted from other images (often the shorter-exposure images). Motion in the images can also be estimated in order to remove ghosting artifacts, and other processing can occur to improve the final image of the scene.

As shown in <FIG>, a collection <NUM> of images is captured using the camera of the electronic device <NUM>. Here, the collection <NUM> includes at least three images <NUM>, <NUM>, and <NUM>, each of which can be captured using a different exposure but a common flash strength. For example, the image <NUM> could be captured using the shortest exposure, the image <NUM> could be captured using the longest exposure, and the image <NUM> could be captured using an intermediate exposure between the shortest and longest exposures. Note, however, that other numbers of images (including two images or more than three images) could be captured and other numbers of exposures (including two exposures or more than three exposures) could be used. One or multiple images could be captured at each exposure, and there is no requirement that an equal number of images be captured per exposure.

The image collection <NUM> is provided to an image registration operation <NUM>, which generally operates to align the images <NUM>, <NUM>, and <NUM>. Alignment may be needed if the electronic device <NUM> moves or rotates in between image captures and causes objects in the images to move or rotate slightly, which is common with handheld devices. The images <NUM>, <NUM>, and <NUM> here can be aligned both geometrically and photometrically. In some embodiments, the image registration operation <NUM> can use global Oriented FAST and Rotated BRIEF (ORB) features as local features and global features from a block search to align the images. One example implementation of the image registration operation <NUM> is described below, although other implementations of the image registration operation <NUM> could also be used.

The aligned images are output and processed using an exposure analysis operation <NUM> and a de-ghosting operation <NUM>. The exposure analysis operation <NUM> analyzes the aligned images to generate well-exposedness maps for the aligned images. Each well-exposedness map generally identifies the area or areas of one of the aligned images that are well-exposed (not over-exposed or under-exposed). Different metrics can be used to define the well-exposed portions of the images based on the camera exposures used to capture those images. For instance, different metrics can be defined by different functions, where the functions convert pixel values into well-exposedness values and where the functions are applied to different aligned images. One example implementation of the exposure analysis operation <NUM> is described below, although other implementations of the exposure analysis operation <NUM> could also be used.

The de-ghosting operation <NUM> processes the aligned images to identify motion occurring in the images, such as people or objects moving within the images. In some embodiments, the de-ghosting operation <NUM> divides each of the aligned images into tiles, such as sixteen tiles arranged in a four-by-four grid. The de-ghosting operation <NUM> then processes the tiles to identify motion, where the motion is identified as differences between the tiles. In this way, the de-ghosting operation <NUM> generates motion maps to identify areas in the images where motion is occurring. For instance, each motion map could include black pixels indicating where no motion is detected and white pixels indicating where motion is detected. The de-ghosting operation <NUM> can also equalize the images to account for the different camera exposures/camera sensitivities used to capture the images. One example implementation of the de-ghosting operation <NUM> is described below, although other implementations of the de-ghosting operation <NUM> could also be used.

A multi-scale image blending operation <NUM> receives the aligned images, the well-exposedness maps, and the motion maps and uses this information to generate one or more blended images. Each blended image can include or be based on portions of different images. For example, a blended image could be formed by selecting one of the images (such as an image <NUM> captured with an intermediate exposure) as a reference image and replacing blown-out or other portions of the reference image using or based on corresponding portions from other images. As a particular example, over-exposed portions of the image <NUM> can typically be replaced with or using corresponding portions of the image <NUM> when the image <NUM> is captured using a shorter exposure. The blending can also account for motion in the images, such as by avoiding the insertion of a moving object from one image in the wrong position in the reference image. In some embodiments, the blending represents a weighted blending of synthesized images across multiple scales, where blending maps are used as the weights and are based on a composite of the well-exposedness maps and de-ghosting maps. For instance, each of the blending maps could represent a product of one of the well-exposedness maps and one of the de-ghosting maps. One example implementation of the multi-scale blending operation <NUM> is described below, although other implementations of the multi-scale blending operation <NUM> could also be used.

Each blended image can then be subjected to one or more post-processing operations in order to improve the blended image. For example, the blended image can be subjected to an edge-enhanced noise filtering function <NUM>, which generally operates to remove noise and improve the appearances of edges in the blended image. Various techniques for edge enhancement and noise filtering are known in the art. In some embodiments, the filtering function <NUM> can represent a multi-scale de-noising process that is guided by the blending maps, well-exposedness maps, and de-ghosting maps. The filtered blended image can be processed by a contrast enhancement operation <NUM>, which generally operates to increase the overall contrast of the blended image while maintaining natural hue within the blended image. One example implementation of the contrast enhancement operation <NUM> is described below, although other implementations of the contrast enhancement operation <NUM> could also be used.

The output of the process <NUM> is at least one final image <NUM> of the scene. The final image <NUM> generally represents a blend of the original images <NUM>, <NUM>, and <NUM> after processing. As noted above, for example, the final image <NUM> may represent the image selected as the reference image (such as the image <NUM>), with one or more portions of the reference image (such as one or more blown-own regions) replaced or combined with one or more corresponding portions of at least one other image (such as the shorter-exposure image <NUM>). Ideally, the final image <NUM> has both a foreground and a background with more uniform illumination. The illumination need not be completely uniform, but the illumination in the final image <NUM> is more uniform compared to the illumination in at least the reference image.

<FIG> illustrates an example process for an image registration operation <NUM> in the process <NUM> of <FIG> in accordance with this disclosure. As described above, the image registration operation <NUM> is used to align multiple images (such as the images <NUM>, <NUM>, and <NUM>) captured by the electronic device <NUM>. In <FIG>, multiple images, namely a reference image and a non-reference image, are aligned by fitting a transformation matrix H to matched feature points. A pair of matched feature points represents a feature point in one image that is matched to a corresponding feature in the other image. Overall, this helps to compensate for movement of the camera/electronic device <NUM>.

As shown in <FIG>, a reference image Iref and a non-reference image Inonref are provided to a feature detection and matching function <NUM>, which generally operates to identify the feature points in each image and match the feature points common to both images. In this example, the matched feature points are expressed as {pnonref, pref} values. The feature detection and matching function <NUM> can use any suitable technique for identifying and matching feature points, such as ORB feature detection and matching. Various types of feature point detection and matching are known in the art. A first transformation matrix estimation function <NUM> receives the matched feature points {pnonref, pref} and generates an initial estimate of the transformation matrix. The initial estimate represents an initial guess of the transformation matrix that could be used to transform the features points of the non-reference image to match the features points of the reference image. Various types of transformation matrix estimation techniques are known in the art, such as linear estimation.

The reference and non-reference images and the initial estimate of the transformation matrix are provided to a block search function <NUM>. Unlike the feature detection and matching (which matches feature points), the block search function <NUM> attempts to match blocks in the reference and non-reference images after at least one of the images has been transformed using the initial estimate of the transformation matrix. This allows the block search to be guided by the identified feature points. In this example, the matched blocks are expressed as {qnonref, qref} values. The block search function <NUM> can use any suitable technique for identifying and matching blocks.

A second transformation matrix estimation function <NUM> receives the matched feature points {pnonref, pref} and the matched blocks {qnonref, qref} and generates a final estimate of the transformation matrix H. The final estimate ideally represents the best estimate of the transformation matrix to be used to transform the features points and blocks of the non-reference image to match the features points and blocks of the reference image. Once the non-reference image is transformed using the transformation matrix H, the non-reference image is generally aligned with the reference image. Again, various types of transformation matrix estimation techniques are known in the art, such as linear estimation.

Note that the process shown in <FIG> can be repeated for each non-reference image in the image collection <NUM>, typically using the same image from the collection <NUM> as the reference image. The results from performance of the process in <FIG> is ideally a set of images (denoted <NUM>', <NUM>', and <NUM>' below) that are generally aligned with one another. It is possible that one of the images in the set of aligned images still represents the corresponding original image, such as when the image <NUM>' matches the image <NUM> if the image <NUM> is used as the reference image during the process <NUM>.

<FIG> and <FIG> illustrate an example process for an exposure analysis operation <NUM> in the process <NUM> of <FIG> in accordance with this disclosure. As described above, the exposure analysis operation <NUM> is used to identify different areas of aligned versions of images captured by the electronic device <NUM> that are well-exposed and therefore include the most useful and reliable image information. As shown in <FIG>, different metrics <NUM>, <NUM>, and <NUM> are applied to the aligned images <NUM>', <NUM>', and <NUM>'. Different metrics <NUM>, <NUM>, and <NUM> can be used here since the images <NUM>', <NUM>', and <NUM>' are associated with different camera exposures. Thus, different measures can be used to determine whether portions of the different images <NUM>', <NUM>', and <NUM>' are well-exposed. In this way, each image <NUM>', <NUM>', and <NUM>' is converted into a well-exposedness map <NUM>, <NUM>, and <NUM>, respectively. Each well-exposedness map <NUM>, <NUM>, and <NUM> could represent a grayscale image, where brighter colors represent well-exposed areas of the associated image and darker colors represent over- or under-exposed areas of the associated image.

Examples of the metrics <NUM>, <NUM>, and <NUM> that could be used here are shown in <FIG>, where lines <NUM>, <NUM>, and <NUM> respectively represent the functions applied by the metrics <NUM>, <NUM>, and <NUM>. For each function, the corresponding line <NUM>, <NUM>, or <NUM> identifies how actual pixel values in an image <NUM>', <NUM>', or <NUM>' are translated into values contained in the well-exposedness map <NUM>, <NUM>, or <NUM>. For example, the line <NUM> here represents how pixel values in a short-exposure image (such as the image <NUM>') can be converted into corresponding values in the well-exposedness map <NUM>. The line <NUM> here represents how pixel values in a mid-exposure image (such as the image <NUM>') can be converted into corresponding values in the well-exposedness map <NUM>. The line <NUM> here represents how pixel values in a long-exposure image (such as the image <NUM>') can be converted into corresponding values in the well-exposedness map <NUM>. Because the pixel values in different images of the same scene can vary in a generally known manner as the camera exposure changes, it is possible to predict which pixel values are more likely to represent well-exposed areas of images captured at different exposures. Note, however, that the functions shown in <FIG> are examples only, and other metrics could be used to convert an image into a well-exposedness map.

<FIG>, <FIG>, <FIG>, <FIG>, and <FIG> illustrate an example process for an image de-ghosting operation <NUM> in the process <NUM> of <FIG> in accordance with this disclosure. As described above, the image de-ghosting operation <NUM> is used to identify motion in aligned versions of images captured by the electronic device <NUM>. As shown in <FIG>, the image de-ghosting operation <NUM> generally includes operations performed by a reference frame block <NUM> and a main block <NUM>. The reference frame block <NUM> receives luminance (Y) values of a reference image and a non-reference image and generates a motion multiplier (Mot_Mult) for the two images. The motion multiplier controls how aggressively the main block <NUM> in the image de-ghosting operation <NUM> will be in terms of rejecting pixels with high difference as motion. The main block <NUM> receives the motion multiplier, the luminance values of the reference and non-reference images, and chrominance values (U and V) of the reference and non-reference images, along with any desired tuning parameters (such as a noise level estimate denoted Sig_Est). The noise level estimate can be based on the ISO level of the camera during the capture of the images. The main block <NUM> uses this information to generate a de-ghosting map <NUM> for the two images. The de-ghosting map <NUM> (also referred to as a motion map) identifies areas in the two images where motion is occurring and should be removed, thereby identifying the expected motion and noise level in the images.

<FIG> illustrates an example implementation of the reference frame block <NUM> in <FIG>. As shown in <FIG>, the reference frame block <NUM> includes downscaling functions <NUM> and <NUM>. The downscaling function <NUM> receives the luminance values Yref of the reference image and downscales the luminance values to produce downscaled luminance values Yref_DS. Similarly, the downscaling function <NUM> receives the luminance values Ynonref of the non-reference image and downscales the luminance values to produce downscaled luminance values Ynonref_DS. The downscaling allows less data to be processed in subsequent operations, which can help to speed up the subsequent operations. Any suitable amount of downscaling can be used, such as by downscaling the data by a factor of four. However, downscaling is not necessarily required here.

A difference function <NUM> identifies the differences between the downscaled luminance values (or of the original luminance values) on a pixel-by-pixel basis. Assuming there is no movement between the two images and proper equalization of the images' exposures, the difference function <NUM> outputs a difference map identifying only the differences between the images, which (ideally) represent motion within the images. For example, the difference map could have darker pixels indicating little difference between the image pixel values and brighter pixels indicating more differences between the image pixel values. A histogram function <NUM> generates a histogram based on the difference map, which quantifies motion statistics within a tile.

A threshold/transfer function <NUM> receives the motion statistics from the histogram function <NUM> and the noise level estimate Sig_Est. The threshold/transfer function <NUM> uses the noise level estimate to identify when differences detected in the images are actually representative of motion in the images. The output of the threshold/transfer function <NUM> is a motion multiplier <NUM>.

<FIG> illustrates an example implementation of the main block <NUM> in <FIG>. As shown in <FIG>, the main block <NUM> includes an edge strength filter <NUM> and a main sub-block <NUM>. The edge strength filter <NUM> receives the luminance values Yref of the reference image, the noise level estimate Sig_Est, and the motion multiplier Mot_Mult and generates a norm map, which is used by the main sub-block <NUM>. One example implementation of the edge strength filter <NUM> is described below, although other implementations of the edge strength filter <NUM> could also be used. The main sub-block <NUM> receives the luminance and chrominance values YUVref and YUVnonref of the reference and non-reference images, along with the norm map. The main sub-block <NUM> uses this information to generate the de-ghosting map <NUM>. One example implementation of the main sub-block <NUM> is described below, although other implementations of the main sub-block <NUM> could also be used.

<FIG> illustrates an example implementation of the edge strength filter <NUM> of the main block <NUM> in <FIG>. As shown in <FIG>, the edge strength filter <NUM> includes a downscaling function <NUM>, which receives the luminance values Yref of the reference image and downscales the luminance values to produce downscaled luminance values Yref_DS. Any suitable downscaling can be used here (such as downscaling by a factor of four), although no downscaling may be needed. The downscaled luminance values Yref_DS are passed through a high-pass filter <NUM> to produce edge values (denoted YESF), which represent rough edges in the scene. The edge values are passed through a low-pass filter <NUM> to produce filtered edge values (denoted FilterESF), which represent smoothed edges in the scene. The high-pass filter <NUM> represents any suitable high-pass filter for filtering pixel values, such as a <NUM>×<NUM> high-pass filter. The low-pass filter <NUM> represents any suitable low-pass filter for filtering pixel values, such as a <NUM>×<NUM> low-pass filter.

The filtered edge values are provided to an add/shift/multiply function <NUM>, which also receives the noise level estimate Sig_Est and the motion multiplier Mot_Mult. The add/shift/multiply function <NUM> operates to generate the norm map using this information, where the norm map is used to normalize the motion due to pixel differences within a tile as described below. The add/shift/multiply function <NUM> can use the filtered edge values FilterESF, noise level estimate Sig_Est, and motion multiplier Mot_Mult in any suitable manner to generate the norm map. In some embodiments, the add/shift/multiply function <NUM> generates the norm map by performing the following calculation, although other suitable calculations could also occur.

<FIG> illustrates an example implementation of the main sub-block <NUM> of the main block <NUM> in <FIG>. As shown in <FIG>, the main sub-block <NUM> includes difference functions <NUM> and <NUM>. The difference function <NUM> identifies the differences Ydiff between the luminance values Yref and Ynonref of the reference and non-reference images, and the difference function <NUM> identifies the differences Udiff and Vdiff between the chrominance values UVref and UVnonref of the reference and non-reference images. The differences Ydiff in the luminance values are provided to an average/downscale function <NUM>, which averages sets of luminance value differences to downscale the size of the luminance value differences and produce downscaled luminance value differences Ydiff_DS. Again, any suitable downscaling can be used here (such as downscaling by a factor of four), although no downscaling may be needed.

A sum/cap function <NUM> receives the downscaled luminance value differences Ydiff_DS and the chrominance value differences Udiff and Vdiff and operates to generate the difference map, which identifies the differences between the images. The sum/cap function <NUM> can use the downscaled luminance value differences Ydiff_DS and chrominance value differences Udiff and Vdiff in any suitable manner to generate the difference map. In some embodiments, the sum/cap function <NUM> generates the difference map by performing the following calculation, although other suitable calculations could also occur.

<MAT>
where Diff_map represents the difference map pixel values and Sat_Thr represents a saturation threshold.

The difference map is provided to a low-pass filter (LPF)/divide function <NUM>, which also receives the norm map and two scalar values. One scalar value represents a reference weight Ref_weight, and the other scalar value represents a weight multiplier W_mult. The low-pass filter/divide function <NUM> uses the difference map, norm map, and scalar values to generate the de-ghosting map, which identifies areas in the images where motion is occurring. The low-pass filter/divide function <NUM> can use the difference map, norm map, and scalar values in any suitable manner to generate the de-ghosting map. In some embodiments, the low-pass filter/divide function <NUM> generates the de-ghosting map by calculating the following, although other suitable calculations could also occur.

<MAT>
where Deghost_map represents the de-ghosting map pixel values and LPF() represents a filtering function. The reference weight Ref weight here defines the maximum value that the de-ghosting map pixels can obtain. The weight multiplier W_mult here defines the value that the Filt_Mot value is multiplied by in order to identify the amount to subtract from the reference weight Ref_weight when motion is present. Larger values of the weight multiplier W_mult therefore result in larger values subtracted from the reference weight Ref_weight, resulting in more motion being detected.

Note that the process shown in <FIG>, <FIG>, <FIG>, <FIG>, and <FIG> can be repeated for each non-reference image in the collection of aligned images, typically using the same image from the collection as the reference image. The results from performance of the process in <FIG>, <FIG>, <FIG>, <FIG>, and <FIG> is ideally a set of de-ghosting maps <NUM> that identify all of the motion between the non-reference images and the reference image (or at least all motion exceeding a threshold).

<FIG> and <FIG> illustrate an example process for an image blending operation <NUM> in the process <NUM> of <FIG> in accordance with this disclosure. As described above, the image blending operation <NUM> is used to blend different portions of images captured by the electronic device <NUM>. As shown in <FIG>, the image blending operation <NUM> receives different images <NUM>', <NUM>', and <NUM>' (which represent the aligned versions of the images <NUM>, <NUM>, and <NUM>). The image blending operation <NUM> also receives three blending maps W1, W2, and W3 since there are three images being used here, although other numbers of blending maps can be used if two images or more than three images are being blended. As noted above, the blending maps can be based on or can be a composite of (such as products of) the well-exposedness maps and the de-ghosting maps generated earlier.

One of the images <NUM>', <NUM>', and <NUM>' in <FIG> is treated as the reference image. In this example, the image <NUM>' is the reference image, which is consistent with earlier uses of the images <NUM> as the reference image. The image <NUM>' is provided to a multiplier function <NUM>, which multiplies the pixels of the image <NUM>' by weights in the blending map W1. A histogram match function <NUM> generates a version of the image <NUM>' using the image <NUM>' as a reference. Effectively, a transfer function is applied to the image <NUM>' in order to make the histogram of the image <NUM>' match the histogram of the image <NUM>' as closely as possible. The resulting version of the image <NUM>' is provided to a multiplier function <NUM>, which multiplies the pixels of the resulting version of the image <NUM>' by weights in a blending map calculated as <NUM>-W1 (assuming each weight in the blending map has a value between zero and one, inclusive). The results from the multiplier functions <NUM> and <NUM> are summed by an adder function <NUM>. This alpha-blends the image <NUM>' and a version of the image <NUM>', synthesizing a new image that helps to avoid ghosting artifacts by removing motion between the images. Assuming the image <NUM>' has a shorter exposure than the image <NUM>', the new synthesized image may be referred to as a new short-exposure image.

Similarly, the image <NUM>' is provided to a multiplier function <NUM>, which multiplies the pixels of the image <NUM>' by weights in the blending map W3. A histogram match function <NUM> generates a version of the image <NUM>' using the image <NUM>' as a reference. Effectively, a transfer function is applied to the image <NUM>' in order to make the histogram of the image <NUM>' match the histogram of the image <NUM>' as closely as possible. The resulting version of the image <NUM>' is provided to a multiplier function <NUM>, which multiplies the pixels of the resulting version of the image <NUM>' by weights in a blending map calculated as <NUM>-W3 (assuming each weight in the blending map has a value between zero and one, inclusive). This alpha-blends the image <NUM>' and a version of the image <NUM>', synthesizing a new image that helps to avoid ghost artifacts by removing motion between the images. Assuming the image <NUM>' has a longer exposure than the image <NUM>', the new synthesized image may be referred to as a new long-exposure image.

The blended image output from the adder <NUM> (such as the new short-exposure image) is provided to a multiplier function <NUM>, which pyramid multiplies the pixels of the new short-exposure image by the weights in the blending map W1. The image <NUM>' (as the reference image) is provided to a multiplier function <NUM>, which pyramid multiplies the pixels of the image <NUM>' by the weights in the blending map W2. The blended image output from the adder <NUM> (such as the new long-exposure image) is provided to a multiplier function <NUM>, which pyramid multiplies the pixels of the new long-exposure image by the weights in the blending map W3. This weights the three images according to the three blending maps W1, W2, and W3, respectively. The results are combined in a pyramid add operation <NUM>, which combines the results to produce a final image. Among other things, the multiplier functions <NUM>, <NUM>, and <NUM> and the add operation <NUM> operate to pyramid blend the images to brighten dark regions (such as the background) and recover saturated regions (such as in the foreground) of the original images <NUM>, <NUM>, and <NUM>.

An example of the pyramid blending of a new short-exposure image, an image <NUM>', and a new long-exposure image is shown in <FIG>. In this example, the image <NUM>' is referred to as a "medium" image since its exposure is between the exposures of the images <NUM>' and <NUM>'. As shown in <FIG>, the image <NUM>' is decomposed into a Laplacian pyramid <NUM>, and the blending map W2 is decomposed into a Gaussian pyramid <NUM>. The decomposition of an image into a Laplacian pyramid can occur by multiplying the image data with a set of transform functions. The blending map W2 is based on the well-exposedness map and the de-ghosting map associated with the original image <NUM>. The decomposition of a blending map into a Gaussian pyramid can occur by weighting the blending map using Gaussian averages. Each of the pyramids <NUM> and <NUM> here is a multi-scale pyramid representing an image or blending map at multiple resolution levels or scales. The levels or scales of the pyramids <NUM> and <NUM> are multiplied together as shown in <FIG>, which represents the multiplier function <NUM>. Optionally, at least some of the levels or scales of the pyramids <NUM> and <NUM> can be multiplied by a halo-control term, which is done for halo suppression.

Similar operations occur for the new short- and long-exposure images. In this example, the new short-exposure image is decomposed into a Laplacian pyramid <NUM>, and the blending map W1 is decomposed into a Gaussian pyramid <NUM>. The blending map W1 is based on the well-exposedness map and the de-ghosting map associated with the original image <NUM>. The levels or scales of the pyramids <NUM> and <NUM> are multiplied together (which represents the multiplier function <NUM>), and optionally at least some of the levels or scales of the pyramids <NUM> and <NUM> can be multiplied by a halo-control term for halo suppression. Also in this example, the new long-exposure image is decomposed into a Laplacian pyramid <NUM>, and the blending map W3 is decomposed into a Gaussian pyramid <NUM>. The blending map W3 is based on the well-exposedness map and the de-ghosting map associated with the original image <NUM>. The levels or scales of the pyramids <NUM> and <NUM> are multiplied together (which represents the multiplier function <NUM>), and optionally at least some of the levels or scales of the pyramids <NUM> and <NUM> can be multiplied by a halo-control term for halo suppression.

The resulting products of the pyramids <NUM> and <NUM>, pyramids <NUM> and <NUM>, and pyramids <NUM> and <NUM> are summed at each level or scale (which represents the add operation <NUM>) to produce a blended image pyramid <NUM>. The various levels or scales of the blended image pyramid <NUM> can then be collapsed or recomposed to produce a blended image of a scene, where the blended image represents a blended version of the new short-exposure image, the image <NUM>', and the new long-exposure image. Ideally, the blended image includes or is based on well-exposed portions of the scene from the image <NUM>' and from the new short- and long-exposure images. This may allow, for example, brighter portions of the background from the image <NUM>' to be combined with well-exposed portions of the foreground in the image <NUM>' in order to produce a blended image with more uniform illumination.

<FIG> illustrates an example process for a contrast enhancement operation <NUM> in the process of <FIG> in accordance with this disclosure. As described above, the contrast enhancement operation <NUM> increases the overall contrast of the blended image while maintaining natural hue within the blended image. As shown in <FIG>, the contrast enhancement operation <NUM> includes a histogram equalization function <NUM>, which generally adapts a blended image produced to a suitable scene. In some embodiments, the histogram equalization function <NUM> uses a global version of contrast-limited adaptive histogram equalization (CLAHE) to improve contrast of the blended image in the luminance domain.

In particular embodiments, histogram equalization is applied on top of the tone curve for the blended image, where the parameters to the histogram equalization function <NUM> include a clip limit, a minimum value, and a maximum value. The clip limit controls the threshold above which histogram entries are redistributed to other areas of the histogram. In some cases, the clip limit can have a typical useful range between <NUM> and <NUM>. The minimum value represents a contrast control parameter defining the percentage below which pixels are clipped at a value of zero. In some cases, the minimum value can have a typical useful range between <NUM> and <NUM>. The maximum value represents a contrast control parameter defining the percentage above which pixels are clipped at a value of <NUM>. In some cases, the maximum value can have a typical useful range between <NUM> and <NUM>.

The contrast enhancement operation <NUM> also includes a chroma gain function <NUM> and a hue correction function <NUM>. The chroma gain function <NUM> generally operates to identify the gain applied to the luminance values by the histogram equalization function <NUM> and to apply the same gain to the chrominance values of the blended image. This can be done to help avoid color desaturation. However, a visible artifact can be created when applying a chroma gain globally in the blended image. In particular, there can be a global shift of hue towards red when applying a chroma gain globally. Hence, the hue correction function <NUM> can be applied to correct this global shift. The output of the hue correction function <NUM> can represent a final image <NUM> of a scene being captured using the electronic device <NUM>.

Although <FIG> illustrates one example of a process <NUM> for multi-scale blending of images in a mobile electronic device and <FIG> illustrate examples of operations in the process <NUM> of <FIG>, various changes may be made to <FIG>. For example, while shown as sequences of steps, various operations shown in <FIG> could overlap, occur in parallel, occur in a different order, or occur any number of times. Also, the specific operations shown in <FIG> are examples only, and other techniques could be used to perform each of the operations shown in <FIG>.

It should be noted that the operations shown in <FIG> can be implemented in an electronic device <NUM> in any suitable manner. For example, in some embodiments, the operations shown in <FIG> can be implemented or supported using one or more software applications or other software instructions that are executed by the processor <NUM> of the electronic device <NUM>. In other embodiments, at least some of the operations shown in <FIG> can be implemented or supported using dedicated hardware components. In general, the operations shown in <FIG> can be performed using any suitable hardware or any suitable combination of hardware and software/firmware instructions.

It should also be noted that the operations shown in <FIG> are described above as being performed using a specific implementation of the electronic device <NUM>, but a number of modifications can be made based on the actual electronic device <NUM> being used. For example, the electronic device <NUM> could include or use a single camera or multiple cameras. If a single camera is used, multiple images of a scene could be captured sequentially, such as in one or more fast bursts. If multiple cameras are used, it may be possible to capture multiple images concurrently or in an overlapping manner, such as by capturing multiple images of a scene at the same time but with different camera exposures using different cameras. If needed, multiple images of the scene could still be captured sequentially using at least one of the multiple cameras. Assuming the geometry of the multiple cameras is known ahead of time, this geometry can be used to help align the images captured by the cameras or perform other functions. As another example, the electronic device <NUM> is described above as performing various operations using YUV image data. However, data in other domains (such as RGB data) could also be used or processed. As a third example, the techniques described in this patent document could be combined with conventional HDR image processing algorithms, such as in a software library used by the electronic device <NUM>. This may allow a user of the electronic device <NUM> to select between different image processing algorithms, such as based on the specific situation or based on user preference.

<FIG> illustrates an example method <NUM> for multi-pair image analysis and multi-scale blending in accordance with this disclosure. For ease of explanation, the method <NUM> shown in <FIG> is described as being performed using the electronic device <NUM> of <FIG> and the techniques shown in <FIG>. However, the method <NUM> shown in <FIG> could be used with any other suitable electronic device and in any suitable system, and various steps in the method <NUM> may or may not occur using the operations and functions shown in <FIG>.

As shown in <FIG>, multiple ambient images of a scene are captured using an electronic device and without using a flash at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> receiving a capture request <NUM> and causing at least one camera of the electronic device <NUM> to capture the ambient images <NUM> of the scene. This could also include the processor <NUM> of the electronic device <NUM> controlling the camera(s) to use different exposures when capturing the ambient images <NUM>. Multiple flash images of the scene are captured using the electronic device and while using a pilot flash sequence at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> causing the at least one camera of the electronic device <NUM> to capture the flash images <NUM> of the scene. This could also include the processor <NUM> of the electronic device <NUM> controlling the flash <NUM> to produce the pilot flash sequence (possibly at a predefined flash strength) and controlling the camera(s) to use different exposures when capturing the flash images <NUM>.

Multiple pairs of the captured images are analyzed to estimate the exposure differences obtained using the flash at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> processing multiple pairs of images (each pair including one of the ambient images <NUM> and one of the flashing images <NUM> having a common exposure time) to identify the exposure differences between each pair of images. Different pairs of images can be captured using different camera exposures. As a specific example, each pair of images could be processed by dividing the pixel values in the images, converting the quotients into a logarithmic domain, applying a rectifier linear unit operation, averaging the resulting values, and performing an edge-preserving filtering of the averaged values. As another specific example, each pair of images could be processed using an artificial intelligence function (such as a convolutional neural network <NUM>). An appropriate flash strength for the scene is identified using the exposure differences at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> mapping the identified exposure differences to the appropriate flash strength. As noted above, the mapping can be based on a number of factors.

The flash of the electronic device is fired at the determined flash strength and additional images of the scene are captured using the electronic device at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> controlling the flash <NUM> to fire at the appropriate flash strength determined earlier. This could also include the processor <NUM> of the electronic device <NUM> causing the at least one camera of the electronic device <NUM> to capture the additional images <NUM>, <NUM>, and <NUM> of the scene. The additional images <NUM>, <NUM>, and <NUM> can be captured using a different exposure but the same common flash strength. The additional images are aligned and pre-processed at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> aligning the additional images <NUM>, <NUM>, and <NUM> using feature point detection and matching and block searching. This could also include the processor <NUM> of the electronic device <NUM> performing exposure analysis and de-ghosting of the aligned images <NUM>', <NUM>', and <NUM>'.

The aligned and pre-processed images are then blended. In this example, the blending occurs by generating multi-scale representations of images after alignment and processing at step <NUM>, and the multi-scale representations are blended to produce a blended image of the scene at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> decomposing one of the images selected as a reference (such as the image <NUM>") into a Laplacian pyramid and decomposing the associated blending map into a Gaussian pyramid. This could also include the processor <NUM> of the electronic device <NUM> generating one or more synthesized images based on one or more processed images (such as new versions of the image <NUM>" based on the images <NUM>" and <NUM>"), decomposing the synthesized image(s) into one or more Laplacian pyramids, and decomposing the associated blending map(s) into one or more Gaussian pyramids. This can further include the processor <NUM> of the electronic device <NUM> multiplying each Laplacian pyramid by the associated Gaussian pyramid, applying any desired halo correction factors, and summing the results at each level of the multi-scale representations. In addition, this can include the processor <NUM> of the electronic device <NUM> collapsing to summed results to produce a blended image of the scene. Ideally, the blended image of the scene has a more uniform illumination compared to any of the original images.

Any desired post-processing of the blended image occurs at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> performing an edge-enhanced noise filtering function <NUM> and/or a contrast enhancement operation <NUM> on the blended image of the scene. The output of the post-processing is a final image of the scene, which can be stored, output, or used in some manner at step <NUM>. This could include, for example, the processor <NUM> of the electronic device <NUM> displaying the final image of the scene on the display <NUM> of the electronic device <NUM>. This could also include the processor <NUM> of the electronic device <NUM> saving the final image of the scene to a camera roll stored in a memory <NUM> of the electronic device <NUM>. This could further include the processor <NUM> of the electronic device <NUM> attaching the final image of the scene to a text message, email, or other communication to be transmitted from the electronic device <NUM>. Of course, the final image of the scene could be used in any other or additional manner.

Although <FIG> illustrates one example of a method <NUM> for multi-pair image analysis and multi-scale blending, various changes may be made to <FIG>. For example, while shown as a series of steps, various steps in <FIG> could overlap, occur in parallel, occur in a different order, or occur any number of times. Also, it is not necessary that the techniques for multi-pair image analysis described above be used with the techniques for multi-scale blending described above. It is possible, for instance, to perform the multi-pair image analysis to identify and fire the flash <NUM> without then performing the multi-scale blending. Similarly, it is possible to perform the multi-scale blending based on images captured using a flash strength that is not based on the multi-pair image analysis.

<FIG>, <FIG> illustrate example results that can be obtained using multi-pair image analysis and multi-scale blending in accordance with this disclosure. In <FIG>, an image <NUM> of a scene that includes a darker foreground with an object (a teddy bear resting on a table) and a brighter background is captured, where no flash is used. As can be seen here, the brighter background is generally well-illuminated, while the foreground with the object is dark and under-exposed. In <FIG>, an image <NUM> of the same scene is captured, but this time a standard flash is used. As can be seen here, the brighter background is now darker, while the foreground with the object is very bright and likely considered over-exposed. In <FIG>, an image <NUM> of the same scene is captured, but this time a standard flash and standard HDR image processing are used. As can be seen here, the background is still bright, and slight improvements have been made in illuminating the foreground. Unfortunately, the foreground still has significant illumination differences with the background.

In <FIG>, an image <NUM> of the same scene is captured using the approaches described above for multi-pair image analysis and multi-scale blending. As can be seen here, the background of the scene is still well-illuminated, but the background has not been darkened as was done in the standard flash image <NUM> in <FIG>. Moreover, the foreground of the scene has been brightened compared to the original non-flash image <NUM> in <FIG>, but the foreground is not over-exposed as in the standard flash image <NUM> in <FIG>. Also, the foreground of the scene has a more consistent illumination compared to the background, which is not achieved using the standard HDR image processing for the image <NUM> in <FIG>. The image <NUM> is therefore more aesthetically pleasing overall compared to any of the images <NUM>, <NUM>, and <NUM>.

Although <FIG>, <FIG> illustrate examples of results that can be obtained using multi-pair image analysis and multi-scale blending, various changes may be made to <FIG>, <FIG>. For example, <FIG>, <FIG> are merely meant to illustrate one example of the type of results that could be obtained using the approaches described in this disclosure. Obviously, standard, flash, and HDR images can vary widely, and the results obtained using the approaches described in this patent document can also vary widely depending on the circumstances.

Claim 1:
A method comprising:
capturing multiple ambient images (<NUM>) of a scene using at least one camera of an electronic device (<NUM>) and without using a flash of the electronic device (<NUM>);
capturing multiple flash images (<NUM>) of the scene using the at least one camera of the electronic device (<NUM>) and during firing of a pilot flash sequence using the flash;
analyzing multiple pairs of images to estimate exposure differences obtained using the flash , wherein each pair of images includes one of the ambient images (<NUM>) and one of the flash images (<NUM>) that are both captured using a common camera exposure, and wherein different pairs of images are captured using different camera exposures;
determining a flash strength for the scene based on the estimate of the exposure differences; and
firing the flash based on the determined flash strength,
wherein the method further comprises:
capturing additional images (<NUM>;<NUM>;<NUM>) of the scene using the at least one camera of the electronic device (<NUM>), at least some of the additional images (<NUM>;<NUM>;<NUM>) captured at the determined flash strength and using different camera exposures; and
blending the additional images (<NUM>;<NUM>;<NUM>) to produce a blended image of the scene.