Patent Description:
A Bayer color filter array (CFA) pattern is widely used in mobile imaging sensors. However, its limited sensitivity makes it a challenge to use under extreme low-light conditions. Alternatively, a red-green-blue-white (RGBW) sensor includes a CFA pattern having a white color filter along with the red, green, and blue filters. The RGBW CFA pattern is more sensitive and offers a higher signal-to-noise ratio (SNR). A drawback of RGBW is its over-sensitivity of the white (W) channel under high-light conditions, assuming the automatic exposure (AE) logic remains calibrated for RGB.

<CIT> describes a system and method for dynamic selection of reference image frame. <NPL> proposes a rank minimization-based color interpolation method with a colorization constraint for the RGBW format with a large number of W pixels. <CIT> describes an image pickup apparatus, image processing method and computer program.

This disclosure provides a system and method for multi-exposure, multi-frame blending of red-green-blue-white (RGBW) images.

Accordance with an aspect of the disclosure, a method as defined by claim <NUM> of the claims appended hereto is provided.

According to an embodiment, the method may include aligning the multiple images to generate motion vectors. According to an embodiment, the method may include converting the images into the multi-channel frames using the motion vectors. According to an embodiment, the method may include equalizing the multi-channel frames to a common brightness level. According to an embodiment, the method may include grouping and blending the multi-channel frames.

According to an embodiment, the method may include fusing the white channel with red, green, and blue channels in the saturation-free RGBW frame to generate a three-channel RGB frame. According to an embodiment, the method may include converting the three-channel RGB frame from a first bit depth to a second bit depth. According to an embodiment, the method may include post-processing the converted three-channel RGB frame to generate the final image.

According to an embodiment, the method may include, in estimating the saturation across the exposure differences between the different exposure levels, a saturation level at a current exposure level depends on an exposure value of a next exposure level with respect to a reference exposure level.

According to an embodiment, the method may include multiplying pairs of the generated motion maps and saturation maps at the different exposure levels to generate combined maps. According to an embodiment, the method may include combining the combined maps with the multi-channel frames at the different exposure levels.

According to an embodiment, the method may include performing saturation clipping on pairs of the multi-channel frames. According to an embodiment, the method may include converting red, green, and blue channels of the multi-channel frames to Y, U, and V channels. According to an embodiment, the method may include equalizing a brightness of the white channel to correspond to a brightness of the Y channel. According to an embodiment, the method may include performing guided filtering on the Y channel using the white channel as the guidance signal.

Accordance with an aspect of the disclosure, an electronic device as defined by claim <NUM> of the claims appended hereto is provided.

According to an embodiment, the electronic device may be configured to align the multiple images to generate motion vector. According to an embodiment, the electronic device may be configured to convert the images into the multi-channel frames using the motion vectors. According to an embodiment, the electronic device may be configured to equalize the multi-channel frames to a common brightness level. According to an embodiment, the electronic device may be configured to group and blend the multi-channel frames.

According to an embodiment, the electronic device may include the at least one processor. The at least one processor may be configured to align the multiple images to generate motion vectors. The at least one processor may be configured to convert the images into the multi-channel frames using the motion vectors. The at least one processor may be configured to equalize the multi-channel frames to a common brightness level. The at least one processor may be configured to group and blend the multi-channel frames.

According to an embodiment, the electronic device may include the at least one processor. The at least one processor may be configured to fuse the white channel with red, green, and blue channels in the saturation-free RGBW frame to generate a three-channel RGB frame. The at least one processor may be configured to convert the three-channel RGB frame from a first bit depth to a second bit depth. The at least one processor may be configured to post-process the converted three-channel RGB frame to generate the final image.

According to an embodiment, the electronic device may include the at least one processor.

According to an embodiment, the electronic device may include, in estimating the saturation across the exposure differences between the different exposure levels, a saturation level at a current exposure level depends on an exposure value of a next exposure level with respect to a reference exposure level.

According to an embodiment, the electronic device may include the at least one processor. The at least one processor may be configured to multiply pairs of the generated motion maps and saturation maps at the different exposure levels to generate combined maps. The at least one processor may be configured to combine the combined maps with the multi-channel frames at the different exposure levels.

According to an embodiment, the electronic device may include the at least one processor. The at least one processor may be configured to perform saturation clipping on pairs of the multi-channel frames. The at least one processor may be configured to convert red, green, and blue channels of the multi-channel frames to Y, U, and V channels. The at least one processor may be configured to equalize a brightness of the white channel to correspond to a brightness of the Y channel. The at least one processor may be configured to perform guided filtering on the Y channel using the white channel as the guidance signal.

Accordance with an aspect of the disclosure, a machine-readable medium contains instructions that when executed cause at least one processor of an electronic device to generate multi-channel frames at different exposure levels from multiple images of a scene obtained using at least one RGBW image sensor. The medium also contains instructions that when executed cause the at least one processor to generate multiple motion maps by estimating motion across exposure differences between the different exposure levels using a white channel of the multi-channel frames as a guidance signal. The medium further contains instructions that when executed cause the at least one processor to generate multiple saturation maps by estimating saturation across the exposure differences between the different exposure levels. The medium also contains instructions that when executed cause the at least one processor to use the generated motion maps and saturation maps to recover saturations from the different exposure levels and generate a saturation-free RGBW frame. In addition, the medium contains instructions that when executed cause the at least one processor to process the saturation-free RGBW frame to generate a final image of the scene.

Othertechnical features may be readily apparent to one skilled in the art from the following figures, descriptions, and claims.

Acomputer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. A computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.

As used here, terms and phrases such as "have," "may have," "include," or "may include" a feature (like a number, function, operation, or component such as a part) indicate the existence of the feature and do not exclude the existence of other features. Also, as used here, the phrases "A or B," "at least one of A and/or B," or "one or more of A and/or B" may include all possible combinations of A and B. For example, "A or B," "at least one of A and B," and "at least one of A or B" may indicate all of (<NUM>) including at least one A, (<NUM>) including at least one B, or (<NUM>) including at least one A and at least one B. Further, as used here, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate different user devices from each other, regardless of the order or importance of the devices. A first component may be denoted a second component and vice versa without departing from the scope of this disclosure.

It will be understood that, when an element (such as a first element) is referred to as being (operatively or communicatively) "coupled with/to" or "connected with/to" another element (such as a second element), it can be coupled or connected with/to the other element directly or via a third element. In contrast, it will be understood that, when an element (such as a first element) is referred to as being "directly coupled with/to" or "directly connected with/to" another element (such as a second element), no other element (such as a third element) intervenes between the element and the other element.

As used here, the phrase "configured (or set) to" may be interchangeably used with the phrases "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of" depending on the circumstances. The phrase "configured (or set) to" does not essentially mean "specifically designed in hardware to. " Rather, the phrase "configured to" may mean that a device can perform an operation together with another device or parts. For example, the phrase "processor configured (or set) to perform A, B, and C" may mean a generic-purpose processor (such as a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (such as an embedded processor) for performing the operations.

The terms and phrases as used here are provided merely to describe some embodiments of this disclosure but not to limit the scope of other embodiments of this disclosure. All terms and phrases, including technical and scientific terms and phrases, used here have the same meanings as commonly understood by one of ordinary skill in the art to which the embodiments of this disclosure belong. It will be further understood that terms and phrases, such as those defined in commonly-used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined here. In some cases, the terms and phrases defined here may be interpreted to exclude embodiments of this disclosure.

Examples of an "electronic device" according to embodiments of this disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of an electronic device include a smart home appliance. Examples of the smart home appliance may include at least one of a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washer, a drier, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (such as SAMSUNG HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or speaker with an integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a gaming console (such as an XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame. Still other examples of an electronic device include at least one of various medical devices (such as diverse portable medical measuring devices (like a blood sugar measuring device, a heartbeat measuring device, or a body temperature measuring device), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, a sailing electronic device (such as a sailing navigation device or a gyro compass), avionics, security devices, vehicular head units, industrial or home robots, automatic teller machines (ATMs), point of sales (POS) devices, or Internet of Things (IoT) devices (such as a bulb, various sensors, electric or gas meter, sprinkler, fire alarm, thermostat, street light, toaster, fitness equipment, hot water tank, heater, or boiler). Other examples of an electronic device include at least one part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or various measurement devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that, according to various embodiments of this disclosure, an electronic device may be one or a combination of the above-listed devices. According to some embodiments of this disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed here is not limited to the above-listed devices and may include new electronic devices depending on the development of technology.

In the following description, electronic devices are described with reference to the accompanying drawings, according to various embodiments of this disclosure. As used here, the term "user" may denote a human or another device (such as an artificial intelligent electronic device) using the electronic device.

Definitions for other certain words and phrases may be provided throughout this patent document.

None of the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claim scope. The scope of patented subject matter is defined only by the claims.

For a more complete understanding of this disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts:.

<FIG>, discussed below, and the various embodiments of this disclosure are described with reference to the accompanying drawings. However, it should be appreciated that this disclosure is not limited to these embodiments and all changes and/or equivalents or replacements thereto also belong to the scope of this disclosure, so far as they fall within the scope of the appended claims.

As discussed above, a Bayer color filter array (CFA) pattern is widely used in mobile imaging sensors. However, its limited sensitivity makes it a challenge to use under extreme low-light conditions. This is because a Bayer CFA pattern passes different spectra of light, namely red (R), green (G), and blue (B) spectra, to separate pixels. As a result, the red, green, and blue pixels are exposed at a similar level, and all pixels can be under-exposed in extreme low-light conditions. Alternatively, a red-green-blue-white (RGBW) sensor includes a CFA pattern having a white color filter along with the red, green, and blue filters. Stated differently, the RGBW CFA pattern also passes white light, which covers the entire spectrum of the red, green, and blue spectra. Hence, white pixels are exposed more than the red, green, and blue pixels. The RGBW CFA pattern is more sensitive and offers a higher signal-to-noise ratio (SNR). A drawback of RGBW is its over-sensitivity of the white (W) channel under high-light conditions, assuming the automatic exposure (AE) logic remains calibrated for RGB. Conventional techniques are typically limited to single-frame RGBW, which focuses on its sensitivity and resolution/SNR improvement, while largely ignoring the over-sensitive nature of the white channel (by assuming either the exposure is well-conditioned or the dynamic range is sufficiently low).

This disclosure provides systems and methods for multi-exposure, multi-frame blending of RGBW images. As described in more detail below, the systems and methods provide a multi-frame, multi-exposure imaging pipeline where multiple images are captured by an RGBW sensor at different exposures and blended so as to retain the SNR benefit of RGBW while avoiding the issue of overexposed or saturated pixels in a subset of channels. Using multi-frame, multi-exposure captures, a saturation/over-exposure analysis, and recovery for RGB and W channels, the disclosed systems and methods provide improved levels of exposure for all four RGBW channels. Note that while some of the embodiments discussed below are described in the context of image processing in consumer electronic devices such as smart phones, this is merely one example, and it will be understood that the principles of this disclosure may be implemented in any number of other suitable contexts.

<FIG> illustrates an example network configuration <NUM> including an electronic device according to this disclosure. The embodiment of the network configuration <NUM> shown in <FIG> is for illustration only. Other embodiments of the network configuration <NUM> could be used without departing from the scope of this disclosure.

According to embodiments of this disclosure, an electronic device <NUM> is included in the network configuration <NUM>. The electronic device <NUM> can include at least one of a bus <NUM>, a processor <NUM>, a memory <NUM>, an input/output (I/O) interface <NUM>, a display <NUM>, a communication interface <NUM>, or a sensor <NUM>. In some embodiments, the electronic device <NUM> may exclude at least one of these components or may add at least one other component. The bus <NUM> includes a circuit for connecting P45498EPN1 - OA1 the components <NUM>-<NUM> with one another and for transferring communications (such as control messages and/or data) between the components.

The processor <NUM> includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor <NUM> is able to perform control on at least one of the other components of the electronic device <NUM> and/or perform an operation or data processing relating to communication. In some embodiments, the processor <NUM> can be a graphics processor unit (GPU). As described in more detail below, the processor <NUM> may perform one or more operations for multi-exposure, multi-frame blending of RGBW images.

The memory <NUM> can include a volatile and/or non-volatile memory. For example, the memory <NUM> can store commands or data related to at least one other component of the electronic device <NUM>. According to embodiments of this disclosure, the memory <NUM> can store software and/or a program <NUM>. The program <NUM> includes, for example, a kernel <NUM>, middleware <NUM>, an application programming interface (API) <NUM>, and/or an application program (or "application") <NUM>. At least a portion of the kernel <NUM>, middleware <NUM>, or API <NUM> may be denoted an operating system (OS).

The kernel <NUM> can control or manage system resources (such as the bus <NUM>, processor <NUM>, or memory <NUM>) used to perform operations or functions implemented in other programs (such as the middleware <NUM>, API <NUM>, or application <NUM>). The kernel <NUM> provides an interface that allows the middleware <NUM>, the API <NUM>, or the application <NUM> to access the individual components of the electronic device <NUM> to control or manage the system resources. The application <NUM> may support one or more functions for multi-exposure, multi-frame blending of RGBW images as discussed below. These functions can be performed by a single application or by multiple applications that each carry out one or more of these functions. The middleware <NUM> can function as a relay to allow the API <NUM> or the application <NUM> to communicate data with the kernel <NUM>, for instance. A plurality of applications <NUM> can be provided. The middleware <NUM> is able to control work requests received from the applications <NUM>, such as by allocating the priority of using the system resources of the electronic device <NUM> (like the bus <NUM>, the processor <NUM>, or the memory <NUM>) to at least one of the plurality of applications <NUM>. The API <NUM> is an interface allowing the application <NUM> to control functions provided from the kernel <NUM> or the middleware <NUM>. For example, the API <NUM> includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.

The I/O interface <NUM> serves as an interface that can, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic device <NUM>. The I/O interface <NUM> can also output commands or data received from other component(s) of the electronic device <NUM> to the user or the other external device.

The display <NUM> includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display <NUM> can also be a depth-aware display, such as a multi-focal display. The display <NUM> is able to display, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display <NUM> can include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.

The communication interface <NUM>, for example, is able to set up communication between the electronic device <NUM> and an external electronic device (such as a first electronic device <NUM>, a second electronic device <NUM>, or a server <NUM>). For example, the communication interface <NUM> can be connected with a network <NUM> or <NUM> through wireless or wired communication to communicate with the external electronic device. The communication interface <NUM> can be a wired or wireless transceiver or any other component for transmitting and receiving signals.

The wireless communication is able to use at least one of, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (<NUM>), millimeter-wave or <NUM> wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection can include, for example, at least one of a universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard <NUM> (RS-<NUM>), or plain old telephone service (POTS). The network <NUM> or <NUM> includes at least one communication network, such as a computer network (like a local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The electronic device <NUM> further includes one or more sensors <NUM> that can meter a physical quantity or detect an activation state of the electronic device <NUM> and convert metered or detected information into an electrical signal. For example, one or more sensors <NUM> include one or more cameras or other imaging sensors for capturing images of scenes, such as at least one red-green-blue-white (RGBW) image sensor configured to obtain multiple images of scenes. The sensor(s) <NUM> can also include one or more buttons for touch input, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red green blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasound sensor, an iris sensor, or a fingerprint sensor. The sensor(s) <NUM> can further include an inertial measurement unit, which can include one or more accelerometers, gyroscopes, and other components. In addition, the sensor(s) <NUM> can include a control circuit for controlling at least one of the sensors included here. Any of these sensor(s) <NUM> can be located within the electronic device <NUM>.

The first external electronic device <NUM> or the second external electronic device <NUM> can be a wearable device or an electronic device-mountable wearable device (such as an HMD). When the electronic device <NUM> is mounted in the electronic device <NUM> (such as the HMD), the electronic device <NUM> can communicate with the electronic device <NUM> through the communication interface <NUM>. The electronic device <NUM> can be directly connected with the electronic device <NUM> to communicate with the electronic device <NUM> without involving with a separate network. The electronic device <NUM> can also be an augmented reality wearable device, such as eyeglasses, that include one or more cameras.

The first and second external electronic devices <NUM> and <NUM> and the server <NUM> each can be a device of the same or a different type from the electronic device <NUM>. According to certain embodiments of this disclosure, the server <NUM> includes a group of one or more servers. Also, according to certain embodiments of this disclosure, all or some of the operations executed on the electronic device <NUM> can be executed on another or multiple other electronic devices (such as the electronic devices <NUM> and <NUM> or server <NUM>). Further, according to certain embodiments of this disclosure, when the electronic device <NUM> should perform some function or service automatically or at a request, the electronic device <NUM>, instead of executing the function or service on its own or additionally, can request another device (such as electronic devices <NUM> and <NUM> or server <NUM>) to perform at least some functions associated therewith. The other electronic device (such as electronic devices <NUM> and <NUM> or server <NUM>) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device <NUM>. The electronic device <NUM> can provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example. While <FIG> shows that the electronic device <NUM> includes the communication interface <NUM> to communicate with the external electronic device <NUM> or server <NUM> via the network <NUM> or <NUM>, the electronic device <NUM> may be independently operated without a separate communication function according to some embodiments of this disclosure.

The server <NUM> can include the same or similar components <NUM>-<NUM> as the electronic device <NUM> (or a suitable subset thereof). The server <NUM> can support to drive the electronic device <NUM> by performing at least one of operations (or functions) implemented on the electronic device <NUM>. For example, the server <NUM> can include a processing module or processor that may support the processor <NUM> implemented in the electronic device <NUM>. As described in more detail below, the server <NUM> may perform one or more operations to support techniques for multi-exposure, multi-frame blending of RGBW images.

Although <FIG> illustrates one example of a network configuration <NUM> including an electronic device <NUM>, various changes may be made to <FIG>. For example, the network configuration <NUM> could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. Also, while <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

<FIG> illustrates an example process <NUM> for multi-exposure, multi-frame blending of RGBW images according to this disclosure. Note that the process <NUM> is described as being performed using one or more components of the electronic device <NUM> described above. However, this is merely one example, and the process <NUM> could be performed using any other suitable device(s), such as when implemented using the server <NUM>.

As shown in <FIG>, in the process <NUM>, the electronic device <NUM> obtains multiple images <NUM> of a scene. For example, the electronic device <NUM> can obtain the images <NUM> in a burst capture operation, meaning the images <NUM> can be obtained in rapid succession or at or near the same time. The capture operation may be performed in response to an event, such as a user actuating a shutter control or image capture control. In some embodiments, the images <NUM> are captured using a common image sensor <NUM> of the electronic device <NUM>, such as a camera having an RGBW sensor. In other embodiments, the images <NUM> are captured using multiple image sensors <NUM> of the electronic device <NUM>. In the process <NUM>, the images <NUM> can be multi-exposure, multi-frame, one-channel (mosaic) raw RGBW images at a twelve-bit depth per pixel (which can be identified as RGBW 1C 12b). However, this is merely one example, and images having other depths or numbers of channels are within the scope of this disclosure.

The electronic device <NUM> performs an alignment operation <NUM> on the images <NUM> to generate motion vectors. Alignment generally refers to aligning different image frames so that common points in the image frames are aligned. This may be useful or desirable since later blending operations may be more successful when the image frames are well-aligned. The alignment operation <NUM> can include any suitable alignment process, technique, or algorithm for generating motion vectors from images.

The electronic device <NUM> uses the motion vectors generated during the alignment operation <NUM> as inputs to a demosaic-warp operation <NUM>. The electronic device <NUM> performs the demosaic-warp operation <NUM> to convert the mosaic frames of the image <NUM> into multi-channel frames (such as four-channel frames). In the demosaic-warp operation <NUM>, the exposure levels of the demosaiced frames can be, for example, EV-<NUM>, EV-<NUM>, and EV0, although other combinations of exposure levels are possible. The demosaic-warp operation <NUM> can include any suitable process, technique, or algorithm for converting mosaic frames into multi-channel frames.

The electronic device <NUM> performs an equalization operation <NUM> on the multi-channel frames in order to equalize the frames to a common brightness level. In some embodiments, the selected common brightness level is the brightness of the EV0 frames, although other brightness levels could be selected for the common brightness level. During the equalization operation <NUM>, the other frames (such as the EV-<NUM> and EV-<NUM> frames) are equalized to the brightness of the target frame (such as the EV0 frames). In some cases, this results in at most a sixteen-bit depth per pixel. The EV0 frame is already at the desired brightness level, so the EV0 frames may be unchanged in the equalization operation <NUM> (although again this depends on the desired brightness level). The equalization operation <NUM> can include any suitable process, technique, or algorithm for equalizing brightness levels in multi-channel frames.

The electronic device <NUM> groups the multi-channel frames by exposure level and provides the groups of multi-channel frames as inputs to single-exposure (SE) blending operations <NUM>, <NUM>, and <NUM>. Each of the SE blending operations <NUM>, <NUM>, and <NUM> is performed on a group of multi-channel frames having a common exposure level. For example, the SE blending operation <NUM> may be performed on the multi-channel frames at the EV-<NUM> exposure, the SE blending operation <NUM> may be performed on the multi-channel frames at the EV-<NUM> exposure, and the SE blending operation <NUM> may be performed on the multi-channel frames at the EV0 exposure. In each of the SE blending operations <NUM>, <NUM>, and <NUM>, the multi-channel frames of the corresponding group are blended to generate a single blended image frame. The SE blending operations <NUM>, <NUM>, and <NUM> represent any suitable process, technique, or algorithm for blending multiple multi-channel frames into a single blended frame. The outputs of the SE blending operations <NUM>, <NUM>, and <NUM> are a blended, four-channel, 16xEV-<NUM>, RGBW image frame; a blended, four-channel, 4xEV-<NUM>, RGBW image frame; and a blended, four-channel, EV0 RGBW image frame, respectively, as shown in <FIG>.

The electronic device <NUM> also performs W-guided deghosting operations <NUM> and <NUM>. The W-guided deghosting operations <NUM> and <NUM> are performed to identify motion regions among cross-exposure image frames so that blending in those regions can be suppressed. This is useful since blending in motion regions can lead to ghost artifacts. The W-guided deghosting operations <NUM> and <NUM> therefore operate to estimate motion across exposure differences. To account for motion between cross-exposure frames, the frames are input to the W-guided deghosting operations <NUM> and <NUM> in a forward, pair-wise manner. For example, the 16xEV-<NUM> and 4xEV-<NUM> image frames may be inputs for the W-guided deghosting operation <NUM>, and the 4xEV-<NUM> and EV0 image frames may be inputs for the W-guided deghosting operation <NUM>. As discussed in greater detail below, the outputs of the W-guided deghosting operations <NUM> and <NUM> are motion maps for the image frame pairs.

The W-guided deghosting operations <NUM> and <NUM> can be performed in any suitable manner. <FIG> illustrates an example implementation of W-guided deghosting operations <NUM> and <NUM> in the process <NUM> of <FIG> according to this disclosure. As shown in <FIG>, the W-guided deghosting operations <NUM> and <NUM> receive the image frame pairs, which are RGBW 4C 16b frames in this example. The electronic device <NUM> provides the frames as inputs to a saturation clipping operation <NUM>. Since deghosting is performed on cross-exposure image frames, a saturation limit is established to ignore pixel differences due to dynamic range differences. The electronic device <NUM> performs the saturation clipping operation <NUM> to differentiate motion-based and range-based pixel differences. As a particular example, the saturation clipping operation <NUM> may set the saturation limit at an exposure level (such as EV-<NUM>) that depends on the exposure value of the next exposure level (such as EV-<NUM>) with respect to the reference exposure level (such as EV0). This can be expressed mathematically as follows. <MAT> <MAT> where Ix is the input image frame at the given exposure level, Iclips,x is the clipped image frame, and tsatref is the saturation threshold at the reference EV level (such as EV0). Equation (<NUM>) corresponds to the saturation clipping operation <NUM> for the 16xEV-<NUM> image frames, while Equation (<NUM>) corresponds to the saturation clipping operation <NUM> for the 4xEV-<NUM> image frames. The electronic device <NUM> also applies a gamma curve with initial linearity <NUM> on each frame to brighten up contents while avoiding the amplification of dark area noise. The electronic device <NUM> can apply any suitable gamma curve on each frame.

Unlike conventional RGB deghosting techniques, the W-guided deghosting operations <NUM> and <NUM> can achieve higher motion estimation accuracy by exploiting the high-SNR white channel as a guidance signal. Here, the electronic device <NUM> separately converts each of the red, green, and blue channels of the image frames into Y, U, and V channels using a RGB2YUV conversion operation <NUM>. The outputs of the RGB2YUV conversion operation <NUM> are Y, U, and V channels of each image frame. The electronic device <NUM> can perform any suitable process, technique, or algorithm for converting RGB channels to YUV channels. In some embodiments, the conversion operation <NUM> can be expressed mathematically as follows.

In addition, since the white channel is at a different brightness level than the RGB/YUV channels, the white channel can be equalized in brightness before being used as the guidance signal for filtering the Y channel. Accordingly, the electronic device <NUM> performs a brightness equalization operation <NUM> on the W channel. Various techniques for the brightness equalization operation <NUM> may be used. In some embodiments, using the W channel and the Y channel as inputs, the electronic device <NUM> performs the brightness equalization operation <NUM> as follows. <MAT> where Weq is the equalized white channel, and Ymedian and Wmedian are the median values of the Y and W channels respectively.

The electronic device <NUM> obtains an improved Y channel (identified as YGF) by applying a guided filter operation <NUM> to the Y channel with Weq as the guidance signal. The guided filter operation <NUM> is based on a local linear model between the filtering output q and the guidance image I at every pixel i. In some embodiments, this can be expressed as follows. <MAT> where ak, bk are linear coefficients to be optimized for a local window wk centered at pixel k. The filtering output can be optimized to be closest to the input p in a mean-square sense. For example, in some embodiments, ak and bk can be expressed as follows. <MAT> <MAT> where <MAT> is the local variance of I in wk. Of course, this is merely one example of the guided filter operation <NUM>. Other suitable processes, techniques, or algorithms are possible and within the scope of this disclosure.

The electronic device <NUM> further performs a YUV deghosting operation <NUM> using the YGF channel and the original U and V channels of the image frames. During the YUV deghosting operation <NUM>, the electronic device <NUM> processes the YGF, U, and V channels to produce more accurate motion maps MM.

Returning to <FIG>, in addition to performing the deghosting operations <NUM> and <NUM>, the electronic device <NUM> performs multiple saturation operations <NUM>, <NUM>, <NUM>, and <NUM>. The saturation operations <NUM>, <NUM>, <NUM>, and <NUM> are performed to estimate saturated pixels across exposure differences for both RGB and W channels, thus accounting for saturation between cross-exposure frames. In the saturation operations <NUM>, <NUM>, <NUM>, and <NUM>, the electronic device <NUM> processes the frames using saturation analysis in a forward, pair-wise manner. Unlike conventional RGB saturation analysis techniques, the saturation analysis in the process <NUM> is performed differently for RGB and W channels. Thus, the saturation analyses for RGB and W channels are performed separately, resulting in an RGB branch (including the saturation operations <NUM> and <NUM>) and a W branch (including the saturation operations <NUM> and <NUM>). Both saturation analyses, however, can follow the same principle: the saturation level at the current exposure level (such as EV-<NUM>) depends on the exposure value of the next exposure level (such as EV-<NUM>) with respect to the reference exposure level (such as EV0).

As can be seen in this example, the saturation operations <NUM>, <NUM>, <NUM>, and <NUM> have different input pairs. In this example, the inputs to the saturation operation <NUM> are the 4xEV-<NUM> and EV0 frames output from the SE blending operations <NUM> and <NUM>, the inputs to the saturation operation <NUM> are the 16xEV-<NUM> and 4xEV-<NUM> frames output from the SE blending operations <NUM> and <NUM>, the inputs to the saturation operation <NUM> are the 16xEV-<NUM> and 4xEV-<NUM> frames output from the SE blending operations <NUM> and <NUM>, and the inputs to the saturation operation <NUM> are the 4xEV-<NUM> and EV0 frames output from the SE blending operations <NUM> and <NUM>. The outputs of the saturation operations <NUM>, <NUM>, <NUM>, and <NUM> are saturation maps MS of the input pairs. In some embodiments, the saturation operation <NUM> generates a saturation map MS,<NUM>×EV-<NUM> as follows. <MAT> Also, in some embodiments, the saturation operation <NUM> generates a saturation map MS,<NUM>×EV-<NUM> as follows: <MAT> Further, in some embodiments, the saturation operation <NUM> generates a saturation map MS,<NUM>×EV-<NUM> as follows: <MAT> In addition, in some embodiments, the saturation operation <NUM> generates a saturation map MS,<NUM>×EV-<NUM> as follows: <MAT> In Equations (<NUM>)-(<NUM>), IW and IRGB represent the white and RGB channels, respectively, of the image frames, σsatref is the saturation standard deviation at the reference EV level (such as EV0), and tsatref is the saturation threshold at the reference EV level.

Using the motion maps generated in the deghosting operations <NUM> and <NUM> and the saturation maps generated in the saturation operations <NUM>, <NUM>, <NUM>, and <NUM>, the electronic device <NUM> performs backward multi-exposure (ME) blending operations <NUM> and <NUM>. The electronic device <NUM> performs the backward ME blending operation <NUM> to achieve saturation recovery in the RGB branch, and the electronic device <NUM> performs the backward ME blending operation <NUM> to achieve saturation recovery in the W branch. The backward ME blending operations <NUM> and <NUM> are similar but slightly different processes due to the inclusion of the white channel in the backward ME blending operation <NUM>. The outputs after RGB and W saturation recovery are concatenated to form a saturation-free RGBW frame as discussed below.

The backward ME blending operations <NUM> and <NUM> can be performed in any suitable manner. <FIG> illustrates an example implementation of backward multi-exposure blending operations <NUM> and <NUM> in the process <NUM> of <FIG> according to this disclosure. As shown in <FIG>, the backward ME blending operations <NUM>, <NUM> recover saturations incrementally from the multiple exposure levels available, starting at the highest EV level (such as EV0) and working to the lowest EV (such as EV-<NUM>) (hence the term backward). This incremental approach allows for a more graceful transition between exposure levels in a multi-exposure system, resulting in more natural HDR images.

In this example, the motion map MM,<NUM>×EV-<NUM> and the saturation map MS,<NUM>×EV-<NUM> are combined using a multiplier operator 405a to generate a combined map M<NUM>×EV-<NUM>. The combined map M<NUM>×EV-<NUM> is combined with the image frame I<NUM>×EV-<NUM> using a multiplier operator 405b. The combined map M<NUM>×EV-<NUM> is also input to a complement operator 410a, which performs a two's complement operation (such as <NUM>-x) on the combined map M<NUM>×EV-<NUM>. The motion map MM,<NUM>×EV-<NUM> and the saturation map MS,<NUM>×EV-<NUM> are combined using a multiplier operator 405c to generate a combined map M<NUM>×EV-<NUM>. The combined map M<NUM>×EV-<NUM> is combined with the image frame I<NUM>×EV-<NUM> using a multiplier operator 405d. The combined map M<NUM>×EV-<NUM> is also input to a complement operator 410b, which performs a two's complement operation on the combined map M<NUM>×EV-<NUM>. The image frame IEV0 is combined with the output of the complement operator 410b using a multiplier operator 405e. That output is added to the output of the multiplier operator 405d using a summation operator 415a. The output of the summation operator 415a is combined with the output of the complement operator 410a using a multiplier operator 405f. That output is added to the output of the multiplier operator 405b using a summation operator 415b. The output of the summation operator 415b is either the RGB channels or the W channel of the saturation-recovered image frame, depending on the whether the backward ME blending operation <NUM> or the backward ME blending operation <NUM> is performed.

Returning to <FIG>, the electronic device <NUM> performs an interband denoise and sharpening operation <NUM> to fuse the W channel with the RGB channels in the four-channel RGBW image frame to create a three-channel RGB image frame. In some cases, the three-channel RGB image frame may still have a high dynamic range, such as a sixteen-bit depth. The interband denoise and sharpening operation <NUM> represents any suitable process, technique, or algorithm for fusing a four-channel RGBW image frame into a three-channel RGB image frame.

The electronic device <NUM> performs a tone mapping operation <NUM> on the three-channel RGB frame to convert the frame with a display dynamic range (such as a sixteen-bit depth) to a lower bit depth (such as an eight-bit depth). Various techniques may be used to perform the tone mapping operation <NUM>. The electronic device <NUM> also performs one or more post-processing operations <NUM> on the three-channel RGB frame with the lower bit depth in order to improve the image quality. Various post-processing techniques may be used in the post-processing operations <NUM>.

Note that the operations and functions shown in <FIG> can be implemented in an electronic device <NUM>, server <NUM>, or other device in any suitable manner. For example, in some embodiments, the operations and functions shown in <FIG> can be implemented or supported using one or more software applications or other software instructions that are executed by the processor <NUM> of the electronic device <NUM>, server <NUM>, or other device. In other embodiments, at least some of the operations and functions shown in <FIG> can be implemented or supported using dedicated hardware components. In general, the operations and functions shown in <FIG> can be performed using any suitable hardware or any suitable combination of hardware and software/firmware instructions.

Although <FIG> illustrate one example of a process <NUM> for multi-exposure, multi-frame blending of RGBW images and related details, various changes may be made to <FIG>. For example, while the motion maps determined by the W-guided deghosting operations <NUM> and <NUM> are shared between the RGB and W branches, the motion maps for RGB and W can be computed separately in other embodiments. For instance, RGB signals can be converted to YUV for the YUV deghosting operation <NUM>. The YUV deghosting operation <NUM> can also treat the W signal as Y and ignore the chroma effect from U and V. Also, while shown as a specific sequence of operations, various operations shown in <FIG> could overlap, occur in parallel, occur in a different order, or occur any number of times (including zero times). Further, the specific operations shown in <FIG> are examples only, and other techniques could be used to perform each of the operations shown in <FIG>. In addition, the W-guided deghosting operations and the backward multi-exposure blending operations may be implemented in any other suitable manner.

<FIG> illustrates an example method <NUM> for multi-exposure, multi-frame blending of RGBW images according to this disclosure. For ease of explanation, the method <NUM> shown in <FIG> is described as involving the use of the process <NUM> shown in <FIG> and the electronic device <NUM> shown in <FIG>. However, the method <NUM> shown in <FIG> could be used with any other suitable electronic device (such as the server <NUM>) and in any suitable system.

As shown in <FIG>, multiple images of a scene are obtained using at least one RGBW image sensor at step <NUM>. This could include, for example, the electronic device <NUM> capturing the images <NUM> of the scene using at least one RGBW image sensor <NUM>.

The multiple images are aligned to generate motion vectors at step <NUM>. This could include, for example, the electronic device <NUM> performing the alignment operation <NUM> on the images <NUM>. The images are converted into multi-channel frames using the motion vectors at step <NUM>. This could include, for example, the electronic device <NUM> performing the demosaic-warp operation <NUM> to convert the image <NUM> into multi-channel frames. The multi-channel frames are equalized to a common brightness level at step <NUM>. This could include, for example, the electronic device <NUM> performing the equalization operation <NUM> on the multi-channel frames in order to equalize the frames to a common brightness level.

The multi-channel frames are grouped and blended at the different exposure levels at step <NUM>. This could include, for example, the electronic device <NUM> performing the SE blending operations <NUM>, <NUM>, and <NUM> on a group of multi-channel frames having a common exposure level. The operations and functions shown in stage <NUM> through <NUM> generate multi-channel frames at different exposure levels from the images.

Motion across exposure differences between the different exposure levels is estimated using a white channel of the multi-channel frames as a guidance signal in order to generate multiple motion maps at step <NUM>. This could include, for example, the electronic device <NUM> performing the W-guided deghosting operations <NUM> and <NUM> to generate the motion maps. Saturation across the exposure differences between the different exposure levels is estimated to generate multiple saturation maps at step <NUM>. This could include, for example, the electronic device <NUM> performing the saturation operations <NUM>, <NUM>, <NUM>, and <NUM> to generate the saturation maps.

Using the generated motion maps and saturation maps, saturations are recovered from the different exposure levels and a saturation-free RGBW frame is generated at step <NUM>. This could include, for example, the electronic device <NUM> performing the backward ME blending operations <NUM> and <NUM>. The W channel is fused with the RGB channels in the saturation-free RGBW frame to generate a three-channel RGB frame at step <NUM>. This could include, for example, the electronic device <NUM> performing the interband denoise and sharpening operation <NUM> on the RGBW image frame. The three-channel RGB frame is converted from a first bit depth to a second bit depth at step <NUM>. This could include, for example, the electronic device <NUM> performing the tone mapping operation <NUM> on the three-channel RGB image frame. Post-processing is performed on the converted three-channel RGB frame to generate the final image at step <NUM>. This could include, for example, the electronic device <NUM> performing one or more post-processing operations <NUM> on the three-channel RGB image frame.

Although <FIG> illustrates one example of a method <NUM> for multi-exposure, multi-frame blending of RGBW images, various changes may be made to <FIG>. For example, while shown as a series of steps, various steps in <FIG> could overlap, occur in parallel, occur in a different order, or occur any number of times.

Claim 1:
A method (<NUM>) comprising:
obtaining (<NUM>) multiple multi-exposure images of a scene using at least one red-green-blue-white, RGBW, image sensor;
generating (<NUM>) single multi-channel frames at different exposure levels from the images;
generating multiple motion maps by estimating (<NUM>) motion across exposure differences between the multi-channel frames at different exposure levels using a white channel of the multi-channel frames as a guidance signal;
generating multiple saturation maps by estimating (<NUM>) saturated pixels across the exposure differences between the multi-channel frames at different exposure levels for the red, green and blue channels of the multi-channel frames and separately for the white channel of the multi-channel frames;
using (<NUM>) the generated motion maps and saturation maps to achieve red, green and blue saturation recovery and white saturation recovery from the different exposure levels and concatenating outputs to generate a saturation-free RGBW frame; and
processing the saturation-free RGBW frame to generate (<NUM>) a final image of the scene.