Method and apparatus for generating HDR images

A method for generating a high-dynamic-range (HDR) image using an imaging device is provided. The method includes capturing a long-exposure (LE) image, a short-exposure (SE) image, and an auto-exposure (AE) image of a scene, estimating motion information in the SE image and the LE image based on a reference image which is determined based on an image statistics parameter, aligning the SE image and the LE image using the motion information, generating a pixel-weight coefficient for each of the SE image, the LE image, and the AE image, generating an overlapped region mask corresponding to an overlapped region in each of the SE image, the LE image and the LE image, determining a modified pixel-weight coefficient in the overlapped region mask and correcting a brightness difference, and generating an HDR image from the SE image, the LE image and the AE image using the modified at least one pixel-weight coefficient.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119(a) of an Indian provisional patent application number 201741013144, filed on Apr. 12, 2017, in the Indian Patent Office, and of an Indian patent application number 201741013144, filed on Mar. 6, 2018, in the Indian Patent Office, the disclosure of each of which is incorporated by reference herein in its entirety.

BACKGROUND

The disclosure relates to a content processing system. More particularly, the disclosure relates to a method and apparatus for generating a high-dynamic-range (HDR) image of a scene.

2. Description of the Related Art

A fisheye lens is an ultra-wide-angle lens that produces a strong visual distortion intended to create a wide panoramic or hemispherical image of an exposed scene, called as a fisheye image. The fisheye lens utilizes a special mapping for projecting the exposed scene into a characteristic convex non-rectilinear appearance. An imaging device captures a high-dynamic-range (HDR) image including the fisheye images of a front scene and a rear scene.

Thus, it is desired to address the above mentioned disadvantages or other shortcomings or at least provide a useful alternative.

SUMMARY

Aspects of the disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the disclosure is to provide an apparatus and method for generating a high-dynamic-range (HDR) image.

In accordance with an aspect of the disclosure, a method for generating an HDR image of a scene using an imaging device is provided. The method includes capturing a long-exposure (LE) image, a short-exposure (SE) image, and an auto-exposure (AE) image of the scene. Further, the method includes estimating motion information in the SE image and the LE image based on a reference image. The motion information include at least one of a translation and a rotation of the imaging device in undistorted regions of the AE image and the at least one of the SE image and the LE image. The reference image is determined based on an image statistics parameter. Further, the method includes aligning individually the SE image and the LE image using the estimated motion information. Further, the method includes generating a pixel weight coefficient for each of the SE image, the LE image and the AE image. Further, the method includes generating an overlap region mask corresponding to an overlap region in each of the SE image, the LE image and the AE image. Further, the method includes modifying the pixel weight coefficient in the generated overlapped region mask to correct a brightness difference. Further, the method includes fusing the SE image, the LE image and the AE image using the modified pixel-weight coefficient to generate the HDR image. Further, the method includes storing the HDR image.

DETAILED DESCRIPTION

In an embodiment, estimating the motion information in the short-exposure (SE) image and the long-exposure (LE) image based on the reference image includes detecting feature-points in undistorted regions of the reference image and at least one of the SE image and the LE image, and determining pairwise matches in the undistorted region based on the detected feature-points.

In an embodiment, the brightness difference is corrected by determining a distance between each pixel location from a center of the auto-exposure (AE) image, detecting whether the distance meets a threshold, and performing one of modulating a high-dynamic-range (HDR) weight of each of the SE image and the LE image in response to determining that the distance meets the threshold, and using a default weight of each of the SE image and the LE image and the AE image in response to determining that the distance does not meet the threshold.

In an embodiment, the HDR weight of each of the SE image and the LE image is modulated by suppressing weights of the LE image and the SE image and enhancing weights of the AE image based on at least one of a distance from a center of the AE image and pixel difference between a front view and a rear view.

In an embodiment, the imaging device is configured to capture the LE image, the SE image, and the AE image in an HDR mode.

In an embodiment, the HDR image is preferably a 360-degree (360-degree wide) image.

Accordingly, the various embodiments herein provide a method for generating an HDR image of a scene in an imaging device. The method includes capturing an LE image, an SE image, and an AE image of the scene. Further, the method includes estimating motion information in the SE image and the LE image based on a reference image. The reference image is determined based on an image statistics parameter. Further, the method includes aligning individually the SE image and the LE image using the estimated motion information. Further, the method includes generating a pixel weight coefficient for each of the SE image, the LE image and the AE image. Further, the method includes fusing the SE image, the LE image and the AE image using the pixel-weight coefficient to generate the 360-degree wide HDR image. Further, the method includes storing the HDR image.

Accordingly the various embodiments herein provide a method for generating an HDR image of a scene in an imaging device. The method includes capturing an HDR image of the scene. Further, the method includes correcting at least one of an alignment error and a brightness error in the captured HDR image. Further, the method includes generating the HDR image based on the correction. Further, the method includes storing the HDR image.

Accordingly the various embodiments herein disclose an imaging device for generating an HDR image of a scene. The imaging device includes at least one sensor, an alignment controller, a weight-map generator, a brightness correction controller, a fusion controller and a memory. Further, the sensor is configured to capture an LE image, an SE image, and an AE image of the scene. Further, the alignment controller is configured for estimating a motion information in the SE image and the LE image based on a reference image, and aligning individually the SE image and the LE image using the estimated motion information, wherein the motion information include at least one of a translation and a rotation of the imaging device in undistorted regions of the AE image and the at least one of the SE image and the LE image, and wherein the reference image is determined based on an image statistics parameter. Further, the weight-map generator is configured for generating a pixel weight coefficient for each of the SE image, the LE image and the AE image. Further, the brightness correction controller is configured for generating an overlap region mask corresponding to an overlap region in each of the SE image, the LE image and the AE image, and modifying the pixel weight coefficient in the generated overlapped region mask to correct a brightness difference. Further, the fusion controller is configured for fusing the SE image, the LE image and the AE image using the modified pixel-weight coefficient to generate the HDR image. Further, the memory is configured to store the HDR image.

Accordingly the various embodiments herein disclose an imaging device for generating an HDR image of a scene. The imaging device includes at least one sensor, an alignment controller, a weight-map generator, a fusion controller and a memory. Further, the sensor is configured for capturing a LE image, a SE image, and an AE image of the scene. Further, the alignment controller is configured for estimating a motion information in the SE image and the LE image based on a reference image, and aligning individually the SE image and the LE image using the estimated motion information. Further, the weight-map generator is configured for generating a pixel weight coefficient for each of the SE image, the LE image and the AE image. Further, the fusion controller is configured for fusing the SE image, the LE image and the AE image using the pixel-weight coefficient to generate the HDR image. Further, the memory is configured for storing the HDR image.

Accordingly, the various embodiments herein disclose an imaging device for generating an HDR image of a scene. The imaging device includes at least one sensor, an HDR image generator and a memory. Further, the sensor is configured for capturing a LE image, a SE image, and an AE image of the scene. Further, the HDR image generator is configured for correcting at least one of an alignment error and a brightness error in the captured HDR image, and generating the HDR image based on the correction. Further, the memory is configured for storing the HDR image.

Accordingly the various embodiments herein provide a method for generating an HDR (preferably 360-degree covering) image of a scene using an imaging device. The method includes capturing a LE image, a SE image, and an AE image of the scene. Further, the method includes estimating a motion information in the SE image and the LE image based on a reference image, wherein the motion information include at least one of a translation and a rotation of the imaging device in undistorted regions of the AE image and the at least one of the SE image and the LE image, and wherein the reference image is determined based on an image statistics parameter. Further, the method includes aligning individually the SE image and the LE image using the estimated motion information. Further, the method includes generating a pixel weight coefficient for each of the SE image, the LE image and the AE image. Further, the method includes generating an overlap region mask corresponding to an overlap region in each of the SE image, the LE image and the AE image. Further, the method includes modifying the pixel weight coefficient in the generated overlapped region mask to correct a brightness difference. Further, the method includes fusing the SE image, the LE image and the AE image using the modified pixel-weight coefficient to generate the HDR image. Further, the method includes storing the HDR image.

The method can be used to generate the HDR image with better quality by processing the SE image, the LE image and the AE image in a fish-eye domain itself without any rectilinear conversion. This results in generating the HDR image in a quick manner. The method can be used to generate the HDR image by processing the SE image, the LE image and the AE image in the fish-eye domain itself with less computational power.

The proposed method can be used to generate the fisheye HDR images within the imaging device without using high end hardware elements. The proposed method can be used to generate the fisheye HDR images within the imaging device in a cost effective manner.

The imaging device can be used to generate the fisheye HDR image of the scene comprising a moving object. The imaging device aligns the SE image and LE image of the scene to compensate the motion of an object in the scene while capturing the images. Thus a blurred image formation can be removed in the 360-degree HDR image.

In the proposed method, the formation of a visible line in an overlapping region of stitched rectilinear HDR images of the front scene and the rear scene is removed by providing a brightness correction at the overlapping region. Thus the HDR images attain a better quality by maintaining the continuity of the objects in the stitched rectilinear HDR images of the front scene and the rear scene, without making any pixel misalignment or color imperfections.

In photography, bracketing is a general technique of taking several shots of the same scene using a different camera settings for exposure, flash, white balance, international organization for standardization (ISO) levels, focus etc. HDR imaging is an image processing technique to increase a dynamic range of the scene by fusing/merging the exposure bracketed images. The exposure bracketed images are captured with different exposure time in subsequent images. The exposure time refers to an effective amount of light hitting on a sensor. In an example, a LE image might be considered as over-exposed image to create a bright image, while the SE image might be considered as under-exposed to create a dark image. An AE might be considered as medium-exposed image to create the image with the bright and dark balance. The HDR imaging presents a more versatile and natural image representation in line with human vision.

An HDR image can be generated by merging the exposure bracketed images from the same camera for the same scene. However, due to hand shaking or environmental conditions, some camera motion (typically rotational and/or translational) should be expected between each picture. Hence, a misalignment of the exposure bracketed images may occur due to camera motion which results in a blurry HDR image.

Implementing the HDR imaging to generate a fisheye HDR image on an imaging device, which has a limited computational power and memory, has various challenges associated with it.

Before fusing the exposure bracketed images in the imaging device, the images have to be converted into the corresponding rectilinear images, to generate a rectilinear HDR image. This results in an excess of time consumption on the imaging device having an accompanying processor which has low computational power and memory.

Due to a low working space memory in the imaging device, the LE image, the AE image and the SE image of the scene are processed independently to generate the fisheye HDR image of the scene. This can cause a brightness mismatch in the fisheye HDR images of the front scene and the rear scene.

Referring now to the drawings, and more particularly toFIGS. 1 through 19, there are shown various embodiments.

FIG. 1illustrates an overview of a system for generating a rectilinear HDR image from exposure bracketed fisheye images according to an embodiment of the disclosure.

Referring toFIG. 1, the system includes an imaging device100and an electronic device200. The exposure bracketed fisheye images include an LE image of the scene, an AE image of the same scene, and an SE image of the same scene.

The LE image, the AE image and the SE image of the scene from the imaging device100are transferred to the electronic device200. The electronic device200converts the LE image, the AE image and the SE image to corresponding rectilinear-LE image, rectilinear-AE image and rectilinear-SE image respectively. Further, the rectilinear-LE image, the rectilinear-AE image and the rectilinear-SE image are combined together by an HDR controller202in the electronic device200to form the rectilinear HDR image of the scene. Further, a rectilinear image can be generated by stitching the rectilinear HDR images in the electronic device200.

FIG. 2illustrates a fisheye image of a front scene of an imaging device according to an embodiment of the disclosure.

Referring toFIG. 2, the fisheye lens uses the special mapping (e.g., equisolidangle projection) for projecting the exposed scene into the characteristic convex non-rectilinear appearance. Hence, the straight lines in the real world have become curvy in the fisheye image of the scene.

FIG. 3illustrates a visual distortion in various regions in a fisheye image according to an embodiment of the disclosure.

Referring toFIG. 3, while moving away from the central region of the fisheye image, the distortion is gradually increased and maximum at an edge part of the fisheye image. Further, a pure translational motion parallel to a plane has a different impact based on distance from an image center.

Further, the pixels away from the image center have more spherical distortion than other regions in the fish-eye image. Thus, image alignment in the fish-eye domain has following technical challenges such as estimating the parameters of motion-model (translation and rotation), and performing the alignment in the fish-eye domain considering spherical distortion aspects.

FIG. 4illustrates an overlapping region in fisheye images of a front scene and a rear scene of an imaging device according to an embodiment of the disclosure.

Referring toFIG. 4, the region between an outer circle and an inner circle represent the overlapping region in both the fisheye images of the front scene and the rear scene of the imaging device100. The brightness mismatch is formed at this region during the generation of HDR image.

FIG. 5illustrates a visible line in stitched rectilinear HDR images of a front scene and a rear scene according to an embodiment of the disclosure.

Referring toFIG. 5, due to a low working space memory in the imaging device100, the LE image, the AE image and the SE image of the scene are processed independently to generate the fisheye HDR image of the front and rear scenes. This may cause the brightness mismatch at the overlapping region in the HDR images of the front scene and the rear scene upon stitching the rectilinear HDR images and to form the stitched rectilinear HDR image. Thus, the stitched rectilinear HDR image has the visible line at the overlapping region.

FIG. 6illustrates an overview of a system for generation of a rectilinear HDR image according to an embodiment of the disclosure.

Referring toFIG. 6, in an embodiment, a system1000includes an imaging device100and an electronic device200. The imaging device100can comprise, for example, but not limited to, a camera device, a Gear360device (i.e., a head mounted display (HIVID) for virtual or augmented reality, etc.), or the like. The electronic device200can be, for example, but not limited to, a smartphone, a tablet computer, a mobile device, a personal digital assistant (PDA), a multimedia device, a video device, or the like.

Referring toFIG. 6, the imaging device100includes a sensor (not shown) and an HDR image generator110. The sensor is configured to capture the LE image, the SE image, and an AE image of the scene. After capturing the LE image, the SE image, and the AE image of the scene, the HDR image generator110is configured to generate the HDR image based on the LE image, the SE image, and the AE image.

In an embodiment of the disclosure, the HDR image generator110may estimate motion information in the SE image and the LE image based on a reference image. Further, the HDR image generator110may align individually the SE image and the LE image using the estimated motion information.

In an embodiment of the disclosure, the motion information includes at least one of a translation and a rotation of the imaging device100in undistorted regions of the AE image and the at least one of the SE image and the LE image.

In an embodiment, the motion information in the SE image and the LE image is estimated by detecting feature-points in undistorted regions of in the AE image as the reference image and at least one of the SE image and the LE image, and determining pairwise matches in the undistorted region based on the detected feature-points.

In the embodiment of the disclosure, the reference image is determined based on an image statistics parameter (e.g., mean brightness value or the like).

After aligning the SE image and the LE image, the HDR image generator110may generate a pixel weight coefficient for each of the SE image, the LE image and the AE image.

Further, the HDR image generator110may generate an overlap region mask corresponding to the overlap region in each of the SE image, the LE image and the AE image, and modify the pixel weight coefficient in the generated overlapped region mask to correct a brightness difference.

After correcting the brightness difference, the HDR image generator110may fuse or combine the SE image, the LE image and the AE image using the pixel-weight coefficient to generate a fused image.

In an embodiment of the disclosure, the brightness difference is corrected by determining a distance between each pixel location from a center of the AE image, detecting whether the distance meets a threshold, and modulating an HDR weight of each of the SE image and the LE image in response to determining that the distance meets the threshold.

In an embodiment of the disclosure, the brightness difference is corrected by determining the distance between each pixel location from the center of the AE image, detecting whether the distance meets the threshold, and using a default weight of each of the SE image and the LE image and the AE image in response to determining that the distance does not meets a predetermined threshold.

In an embodiment, the HDR weight of each of the SE image and the LE image is modulated by suppressing weights of the LE image and the SE image and enhancing weights of the AE image based on the distance from the center of the AE image.

After generating the HDR image, the imaging device100stores the HDR image. Further, the HDR image is transferred to the electronic device200to convert the HDR image to the rectilinear HDR image.

The imaging device100can be operated in a single lens mode or in a dual lens mode. In the single lens mode, while capturing the scene, either the front sensor or the rear sensor becomes active to capture the scene by the imaging device100. In the dual lens mode, both the front and rear sensors become active to capture the scene by the imaging device100.

In an embodiment of the disclosure, the imaging device100may capture the LE image, the SE image, and the AE image in an HDR mode.

AlthoughFIG. 6shows the hardware components of the system1000but it is to be understood that other embodiments are not limited thereon. In other embodiments, the system1000may include less or more number of components. Further, the labels or names of the components are used only for illustrative purpose and does not limit the scope of the disclosure. One or more components can be combined together to perform same or substantially similar function to generate the HDR image. For example, the image generator may be implemented as at least one hardware processor.

FIG. 7is a block diagram illustrating an imaging device according to various embodiments of the disclosure.

Referring toFIG. 7, in an embodiment of the disclosure, the imaging device100includes the HDR image generator110, the sensor120, a processor130, a memory140, and a communicator150. The HDR image generator110includes an alignment controller111, a weight-map generator112, a brightness correction controller113, and a fusion controller114. The processor130, communicator150, and the HDR image generator may be implemented as at least one hardware processor(s).

The sensor120may capture the LE image, the SE image, and the AE image of the scene. After capturing the LE image, the SE image, and the AE image of the scene, the alignment controller111may estimate the motion information in the SE image and the LE image based on the AE image as the reference image. Further, the alignment controller111may align individually the SE image and the LE image using the estimated motion information. The motion information includes at least one of the translation and the rotation of the imaging device100in undistorted regions of the AE image and the at least one of the SE image and the LE image.

In an embodiment of the disclosure, the alignment controller111may detect the feature-points in the undistorted regions of in the AE image as the reference image and at least one of the SE image and the LE image. Based on the detected feature-points, the alignment controller111may determine the pairwise matches in the undistorted region. Based on the pairwise matches, the alignment controller111may estimate the motion information.

Further, weight-map generator112may generate the pixel weight coefficient for each of the SE image, the LE image and the AE image. The brightness correction controller113is configured to generate the overlap region mask corresponding to the overlap region in each of the SE image, the LE image and the AE image. Further, the brightness correction controller113may modify the pixel weight coefficient in the generated overlapped region mask to correct a brightness difference.

In an embodiment of the disclosure, the weight-map generator112determines the pixel weight coefficient for each of the LE image, the AE image and the SE image using a look-up table (LUT).

In an embodiment of the disclosure, the weight-map generator112may determine a ghost map for each of the LE image, the AE image and the SE image. The pixel weight coefficient of each of the LE, SE and AE images are modified based on the computed ghost map distribution for each images.

The weight-map generator112determines a modified pixel weight coefficient for each of the LE image, the AE image and the SE image by adjusting the pixel weight coefficient based on the computed ghost map. The weight-map generator112generates the LE image, the AE image and the SE image with the modified pixel weight coefficients for of the LE image, the AE image and the SE image. The weight-map generator112transfers the LE image, the AE image and the SE image having the modified pixel weight coefficient to the brightness correction controller113.

In an embodiment of the disclosure, the brightness correction controller113may determine a distance between each pixel location from the center of the AE image. Further, the brightness correction controller113may detect whether the distance meets a predetermined threshold. The threshold is set by the user or an original equipment manufacturer (OEM) in advance. Further, the brightness correction controller113may modulate the HDR weight of each of the SE image and the LE image in response to a determination that the distance meets the predetermined threshold.

In an embodiment of the disclosure, the brightness correction controller113may determine a distance between each pixel location from the center of the AE image. Further, the brightness correction controller113may detect whether the distance meets a predetermined threshold. Further, the brightness correction controller113may utilize the default weight of each of the SE image and the LE image in response to a determination that the distance does not meets the predetermined threshold.

In an embodiment of the disclosure, the HDR weight of each of the SE image and the LE image is modulated by suppressing weights of the LE image and the SE image and enhancing weights of the AE image based on the distance from the center of the AE image.

In an embodiment of the disclosure, the brightness correction controller113may correct the brightness in the LE image, the AE image and the SE image having the modified pixel weight coefficient. The brightness correction controller113modifies the pixel weight coefficient with the normal/default weights to correct the brightness in the LE image, the AE image and the SE image in the single lens mode.

In an embodiment of the disclosure, the brightness correction controller113may modify the pixel weight coefficient in an overlapping region with the modulated weights to correct the brightness difference in the overlapping region in the LE image and the SE image of the front and rear scenes in the dual lens mode.

The brightness correction controller113can transfer the LE image, the AE image and the SE image to the fusion controller114after performing brightness correction.

Further, the fusion controller may fuse the SE image, the LE image and the AE image using the modified pixel-weight coefficient to generate an HDR image.

After generating the HDR image, the memory140stores the HDR image.

Further, the HDR image is transferred to an electronic device200to convert the HDR image in the fisheye domain to the rectilinear HDR image.

The fusion controller114merges/fuses the LE image, AE image and SE image to generate the HDR image.

The HDR image generator110can be coupled to the sensor120, the processor130, the memory140and the communicator150. The processor130may execute instructions stored in the memory140and perform various operations.

The memory140stores instructions to be executed by the processor130. The memory140also stores the captured LE image, AE image and SE image of the scene. Further, the memory140receives and stores the HDR image generated from the HDR image generator110. The memory140may include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories.

In addition, the memory140may, in some examples, be considered a non-transitory storage medium. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted that the memory140is non-movable. In some examples, the memory140can be configured to store larger amounts of information than the memory. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).

The communicator150may enables the imaging device100to communicate with the electronic device200via a wired or a wireless communications. The imaging device100transmits the generated HDR image to the electronic device200through the communicator150.

In an embodiment of the disclosure, the reference image as the AE image is merely used for explanation purpose. But, in the implementation, any of the input images can be selected as the reference image based on a reference selection criterion. The reference selection criterion is set by the user or the OEM in advance.

AlthoughFIG. 7shows the hardware components of the imaging device100but it is to be understood that other embodiments are not limited thereon. In other embodiments, the imaging device100may include less or more number of components. Further, the labels or names of the components are used only for illustrative purpose and does not limit the scope of the disclosure. One or more components can be combined together to perform same or substantially similar function to generate the HDR image.

FIG. 8illustrates a diagram of various operations involved in generation of an HDR image according to an embodiment of the disclosure.

Referring toFIG. 8, initially, the imaging device100may capture the LE image, the AE image and the SE image of the scene using the sensor120in operation810. In operation820, the sensor120transfers the captured LE, AE and SE images of the scene to the alignment controller111in the imaging device100. The alignment controller111determines the motion model for the LE image and the SE image with respect to the reference AE image to estimate the motion information of the imaging device100during capturing the LE image, the AE image and the SE image. The motion information is used to determine the motion model for the LE image and the SE image. The motion information is estimated by performing the pairwise match of the motion information of the feature points in the LE image and the SE image with reference to motion information of the feature points in the AE image.

Further the alignment controller111aligns the LE image and the SE image by estimating a new pixel location for each pixel in the LE image and the SE image based on the motion model. The new pixel location of each pixel in the LE image and the SE image are used to generate the aligned LE and SE images.

In operation830, the LE image and the SE image and the AE image are transferred to the weight-map generator112. The weight-map generator112determines the pixel weight coefficient for each of the LE image, the AE image and the SE image using the LUT. Further, the weight-map generator112determines the ghost map for each of the LE image, the AE image and the SE image.

The weight-map generator112determines the modified pixel weight coefficient for each of the LE image, the AE image and the SE image by adjusting the pixel weight coefficient based on the corresponding computed ghost map. Finally, the weight-map generator112generates the LE image, the AE image and the SE image with modified pixel weight coefficients.

In operations840and850, the LE image, the AE image and the SE image having the modified pixel weight coefficient are further transferred to the brightness correction controller113. The brightness correction controller113detects the lens mode (i.e., single lens mode or dual lens mode) of the imaging device100. The brightness correction controller113uses the pixel weight coefficient with the normal/default weights, upon detecting the lens mode as the single lens mode.

Upon detecting the lens mode as the dual lens mode, the brightness correction controller113modifies the pixel weight coefficient with the modulated weights in the overlapping region of the LE image, the AE image and the SE image of the scenes, to correct the brightness difference in the overlapping regions.

In operation850, the LE image, the AE image and the SE image from the brightness correction controller113are transferred to the fusion controller114. The fusion controller114merges/fuses the LE image, the AE image and the SE image to generate an HDR image in operation860. The foregoing operations can be performed within the imaging device100, so that the operations avoid the intermediate domain conversion, so as to reduce processing time for generating the HDR image. The foregoing operations can be used to generate the HDR image on the imaging device100with less computational power.

FIG. 9Ais a flow diagram illustrating a method for generating an HDR image according to various embodiments of the disclosure.

Referring toFIG. 9A, a flow diagram S9000illustrates generating an HDR image. In operation S9002, the LE image, the SE image, and the AE image of the scene are captured by the sensor120of the imaging device100. In operation S9004, the method includes estimating the motion information in the LE image and the SE image based on the AE image as a reference image. The motion information include at least one of the translation and the rotation of the imaging device100in the undistorted regions of the AE image and the at least one of the SE image and the LE image. In an embodiment of the disclosure, the method allows the alignment controller111to estimate the motion information in the LE image and the SE image based on the AE image as the reference image.

In operation S9006, the method includes aligning individually the SE image and the LE image using the estimated motion information. In an embodiment of the disclosure, the method allows the alignment controller111to align individually the SE image and the LE image using the estimated motion information.

In operation S9008, the method includes generating the pixel weight coefficient for each of the SE image, the LE image and the AE image. In an embodiment of the disclosure, the method allows the weight-map generator112to generate the pixel weight coefficient for each of the SE image, the LE image and the AE image.

In operation S9010, the method includes generating the overlap region mask corresponding to the overlap region in each of the SE image, the LE image and the AE image. In an embodiment of the disclosure, the method allows the brightness correction controller113to generate the overlap region mask corresponding to the overlap region in each of the SE image, the LE image and the AE image. In operation S9012, the method includes modifying the pixel weight coefficient in the generated overlapped region mask to correct the brightness difference. In an embodiment of the disclosure, the method allows the brightness correction controller113to modify the pixel weight coefficient in the generated overlapped region mask to correct the brightness difference.

In operation S9014, the method includes fusing the SE image, the LE image and the AE image using the pixel-weight coefficient to generate the HDR image. In an embodiment of the disclosure, the method allows the fusion controller114to fuse the SE image, the LE image and the AE image using the pixel-weight coefficient to generate the HDR image. In operation S9016, the method includes storing the HDR image. In an embodiment of the disclosure, the method allows the memory140to store the HDR image.

FIG. 9Bis a flow diagram illustrating a method for generating an HDR image according to various embodiments of the disclosure.

Referring toFIG. 9B, another example flow diagram S9200for generating an HDR image is illustrated. In operation S9202, the method includes capturing the LE image, the SE image, and the AE image of the scene. In an embodiment of the disclosure, the method allows the sensor120to capture the LE image, the SE image, and the AE image of the scene. In operation S9204, the method includes estimating the motion information in the SE image and the LE image based on the reference image. The reference image is determined based on the image statistics parameter. In an embodiment of the disclosure, the method allows the alignment controller111to estimate the motion information in the SE image and the LE image based on the reference image, wherein the reference image is determined based on the image statistics parameter.

In operation S9206, the method includes aligning individually the SE image and the LE image using the estimated motion information. In an embodiment of the disclosure, the method allows the alignment controller111to align individually the SE image and the LE image using the estimated motion information. In operation S9208, the method includes generating the pixel weight coefficient for each of the SE image, the LE image and the AE image. In an embodiment of the disclosure, the method allows the weight-map generator112to generate the pixel weight coefficient for each of the SE image, the LE image and the AE image.

In operation S9210, the method includes fusing the SE image, the LE image and the AE image using the pixel-weight coefficient to generate the HDR image. In an embodiment of the disclosure, the method allows the fusion controller114to fuse the SE image, the LE image and the AE image using the pixel-weight coefficient to generate the HDR image. In operation S9212, the method includes storing the HDR image.

FIG. 9Cis a flow diagram illustrating a method for generating an HDR image by correcting an alignment error and a brightness error according to various embodiments of the disclosure.

Referring toFIG. 9C, another example flow diagram S9400for generating an HDR image is illustrated. In operation S9402, the method includes capturing the HDR image of the scene. In an embodiment of the disclosure, the method allows the sensor120to capture the HDR image of the scene. In operation S9404, the method includes correcting at least one of the alignment error and the brightness error in the captured HDR image. In an embodiment of the disclosure, the method allows the HDR image generator110to correct at least one of the alignment error and the brightness error in the captured HDR image.

In operation S9406, the method includes generating the HDR image based on the correction. In an embodiment, the method allows the HDR image generator110to generate the HDR image based on the correction. In operation S9408, the method includes storing the HDR image. In an embodiment of the disclosure, the method allows the sensor120to the memory140to store the HDR image.

FIG. 10is a flow diagram illustrating a method for aligning a source image with respect to a reference image according to various embodiments of the disclosure.

Referring toFIG. 10, another example flow diagram S1000illustrates aligning a source image based on a reference image. In an embodiment of the disclosure, the source image includes the LE image and the SE image of the scene. Further, the reference image includes the AE image of the same scene. The captured source image and the captured reference image are transferred to the alignment controller111from the sensor120to align the source image.

In operation S1002, the method includes selecting the undistorted central region in the source image and the reference image. In an embodiment of the disclosure, the method allows the alignment controller111to select the undistorted central region in the source image and the reference image. In operation S1004, the method includes determining the motion model for the source image. The motion model for the source image is determined based on the motion information of the feature points in the selected undistorted region in the source image and the reference image. In an embodiment, the method allows the alignment controller111to estimate the motion model for the source image.

In operation S1006, the method includes applying the motion model on the source image to obtain the aligned source image. In an embodiment of the disclosure, the method allows the alignment controller111to apply the motion model on the source image. In operation S1008, the method includes obtaining the aligned source image.

FIG. 11is a flow diagram illustrating a method for determining a motion model for aligning a source image according to various embodiments of the disclosure.

Referring toFIG. 11, an example flow diagram S1100for generating a motion model is illustrated. In operation S1102, the method includes detecting the feature-points in the undistorted regions in the AE image as the reference image and at least one of the SE image and the LE image. In an embodiment, the method allows the alignment controller111to detect the feature-points in the undistorted regions in the AE image as the reference image and at least one of the SE image and the LE image.

In operation S1104, the method includes determining the pairwise matches in the undistorted region based on the detected feature-points. In an embodiment of the disclosure, the method allows the alignment controller111to determine the pairwise matches in the undistorted region based on the detected feature-points in the source image and the reference image. In operation S1106, the method includes estimating the motion information based on the pairwise matches. In an embodiment of the disclosure, the method allows the alignment controller111to estimate the motion information based on the pairwise matches.

In operation S1108, the method includes estimating the motion model for the source image based on the motion information. In an embodiment of the disclosure, the method allows the alignment controller111to estimate the motion model based on the motion information.

FIG. 12is an example illustration in which various operations are explained for aligning a source image with respect to a reference image according to an embodiment of the disclosure.

Referring toFIG. 12, the source image includes the LE image and the SE image of the scene. Further, the reference image includes the AE image of the same scene. In operations1201and1203, the source image and the reference image are selected to estimate the motion information of the feature points in the source image.

The impact of distortion is unaffected at a central region of the images, while moving away from the central region of the images, and, the impact of distortion is gradually increased and maximum at the edge part of the images.

In operation1205, the undistorted central region is selected from the source image and the reference image. Further, the feature points in the selected region of the source image and the reference image are detected.

In operation1207, the motion information is estimated by performing the pairwise match of the motion vectors of the feature points in the source image with respect to the corresponding motion vectors of feature points in the reference image. In operation1209, the motion model for the source image is generated using the estimated motion information. In an embodiment of the disclosure, the motion information of each feature point in the LE image and SE image are estimated by matching a motion vectors of the feature points in the LE image and the SE image with the corresponding motion vectors of the feature points in the AE image.

In operation1211, the motion model is applied on the source image to compute the new pixel location of each pixel in the source image. Based on the motion-model, each pixel location is shifted to a new pixel location in the fisheye view. The new pixel location is computed using a translational model. Further, the new pixel location is computed considering spherical projection aspects and is approximated using a taylor series expansion for fast and simpler calculation.

In operation1213, the source image is aligned based on the motion model.

FIG. 13illustrates a comparison of images output from two types of image-alignment pipelines according to an embodiment of the disclosure.

Referring toFIG. 13, the output image1309from the proposed method is formed in consideration of radial distortions and the spherical projection aspects for the motion estimation of a motion estimation model1305and image compensation1307during aligning the source image1301with reference to the reference image1303.

The output image1315in the traditional method is formed by considering pin-hole camera projection aspects for the motion estimation model1311and the image compensation1313during aligning the source image. Further, the traditional method does not consider the radial distortions and the spherical projection aspects for the motion estimation and compensation during aligning the source image with reference to the reference image.

FIG. 14is a flow diagram illustrating a method for generating a modified pixel weight coefficient for an LE image, an AE image, and an SE image according to an embodiment of the disclosure.

Referring toFIG. 14, a flow diagram1400is illustrated for determining the modified pixel weight coefficients. In operation1402, Ghost-Map Computation is provided to estimate local motion regions between the AE-LE pair and the AE-SE pair and generate a ghost-map for each pair respectively. The generated ghost-map is further diffused using a guided filter. This output is fed to a pyramid diffusion module in a weight map computation module.

In operation1404, weight-map computation is provided. The computation of weight map for each image is done so as to bring the best exposed regions of each of the images in the HDR output. The weight maps are generated in a way that the HDR fused image should not contain over-saturated and under-saturated pixels. The regions which are over-exposed are better captured in the SE image, regions which are under-exposed are better captured in the LE image and the regions which are near mid intensity value are best captured in the AE image. The weight maps are estimated using a LUT and then normalized.

These normalized weight-maps are diffused smoothly to remove any artifacts which can occur because of discontinuous fusion weights in the composite image. In addition, it is required to blend or mix the information from the ghost map in the SE and LE weight maps. The weight maps for each of the images (i.e., LE and SE) are modified based on the gradient difference maps obtained in the calculation of ghost map as follows:

1. If in a region there is a local motion between only AE and LE, then the weights of LE are suppressed in that region,

2. In a region where there is a local motion between only AE and SE, then the weights of SE are suppressed in that region, and

3. If both LE and SE images have local motion with respect to AE then the weights of both LE and SE are suppressed in that region.

In this way, the weight maps of LE and SE are modified based on the local motion detection. The resulting weight maps are again normalized and further used for weighted composition of the HDR image.

In operation1406, pyramid level estimation is provided. Based on the input images, a number of pyramid levels are estimated based on the input resolution and is fed to a pyramid diffusion module in the weight map computation module.

FIG. 15is a flow diagram illustrating a method for generating a modified pixel weight coefficient for an LE image, an AE image, and an SE image according to an embodiment of the disclosure.

Referring toFIG. 15, a flow diagram1500illustrates determining modified pixel weight coefficients. In operation1502, the AE image and the aligned LE and SE images, are transferred from the alignment controller111to weight-map generator112. In operation1504, the pixel weight coefficient is computed for each of the LE image, the AE image and the SE image using the LUT by the weight-map generator112. In operation1506, the ghost map is computed for the LE image, the AE image and the SE image images in the weight-map generator112. In operation1508, the computed pixel weight coefficient of each of the LE image, the AE image and the SE image are modified based on the computed ghost map distribution for corresponding images. In operation1510, the weight-map generator112outputs the LE image, the AE image and the SE image images having modified pixel weight coefficient.

FIG. 16is the flow diagram illustrating a method for correcting brightness at an overlapping region in an HDR image according to an embodiment of the disclosure.

Referring toFIG. 16, a flow diagram1500illustrates that the LE image, the AE image and the SE image have the modified pixel weight coefficient and are transferred to the brightness correction controller113in the imaging device100. In operation S1602, the method includes determining a distance between each pixel location from the center of the AE image. In an embodiment of the disclosure, the method allows brightness correction controller113to determine the distance between each pixel location from the center of the AE image.

In operation S1604, the method includes detecting whether the distance meets a predetermined threshold. In an embodiment of the disclosure, the method allows brightness correction controller113to determine whether the distance meets the threshold. In operation S1606, the method includes modulating the HDR weight of each of the SE image and the LE image in response to determining that the distance meets the threshold. In an embodiment of the disclosure, the method allows brightness correction controller113to modulate the HDR weight of each of the SE image and the LE image in response to a determination that the distance meets the predetermined threshold.

In operation S1608, the method includes using the default weight of each of the SE image and the LE image and the AE image in response to a determination that the distance does not meet the predetermined threshold. In an embodiment, the method allows brightness correction controller113to use the default weight of each of the SE image and the LE image and the AE image in response to determining that the distance does not meets the threshold.

FIGS. 17A and 17Billustrate a comparison of brightness correction in an image due to modulated weights and normal weights according to an embodiment of the disclosure.

Referring toFIG. 17A, the image having pixel weight coefficient with normal/default weight is shown. The pixel weight coefficient is reduced after a modulation at a corresponding pixel position. Referring toFIG. 17B, the pixel weight coefficient of the same image is modulated at the overlapping region. The brightness of the image in the overlapping region is reduced inFIG. 17Bas compared toFIG. 17A.

FIGS. 18A and 18Billustrate a comparison of brightness differences in a stitched image according to an embodiment of the disclosure.

Referring toFIG. 18A, the stitched rectilinear HDR image without brightness correction is shown. The marked region in the image is one of the stitched region. Due to the brightness mismatch, the visible line is formed at the stitched region in the image. Referring toFIG. 18B, the stitched rectilinear HDR image with the brightness correction is shown. The visible line seen inFIG. 18Ais absent in the corresponding marked region inFIG. 18B. The weights for the pixel weight coefficient in the overlapping region of two HDR images (front and rear) are modulated before stitching.

FIG. 19illustrates an example scenario of generating a 360-degree landscape HDR image using an imaging device according to an embodiment of the disclosure.

Referring toFIG. 19, the imaging device100includes two sensors comprising front sensor120A and rear sensor120B, which are located each at a front side and a rear side of the imaging device100.

In an embodiment of the disclosure, a front sensor120A and a rear sensor120B capture the HDR image in a fisheye domain.

Further the imaging device100includes a display, a pair of mode selection button and an image capturing button. In operation1901, the user may select the landscape HDR capturing mode in the imaging device100by pressing the mode selection button to select the landscape HDR capturing mode. Further, the user may select the dual lens mode by pressing the mode selection button, to enable both of the front sensor120A and the rear sensor120B to capture the front and the rear image together. In operation1903, the user can view the selection mode in the display of the imaging device100. In operation1904, the user can initiate the image capturing by pressing the image capture button.

Further, the imaging device100may captures the HDR image upon pressing the image capturing button.

In operation1905, the imaging device100transmits the HDR images to the electronic device200(e.g., smartphone, personal computer etc.). In operation1906, the user can select the received HDR images to stitch the front HDR image and the rear HDR image. In operation1907, the selected images are used to generate the stitched rectilinear HDR image. The received HDR images are converted to rectilinear form before performing stitching.