Patent Description:
Conventionally, dynamic range of real world is very large, usually more than five orders of magnitude. Consequently, the dynamic range of everyday scenes can hardly be recorded by a conventional sensor. Therefore, some portions of the picture can be over-exposed or under-exposed. In recent years, high dynamic range (HDR) imaging techniques make it possible to reconstruct the radiance map that covers the full dynamic range by combining multiple exposures of the same scene (exposure bracketing). Moreover, these techniques usually estimate the camera response function, and then further estimate the radiance of each pixel.

Notably, current HDR solutions can be classified into single-exposure and multi-exposure methods that only rely on standard image sensors; event-based methods that only use event sensor; and hybrid methods that utilize both. Firstly, single-exposure methods infer an HDR image or stack of differently exposed LDR images from a single LDR image and can therefore be applied to legacy LDR content. However, they still essentially hallucinate details in saturated regions from the surrounding non-saturated context, and, thus, are not suitable for commercial applications. Secondly, a simple solution to the alignment problem is to reject moving object pixels or use robust exposure fusion. However, these methods often fail to identify moving objects, and are unable to reconstruct them in HDR. Notably, event-based methods reconstruct intensity frames from events using deep learning. Additionally, these methods do not explicitly target HDR imaging, but they rather reconstruct HDR-like images as a consequence of using events. However, reconstruction of an image from its spatial gradients is an ill-posed problem, especially when considering the instability of the contrast threshold. Therefore, event-based methods often produce images with incorrect global contrast. Another solution is the use of hybrid HDR method that combines an LDR image captured by a high-resolution frame camera with events acquired by an auxiliary event camera. However, they still rely on the chrominance of the LDR image when it comes to color, as most event cameras currently do not provide color information.

Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with the conventional methods of HDR imaging.

<CIT> discloses an imaging for fusing diverse information for a scene at the same time. The method comprises the steps of acquiring visible light information collected by a visible light sensor, infrared radiation information collected by an infrared sensor and event flow collected by an event sensor at the same time in the same scene; determining first internal and external parameter information between the visible light sensor and the infrared sensor and second internal and external parameter information between the visible light sensor and the event sensor; and fusing any two or three of the visible light information, the infrared radiation information and the event flow according to the first internal and external parameter information and the second internal and external parameter information to generate target image information.

<NPL> discloses the reconstruction of intensity images from event streams directly from data instead of relying on any hand-crafted priors. To this end, a novel recurrent network is proposed to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training a perceptual loss is used to encourage reconstructions to follow natural image statistics.

<NPL> discloses the creation of images/videos from an adjustable portion of an event data stream. The stacks of space-time coordinates of events are used as inputs and the network is trained to reproduce images based on the spatio-temporal intensity changes.

<NPL> discloses a convolutional recurrent neural network which takes a sequence of neighboring events to reconstruct high speed HDR videos.

The present disclosure seeks to provide a high dynamic range, HDR, imaging device. The present disclosure seeks to provide a solution to the existing problem regarding alignment of low dynamic range (LDR) images, for example, brightness constancy needed for optical flow estimation, saturation, blur, and noise. An aim of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in prior art, and provides an improved HDR imaging device.

The object of the present disclosure is achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims.

In one aspect, the present disclosure provides a high dynamic range, HDR, imaging device comprising: an image sensor configured to capture a plurality of low dynamic range, LDR, images; an aligned event sensor with pixels configured to capture a plurality of events between the plurality of LDR images, wherein each pixel is configured to trigger an event when a change of intensity of the pixel crosses a threshold; an image encoder module configured to generate an image feature representation based on each of the LDR images; an event encoder module configured to generate an event feature representation based on a subset of the plurality of events associated with each of the LDR images; a feature alignment module configured to: estimate per-pixel offsets between the image feature representation of a specified LDR image and the image feature representations of each other LDR image, based on the image and event feature representations of each LDR image, and generate a plurality of aligned feature representations by applying deformable convolutions or feature warping to the image feature representations of each other LDR image based on the estimated offsets; a fusion module configured to generate a fused feature representation where each pixel is selected from one or more of the plurality of aligned feature representations locally or non-locally; and an image decoder module configured to generate a HDR image based on the fused feature representation.

Beneficially, such an arrangement can recover structure and color from actual measurements in the saturated parts of the LDR image without hallucination. Additionally, it is capable of estimating true (camera and scene) motion by combining information from images and events, making it more robust to ghosting artifacts and blurriness. Moreover, HDR RGB images of high quality and high spatial resolution are produced, since the arrangement combines the event sensor with a high-quality RGB camera that captures a burst of bracketed exposures. Furthermore, the present disclosure combines motion information from both images and events, and perform pyramid, iterative, and deformable convolution alignment at the feature level.

In an implementation form, the image sensor is configured to generate a burst of three LDR images including a mid-exposed LDR image followed by a short-exposed LDR image followed by a long-exposed LDR image, wherein the specified LDR image is the mid- exposed LDR image.

Beneficially, this enables to reveal the details of an image in the darkest of shadows that requires high exposures, while preserving detail in very bright situations that requires very low exposures.

In an implementation form, the image sensor and aligned event sensor are aligned with beam splitter, or wherein the image sensor and aligned event sensor are integrated in a hybrid sensor.

Beneficially, such an arrangement helps to perform pyramid, iterative, and deformable convolution alignment at the feature level. Additionally, the present disclosure aligned with the beam splitter or integrated in a hybrid sensor are temporarily synchronized and optically aligned, including focus and field of view.

In an implementation form, a pre-processing module configured to de-noise and/or de-blur each LDR image based on the associated subset of the plurality of events.

Beneficially, de-blurring of the LDR image is a recovering process that recovers a sharp latent LDR image from a blurred LDR image.

In an implementation form, the image encoder module and event encoder module generate feature representations at a plurality of scales.

Beneficially, this enables the early layers to learn low-scale spatial features like texture, edges, or boundaries etc. while the deep layers learn high-scale semantic features which are close to the provided labels.

In an implementation form, the image encoder module and event encoder module generate feature representations at three scales.

Beneficially, the high-scale, mid-scale and low-scale semantic features help in locating the camera or scale motion accurately. Additionally, they also provide the spatial context information around the camera or scale motion.

In an implementation form, the event encoder module includes a convolutional LSTM module for each scale, each configured to integrate the event feature representations between the specified LDR image and each other LDR image.

Beneficially, at the end of the pyramid event encoder module there are convolutional LSTM modules, one for each scale, that integrate the per-scale event feature representations between the time frames, which is required to align low and high exposure LDR images to the mid exposure LDR image. Additionally, combining the advantages of convolutional neural networks (CNN) that can extract effective features from the data, and long short-term memory (LSTM) which can detect long range dependencies across time that are associated with the camera or scene motion.

In an implementation form, the feature alignment module first estimates offsets and generates aligned feature representations for the smallest scale, and is configured to estimate the offsets for each subsequent scale based at least in part on the offsets for the preceding scale.

Beneficially, estimating the offsets for different scales in this way can ensure alignment is performed in a coarse-to-fine manner, and increase the receptive field of the network in cases of larger motions.

In an implementation form, the feature alignment module is configured to estimate the offsets using a lookup network configured to extract a plurality of per-pixel features, build multi-scale 4D correlation volumes for all pairs of pixels, and iteratively perform a lookup on the correlation volumes to update a flow field with values for the offsets.

Beneficially, the correlation volumes help establish feature correlations across the entirety of the image, thus allowing for compensation of larger motions.

In an implementation form, the image encoder module and event encoder module are pyramid encoder modules each including <NUM> to <NUM> residual blocks.

Beneficially, residual blocks allow memory (or information) to flow from initial to last layers. Additionally, the residual blocks create an identity mapping to activations earlier in the network to thwart the performance degradation problem associated with deep neural architectures. Furthermore, this helps in tackling the vanishing gradient problem using identity mapping.

In an implementation form, the feature alignment module is configured to apply the feature-level deformation or warping in a coarse to fine manner.

Beneficially, such an arrangement works on different scales and produces a series of warping fields, capturing flow on different scales from coarse to fine. Additionally, this enables efficient inference possible while minimizing loss of accuracy. Furthermore, a coarse-to-fine approach performs inference at successively finer level. It uses the results from the coarser stages, where inference is faster, to guide and speed up inference at the more refined levels.

In an implementation form, the feature alignment module is supervised by generating auxiliary reconstructions at a plurality of scales and comparing the auxiliary reconstruction for each scale with a ground truth HDR image.

Beneficially, this allows for deep supervision of the deformable alignment of each LDR image, ensuring that the proper offsets will be learned of each LDR image.

In an implementation form, the image decoder module includes <NUM> to <NUM> residual blocks.

In an implementation form, the image decoder module includes a skip connection of the image feature representation of the specified LDR image from the image encoder module.

Beneficially, the skip connection ensures alignment with the specified LDR image, that is the mid-exposed LDR image, and pushes the network to only learn the missing information that needs to be filled from the other LDR images, that is short- and long-exposed LDR images.

In one aspect, the present disclosure provides a method of generating a high dynamic range, HDR, image, comprising: capturing, using an image sensor, a plurality of low dynamic range, LDR, images; capturing, using an aligned event sensor, a plurality of events between the plurality of LDR images, by triggering an event from each pixel of the aligned event sensor when a change of intensity of the pixel crosses a threshold; generating, using an image encoder module, an image feature representation based on each of the LDR images; generating, using an event encoder module, an event feature representation based on a subset of the plurality of events associated with each of the LDR images; estimating, using a feature alignment module, per-pixel offsets between the image feature representation of a specified LDR image and the image feature representations of each other LDR image, based on the image and event feature representations of each LDR image; generating, using the feature alignment module, a plurality of aligned feature representations by applying deformable convolutions or feature warping to the image feature representations of each other LDR image based on the estimated offsets; generating, using a fusion module, a fused feature representation where each pixel is selected from one or more of the plurality of aligned feature representations locally or non-locally; and generate, using an image decoder module, a HDR image based on the fused feature representation.

Beneficially, such an arrangement can recover structure and color from actual measurements in the saturated parts of the LDR image without hallucinating it. Additionally, it is capable of estimating true (camera and scene) motion by combining information from images and events, making it more robust to ghosting artifacts and blurriness. Moreover, HDR RGB images of high quality and high spatial resolution are produced, since the arrangement combines the event sensor with a high-quality RGB camera that captures a burst of bracketed exposures. Furthermore, the present disclosure combines motion information from both images and events, and perform pyramid, iterative, and deformable convolution alignment at the feature level.

In an implementation form, the method further comprises the image sensor configured to generate a burst of three LDR images including a mid-exposed LDR image followed by a short-exposed LDR image followed by a long-exposed LDR image, wherein the specified LDR image is the mid exposed LDR image.

In an implementation form, the method further comprises the image sensor and aligned event sensor aligned with beam splitter, or wherein the image sensor and aligned event sensor are integrated in a hybrid sensor.

In an implementation form, the method further comprises a pre-processing module configured to de-noise and/or de-blur each LDR image based on the associated subset of the plurality of events.

In an implementation form, the method further comprises the image encoder module and event encoder module to generate feature representations at a plurality of scales.

In an implementation form, the method further comprises the image encoder module and event encoder module to generate feature representations at three scales.

Beneficially, the high-scale, mid-scale and low scale semantic features help in locating the camera or scene motion accurately. Additionally, they also provide the spatial context information around the camera or scene motion.

In an implementation form, the method further comprises the event encoder module to include a convolutional LSTM module for each scale, each configured to integrate the event feature representations between the specified LDR image and each other LDR image.

In an implementation form, the method further comprises the feature alignment module to first estimates offsets and generates aligned feature representations for the smallest scale and is configured to estimate the offsets for each subsequent scale based at least in part on the offsets for the preceding scale.

In an implementation form, the method further comprises the feature alignment module configured to estimate the off-sets using a lookup network configured to extract a plurality of per-pixel features, build multi-scale 4D correlation volumes for all pairs of pixels, and iteratively perform a lookup on the correlation volumes to update a flow field with values for the offsets.

In an implementation form, the method further comprises wherein the image encoder module and event encoder module are pyramid encoder modules each including <NUM> to <NUM> residual blocks.

In an implementation form, the method further comprises wherein the feature alignment module is configured to apply the feature-level deformation or warping in a coarse to fine manner.

In an implementation form, the method further comprises wherein the feature alignment module is supervised by generating auxiliary reconstructions at a plurality of scales and comparing the auxiliary reconstruction for each scale with a ground truth HDR image.

In an implementation form, the method further comprises wherein the image decoder module includes <NUM> to <NUM> residual blocks.

In an implementation form, the method further comprises wherein the image decoder module includes a skip connection of the image feature representation of the specified LDR image from the image encoder module.

In one aspect, the present disclosure provides a computer-readable medium comprising instructions which, when executed by a processor, cause the processor to perform the method of any one of claims <NUM> to <NUM>.

Beneficially, the method of present disclosure may be performed using a processor. Additionally, the processor may include one or more individual processors, processing devices and elements arranged in various architectures for responding to and processing the instructions that drive the system.

It has to be noted that all devices, elements, circuitry, units and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof.

<FIG> is a block diagram of a high dynamic range, HDR, imaging device, in accordance with an embodiment of the present disclosure. With reference to <FIG>, there is shown an HDR imaging device <NUM>. The HDR imaging device <NUM> comprises an image sensor <NUM> to capture a plurality of low dynamic range, LDR, images <NUM>. Furthermore, an aligned event sensor <NUM> is configured to capture a plurality of events <NUM> between the plurality of LDR images <NUM>. Additionally, an image encoder module <NUM> is configured to generate an image feature representation based on each of the plurality of the LDR images <NUM>. Moreover, an event encoder module <NUM> is configured to generate event feature representation based on a subset of the plurality of events <NUM>. A feature alignment module <NUM> is configured to estimate and generate a plurality of aligned feature representations. A fusion module <NUM> is configured to generate a fused feature representation. An image decoder module <NUM> is configured to generate a HDR image <NUM>.

<FIG> is a schematic illustration depicting configuration of an HDR image, in accordance with an embodiment of the present disclosure. <FIG> is in conjunction with <FIG>. With reference to <FIG>, there is shown an HDR image <NUM>. The HDR image <NUM> comprises three LDR images including a mid-exposed LDR image <NUM>, followed by a short-exposed LDR image <NUM>, further followed by a long-exposed LDR image <NUM>.

The HDR imaging device <NUM> uses HDR imaging, wherein HDR imaging is a set of techniques used to reproduce a greater range of luminosity than that which is possible with standard photographic techniques. Herein, the term "dynamic range" refers to the ratio between maximum and minimum of tonal value in an HDR image <NUM>. Furthermore, the HDR imaging device <NUM> may be a camera, a smartphone, television sets, personal digital assistants and so forth.

The image sensor <NUM> is configured to capture the plurality of LDR images <NUM>. Herein, an image sensor <NUM> detects and conveys information by acquiring the plurality of LDR images <NUM> under different exposures and fuse them to produce the HDR image <NUM>. Typically, the image sensor <NUM> converts incoming light into an electrical signal that can be viewed, analyzed and stored. Herein, the image sensor <NUM> is used in electronic HDR imaging device <NUM> of both analog and digital types, such as for example, but not limited to, digital cameras, camera modules, camera phones, optical mouse devices, medical imaging equipment. Furthermore, the image sensor <NUM> are of two main types i.e., charge-coupled device (CCD) and active-pixel sensor which is a complementary metal-oxide-semiconductor (CMOS) sensor. Herein, the image sensor <NUM> used is an RGB sensor that helps the HDR imaging device <NUM> analyze a scene being captured and determines amount of light needed to produce a well-exposed HDR image <NUM>. Typically, the HDR image <NUM> stores real-scene luminance information corresponding to ability of human visual system to see simultaneously in a particular scene. In this sense, since the HDR image <NUM> hold true color and dynamic range information of the original scene, their processing, manipulation, representation, and other operations will no longer be limited to the number of bits used to describe each pixel. Typically, the image sensor <NUM> captures the plurality of LDR images <NUM> and gathers data on brightness of the plurality of LDR images <NUM>, then optimizes exposure by adjusting shutter speed, aperture and ISO sensitivity accordingly. Additionally, an image with the dynamic range lower than <NUM>:<NUM> is considered to be the LDR image.

In accordance with an embodiment, the image sensor <NUM> is configured to generate a burst of three LDR images including a mid-exposed LDR image <NUM> followed by a short-exposed LDR image <NUM> followed by a long-exposed LDR image <NUM>, wherein the specified LDR image is the mid exposed LDR image <NUM>. Herein, the image sensor <NUM> captures a burst of bracketed LDR images at consecutive time corresponding to mid exposure time, low exposure time and high exposure time. Particularly, bracketing is a technique wherein the plurality of LDR images <NUM> are taken more than once using different settings for different exposures, wherein exposure is amount of light per unit area reaching a frame of photographic film or surface of the image sensor <NUM>, as determined by shutter speed, lens aperture and scene luminance. Furthermore, the three LDR images are captured by a handheld camera, wherein the handheld camera is an RGB camera that comprises a significant camera motion among the mid-exposed LDR image <NUM>, the short-exposed LDR image <NUM> and the long-exposed LDR image <NUM>. Herein, the RGB camera collects visible light and coverts it to an electrical signal, then organizes that information to render images and video streams. Furthermore, the RGB camera is designed to create images that replicate human vision, capturing light in red, green and blue wavelengths for accurate color representation.

The aligned event sensor <NUM> with pixels configured to capture a plurality of events <NUM> between the plurality of LDR images <NUM>, wherein each pixel is configured to trigger an event when a change of intensity of the pixel crosses a threshold. Herein, the aligned event sensor <NUM> with pixels independently respond to changes in intensity, such as brightness as they occur. Typically, each pixel stores a reference intensity level, and continuously compares it to a current level of intensity. Subsequently, in case change of intensity crosses the threshold, that pixel resets to the reference intensity level and triggers an event. Herein, the plurality of events <NUM> is a stream of asynchronous events, wherein each single event comprises triggering time of the event, spatial position of the event, and polarity i.e., positive polarity or negative polarity. Furthermore, data from the plurality of events <NUM> and the plurality of LDR images <NUM> are temporally synchronized and spatially aligned and comprises a resolution as though they come from one image sensor <NUM>. Herein, the plurality of events <NUM> are temporally dense, and carry information about true fine-grained motion in-between frames of the plurality of LDR images <NUM>, without suffering from motion blur and saturation. Furthermore, the plurality of LDR images <NUM> are not bound by assumption of constant brightness to estimate motion, in contrast to the plurality of LDR images <NUM> that need to be exposure compensated. Notably, the HDR imaging device <NUM> combines both the plurality of LDR images <NUM> and the plurality of events <NUM> to estimate robust motion information from complementary sources, thereby facilitating LDR image alignment.

In accordance with an embodiment, the image sensor <NUM> and the aligned event sensor <NUM> are aligned with beam splitter, or wherein the image sensor <NUM> and the aligned event sensor <NUM> are integrated into a hybrid sensor. Herein, the beam splitter is an optical device that splits a beam of light in two. Furthermore, the HDR imaging device <NUM> combines the RGB camera with the aligned event sensor <NUM> to perform HDR imaging that is multi-bracketed in nature. Notably, high quality bracketed plurality of LDR images <NUM> is recorded from the image sensor <NUM>. Additionally, hybrid sensors enhance the luminance of the plurality of LDR images <NUM> using added information from the plurality of events <NUM>. Furthermore, this approach aims to merge the advantages of both the image sensor <NUM> and the aligned event sensor <NUM> in one unique design. Furthermore, the stream of plurality of events <NUM> from the event sensor <NUM> is geometrically aligned with the image sensor <NUM>. Additionally, the image sensor <NUM> and the aligned event sensor <NUM> are temporarily synchronized and optically aligned.

In accordance with an embodiment, the HDR imaging device <NUM> comprises a pre-processing module configured to de-noise and/or de-blur each LDR image based on the associated subset of the plurality of events <NUM>. Herein, noise in each LDR image may be caused by different intrinsic and extrinsic conditions which as often not possible to avoid in practical situations. Furthermore, de-blurring of the each LDR image is a recovering process that recovers a sharp latent LDR image from a blurred LDR image, wherein the blurring may be caused due to shaking of the HDR imaging device <NUM> or motion of object whose image is being taken. Subsequently, each LDR image undergoes a deformable alignment via multi-scale auxiliary reconstruction losses, thereby helping the HDR imaging device <NUM> with better alignment at each scale.

The image encoder module <NUM> configured to generate an image feature representation based on each of the LDR images. Herein, the image encoder module <NUM> compresses each of the LDR images into a latent space representation and encodes the input LDR images as a compressed representation in a reduced dimension. Moreover, the image encoder module <NUM> helps to learn important hidden features present in the input data, in the process to reduce the reconstruction error. Furthermore, during encoding, a new set of combinations of original features is generated.

The event encoder module <NUM> is configured to generate an event feature representation based on a subset of the plurality of events <NUM> associated with each of the LDR images. Simultaneously, the events in between bracketed exposures of the LDR images, namely the mid-exposed LDR image <NUM>, the short-exposed LDR image <NUM>, the long-exposed LDR image <NUM> are split and fed into the feature alignment module <NUM>. Furthermore, the event encoder module <NUM> generates a multi-scale feature representation.

In accordance with an embodiment, the image encoder module <NUM> and the event encoder module <NUM> are pyramid encoder modules each including <NUM> to <NUM> residual blocks. Herein, pyramid encoding is the multi-scale representation of the plurality of LDR images <NUM>, wherein each of the LDR image is subjected to repeated down-sampling. Furthermore, the pyramid encoder module predicts global and local motion.

In accordance with an embodiment, the image encoder module <NUM> and the event encoder module <NUM> generate feature representation at a plurality of scales. Notably, due to the intrinsic hierarchical characteristics of convolutional neural networks (CNN), multi-scale hierarchical feature learning can be achieved via architecture design of CNNs. Additionally, inside a CNN, the early layers learn low-scale spatial features like texture, edges, or boundaries etc. while the deep layers learn high-scale semantic features. Herein, the plurality of scales refers to resizing of the plurality of LDR images <NUM>. Furthermore, in order to procure an HDR image <NUM>, the feature representation should include high-level semantic features as well as low-level spatial features, as the low-level spatial features provide spatial context information per pixel.

In accordance with an embodiment, the image encoder module <NUM> and the event encoder module <NUM> generate feature representations at three scales. Herein, the feature representation is carried out for consecutive time, which is required to align each LDR image and generate feature representations at three scales, namely the short-exposed LDR image <NUM> and the long-exposed LDR image <NUM> to the mid-exposed LDR image <NUM>. In accordance with an embodiment, the event encoder module <NUM> includes a convolutional long short-term memory (LSTM) module for each scale, each configured to integrate the event feature representations between the specified LDR image and each other LDR image. Herein, the convolutional LSTM module are operably coupled at the end of the pyramid encoder module to integrate the per-scale event feature representations between consecutive time frames, which is required to align the short-exposed LDR image <NUM> and long-exposed LDR image <NUM> to the mid-exposure LDR image <NUM>.

The image decoder module <NUM> is configured to generate the HDR image <NUM> based on the fused feature representation. Herein, the aligned feature representations of the plurality of LDR images <NUM> and the plurality of events <NUM> are passed into the fusion module <NUM> that selects per-pixel the aligned feature representations from all different sources. Furthermore, the image decoder module <NUM> converts encoded LDR image back to an uncompressed bitmap which can then be rendered on the HDR imaging device <NUM>.

In accordance with an embodiment, the image decoder module <NUM> includes <NUM> to <NUM> residual blocks.

In accordance with an embodiment, the image decoder module <NUM> includes a skip connection of the image feature representation of the specified LDR image from the image encoder module <NUM>. Herein, the skip connection ensures alignment with the specified LDR image, that is the mid-exposed LDR image, and pushes the network to only learn the missing information that needs to be filled from the other LDR images, that is short- and long-exposed LDR images.

<FIG> is a block diagram of a feature alignment module, in accordance with an embodiment of the present disclosure. With reference to <FIG>, there is shown a feature alignment module <NUM> for a pair of specified LDR images, namely a first specified LDR image 202A and a second specified LDR image 202B and their corresponding event feature representation, namely a first event feature representation 204A and a second event feature representation 204B. The feature alignment module <NUM> generates a plurality of aligned feature representations 206A, 206B by applying deformable convolutions 208A, 208B to the image feature representations of the first specified LDR image 202A and the second specified LDR image 202B. Furthermore, the HDR imaging device generates aligned feature representations 206A and 206B for scale, wherein the scale comprises large scale 210A and 210B, and small scale 212A and 212B. Additionally, the feature alignment module <NUM> is configured to estimate offsets using a lookup network, namely a low look up network 214A and a high look up network 214B. Moreover, the feature alignment module <NUM> is supervised by generating auxiliary reconstructions <NUM>.

<FIG> is a block diagram of architecture of the low lookup network, in accordance with an embodiment of the present disclosure. <FIG> is in conjunction with <FIG>. With reference to <FIG>, there is shown an overview of the low lookup network 214A. The low lookup network 214A uses the feature alignment module <NUM> to estimate offsets and generate aligned feature representations 206A for the small scale 212A. The low lookup network 214A extracts per-pixel features <NUM>, builds a multi-scale 4D correlation volume <NUM> for all pairs of pixels, and iteratively update a flow field through a recurrent unit <NUM> that performs lookups <NUM> on the multi-scale 4D correlation volume <NUM>. Furthermore, the deformable convolutions 208A, 208B of image feature representation of the first specified LDR image 202A and the first event feature representation 204A are concatenated into a context net <NUM>.

The feature alignment module <NUM> is configured to estimate per-pixel offsets between the image feature representation of a specified LDR image, namely the first specified LDR image 202A and the second specified LDR image 202B and the image feature representations of each other LDR image, based on the image and event feature representations, namely the first event feature representation 204A and the second event feature representation 204B of each LDR image. Herein, the per-pixel offset is a technique for measuring displacement between the image feature representation of the specified LDR image. Additionally, the feature alignment module <NUM> is iterative and deformable in nature.

The feature alignment module <NUM> is configured to generate a plurality of aligned feature representations 206A, 206B by applying deformable convolutions 208A, 208B or feature warping to the image feature representations of each other LDR image based on the estimated offsets. Herein, deformable convolution 208A, 208B refers to adding two-dimensional (2D) offsets to a regular grid sampling locations in standard convolution. Additionally, this enables free form deformation of the sampling grid. Moreover, the offsets are learned from the preceding feature maps, via additional convolutional layers. Herein, the offsets are used to apply deformable convolutions 208A, 208B to features of the second specified LDR image 202B at the small scale 212B and align them to features of the first specified LDR image 202A. Subsequently, same procedure is repeated for the large scale 210A, 210B with starting point being results coming from the small scale 212B.

In accordance with an embodiment, the feature alignment module <NUM> first estimates offsets and generates aligned feature representations 206A, 206B for the smallest scale, and is configured to estimate the offsets for each subsequent scales, namely the large scale 210A, 210B and the small scale 212A, 212B, based at least in part on the offsets for the preceding scale. Herein the LDR images are the first specified LDR image 202A and the second specified LDR image 202B and their corresponding event feature representation, namely the first event feature representation 204A and the second event feature representation 204B may be processed at two scales i.e., large scale 210A and 210B, and small scale 212A and 212B. Subsequently, the image feature representations of the first specified LDR image 202A and the second specified LDR image 202B are concatenated with the first event feature representation 204A and the second event feature representation 204B at small scale 212A and 212B which are passed to the low lookup network 214A. Furthermore, the offsets of the deformable convolutions 208A, 208B at each scale, are computed jointly from feature representations of the first specified LDR image 202A and the second specified LDR image 202B and their corresponding event feature representation, namely the first event feature representation 204A and the second event feature representation 204B. Alternatively, the offsets may be estimated separately, once from the image feature representation of the first specified LDR image 202A and the second specified LDR image 202B alone, and another from the first event feature representation 204A and the second event feature representation 204B resulting in a pair of aligned features representations 206A, 206B.

In accordance with an embodiment, the feature alignment module <NUM> is configured to estimate the offsets using a lookup network configured to extract a plurality of per-pixel features <NUM>, build multi-scale <NUM>-dimensional (4D) correlation volumes <NUM> for all pairs of pixels, and iteratively perform a lookup on the correlation volumes to update a flow field with values for the offsets. Herein, the iteration is performed using a recurrent unit <NUM> to update the flow field. Additionally, the lookup network comprises a low lookup network 214A and a high lookup network 314B built upon Raft Consensus Algorithm or Raft protocol, wherein Raft protocol solves a problem of getting multiple servers to agree on a shared state even while facing failures. Furthermore, the concatenation of the first specified LDR image 202A and the second specified LDR image 202B and their corresponding event feature representation, namely the first event feature representation 204A and the second event feature representation 204B at the small scale 212A are passed to the low lookup network 214A that estimates the offsets of a <NUM>-by-<NUM> kernel of deformable convolutions 208A and 208B at that scale. Herein, the multi-scale 4D correlation volumes <NUM> is a technique for full 4D strain and deformation measurements. Moreover, the multi-scale 4D correlation volumes <NUM> imports volume LDR images of component in reference and deformable states and is able to calculate full 4D displacement and strain map.

In accordance with an embodiment, the feature alignment module <NUM> is configured to apply the feature-level deformation or warping in a coarse to fine manner. Herein, warping is the process of taking information from a target location in a target image and moving it to a reference location in the reference image. It aims to counteract motion and temporally align images at a certain time frame. Furthermore, the HDR imaging approaches performing warping at image level to align the LDR images before merging them into the HDR. Furthermore, the HDR imaging device uses a pyramid encoder module comprising deformable convolution 208A, 208B which predicts global and local motion in a coarse to fine manner. Notably, a coarse-to-fine approach performs inference at successively finer level. Additionally, it uses the results from the coarser stages, where inference is faster, to guide and speed up inference at the more refined levels.

In accordance with an embodiment, the feature alignment module <NUM> is supervised by generating auxiliary reconstructions at a plurality of scales and comparing the auxiliary reconstruction for each scale with a ground truth HDR image. Herein, the ground truth HDR image is the desired outcome of the algorithm. Furthermore, the feature alignment module <NUM> generates auxiliary reconstructions at the plurality of scales, as there is no requirement of additional annotations other than the ground truth HDR image, but down-sampled at different resolutions.

The fusion module is configured to generate a fused feature representation where each pixel is selected from one or more of the plurality of aligned feature representations 208A, 208B. Herein, the aligned feature representations 208A, 208B are passed into the fusion module in case of estimating offsets jointly from the plurality of LDR images and the plurality of events, or in case of estimating the plurality of LDR images and the plurality of events separately. Typically, the fusion module may be a simple <NUM>-by-<NUM> convolutional block, or as complicated as specialized attention modules, for instance spatial attention, spatio-temporal attention and so forth. Optionally, the fusion module may include unaligned feature representations from the bracketed LDR images, wherein the bracketed LDR images are without linearization and exposure compensation. Particularly, an integrated event feature representation, after processing them in reverse order with an extra dedicated convolutional LSTM module that aligns the event feature representations at consecutive times.

<FIG> is an operation of generating HDR image, in accordance with an exemplary embodiment of the present disclosure. With reference to <FIG>, there is shown multiple LDR images taken under different exposures, such as mid-exposed image <NUM>, short-exposed image <NUM> and long-exposed image <NUM>, fused into the HDR image <NUM>. The fusion also includes events in-between exposures such as, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and <NUM> triggered by an aligned event camera. The multiple LDR images, such as <NUM>, <NUM> and <NUM> are taken by the image sensor and is in synchronization with the aligned event sensor.

<FIG> & <FIG> are flowchart depicting steps of a method of generating a HDR image, in accordance with an embodiment of the present disclosure. With reference to <FIG>, there is shown a method <NUM> of generating the HDR image. The method <NUM> includes steps <NUM>-to-<NUM>.

The method <NUM> introduces an efficient way of generating the HDR image. The method <NUM> is described in detail in following steps. At step <NUM>, the method <NUM> comprises capturing, using an image sensor, a plurality of low dynamic range, LDR, images. At step <NUM>, the method <NUM> comprises capturing, using an aligned event sensor, a plurality of events between the plurality of LDR images, by triggering an event from each pixel of the aligned event sensor when a change of intensity of the pixel crosses a threshold. At step <NUM>, the method <NUM> comprises generating, using an image encoder module, an image feature representation based on each of the LDR images. At step <NUM>, the method <NUM> comprises generating, using an event encoder module, an event feature representation based on a subset of the plurality of events associated with each of the LDR images. At step <NUM>, the method <NUM> comprises estimating, using a feature alignment module, per-pixel offsets between the image feature representation of a specified LDR image and the image and event feature representations of each other LDR image, based on the image and event feature representations of each LDR image. At step <NUM>, the method <NUM> comprises generating, using the feature alignment module, a plurality of aligned feature representations by applying deformable convolutions or feature warping a feature-level deformation to the image and event feature representations of each other LDR image based on the estimated offsets. At step <NUM>, the method <NUM> comprises generating, using a fusion module, a fused feature representation where each pixel is selected from one or more of the plurality of aligned feature representations. At step <NUM>, the method <NUM> comprises generate, using an image decoder module, a HDR image based on the fused feature representation.

In accordance with an embodiment, the method comprises the image sensor that is configured to generate a burst of three LDR images including a mid-exposed LDR image followed by a short-exposed LDR image followed by a long-exposed LDR image, wherein the specified LDR image is the mid exposed LDR image.

In accordance with an embodiment, the image sensor and aligned event sensor aligned with beam splitter, or wherein the image sensor and aligned event sensor are integrated in a hybrid sensor.

In accordance with an embodiment, a pre-processing module configured to de-noise and/or de-blur each LDR image based on the associated subset of the plurality of events.

In accordance with an embodiment, the image encoder module and event encoder module generate feature representations at a plurality of scales.

In accordance with an embodiment, the image encoder module and event encoder module generate feature representations at three scales.

In accordance with an embodiment, the event encoder module includes a convolutional LSTM module for each scale, each configured to integrate the event feature representations between the specified LDR image and each other LDR image.

In accordance with an embodiment, the feature alignment module first estimates offsets and generates aligned feature representations for the smallest scale, and is configured to estimate the offsets for each subsequent scale based at least in part on the offsets for the preceding scale.

In accordance with an embodiment, the feature alignment module is configured to estimate the offsets using a lookup network configured to extract a plurality of per-pixel features, build multi-scale 4D correlation volumes for all pairs of pixels, and iteratively per-form a lookup on the correlation volumes to update a flow field with values for the offsets.

In accordance with an embodiment, the image encoder module and event encoder module are pyramid encoder modules each including <NUM> to <NUM> residual blocks.

In accordance with an embodiment, the feature alignment module is configured to apply the feature-level deformation or warping in a coarse to fine manner.

In accordance with an embodiment, the feature alignment module is supervised by generating auxiliary reconstructions at a plurality of scales and comparing the auxiliary reconstruction for each scale with a ground truth HDR image.

In accordance with an embodiment, the image decoder module includes <NUM> to <NUM> residual blocks.

In accordance with an embodiment, the image decoder module includes a skip connection of the image feature representation of the specified LDR image from the image encoder module.

The computer-readable medium comprising instructions which, when executed by a processor, cause the processor to perform the method of the present disclosure. Herein, the term "computer-readable medium" is a medium capable of storing data in a format readable and executable by the processor. Furthermore, the computer-readable medium may include magnetic media such as magnetic disks, cards, tapes, and drums, punched cards and paper tapes, optical discs, barcodes and magnetic ink characters. Additionally, common computer-readable medium technologies include magnetic recording, processing waveforms, and barcodes. Moreover, the term "processor" relates to a computational element that is operable to respond to and processes instructions that drive the computer-readable medium. Optionally, the processor includes, but is not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or any other type of processing circuit. Furthermore, the term "processor" may refer to one or more individual processors, processing devices and various elements associated with a processing device that may be shared by other processing devices. Additionally, the one or more individual processors, processing devices and elements are arranged in various architectures for responding to and processing the instructions that drive the system.

Claim 1:
A high dynamic range, HDR, imaging device (<NUM>) comprising:
an image sensor (<NUM>) configured to capture a plurality of low dynamic range, LDR, images (<NUM>);
an aligned event sensor (<NUM>) with pixels configured to capture a plurality of events (<NUM>) between the plurality of LDR images (<NUM>), wherein each pixel is configured to trigger an event when a change of intensity of the pixel crosses a threshold;
an image encoder module (<NUM>) configured to generate an image feature representation based on each of the LDR images;
an event encoder module (<NUM>) configured to generate an event feature representation based on a subset of the plurality of events (<NUM>) associated with each of the LDR images;
a feature alignment module (<NUM>, <NUM>) configured to:
estimate per-pixel offsets between the image feature representation of a specified LDR image and the image feature representations of each other LDR image, based on the image and event feature representations of each LDR image, and
generate a plurality of aligned feature representations (206A, 206B) by applying deformable convolutions (208A, 208B) or feature warping to the image feature representations of each other LDR image based on the estimated offsets;
a fusion module (<NUM>) configured to generate a fused feature representation where each pixel is selected from one or more of the plurality of aligned feature representations (206A, 206B); and
an image decoder module (<NUM>) configured to generate a HDR image (<NUM>) based on the fused feature representation.