Systems and methods for multispectral image demosaicking using deep panchromatic image guided residual interpolation

Described herein are systems and embodiments for multispectral image demosaicking using deep panchromatic image guided residual interpolation. Embodiments of a ResNet-based deep learning model are disclosed to reconstruct the full-resolution panchromatic image from multispectral filter array (MSFA) mosaic image. In one or more embodiments, the reconstructed deep panchromatic image (DPI) is deployed as the guide to recover the full-resolution multispectral image using a two-pass guided residual interpolation methodology. Experiment results demonstrate that the disclosed method embodiments outperform some state-of-the-art conventional and deep learning demosaicking methods both qualitatively and quantitatively.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is filed pursuant to 35 USC § 371 as a US National Phase Application of International Patent Application No. PCT/CN2019/094839, filed on 5 Jul. 2019, entitled “SYSTEMS AND METHODS FOR MULTISPECTRAL IMAGE DEMOSAICKING USING DEEP PANCHROMATIC IMAGE GUIDED RESIDUAL INTERPOLATION,” listing Zhihong Pan, Baopu Li, Yingze Bao, and Hsuchun Cheng as inventors, which patent document is incorporated by reference herein in its entirety and for all purposes.

TECHNICAL FIELD

The present disclosure relates generally to systems and methods for image processing. More particularly, the present disclosure relates to systems and methods for multispectral image demosaicking.

BACKGROUND

Snapshot multispectral imaging based on multispectral filter array (MSFA) has gained popularity recently for its size and speed. The added spectral information has led to its utilization in a broad range of application fields such as remote sensing and medical imaging. To reconstruct the full-resolution multispectral image, all the missing spectral information at each pixel must be estimated based on raw sensor measurements from neighboring pixels. This recovering process is referred to as demosaicking.

Demosaicking is one of the most crucial and challenging steps to reduce artifacts in both spatial and spectral domain. Various demosaicking methods for MSFA cameras have been proposed to continuously improve demosaicked image quality. It is always desirable to minimize the error between demosaicked image and original high-resolution image.

Accordingly, what is needed are systems and methods for multispectral image demosaicking for performance improvement.

SUMMARY

In a first aspect, the present the present disclosure provides a computer-implemented method for demosaicking a multispectral image from a multispectral filter arrays (MSFA) sensor with multiple sub-bands, the method comprising: using a deep neural network and the multispectral image to reconstruct a full resolution deep panchromatic image (DPI) of the multispectral image; for each sub-band of the multispectral image, performing at least one guided residual interpolation based on the reconstructed DPI to obtain a demosaicked sub-band image; and forming a demosaicked multispectral image comprising the demosaicked sub-band images.

In a first aspect, the present the present disclosure provides computer-implemented method for demosaicking a sub-band image in a multispectral image captured by a multispectral filter arrays (MSFA) sensor using one or more processors to cause steps to be performed comprising: using a deep neural network and the multispectral image to reconstruct a full resolution deep panchromatic image (DPI) of the multispectral image; subsampling the DPI relative to the sub-band to obtain a subsampled DPI; obtaining a subsampled residual image from a difference between the sub-band image and the subsampled DPI; using the DPI as a guide image in a guided interpolation on the subsampled residual image to obtain a demosaicked residual image; and adding the demosaicked residual image to the DPI to obtain a first-pass demosaicked sub-band image.

In a third aspect, the present disclosure provides a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes the steps for demosaicking a multispectral image, from a multispectral filter arrays (MSFA) sensor, with multiple sub-bands to be performed comprising: reconstructing, using a deep neural network, the multispectral image to a full resolution deep panchromatic image (DPI); for each sub-band of the multispectral image, performing at least one guided residual interpolation based on the reconstructed DPI to obtain a demosaicked sub-band image; and forming a demosaicked multispectral image comprising the demosaicked sub-band images.

In a fourth aspect, the present disclosure provides a system for demosaicking a multispectral image from a multispectral filter arrays (MSFA) sensor with multiple sub-bands, the system comprising: at least one processor; and a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform the operations according to the first aspect.

In a fifth aspect, the present disclosure provides a system for demosaicking a sub-band image in a multispectral image captured by a multispectral filter arrays (MSFA) sensor, the system comprising: at least one processor; and a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform the operations according to the second aspect.

DETAILED DESCRIPTION OF EMBODIMENTS

The use of certain terms in various places in the specification is for illustration and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; usage of these terms may refer to a grouping of related services, functions, or resources, which may be distributed or aggregated. An image may be a still image or from a video.

The terms “include,” “including,” “comprise,” and “comprising” shall be understood to be open terms and any lists the follow are examples and not meant to be limited to the listed items. Any headings used herein are for organizational purposes only and shall not be used to limit the scope of the description or the claims. Each reference mentioned in this patent document is incorporated by reference herein in its entirety.

Furthermore, one skilled in the art shall recognize that: (1) certain steps may optionally be performed; (2) steps may not be limited to the specific order set forth herein; (3) certain steps may be performed in different orders; and (4) certain steps may be done concurrently.

Spectral imaging, including both hyperspectral images (HSIs) and multispectral images (MSIs), supplements additional information in spectral domain for each pixel. The added spectral information has led to its utilization in a broad range of application fields such as remote sensing and medical imaging. To measure a 3D spectral data cube on a 2D detector, various scanning techniques, including filter wheel and push broom scanners, have been developed. More recently, multispectral filter array (MSFA) is increasingly utilized in snapshot spectral cameras to capture a MSI in a single shot. Images captured by a MSFA camera have only one value at certain wavelength for each pixel, ordered in a designed MSFA mosaic pattern. To reconstruct the full-resolution multispectral image, all the missing spectral information at each pixel must be estimated based on raw sensor measurements from neighboring pixels. This recovering process is referred to as demosaicking.

Various demosaicking methods for MSFA cameras have been proposed to continuously improve demosaicked image quality. For MSFAs with primary bands, a binary tree-based edge-sensing (BTES) method was first designed to recover secondary bands using information estimated from primary bands. Another 4×4 MSFA took a dominant green band as the guide for residual interpolations to achieve MSI demosaicking. MSFA patterns without dominant bands were also devised and different demosaicking methods emerged in this process. The PPID method estimated a pseudo-panchromatic image first and used it as a guide to smooth its difference with each subsampled band. More recently, some proposed a new demosaicking model based on deep learning and experiments showed significant improvements comparing to PPID.

In this patent document, system and method embodiments using a two-step demosaicking process for MSIs called deep panchromatic image guided residual interpolation (DGRI) are disclosed.FIG. 1depicts architecture of a two-step demosaicking system, according to embodiments of the present disclosure. The system comprises a deep neural network110and a DGRI demosaicking module120. In one or more embodiments, the deep neural network is based on a residual neural network (ResNet) and may be referred as a deep panchromatic image (DPI) model or DPI-Net hereinafter. The DPI model receives a raw mosaic image105given a specific MSFA pattern and reconstructs a DPI115from the raw mosaic image. In one or more embodiments, the DPI model is trained to minimize the error between its output and the reference panchromatic image averaged from all bands of the ground-truth multispectral image. The DGRI demosaicking module120then recovers a full resolution MSI125from the DPI115using guided residual interpolation. In one or more embodiments, the DGRI demosaicking module120uses a two-pass demosaicking method for recovering the full resolution MSI125. The first pass utilizes the DPI as a guide to filter the residual between each subsampled band and the DPI. The second pass takes the initially demosaicked band to further reduce the residual between itself and the subsampled mosaic image. Various detailed embodiments of the disclosed process are described in Section B and the experimental results are shown in Section C.

B. Embodiments for Demosaicking Process

In this section, demosaicking process embodiments are disclosed.

1. Embodiments of Deep Panchromatic Image

FIG. 2depicts architecture of a deep panchromatic image network (DPI-Net), andFIG. 3depicts a process for recovering a full resolution panchromatic image from a multispectral mosaic image using the DPI-Net, according to embodiments of the present disclosure. The DPI-Net recovers a full resolution panchromatic image from a multispectral mosaic image where each pixel is sampled from one of the MSFA filters. In one or more embodiments, the DPI-Net is a convolutional neural network (CNN), and has a total depth of D. The DPI-Net comprises a first block as a projection layer210, one or more middle blocks220, a final convolutional layer230, and a clipping module240. The projection layer210transforms (305) the input 2D mosaic image205to multiple channels215using a set of convolutional and rectified linear unit (ReLU) filters. For the one or more middle blocks (D-2)220, each block consists of two layers222and224of convolutions and parameterized ReLUs (PReLUs) with a shortcut connection226between the block input and block output. In one or more embodiments, batch normalization (BN) was not included in the whole DPI network since it was found that BN layer does not improve the performance of reconstruction. The one or more middle blocks220generate (310) an intermediate output228, which is projected (315) via the final convolutional layer230to a 2D residual image235. The 2D residual image235is subtracted (320) from the input205to obtain a residual subtracted image236. The residual subtracted image236is then clipped (325), using the clipping module240, to a valid pixel value range for the final output245, which is the full resolution panchromatic image.

In one or more embodiments, the two layers222and224of convolutions and PReLUs may or may not have the same kernel size. For example, the first layer222may have a size of 7×7, while the second layer may have a size of 5×5 instead.

In one or more embodiments, the DPI-Net is a residual network which may has certain similarities to the deep denoising network ResDNet (Kokkinos et al., Deep image demosaicking using a cascade of convolutional residual denoising networks, Proceedings of the European Conference on Computer Vision (ECCV), 2018 pp. 317-333) as the mosaic image may be a form of noisy panchromatic image. However, as the difference between the mosaic image and band-averaging panchromatic image has a pre-determined range, the DPI-Net in the present invention document does not need noise variance. Additionally, embodiments of the DPI-Net may adopt three steps, which were demonstrated to be beneficial. First, the size of kernel matters more over the depth of layers. Secondly, a special padding scheme may be used in the first block210, which is a limited rolling shift of the first MSFA pattern on each side. In one or more embodiments, the special padding scheme is a circular padding with the number of padded rows (both up and down) and columns (both left and right) the same as the MSFA pattern. Lastly, during training, a sharpen filter illustrated below may be applied to at least one of the panchromatic images before calculating the loss function because it may provide better gradient descent.

In one or more embodiments, the sharpen filter is applied to both the panchromatic image reconstructed from the DPI-Net and the reference panchromatic images. Therefore, instead of calculating loss function between the panchromatic image reconstructed from the DPI-Net or the reference panchromatic image, the two sharpened panchromatic images are used for loss function calculation.

2. Embodiments of Guided Interpolation for Demosaicking

Guided filtering is a recently proposed method with excellent structure-preserving capabilities. It approximates the filtering as a linear approximation within a small window. As illustrated in Equation (1) below, for a given window w, the filtered value of any pixel (x, y) within the window is approximated as a linear transformation of original value:
Î(x,y)=awG(x,y)+bw,∀(x,y)∈w(1)

where G(x, y) is the guided image value at pixel (x, y) and Î (x, y) is the filtered image value. Using linear regression method, the two coefficients awand bwmay be estimated by minimizing the difference between I(x, y) and Î(x, y) for all pixels in the window w.

In one or more embodiments, the filter is applied to the whole image by sliding the window so that each pixel (x, y) is estimated multiple times for all windows that (x, y) belongs to. In one or more embodiments, the linear transformation coefficients for each pixel is averaged over these estimations as

where |w| is the number of windows one pixel resides in, the same as the number of pixels inside a window. The filtered image may be calculated as
Î(x,y)=a(x,y)*I(x,y)+b(x,y)  (3)

For the above guided filtering, the input image and guide image have the same number of pixels. In one or more embodiments, to apply the same method to MSI demosaicking, each subsampled band needs to be pre-processed using upsampling methods like bilinear interpolation.FIG. 4depicts a process of guided Interpolation for demosaicking, according to embodiments of the present disclosure. In one or more embodiments, for the proposed MSI demosaicking, a new process is put forward for the estimation of a(x, y, λ) and b(x, y, where λ), refers to the wavelength of a specific filter in MSFA. First, assuming the MSFA pattern is of m×n, a sliding window size (m+1)×(n+1) is used (405). The sliding steps are m and n in corresponding directions respectively. For each step the window covers 4 pixels from the sparse grid in each subsampled bands and only these 4 pixels are used (410) to estimate coefficients aw(A) and bw(A). For each pixel (x, y) on the sparse subsampled grid, linear interpolation coefficients a(x, y, λ) and b(x, y, λ) may be obtained (415) using the average of the estimated coefficients for all steps shown in equation (4) below since each pixel is only covered by 4 windows.

The full resolution coefficients a(x, y, and b(x, y, λ) for a subsampled image are obtained (420) from the sparse grid or subset using bilinear interpolation. The last step of the guided interpolation is to obtain (425) a full resolution image for the sub-band from a linear transformation using the guided image I (x, y, λ) and the obtained full resolution coefficients, showing as:
Î(x,y,λ) =a(x,y,λ)*I(x,y,λ)+b(x,y,λ)  (5)

3. Embodiments of Two-Pass Guided Residual Interpolation

FIG. 5graphically depicts a DPI guided residual interpolation (DGRI) demosaicking process, according to embodiments of the present disclosure the DGRI demosaicking process may involve one or more demosaicking passes for recovering the full resolution MSI.FIG. 6depict a process for a first-pass using a DPI as a guide image for guided residual interpolation, according to embodiments of the present disclosure.

As shown in theFIG. 5, an input multispectral mosaic image502may be separated (605) to a set of sparse subsampled or sub-band images, each corresponding to one filter or sub-band in the MSFA. A full resolution deep panchromatic image (DPI) is recovered (610) from the multispectral mosaic image using the DPI-Net512. A subsampled DPI image514is obtained (615) from the recovered DPI. For one sub-band image504(using the subsampled R band image as an example, the sub-band for the sub-band image504corresponds to the subsampled DPI image514), it is first subtracted (620) from the subsampled DPI514to get a sparse residual image516. Using the DPI as a guide image515, this sparse residual image516is interpolated (625) to full resolution to obtain an initial demosaicked residual image517, which is then added back (630) to the DPI to get the first-pass demosaicked image523corresponding to the R band.

In one or more embodiments, additional processing may be applied to further reduce the residual between the first-pass demosaicked image523and the subsampled mosaic image.FIG. 7depict a second pass using the first-pass demosaicked image as a guide image for guided residual interpolation, according to embodiments of the present disclosure.

In the second-pass, the first-pass demosaicked image523is subsampled (705) to obtain a subsampled demosaicked image524. The original sub-band image504is subtracted (710) from the subsampled demosaicked image524to obtain a second-pass subsampled R residual image526. A similar guided interpolation method, using the first-pass demosaicked image523as the guide image525, is further applied to interpolated (715) the second-pass subsampled R residual image526to full resolution to obtain a second-pass demosaicked residual image527. The second-pass demosaicked residue527is added (720) to the first-pass demosaicked image523to get the final (second-pass) demosaicked R image530. In one or more embodiments, the second pass is necessary to minimize the difference between to the demosaicked image and the raw image at the sparse grid of subsampled R band. All other sub-bands may follow the same procedure to recover demosaicked images at other sub-bands, and thus a fully demosaicked MSI may be obtained.

It shall be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using a specific embodiment or embodiments; accordingly, neither these experiments nor their results shall be used to limit the scope of the disclosure of the current patent document.

In one or more experimental settings, to evaluate the performance of the proposed process, the most popular peak-signal-to-noise ratio (PSNR) was adopted to represent the quality of a multispectral image I. A MSI of dimension m×n×w may be defined as I(x, y, λi), where x=1, . . . ,m; y=1, . . . , n; and i=1, . . . , s. λide-notes a wavelength in a s-band multispectral image. I (x, y) corresponds to the spectral reflectance at pixel (x, y) in the image, which is a vector of s elements, one for each λi. To define image quality metrics, the original reference image is denote as Irand the test image acquired by an imaging system is denote as It. To characterize the multispectral image noise, the root mean square error (RMSE) was calculated for each pixel then averaged over all pixels. For each pixel, the RMSE(x, y) calculates the difference of spectral responses between a pixel in the reference image and the corresponding pixel in the test image as below:

Then the PSNR may be derived:

P⁢S⁢N⁢R=20·log1⁢0(MAXI1m×n⁢∑x=1m⁢⁢∑y=1n⁢⁢RMSE⁡(x,y))(7)
where MAXIis maximum possible pixel value of the multispectral image.

In one or more experiments, various multispectral data sets were used for model training and testing. All data sets were resampled to 16 bands with central wavelengths at λi∈{469,480,489,499,513,524,537,551,552,566,580,590,602,613,621,633}(nm). These bands are also the configurations implemented in the top off-the-shelf MSFA-based systems available on the market today, namely XIMEA's xiSpec camera using IMEC's MSFA technology. In experiments in the present patent document, all images were converted to reflectance values between 0 to 1, and segmented to 128×128 patches when used for DPI training and validation.

For the DPI model training, each 16 band MSI patch was processed to generate a mosaic image as the network input by subsampling each band according to a 4×4 MSFA pattern. It was also transformed to a panchromatic image by averaging the 16 bands, which were used to compare with the network output for calculations of loss functions. These patches were randomly separated to a training set and a validation set with a rough 4:1 ratio. The training set was used in back propagation and gradient descent for model convergence, while the validation set was used to find the optimal model based on average PSNR between the network output and the band-averaging panchromatic image for all patches.

In one or more experimental settings, the DPI network depth was set at D=7. A 9×9 kernel size and a number of 24 filters were deployed in the first convolution block. The middle3blocks had the same two layer structure, and a 7×7 and 5×5 kernel size was used for the first and second layer respectively, where the channels re-main as 24. The last block projected the 24 channels to one 2D residual image using a kernel size of 5×5. In one or more experimental settings, all weights were initialized as random numbers as sampled from a normal distribution and the optimization was carried out using adaptive moment estimation (ADAM) (Kingma et al., ADAM: A Method for Stochastic Optimization, arXiv preprint arXiv:1412.6980, 2014). In one or more experimental settings, the training procedure starts with an initial learning rate of 10−3and multiplied by a ratio of 0.95 for every 10 epochs.

D. Some Conclusions

Disclosed herein are system and method embodiment to demosaic multispectral images from MSFA sensors. Embodiments of a deep neural network using multiple ResNet layers are proposed to first recover a panchromatic image in full spatial resolution from a raw mosaic image. To enhance the demosaicking process, embodiments of a two-pass guided residual interpolation method are further used to demosaic each subsampled band separately before stacking together to reconstruct the full resolution MSI image. Experiment results demonstrate that disclosed embodiments outperform start-of-the-art MSFA demosaicking methods visually as well as in terms of PSNR.

FIG. 8depicts a simplified block diagram of a computing device/information handling system (or computing system) according to embodiments of the present disclosure. It will be understood that the functionalities shown for system800may operate to support various embodiments of a computing system—although it shall be understood that a computing system may be differently configured and include different components, including having fewer or more components as depicted inFIG. 8.

As illustrated inFIG. 8, the computing system800includes one or more central processing units (CPU)801that provides computing resources and controls the computer. CPU801may be implemented with a microprocessor or the like, and may also include one or more graphics processing units (GPU)819and/or a floating-point coprocessor for mathematical computations. System800may also include a system memory802, which may be in the form of random-access memory (RAM), read-only memory (ROM), or both.

A number of controllers and peripheral devices may also be provided, as shown inFIG. 8. An input controller803represents an interface to various input device(s)804, such as a keyboard, mouse, touchscreen, and/or stylus. The computing system800may also include a storage controller807for interfacing with one or more storage devices808each of which includes a storage medium such as magnetic tape or disk, or an optical medium that might be used to record programs of instructions for operating systems, utilities, and applications, which may include embodiments of programs that implement various aspects of the present invention. Storage device(s)808may also be used to store processed data or data to be processed in accordance with the invention. The system800may also include a display controller809for providing an interface to a display device811, which may be a cathode ray tube (CRT), a thin film transistor (TFT) display, organic light-emitting diode, electroluminescent panel, plasma panel, or other type of display. The computing system800may also include one or more peripheral controllers or interfaces805for one or more peripherals806. Examples of peripherals may include one or more printers, scanners, input devices, output devices, sensors, and the like. A communications controller814may interface with one or more communication devices815, which enables the system800to connect to remote devices through any of a variety of networks including the Internet, a cloud resource (e.g., an Ethernet cloud, an Fiber Channel over Ethernet (FCoE)/Data Center Bridging (DCB) cloud, etc.), a local area network (LAN), a wide area network (WAN), a storage area network (SAN) or through any suitable electromagnetic carrier signals including infrared signals.