Image super-resolution with reference images from one or more cameras

A super resolution is produced using multiple reference images. Reference images are upsampled and blurred as needed for comparison between images of different resolution. Patches in blurred images are searched to find those patches which can be assembled into vectors for improving feature content over multiple resolution levels. The searches are based on similarity maps. The assembled vectors are concatenated with one or more other vectors, up-converted and then passed through convolutional layers to obtain new feature vectors. A final feature vector is passed through a convolutional layer to obtain the super resolution image.

FIELD

The present disclosure is related to generation of a super resolution image from a source image and multiple reference images.

BACKGROUND

A super resolution image is rendering of a source image with additional information. Generally, the additional information is related to texture.

SUMMARY

A problem exists in providing satisfactory information in low resolution images, such as a wide-angle image captured by a mobile device. Embodiments provide algorithms and devices for enhancing low resolution source images using multiple reference images.

More than 90% of people take photos using a phone, and most of recent advanced mobile devices are embedded with multiple cameras. Embodiments disclosed herein provide super-resolution models which take advantage of the multiple images from the multiple phone cameras.

Also, devices are embedded with high-resolution screens, and thus high-resolution images can make use of these screens to give an improved display, especially when users show the images on UHD (ultra high definition) TV or a high resolution computer monitor.

High-resolution images carry clearer content with fine details, which can help discover interesting moments or objects from the scene.

Provided herein is a method of forming a super resolution image, the method including: generating a plurality of auxiliary images from a plurality of original images; obtaining a plurality of feature vectors from the plurality of original images and from the plurality of auxiliary images; combining, based on a plurality of indices, a higher resolution feature vector with a lower resolution feature vector, and generating the super-resolution image based on the combined feature vectors.

In some embodiments, the method further includes: dividing the plurality of feature vectors into a plurality of patches; and obtaining patch pairs from the plurality of auxiliary images indicating a second plurality of patches of the plurality of original images with matching texture content, wherein the second plurality of patches are associated with one or more higher resolution images of the plurality of original images, the plurality of indices includes a plurality of patch indices, the patch pairs are identified by the plurality of patch indices, and the higher resolution feature vector and the lower resolution feature vector are derived from the plurality of feature vectors at a patch dimension level.

In some embodiments, the plurality of original images includes a low resolution image, a medium resolution image and a high resolution image.

In some embodiments, the plurality of auxiliary images includes a medium resolution image made blurry by a downsample operation and an upsample operation.

In some embodiments, the combining uses a single similarity to assemble an intermediate feature vector and a final feature vector, the plurality of feature vectors includes the intermediate feature vector and the final feature vector, and the combining further includes fusing a final intermediate feature vector with the final feature vector, wherein the final intermediate feature vector is obtained based on an intermediate fusion using the intermediate feature vector.

In some embodiments, the plurality of original images includes a low resolution image, an original medium resolution image and a high resolution image, the plurality of auxiliary images includes a second blurry image and a first blurry image, and the plurality of feature vectors includes a first feature vector and a second feature vector, the method further including: assembling the first feature vector of the high resolution image using features of the second blurry image and based on a first similarity, wherein a first plurality of feature vectors includes the first feature vector; assembling a second feature vector of the original medium resolution image using features of the low resolution image and based on a second similarity, wherein a second plurality of feature vectors includes the second feature vector; fusing the second feature vector with a third feature vector of the low resolution image to obtain a fifth feature vector; fusing the fifth feature vector with a fourth feature vector to obtain the super resolution image, wherein the fourth feature vector is based on the first feature vector and the second feature vector; and outputting the super resolution image to a display device.

In some embodiments, the assembling the first feature vector of the high resolution image includes: downsampling and upsampling the high resolution image to obtain the first blurry image; extracting a first intermediate feature vector from the first blurry image; downsampling and upsampling of the original medium resolution image to obtain the second blurry image; extracting a second intermediate feature vector from the second blurry image; and assembling, based on the first similarity between the first intermediate feature vector and the second intermediate feature vector, a third intermediate feature vector associated with the high resolution image.

In some embodiments, the assembling the second feature vector of a medium resolution image using features of the low resolution image includes: extracting a fourth intermediate feature vector from the low resolution image; assembling, based on the second similarity between the fourth intermediate feature vector and the second intermediate feature vector, two feature vectors, wherein the two feature vectors are: i) the second feature vector associated with the original medium resolution image, and ii) a fifth intermediate feature vector associated with the original medium resolution image and the high resolution image.

In some embodiments, the method also includes: calculating a first similarity map including a plurality of first inner product values, wherein the plurality of first inner product values are based on features extracted from the first blurry image and the second blurry image, wherein the plurality of first inner product values includes the first similarity.

In some embodiments, the method also includes calculating a second similarity map including a plurality of second inner product values, wherein the plurality of second inner product values are based on features extracted from the low resolution image and the second blurry image, wherein the plurality of second inner product values includes the second similarity.

In some embodiments, the low resolution image is a first upsampled version of a source image, wherein the source image is obtained using a wide-angle lens.

In some embodiments, the original medium resolution image is a second upsampled version of a regular resolution image, wherein the regular resolution image is obtained with a typical focal length lens.

In some embodiments, the high resolution image is obtained with a telephoto lens.

In some embodiments, the method also includes training a feature extractor and a super resolution network using the low resolution image, the original medium resolution image and the high resolution image.

In some embodiments, the method also includes dividing the first blurry image into a first plurality of patches including a first patch, wherein the first intermediate feature vector corresponds to the first patch; dividing the second blurry image into a second plurality of patches including a second patch, wherein the second intermediate feature vector corresponds to the second patch; and dividing the high resolution image into a third plurality of patches including a third patch, wherein the first feature vector corresponds to the third patch, wherein the first similarity between the first intermediate feature vector and the second intermediate feature vector, is configured to indicate that the first patch is similar to the second patch, and the assembling the first feature vector of the high resolution image is based on the third patch corresponding with the second patch.

Also provided herein is an apparatus for forming a super resolution image, the apparatus including: one or more memories configured to store instructions; and one or more processors, wherein execution of the instructions by the one or more processors are configured to cause the apparatus to: generate a plurality of auxiliary images from a plurality of original images; obtain a plurality of feature vectors from the plurality of original images and from the plurality of auxiliary images; combine, based on a plurality of indices, a higher resolution feature vector with a lower resolution feature vector, and generate the super-resolution image based on the combined feature vectors.

Also provided herein is a non-transitory computer readable medium configured to store instructions, wherein execution of the instructions by one or more processors of a computer are configured to cause the computer to: generate a plurality of auxiliary images from a plurality of original images; obtain a plurality of feature vectors from the plurality of original images and from the plurality of auxiliary images; combine, based on a plurality of indices, a higher resolution feature vector with a lower resolution feature vector, and generate the super-resolution image based on the combined feature vectors.

DETAILED DESCRIPTION

Images captured by different cameras have different view ranges, different resolutions and usually same image size.

There are three cameras in an example mobile device (such as a mobile phone): wide-angle (low-resolution wide-range image), Regular (intermediate-resolution middle-range image) and telephoto (high-resolution narrow-range image).

The wide-angle image is referred to herein as a low resolution source image (LS image). The regular resolution image is referred to herein as an intermediate-resolution reference image (IR image). The telephoto image is referred to herein as a high resolution reference image (HR image). Examples of an LS image, IR image and HR image are shown inFIG.1.

FIG.1illustrates a mobile device1-1equipped with a telephoto lens1-4, a regular lens1-3and a wide-angle lens1-2. For an example scene, the corresponding images are shown. That is, the LS-image is obtained using the wide-angle lens1-2, the IR image is obtained using regular lens1-3and the HR image is obtained using the telephoto lens1-4. The low-resolution wide-range image, that is the LS image, contains more information for current scene, which can be used to discover interesting shots or objects. However, the image quality could be limited by the original image lower resolution. Embodiments improve the presentation of the LS image using the super resolution algorithms provided herein.

It is hard to directly generate high-quality high-resolution images from a single image, while images from other cameras (i.e., regular, telephoto) can provide useful information to assist the generation process as the images actually cover some content of the low-resolution one but with higher resolution.

Image super-resolution is a significant problem in the computer vision field. Current existing methods can be categorized as single image super-resolution (SISR) and reference-based image super-resolution (RefSR).

Deep-learning-based single image super-resolution (SISR) methods directly regress a high-resolution (HR) image from the low-resolution (LR) input, which might cause blurry effects.

A deep-learning-based reference-based image super-resolution (RefSR) method may utilize a high-resolution image as reference to enhance the super-resolution process, but they usually are both time- and memory-consuming.

Embodiments are motivated by the fact that most mobile devices are equipped with multiple cameras. Embodiments solve challenges of multiple-reference Image super-resolution (MISR) by providing computationally efficient architectures for obtaining a super resolution image from a source image and multiple reference images. The multiple reference images provide additional relevant information for super-resolution. As mentioned above, the wide-angle images are considered as low resolution source input (LS image), the regular camera images and telephoto camera images are intermediate-resolution reference (IR image) and HR reference (HR image).

Embodiments improve over existing SISR models nor RefSR models which are generally time- and memory-expensive.

In contrast, embodiments provide an effective way to transfer relevant texture from multiple reference images at different feature scales to improve the super-resolution performance.

Also, embodiments provide a progressively learning mechanism by first upsampling the IR image with help from the HR image, and then further boosting the super-resolution of the SR image with information from the upsampled IR image.

Thus, embodiments provide an effective way to transfer relevant texture from multiple reference images at different feature scales to improve the super-resolution performance.

Embodiments include a feature extractor, a super-resolution network and a feature map assemble module. In some embodiments, a model takes a naive upsampled image as IR image, a naive upsampled source image as HR image and outputs a super-resolution (source) image. One example of a naïve upsampling technique is bicubic upsampling.

In order to better calculate the similarity between the reference images (also referred to herein as “auxiliary images”) and the source image, the reference images are naïve downsampled-then-upsampled. The resulting reference images are referred to as blurred IR image, that is BIR image and blurred HR image, that is BEM image. The effect matches the same blurry level to the LS image, which is a naive upsampled source image.

A similarity between a patch A and a patch B of features x and y may be found using Eq. 1, where <u,v> is inner product.

A selection of a patch index Piof a patch B in a vector y using a similarity map to best match a patch A in vector x, y is performed using the argmax function as shown in Eq. 2.

In some embodiments, the feature extractor contains three convolution layers that learn features from the input images. The features are used for similarity calculations, relevant feature assembly and super-resolution generation.

Multi-scale features are extracted from the five input images (LS image, IR image, BIR image, HR image and BHR image).

After extracting the features, feature maps are split into patches. Then, patch-wise similarity maps among reference images (i.e., IR image and HR image) and source image are computed by the normalized inner product based on the features of BIR image, BHR image, and the source image (LS image). Two convolutional layers are added to connect the feature extractor and the super-resolution network.

The super-resolution network includes Up-Cony blocks (e.g., 40 RCABs and a sub-pixel convolution layer) to upscale the feature map by a factor of 2. The super-resolution network also includes a convolution layer after each Up-Cony block and feature concatenation.

The assembled feature maps from the previous step are concatenated to the features generated in the super-resolution network at the corresponding scale so that relevant textures from reference images can be used for better image generation.

For each patch on the source image feature map, the feature of the most relevant patch from IR image and the HR image (i.e., the one with highest similarity score) is be used to construct the textured-enhanced feature maps.

Multi-scale features are reassembled based on the similarities calculated at the smaller scale for effectiveness, and then the multi-scale features are fed into the super-resolution network as part of the generation process of the SR image.

Some embodiments, based on the characteristics and relationship among the reference images and the source image, provide a progressive feature transferring mechanism. The progressive feature transfer first upscales the IR image using information from the HR image, and then upscales the LS image with the upscaled IR image features.

For example, considering that the content in HR image is closer to the IR image, a similarity map M1captures this relationship. The content in IR image is closer to the LS image and a similarity map M2captures this relationship. Embodiments extract the relevant features from the HR image based on a similarity map (M1) between BIR image and the BEM image. The selected features are then fed into a super-resolution network at the intermediate level to further improve the resolution at the intermediate level.

Then a similarity (i.e., inner product) is computed to form the similarity map M2using the features of the LS image and BIR image. M2is used to identify features to bring from the super-resolution network at the intermediate level into the source image super-resolution network.

The source image super-resolution network concatenates and feeds forward the assembled IR image features at different scales. A convolutional layer then outputs a super resolution image (SR image) as the high-resolution image for a given source image (LS image).

Embodiments are now explained in further detail with respect toFIGS.2-10.

FIG.2illustrates logic2-9for obtaining an SR image2-5from the LS image, the IR image and the HR image. At operation2-10, blurred reference images BIR image and BHR image are generated. The blurring can be obtained by first applying a naïve downsampling operation and then a naïve upsampling operation. At operation2-20, features are extracted using a feature extractor for each of the five images. The same feature extractor may be applied to all five images.

Referring briefly toFIG.7, operations commencing with the LS image are a part of top flow7-1which includes processing from the LS image through to the SR image5-1. Operations commencing with the IR image and the BIR image are a part of middle flow7-2. Operations commencing with the HR image and BHR image are part of bottom flow7-3. Each flow begins with the same feature extractor. Further description is provided below.

Returning toFIG.2, from the LS image, low resolution feature vectors2-1are obtained. From the IR image and the BIR image, medium resolution feature vectors2-2are obtained. From the HR image and the BHR image, high resolution feature vectors2-3are obtained. All of these feature vectors are directly comparable in terms of scale, based on the upsampling applied to create the LS image and the upsampling used to create the IR image. That is, the scale of the feature vectors is at the granularity of the HR image.

The feature vectors2-1,2-2and2-3are combined at operation2-30to obtain hybrid feature vectors2-4. At operation2-50the low resolution feature vectors2-1and the hybrid feature vectors2-4are combined to generate the SR image2-5.

FIG.3Aprovides an example of an LS image. This image has been upscaled4xfrom an original camera image captured through lens1-2.

FIG.3Bprovides an example of an IR image. This image has been upscaled2xfrom an original camera image captured through lens1-3.

FIG.3Cprovides an example of a BIR image. This image is obtained by applying a naïve downsampling and then naïve upsampling to the IR image ofFIG.3B.

FIG.3Dprovides an example of an HR image. This image is an original camera image captured through lens1-4.

FIG.3Eprovides an example of a BIR image. This image is obtained by applying a naïve downsampling and then naïve upsampling to the HR image ofFIG.3D.

FIG.4illustrates logic4-9with further details of patches and similarity maps. Operations4-10and4-20are similar to operations2-10and2-20ofFIG.2.

Operation4-30forms similarity map M1between medium resolution feature vectors2-2and high resolution feature vectors2-3. Similarity map M2is formed between low resolution feature vectors2-1and medium resolution feature vectors2-2.

Operation4-40produces patches indices4-7by discovering patch pairs using M1and M2. Operation4-50fuses selected feature vectors from unblurred images of different resolution. The operation of fusing refers to concatenating vectors together and applying the result as an input to a convolutional neural network. Concatenation of the vector U=[u1, . . . , uN] with the vector V=[v1, . . . , vN] produces the vector W=[u1, . . . , uN, v1, . . . vN] where U and V are of length N. In general U and V may be of different lengths. Concatenation may be indicated by a double plus symbol.

An example for comparing an LS image with an SR image is provided byFIGS.5A and5B.FIG.5Ais an example LS image.FIG.5Bis an SR image5-1which is an example of SR image2-5produced by logic2-9or by logic4-9.

The quality of SR image5-1, in terms of PSNR and structural similarity measure SSIM, is better than comparative approaches for upsampling and/or super resolution.

Further details specific to the embodiment for producing the SR image5-1are given in logic6-9ofFIG.6and the neural network7-9ofFIG.7.

FIG.6provides logic6-9which uses the similarity maps M1and M2to identify texture from references images (IR image and HR image) and produce the high PSNR of image5-1, as an example.

InFIG.6, at operation6-10, logic6-9forms the low-res similarity map M2by computing a similarity S2between a patch with patch index A of a feature vector of an unblurred image (LS) and a patch at patch index B of a feature vector of a blurred relatively higher resolution image (BIR) for all pairs (A,B).

At operation6-11, logic6-9searches the low-res similarity map M2for a patch with index Bmax with highest similarity to a patch with index A. A is an index which counts through all the patches of a feature vector, e.g., A=1, 2, . . . N where there are N patches in one feature vector. At operation6-12, logic6-9, using all values of A, assembles a feature vector using the found values of Bmax to access the feature vector of an unblurred version (IR) of the blurred relatively higher resolution image (BIR) which is the patch at index Bmax. Concatenate the assembled vector with the feature vector of the unblurred image (LS). As an example of this concatenation fromFIG.7, operation6-12outputs the concatenation of vector3from LS with vector6. Vector6is composed of selected patches of vector2arising from the IR image.

At operation6-13, fusing is performed by passing the concatenated feature vector through a stage of neural network (e.g., through stage7-11of top flow7-1of neural network7-9to produce vector5). Operation6-25, which has two inputs, is then reached. The second input to operation6-25is derived beginning with operation6-20.

At operation6-20, logic6-9forms a high-res similarity map M1by computing a similarity S1between a patch with patch index B of a feature vector of a blurred image (BIR) and a patch at patch index C of a feature vector of a blurred relatively higher resolution image (BHR) for all pairs (B,C). Logic6-9, at operation6-21, then searches the high-res similarity map M1for a patch with index Cmax with highest similarity to a patch with index B. Operation6-22, similar to operation6-12, assembles a feature vector using found values of Cmax to access the feature vector of an unblurred version (HR) of the blurred relatively higher resolution image (BHR) which is a patch at index Cmax with a feature vector of the unblurred image (IR) which is the patch at index B. An example of the output of operation6-22is the concatenation of vector2associated with the IR image with vector1which is made up of selected patches of vector3ifrom the HR image. At operation6-23, logic6-9passes the concatenated feature vector through a stage of the neural network (e.g., stage7-12of middle flow7-2of neural network7-9).

At operation6-24, logic6-9uses Bmax to select and assemble a feature vector output from stage of6-23(e.g., output of7-12, such as vector4ofFIG.7). Bmax was determined at operation6-12for each value of the index A.

At operation6-25, the vectors arising from operation6-13and6-24are concatenated. An example fromFIG.7is concatenation of vector5and vector4.

At operation6-26the concatenated vector from operation6-25is passed through a convolution layer to provide the SR image5-1. An example of the convolution layer is7-13ofFIG.7.

InFIG.7, operations commencing with the LS image are a part of top flow7-1which includes processing from the LS image through to the SR image5-1. Operations commencing with the IR image and the BIR image are a part of middle flow7-2. Middle flow7-2includes processing of the IR image and the BIR image through to the outputs of super resolution network7-31. Operations commencing with the HR image and BHR image are part of bottom flow7-3. Bottom flow7-3includes processing the HR image and the BHR image up to routing vector3iup to vector1based on similarity1.

Top flow7-1ofFIG.7passes the LS image through feature extractor7-31. Feature extractor7-31, in some embodiments has three convolution layers with channel size {16, 16, 32} and a kernel size of (3×3) for each layer and stride {1, 2, 1}. The output of the feature extractor7-31is the vector4i. The vector4iis fed through two convolution layers identified as7-32and produces a vector6iwhich is input to the top flow super resolution network7-33to produce vector3. Vector3is concatenated with vector6of top flow7-1(made up of selected patches of vector2). Specifically, the top flow super resolution network7-33receives indices from similarity map M2to select patches of vector2(forming vector6) and patches of vector4(forming vector7) to improve the SR image5-1.

Vector3and the vector6are concatenated as vector8. Vector8passes through up conversion and a convolution layer (together indicated as7-11inFIG.7) to produce vector5.

Vector5and vector7are concatenated as vector9. Vector9passes through the convolution layer7-13and the result is the SR image5-1.

Considering the architecture of neural network7-9, five images are brought in on the left. The images pass through one or another of feature extractors7-31,7-34and7-35. Intermediate vectors vector1i, vector2i, vector3i, vector4i, vector5i, vector6i, vector7i, vector1, vector2, vector3, vector4, vector5, vector6and vector7are produced, among other vectors. Neural network7-9includes two similarity functions: similarity2operating between the top flow7-1and the middle flow7-2, and similarity1operating between the middle flow7-2and the bottom flow7-3. Each similarity function controls a feature map assemble function which acts as a selector or multiplexer to bring in selected patches to form an assembled feature vector in a different flow.

Details of the middle flow7-2will now be described. The IR image, which is a two times upsampled version of the IR image captured by mobile device1-1, is blurred to produce the BIR image. Both pass through feature extractor7-34, which is the same as feature extractor7-31. The resulting feature vectors are divided into patches. An example patch is shown as patch2inFIG.7(in the middle flow7-2, in vector2i).

The goal of similarity2is to find a patch from vector5i, for example patch5ofFIG.7, which will finally improve SR image5-1by being a best match to patch4of vector4which corresponds to the LS image. To do this, patch4which originated with the LS image is compared with all the patches in vector2iwhich originated with the BIR image. This comparison is indicated by the block similarity2ofFIG.7and the selected patch at two resolutions is passed to the top flow7-1by the function feature map assemble7-42.

That is, the resulting best match from the BIR image is used to indicate the same patch, but in the IR image (patch5of vector5i). After processing by two convolution layers to produce vector7i, up-convolution and the convolution of the middle flow super-resolution network7-37, vector2is obtained. The result of similarity2chooses an instance of vector2, and this is concatenated with vector3in the top flow7-1as described above.

The bottom flow7-3provides an instance of vector3ias vector1, and vector1is concatenated with vector2as vector10. Vector10passes through the up-convolution and convolution layer indicated as7-12inFIG.7to produce vector4. The top flow7-1uses the instance of vector4indicated by similarity2to produce vector7. The same index from M2is used to select from vector2to build vector6and from vector4to build vector7.

The bottom flow7-3passes the BHR image and the HR image through the feature extractor7-35, which is the same as feature extractors7-31and7-34. The BHR image produces vector1iand the HR image produces vector3i. The similarity map M1is used to find a patch similar to patch2of the BIR image. The best matching patch for a given patch2is, for example, patch1of vector1i. The patch with the same index from vector3iis then selected by similarity1and feature map assemble function7-41to occupy the position of vector1in the middle flow7-2.

FIG.8illustrates a neural network8-9of an alternative embodiment. The neural network8-9is similar to the neural network7-9and somewhat simpler. The neural network7-9includes top flow8-1, middle flow8-2and bottom flow8-3. The description ofFIG.8emphasizes differences fromFIG.7, other structural components and actions are similar to those inFIG.7.

The input to the top flow8-1is the IR image without sample change (seeFIG.1) and a blurred image of the IR image without sample change, called BIR image-2to for clarity with respect to the BIR image ofFIG.7. The input to the middle flow8-2is the LS image. The input to the bottom flow8-3is the pair of images BEM image and HR image. Similarity function11and similarity function12are similar to similarity function1and similarity function2ofFIG.7. Likewise the feature map assemble functions ofFIG.8are similar to those ofFIG.7.

The feature extractors ofFIG.8are the same as those ofFIG.7. The feature extractors produce vector11i, vector12i, vector13i, vector14i, vector15i, vector16iand vector17i. Patches are selected from vector11iand vector12iby similarity function11and assembled by feature map assemble function8-15to produce vector12and vector15. Also, patches are selected from vector16iand vector17iby similarity function12and assembled by feature map assemble function8-16to produce vector11and vector14.

The middle flow8-2produces vector10which is then concatenated with vector11and vector12and passed through the up-convolutional layer and convolutional layer indicated as8-11to produce vector13. Vectors13,14and15are then concatenated and pass through the convolutional layer8-13to provide the SR image8-4.

In summary, an embodiment of producing a super resolution image is shown inFIG.8. The method includes extracting a first plurality of feature vectors from an original medium resolution image, wherein the first plurality of feature vectors includes a first intermediate feature vector and a second intermediate feature vector (for example, vector11iand vector12ifrom the IR image without sample change). The method also includes extracting a third intermediate feature vector from a blurry version of the original medium resolution image (for example vector13ifrom the BIR image2). The method also includes extracting a fourth intermediate feature vector from a blurry version of an original low resolution image (for example, vector14ifrom the LS image). The method also includes extracting a fifth intermediate feature vector from a blurry version of an original high resolution image (for example, vector15ifrom the BEM image). The method also includes extracting a second plurality of feature vectors from the original high resolution image, wherein the second plurality of feature vectors includes a sixth intermediate feature vector and a seventh intermediate feature vector (for example vector16iand vector17ifrom the HR image).

The method also includes choosing patches of a first feature vector and a fourth feature vector from the second plurality of feature vectors based on a similarity of patches with respect to the fourth intermediate feature vector (vector14i) to the fifth intermediate feature vector (vector15i); choosing patches of a second feature vector and a fifth feature vector from the first plurality of feature vectors based on a similarity of patches with respect to the third intermediate feature vector (vector13i) to the fourth intermediate feature vector (vector14i).

The method also includes fusing the first feature vector and the second feature vector to form a third feature vector; and fusing the third feature vector, the fourth feature vector and the fifth feature vector to form the super resolution image.

In some embodiments, the models can be used for with only one reference image, corresponding to a scenario in which the mobile device1-1only has two cameras, instead of three cameras. In general, if there is a lack of a reference image (and as a result, a lack of an assembled enhanced feature map), features from the feature extractor are extracted and concatenated to the SR network as the replacement for the missing assembled enhanced feature map.FIG.9is for use with one reference image.

FIG.9includes top flow9-1and bottom flow9-2. The LS image is input to the top flow9-1. The IR image and the BIR image are input to the bottom flow9-2.

In the top flow, the feature extractor9-31obtains a feature vector31i, performs a down-convolution to reduce sample rate and then obtains a feature vector32i. Vector32ipasses through two convolution layers to obtain the vector33i, which is input to the super resolution network9-33after up-convolution to increase sample rate. Vector33ipasses through two convolution layers to produce vector20. Vector20is concatenated with vector21to produce vector28. Vector28passes through up-convolution and then two convolution layers to provide vector22. Vector22is concatenated with vector23to produce vector29. Vector29passes through a final convolution layer to provide the SR image9-3.

In the bottom flow, the IR image and the BIR image pass into the feature extractor9-34to produce intermediate feature vectors21iand22i, respectively. As a brief aside, a convolution layer itself is able to down-adjust the sample rate (perform downsampling), based on the layer settings. In some embodiments, a down-convolution is included in the (first) one of the two convolution layers. The expression “two convolution layers” and “a down-convolution and a convolution layer” are equivalent when down-sampling occurs. Downsampling is also indicated in the figures by a vector representation of a tall narrow rectangle changing to a shorter narrow rectangle. Similarly, up-sampling is implied to have occurred when a shorter narrow rectangle becomes a taller narrow rectangle.

Referring again to intermediate feature vectors21iand22i, these vectors each pass through two convolution layers and produce vectors23iand27i, respectively. A similarity calculation (“Sim” inFIG.9) is applied to each patch in vector32iin comparison with the patches of27i. The resulting index is used to build vector21from selected patches of vector25and build vector23from selected patches of vector27. The assembly of selected patches is performed by the feature map assemble function (marked with “A” inFIG.9).

Returning to the bottom flow9-2, vector23igoes through two convolution layers to produce the vector24. Vector24goes through an up-convolution and this produces vector25. There is no second reference image to provide patches to build up vector26. Vector26is instead obtained as vector27i. In the one-reference model, embodiments use vector27ias vector26. Their values are exactly the same. In this case, although vector27ilooks smaller than vector26, but actually they have the same size. Vectors25and26are then concatenated to form vector30. Vector30passes through an up-convolution and then two convolution layers to obtain vector27. Vector27is used as described above to provide patches to build up vector23.

FIG.10Aillustrates logic10-81for training. At operation10-84, a high resolution image10-1with the same focal length as a low resolution image (that is, capturing the same width and height of a scene) is taken from a training set10-82. Auxiliary images10-83are synthesized from the high resolution training image. Images10-1and10-2ofFIG.10Bare examples of auxiliary images10-83. Simulated images from the camera1-1for testing, image1, image2and image4, are also synthesized from image10-1. At operation10-85, the logic10-81trains feature extractors and super resolution networks. The weights of the feature extractors and super resolution networks are referred to as θ10-87. An example structure used for training is illustrated inFIG.10B.

At operation10-86, a determination is made whether training of the weights is complete. This determination may be based on a mean square error between a reconstructed super resolution image (such as image5-1ofFIG.5BandFIG.10B) and the ground truth super resolution image10-1. If the training is not complete, the logic returns to operation10-84for another and different image10-1. If the training is complete, the logic flows to operation10-88, providing the weights θ10-87.

Operation10-88has inputs of the LS image and one or more reference images10-90. Operation10-88implements the processing of network7-9(FIG.7), network8-9(FIG.8) or network9-9(FIG.9). The output of the operation10-88is an SR image10-91. An example of the SR image10-91is image5-1as shown inFIG.5Bfrom network7-9ofFIG.7.

FIG.10Billustrates a network10-9training. The network10-9is an example implementation for the operation10-85ofFIG.10Afor network7-9ofFIG.7. The neural networks8-9and9-9may be trained in a similar manner.

The layers of the model (weights θ10-87) are trained in part using reconstruction loss LR1, LR2and LR3. The training is also based on dual loss LD1and LD2.

Several images are used for training, as described below. Three scales are shown in the bottom right ofFIG.10B. Image5-1and image10-1correspond to HR scale, that is, the resolution of the HR image (image4). Images10-6,10-2and10-4correspond to IR scale, that is, the resolution of the IR image (image2). Images10-7,10-3and10-5correspond to LS scale, that is the resolution of the LS image (image1).

In terms of data and functional blocks,FIG.10Bincludes the neural network7-9which is being trained, the super resolution ground truth image10-1, the estimated super resolution image5-1, and a function for comparing these using the 1-norm of the difference. Specifically, the 1-norm of the difference between image5-1and image10-1gives the reconstruction loss LR1.

LR2is produced based on an image10-6provided by the network under training after passing vector8through a convolutional layer10-10. The function of the layer10-10is to turn vector8into an image. A 1-norm difference is taken between image10-6and image10-2to provide LR2. Image10-2is a downsampled version of image10-1. Image10-2is a ground truth image at the resolution of the IR image. LR3is produced from image10-7compared with image10-3. Image10-7is obtained by passing vector6ithrough a convolutional layer10-11. Image10-3is a ground truth image at the resolution of the LS image.

Besides the reconstruction losses LR1, LR2and LR3, dual regression losses LD1and LD2are included for training, in some embodiments. A dual regression training scheme contains a primal regression task for super-resolution and a dual regression task to project super-resolved images back to LR images. The primal and dual regression tasks form a closed-loop.

As an example, the reconstruction loss may be expressed as in Eq. 3.

Where the notation IMrefers to the image with reference number M. For example, image5-1is indicated in Eq. 3 as15-1.

The ground truth HR image10-1is naively downsampled by different scale factors to get the image10-2and the image10-3.

A total reconstruction loss is Lrec=LR1+LR2+LR3.

The dual losses are similar to the expressions for the reconstruction losses.

A total dual loss is Ldual=LD1+LD2.

The entire model is trained with loss Ltotal=λ2Lrec+λ1Ldual.

The loss functions are combined to produce Ltotaland this is used to update the weights θ10-87of the neural network being trained (network7-9for the example ofFIG.10B). An Adam optimizer may be used. Adam is an optimization algorithm for stochastic gradient descent for training deep learning models.

A decay rate of the Adam optimizer, may have a value of, for example, 0.9. A is a value set in using the Adam optimizer to combine the losses, and may be, for example 0.1. A learning rate may be initialized at a value of 0.0001 and then exponentially decayed after a number of epochs. The number of epochs may be, for example, 20.

In some embodiments, a VGG19 network and a discriminator network are included to construct the perceptual loss and adversarial loss to enhance the training power. VGG-19 is a convolutional neural network that is 19 layers deep. The VGG19 network operates on image5-1to produce an SR feature vector and operates on image10-1to produce a GT (ground truth) feature vector, and a difference of these features provides a perceptual loss, Lper. A discriminator network produces an estimate of whether the estimated super resolution image, image5-1, is real or fake and an estimate of whether the ground truth image, image10-1, is real or fake; and a difference of these provides the adversarial loss, Ladv.

In this embodiment, the total loss is then given in Eq. 8.
Ltotal=λ4Lper+λ3Ladv+λ2Lrec+λ1LdualEq. 8

The weights θ10-87are then trained using Ltotalof Eq. 8 and the Adam optimizer, in some embodiments.

FIG.11illustrates an exemplary apparatus11-9for implementation of the embodiments disclosed herein. For example,FIG.11illustrates exemplary hardware for implementation of computing devices such as the mobile device1-1, according to some embodiments. The apparatus11-9may be a server, a computer, a laptop computer, a handheld device, or a tablet computer device, for example. Apparatus11-9may include one or more hardware processors11-1. The one or more hardware processors11-1may include an ASIC (application specific integrated circuit), CPU (for example CISC or RISC device), and/or custom hardware. Apparatus11-9also may include wired and/or wireless interfaces11-4. Apparatus11-9also may include a user interface11-5(for example a display screen and/or keyboard and/or pointing device such as a mouse). The apparatus11-9may also include, in addition, a display screen11-17. Apparatus11-9may include one or more volatile memories11-2and one or more non-volatile memories11-3. The components above may be connected by a bus11-6. The one or more non-volatile memories11-3may include a non-transitory computer readable medium storing instructions for execution by the one or more hardware processors11-1to cause apparatus11-9to perform any of the methods of embodiments disclosed herein.