Patent ID: 12210587

DETAILED DESCRIPTION

The following detailed description of example embodiments refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.

FIG.1is a diagram of an environment according to an example embodiment.

Referring toFIG.1, according to an embodiment of the present disclosure, an electronic device101is included in a network environment100. The electronic device101may include at least one of a bus110, a processor120, a memory130, an input/output interface150, a display160, a communication interface170, or an event processing module180. In some embodiments, the electronic device101may exclude at least one of the components or may add another component.

The bus110may include a circuit for connecting the components120to180with one another and transferring communications (e.g., control messages and/or data) between the components.

The processing module120may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor120may perform control on at least one of the other components of the electronic device101, and/or perform an operation or data processing relating to communication.

The memory130may include a volatile and/or non-volatile memory. For example, the memory130may store commands or data related to at least one other component of the electronic device101. According to an embodiment of the present disclosure, the memory130may store software and/or a program140. The program140may include, e.g., a kernel141, middleware143, an application programming interface (API)145, and/or an application program (or “application”)147. At least a portion of the kernel141, middleware143, or API145may be denoted an operating system (OS).

For example, the kernel141may control or manage system resources (e.g., the bus110, processor120, or a memory130) used to perform operations or functions implemented in other programs (e.g., the middleware143, API145, or application program147). The kernel141may provide an interface that allows the middleware143, the API145, or the application147to access the individual components of the electronic device101to control or manage the system resources.

The middleware143may function as a relay to allow the API145or the application147to communicate data with the kernel141, for example. A plurality of applications147may be provided. The middleware143may control work requests received from the applications147, e.g., by allocation the priority of using the system resources of the electronic device101(e.g., the bus110, the processor120, or the memory130) to at least one of the plurality of applications134.

The API145is an interface allowing the application147to control functions provided from the kernel141or the middleware143. For example, the API133may include at least one interface or function (e.g., a command) for filing control, window control, image processing or text control.

The input/output interface150may serve as an interface that may, e.g., transfer commands or data input from a user or other external devices to other component(s) of the electronic device101. Further, the input/output interface150may output commands or data received from other component(s) of the electronic device101to the user or the other external device.

The display160may include, e.g., a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical systems (MEMS) display, or an electronic paper display. The display160may display, e.g., various contents (e.g., text, images, videos, icons, or symbols) to the user. The display160may include a touchscreen and may receive, e.g., a touch, gesture, proximity or hovering input using an electronic pen or a body portion of the user.

For example, the communication interface170may set up communication between the electronic device101and an external electronic device (e.g., a first electronic device102, a second electronic device104, or a server106). For example, the communication interface170may be connected with the network162or164through wireless or wired communication to communicate with the external electronic device.

The first external electronic device102or the second external electronic device104may be a wearable device or an electronic device101-mountable wearable device (e.g., a head mounted display (HMD)). When the electronic device101is mounted in a HMD (e.g., the electronic device102), the electronic device101may detect the mounting in the HMD and operate in a virtual reality mode. When the electronic device101is mounted in the electronic device102(e.g., the HMD), the electronic device101may communicate with the electronic device102through the communication interface170. The electronic device101may be directly connected with the electronic device102to communicate with the electronic device102without involving with a separate network.

The wireless communication may use at least one of, for example, 5G, long term evolution (LTE), long term evolution-advanced (LTE-A), code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection may include at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS).

The network162may include at least one of communication networks, e.g., a computer network (e.g., local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The first and second external electronic devices102and104each may be a device of the same or a different type from the electronic device101. According to an embodiment of the present disclosure, the server106may include a group of one or more servers. According to an embodiment of the present disclosure, all or some of operations executed on the electronic device101may be executed on another or multiple other electronic devices (e.g., the electronic devices102and104or server106). According to an embodiment of the present disclosure, when the electronic device101should perform some function or service automatically or at a request, the electronic device101, instead of executing the function or service on its own or additionally, may request another device (e.g., electronic devices102and104or server106) to perform at least some functions associated therewith. The other electronic device (e.g., electronic devices102and104or server106) may execute the requested functions or additional functions and transfer a result of the execution to the electronic device101. The electronic device101may provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example.

AlthoughFIG.1shows that the electronic device101includes the communication interface170to communicate with the external electronic device104or106via the network162, the electronic device101may be independently operated without a separate communication function, according to an embodiment of the present disclosure.

For example, the event processing server module may include at least one of the components of the event processing module180and perform (or instead perform) at least one of the operations (or functions) conducted by the event processing module180.

The event processing module180may process at least part of information obtained from other elements (e.g., the processor120, the memory130, the input/output interface150, or the communication interface170) and may provide the same to the user in various manners.

Although inFIG.1the event processing module180is shown to be a module separate from the processor120, at least a portion of the event processing module180may be included or implemented in the processor120or at least one other module, or the overall function of the event processing module180may be included or implemented in the processor120shown or another processor. The event processing module180may perform operations according to embodiments of the present disclosure in interoperation with at least one program140stored in the memory130.

FIG.2is a diagram of generating training data and training a super-resolution network. The operations shown inFIG.2may be performed by one or more the electronic device101or the server106.

As shown inFIG.2, the method for training a super-resolution network260may include obtaining a low resolution image210, generating, using a first machine learning model220, a first high resolution image230based on the low resolution image210. According to an example embodiment, the first machine learning model220may use a blind super-resolution method to generate the first high resolution image230. A blind super-resolution process may attempt to infer the unknown blur and downscaling kernel that has produced a given low-resolution (LR) image, and use it to reconstruct its high-resolution version.

According to another example embodiment, the first machine learning model220may use blind super-resolution with iterative kernel correction method.

According to yet another example embodiment, the first machine learning model may first estimate a kernel using a KernelGAN method. The model may then use the estimated kernel in any online optimization method for super-resolution, such as a direct optimization method, a blind super-resolution kernel estimation method using an internal GAN, or a super-resolution method using deep leaning which trains a small image-specific CNN at test time on examples extracted solely from the input image itself.

Further, the method may include generating, using a second machine learning model240, a second high resolution image250based on the first high resolution image230and an unpaired dataset of high resolution images280.

According to an example embodiment, the second machine learning model240may be a domain adaptation module configured to obtain synthetic high resolution images and make them look like real high resolution images. According to an example embodiment, the second machine learning model may use a CycleGAN method for general domain transfer where two generators are trained and a cycle consistency loss is enforced.

Further still, the method may include obtaining a training data set using the low resolution image210and the second high resolution image250; and training the super-resolution network260using the training data set. In this way, the super resolution network260may generate a high resolution image270that is substantially similar to the second high resolution image250based on being trained.

Accordingly, the trained super-resolution network260may be implemented in an electronic device (e.g., a smartphone) to permit the electronic device to perform super-resolution.

The method shown inFIG.2is described in more detail below. The method may include obtaining a set of low-resolution images, and the set of low-resolution images may be represented as:
X={xi}i=1N

The method may include obtaining a set of high-resolution images, and the set of high-resolution images may be represented as:
Y={yi}i=1M

As shown below, W denotes the width of the low-resolution image, H denotes the height of the low-resolution image, and s is a scale factor.
xi∈W×H×3
yi∈sW×sH×3

The method may include learning the following mapping function which is modeled as the deep super-resolution network.
F:X→Y

To train the mapping function F, the method includes generating a paired set of real low-resolution and synthetic high-resolution data:
(X,YS)={(xi,yiS)}i=1N

The method includes generating an initial version of synthetic high-resolution data:
{tilde over (Y)}S={{tilde over (y)}iS}i=1N

Then, the method includes applying a CycleGAN to refine the initial version of the high-resolution data, to obtain the synthetic high-resolution training data.

The method includes training the mapping function F of the super-resolution network in a supervised manner using the paired set:
(X,YS)

In this way, the present disclosure provides a model for unsupervised super-resolution that is free of the limitations described regarding the related art, and performs better than the related art under realistic conditions. In contrast to the related art, the embodiments of the present disclosure do not compute synthetically generated low-resolution images but rather operate in the high-resolution domain directly. Accordingly, the embodiments of the present disclosure can account not just for the unknown blur kernel, but for a multitude of different degradation factors, introduced by a camera system when taking a photograph. Second, the fact that the embodiments do not synthetically create low-resolution images eliminates an additional source of domain gap, that related art systems have to address. Third, because the example embodiments do not need to adapt to any image individually at test time, the embodiments are efficient, while still maintaining good generalization properties.

In this way, the example embodiments generate high-quality pseudo-groundtruth, removing the need for paired training data; explicitly avoid introducing a domain gap between real and synthetically generated low-resolution images; and outperform unpaired super-resolution alternatives, both quantitatively and qualitatively. For instance, the example embodiments are faster because the embodiments do not have to train a model on each image individually at test time, and are truly unsupervised and make no specific kernel assumptions. In this way, the example embodiments avoid any potential artifacts introduced by a domain gap between synthetically generated and real low-resolution images.

FIG.3is a flowchart of a method for training a super-resolution network. The operations shown inFIG.3may be performed by one or more the electronic device101or the server106.

As shown inFIG.3, the process may include obtaining a low resolution image (operation310), and generating, using a first machine learning model, a first high resolution image based on the low resolution image (operation320).

Given a high-resolution image y, its degraded low-resolution version x may be modeled as:
x=(k*y)↓s+n

As shown above, k is the blur kernel, and n is the added noise.

Replacing y for the output of the first stage of the process, y can be recovered using Equation 1 shown below:

y~S*=argminy~S⁢{x-(k*y~S)⁢↓s⁢1⁢+R⁡(y~S)}

As shown above, the first term is the L1 backprojection loss, and R is a regularization term. The former encourages consistency between the reconstructed high-resolution image and the low-resolution observation. The latter is beneficial in dealing with the ill-posed nature of the problem, as there are multiple high-resolution images whose downsampled versions would match x.

In some embodiments, the usage of a regularization term might not be necessary. For instance, {tilde over (y)}S=fθDIP(z), where z∈sW×sH×Cis a random tensor, and fθDIPis a generator network whose weights, θ, are optimized for a particular range input.

An embodiment modifies deep image prior (DIP) by adding a total variation regularization term, with a small weight, which results in the following image loss function:
I(x,k,θ)=∥x−(k*fθDIP(z))↓s∥1+αTV(fθDIP(z))

In a realistic super-resolution scenario, the kernel k is also unknown. An embodiment represents the kernel using a K×K matrix and includes it in the optimization, also adding the following kernel loss to further constrain the optimization:

ℒK⁡(k)=1-∑ij⁢k⁡(i,j)1+1K⁢(xc,yc)-∑ij⁢(k⁡(i,j)·(i,j))∑ij⁢k⁡(i,j)1

As shown above, (xc, yc) denotes the center of the kernel mask. This encourages the kernel to sum to 1 and be properly centered.

The final objective for synthetic data generation is the combination of the image lossIand the kernel lossK:
S(x,k,θ)=I(x,k,θ)+βK(k)

The embodiments minimizeSfor each image xiin the low-resolution dataset, thereby obtaining the paired dataset represented below:
{(xi,{tilde over (y)}iS=fθiDIP(zi))}i=1N

As further shown inFIG.3, the process may include generating, using a second machine learning model, a second high resolution image based on the first high resolution image (operation330) and an unpaired dataset of high resolution images, and obtaining a training data set using the low resolution image and the second high resolution image (operation340).

The second machine learning model may be a domain adaptation module configured to obtain synthetic high resolution images and make them look like real high resolution images. In some embodiments, the second machine learning model may be a CycleGAN that constrains the input and output images to have the same low frequencies. However, in other embodiments, the second machine learning model may use other domain adaptation methods such as methods that match feature distributions in source and target domains, that reweigh or select samples from the source domain, that seek an explicit feature space transformation, that use discriminative classifiers, or the like.

In general, domain adaptation (also called domain translation) refers to a module that converts one type of data into another such as, for example, female faces to male faces, day images to night images, computer-generated graphics to real images, etc.

While DIP can produce a reasonably good high-resolution version of the low-resolution input image, DIP is still limited in the sense that its output is informed only by a single image. As a result, the generated images might not be sharp or “natural” enough to be used as pseudo-ground truth for training a neural network.

To further increase the quality of the synthetic data, the embodiments modify CycleGAN to train a generator G{tilde over (Y)}Ythat performs domain adaptation between the spaces of synthetic DIP-generated images, {tilde over (Y)}, and real high-resolution images, Y.

An objective is to enhance the high frequencies in the output, making it look more natural, without changing the low frequency content. To that end, the embodiments add a low-frequency content preservation loss,LF, defined as the L1 distance between the input and the output of the synthetic-to-real generator, after passing both through a low-pass filter:
LF={tilde over (y)}˜P{tilde over (Y)}[∥G{tilde over (Y)}Y({tilde over (y)})*kDA−{tilde over (y)}*kDA∥1]

As shown above, kDAis a Gaussian blue kernel with standard deviation σDA.

The final domain adaptation loss is provided as:
DA=adv+λcyccyc+λLFLF

As shown above,advis the adversarial loss, andcycis the cycle loss.

Based on the CycleGAN architecture being trained, the embodiments apply G{tilde over (Y)}Yto the images generated by DIP, thereby obtaining a new paired dataset shown below:
{(xi,yiS=G{tilde over (Y)}Y({tilde over (y)}iS)}i=1N

As further shown inFIG.3, the process may include training the super-resolution network using the training data set (operation350).

The super-resolution network may be represented as:
F:W×H×3→sW×sH×3

To train the super-resolution network, the embodiments use a loss function represented as:
f=ηFIDLF+PERC+γGAN

The fidelity term may be modified to operate on low-frequency content, and may be represented as:

ℒFIDLF=1N⁢∑i⁢yiS*kSR-F⁡(xi)*kSRL⁢⁢1

The perceptual loss may be represented as:

ℒPERC=1N⁢∑i⁢Ψ⁡(yiS)-Ψ⁡(F⁡(xi))L⁢⁢1

The GAN loss may be represented as:
GAN

The Gaussian blur kernel may be represented as:
kSR

The Gaussian blur kernel may be represented as:
σSR

In order to reduce the effect of artifacts in the synthetic data on model quality, the embodiments use real HR images as the target domain when training the GAN discriminator, similar to the training of CycleGAN.

FIG.4is a flowchart of performing super-resolution imaging by an electronic device.

As shown inFIG.4, the process may include obtaining, using a camera of the user device, a low resolution image (operation410); inputting the low resolution image into a trained super-resolution network (operation420); obtaining a high resolution image from the trained super-resolution network (operation430); and providing the high resolution image for display (operation440).

A method of performing super-resolution imaging according to an embodiment may generate a pseudo ground truth using blind super-resolution and domain adaptation, and therefore does not require an input of paired training data.

Related methods of image processing use an offline trained model on paired data (general super-resolution). As such, these general super-resolution methods require readily available paired training data. Further, general super-resolution methods that obtain paired training data using simple downscaling will provide low quality outputs when using real input data.

The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

As used herein, the term “component” is intended to be broadly construed as hardware, firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and/or methods were described herein without reference to specific software code—it being understood that software and hardware may be designed to implement the systems and/or methods based on the description herein.

Even though particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.), and may be used interchangeably with “one or more.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.