Patent Description:
With the rapid development of electronics technology, <NUM> and <NUM> high-definition (HD) displays have become standard for home theater systems, tablet computers, and mobile devices. Videos played on mobile devices usually have lower resolutions, because of the limitations associated with the video capturing device, storage space, network bandwidth, and/or data flow. It is therefore important to boost the effective resolution and visual effects of videos before individual frames are displayed on modem devices, in order to take full advantage of the new generations of HD displays.

Existing image resolution enhancement techniques operate primarily on good quality images that do not have noise and artifacts. When noise and blocking artifacts are present in the input video (e.g., videos viewed on mobile devices), the use of the existing techniques often results in further deterioration of the image quality. Additionally, due to computational complexity, current image resolution enhancement techniques do not improve the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) in comparison to conventional interpolation techniques, and, therefore, have a limited use in image processing on mobile devices with HD displays.

The paper<NPL>, uses a single-image super-resolution reconstruction (SISR) method by introducing dense skip connections and Inception-ResNet in deep convolutional neural networks. The paper <NPL>, proposes a residual deep network, called CompNet, for the single image super resolution problem without an excessive increase in the network complexity. The idea behind the proposed network is to compose the residual signal that is more representative of the features produced by the different layers of the network and it is not as sparse.

Various examples are now described to introduce a selection of concepts in a simplified form, which are further described below in the detailed description. The above mentioned problem is solved by the subject matter of the independent claims. Further implementation forms are provided in the dependent claims.

It should be understood at the outset that although an illustrative implementation of one or more embodiments is provided below, the disclosed systems and methods described with respect to <FIG> may be implemented using any number of techniques, whether currently known or not yet in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims.

In the following description, reference is made to the accompanying drawings that form a part hereof, and in which are shown, by way of illustration, specific embodiments which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the inventive subject matter, and it is to be understood that other embodiments may be utilized, and that structural, logical, and electrical changes may be made without departing from the scope of the present claims. The following description of example embodiments is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

The present disclosure is related to video data processing on computing devices. Some aspects relate to changing the image resolution of images using a neural network. Other aspects relate to a real-time video ultra resolution.

As used herein, the terms "low-resolution" (or LR) and "high-resolution" (or HR) in connection with an image are associated with the size of the image (in pixels). For example, if two images depict the same scene and the first image has bigger height and width (in pixels) in comparison to the height and width of the second image, then the first image is referred to as a high-resolution image and the second image is referred to as a low-resolution image.

As used herein, the term "super-resolution" (or SR) refers to a resolution enhancement technique which increases the number of pixels (e.g., via upscaling) and improves the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) compared to conventional interpolation methods.

As used herein, the term "ultra-resolution" (or UR) includes the image resolution enhancement techniques of the SR (i.e., improving the resolution of video frames together with PSNR and SSIM), but also reduces the noise level, removes blocking artifacts commonly caused by video compression, and enhances local contrast (which functionalities are not present in SR).

As used herein, the terms "forward computation" and "backward computation" refer to computations performed at a worker machine in connection with the training of a neural network model (or another type of model). The computations performed during forward and backward computations modify weights based on results from prior iterations (e.g., based on gradients generated at a conclusion of a prior backward computation). A gradient is a measurement of how much output of a worker machine changes per change to the weights of the model that the worker machine is computing. A gradient measures a change in all weights with regard to the change in error. The larger the gradient value, the faster a model can learn.

As used herein, the term "bicubic upsampling" refers to image upsampling using bicubic interpolation. Bicubic interpolation is a technique for interpolating data points on a two-dimensional grid. In connection with image processing, bicubic interpolation considers <NUM> pixels (4x4 pixel matrix), with the interpolation being performed via Lagrange polynomials, cubic splines, or cubic convolution algorithms.

Techniques disclosed herein can be used to improve image resolution on mobile devices in real-time, to achieve ultra-resolution. Functionalities associated with UR are accomplished by a concise artificial neural network (ANN) using a neural network model that extracts and propagates residual image information through a limited number of convolutional layers. More specifically, a residual image is obtained by using the ANN (e.g., using an LR residual image corresponding to an LR input image), which is the difference between an HR image corresponding to the LR input image and a bicubic upsampled version of the LR input image. Processing the LR residual image using the neural network model results in reduced data flow through the convolutional layers, allowing the model to work with a very small number of parameters. The terms "neural network" (or NN) and "artificial neural network" (or ANN) are synonymous and are used interchangeably herein. The terms "convolutional neural network" (or CNN) and "deep neural network" (or DNN) are synonymous and refer to a type of neural network that includes multiple convolutional layers.

Techniques disclosed herein can also be used to train the neural network model with input-output image pairs created from a training image set. An example image in the training set is downsampled and degraded on the input side with simulated noises and blocking artifacts to enable the model to perform SR processing, noise reduction (NR), and artifact removal (AR) to achieve optimal UR processing within mobile devices. The local contrast of the HR image on the output side can be enhanced to boost the fine details discovered during SR processing. In this regard, the neural network model is trained to efficiently perform UR processing with significant visual perception improvements that can be achieved in real-time with acceptable energy consumption by the UR processing.

Prior art techniques perform SR processing using image de-convolution algorithms or example-based sparse coding algorithms. For example, some prior art techniques use a three-layer convolutional model to simulate the procedures of sparse coding for patch extraction, non-linear mapping, and reconstruction, respectively. Such techniques, however, are inefficient, because the input LR image needs to be upsampled to the high resolution before it is fed into the convolutional layers. Consequently, all convolution operations are conducted in the HR space, which contains a significant amount of redundant calculations. Other prior art techniques use an LR input image and perform convolutions in LR space, until the last step, when sub-pixels are combined into an HR image with convolution operations. Such techniques, however, are also slow when working on mobile devices.

A new trend in single image SR processing is using generative adversarial networks (GANs) to generate fine details that are missing in an image. However, the GANs depend on big network capacity to produce good results and thus are not suitable to be deployed on mobile devices. Other prior art SR algorithms designed for videos include optical flow evaluations between successive frames to compensate for motions across frames. Because optical flow evaluations are computation-intensive tasks that are not easier than processing tasks associated with using GANs, those video SR algorithms are also slow to work on mobile devices.

The following are distinctive features of the presently-disclosed techniques for improving image resolution and performing real-time UR processing on mobile devices (which features are not present in the prior art image processing techniques), including: (<NUM>) using ultra-resolution processing for mobile device applications, which enhances the overall visual perception besides the sole task of super resolution; (<NUM>) using a concise neural network model (with a reduced number of layers and parameters) that combines the functions of super resolution, noise reduction, blocking artifact removal, and local contrast enhancement, and can be deployed on mobile devices with real-time performance (a residual image, and not an entire SR image, is processed by the convolutional layers, saving significant processing resources and increasing UR processing speed); (<NUM>) using a fast pixel-shift operation at the output of the convolutional layers to replace a computationally intensive sub-pixel convolutions used in conventional techniques; and (<NUM>) using novel neural network model training techniques that degrade the image quality on the input side and at the same time enhance the image quality on the output side, contributing to more efficient model training and generating of output video frames of superior visual perception.

<FIG> is a block diagram <NUM> illustrating the training of deep learning (DL) model to generate a trained DL model <NUM> using a DL architecture (DLA), according to some example embodiments. In some example embodiments, machine-learning programs (MLPs), including deep learning programs, also collectively referred to as machine-learning algorithms or tools, are utilized to perform operations associated with correlating data or other artificial intelligence (Al)-based functions.

As illustrated in <FIG>, deep learning model training <NUM> is performed within the DLA <NUM> based on training data <NUM> (which can include features). During the deep learning model training <NUM>, features from the training data <NUM> can be assessed for purposes of further training of the DL model. The DL model training <NUM> results in a trained DL model <NUM>. The trained DL model <NUM> can include one or more classifiers <NUM> that can be used to provide assessments <NUM> based on new data <NUM>.

In some aspects, the training data <NUM> can include low-resolution (LR) input images <NUM> and corresponding high-resolution (HR) target output images <NUM>. The LR input images <NUM> and the HR target output images <NUM> are generated as discussed in connection with, e.g., <FIG>, and are used during the DL model training <NUM>, enabling the trained DL model <NUM> to perform the UR-related functionalities discussed herein. More specifically, the LR input images <NUM> and the HR target output images <NUM> are used to train convolutional layers of a neural network model (e.g., neural network model <NUM> of <FIG> which includes convolutional layers <NUM> of <FIG>) to perform real-time UR functions on LR images including LR video frames.

Deep learning is part of machine learning, a field of study that gives computers the ability to learn without being explicitly programmed. Machine learning explores the study and construction of algorithms, also referred to herein as tools, that may learn from existing data, may correlate data, and may make predictions about new data. Such machine learning tools operate by building a model from example training data (e.g., the training data <NUM>) in order to make data-driven predictions or decisions, expressed as outputs or assessments <NUM>. Although example embodiments are presented with respect to a few machine-learning tools (e.g., a deep learning architecture), the principles presented herein may be applied to other machine learning tools.

In some example embodiments, different machine learning tools may be used. For example, Logistic Regression, Naive-Bayes, Random Forest (RF), neural networks, matrix factorization, and Support Vector Machines (SVM) tools may be used during the deep learning model training <NUM> (e.g., for correlating the training data <NUM>).

Two common types of problems in machine learning are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values (for example, is this object an apple or an orange?). Regression algorithms aim at quantifying some items (for example, by providing a value that is a real number). In some embodiments, the DLA <NUM> can be configured to use machine learning algorithms that utilize the training data <NUM> to find correlations among identified features that affect the outcome.

The machine learning algorithms utilize features from the training data <NUM> for analyzing the new data <NUM> to generate the assessments <NUM>. The features include individual measurable properties of a phenomenon being observed and used for training the machine learning model. The concept of a feature is related to that of an explanatory variable used in statistical techniques such as linear regression. Choosing informative, discriminating, and independent features are important for the effective operation of the MLP in pattern recognition, classification, and regression. Features may be of different types, such as numeric features, strings, and graphs. In some aspects, training data can be of different types, with the features being numeric for use by a computing device.

In some aspects, the features used during the DL model training <NUM> can include one or more of the following: LR images (e.g., the LR input images <NUM>); HR images (e.g., the HR target output images <NUM>); sensor data from a plurality of sensors (e.g., audio, motion, image sensors); actuator event data from a plurality of actuators (e.g., wireless switches or other actuators); external information source from a plurality of external sources; timer data associated with the sensor state data (e.g., time sensor data is obtained), the actuator event data, or the external information source data; user communications information; user data; user behavior data, and so forth.

The machine learning algorithms utilize the training data <NUM> to find correlations among the identified features that affect the outcome of assessments <NUM>. In some example embodiments, the training data <NUM> includes labeled data, which is known data for one or more identified features and one or more outcomes. With the training data <NUM> (which can include identified features), the DL model is trained using the DL model training <NUM> within the DLA <NUM>. The result of the training is the trained DL model <NUM>. When the DL model <NUM> is used to perform an assessment, new data <NUM> is provided as an input to the trained DL model <NUM>, and the DL model <NUM> generates the assessments <NUM> as an output. For example, the DLA <NUM> can be deployed at a mobile device and the new data <NUM> can include LR images (e.g., frames from an LR video such as a real-time LR video feed). The DLA <NUM> performs UR functions (e.g., increasing image resolution while reducing noise, removing blocking artifacts, and boosting image contrast) on the LR images to generate HR output images in real time.

<FIG> is a diagram <NUM> illustrating the generation of a trained DL model <NUM> using a neural network model <NUM> trained within a DLA <NUM>, according to some example embodiments. Referring to <FIG>, source data <NUM> can be analyzed by a neural network model <NUM> (or another type of a machine learning algorithm or technique) to generate the trained DL model <NUM> (which can be the same as the trained DL model <NUM>). The source data <NUM> can include a training set of data, such as <NUM>, including data identified by one or more features. As used herein, the terms "neural network" and "neural network model" are interchangeable.

Machine learning techniques train models to accurately make predictions on data fed into the models (e.g., what was said by a user in a given utterance; whether a noun is a person, place, or thing; what the weather will be like tomorrow). During a learning phase, the models are developed against a training dataset of inputs to optimize the models to correctly predict the target output for a given input. Generally, the learning phase may be supervised, semi-supervised, or unsupervised; indicating a decreasing level to which the "correct" outputs are provided in correspondence to the training inputs. In a supervised learning phase, all of the target outputs are provided to the model and the model is directed to develop a general rule or algorithm that maps the input to the output. In contrast, in an unsupervised learning phase, the desired output is not provided for the inputs so that the model may develop its own rules to discover relationships within the training dataset. In a semi-supervised learning phase, an incompletely labeled training set is provided, with some of the outputs known and some unknown for the training dataset.

Models may be run against a training dataset for several epochs, in which the training dataset is repeatedly fed into the model to refine its results (i.e., the entire dataset is processed during an epoch). During an iteration, the model (e.g., a neural network model or another type of machine learning model) is run against a mini-batch (or a portion) of the entire dataset. In a supervised learning phase, a model is developed to predict the target output for a given set of inputs (e.g., source data <NUM>) and is evaluated over several epochs to more reliably provide the output that is specified as corresponding to the given input for the greatest number of inputs for the training dataset. In another example, for an unsupervised learning phase, a model is developed to cluster the dataset into n groups and is evaluated over several epochs as to how consistently it places a given input into a given group and how reliably it produces the n desired clusters across each epoch.

Once an epoch is run, the models are evaluated, and the values of their variables (e.g., weights, biases, or other parameters) are adjusted to attempt to better refine the model in an iterative fashion. As used herein, the term "weights" is used to refer to the parameters used by a machine learning model. During a backward computation, a model can output gradients, which can be used for updating weights associated with a forward computation.

In various aspects, the evaluations are biased against false negatives, biased against false positives, or evenly biased with respect to the overall accuracy of the model. The values may be adjusted in several ways depending on the machine learning technique used. For example, in a genetic or evolutionary algorithm, the values for the models that are most successful in predicting the desired outputs are used to develop values for models to use during the subsequent epoch, which may include random variation/mutation to provide additional data points. One of ordinary skill in the art will be familiar with several other machine learning algorithms that may be applied with the present disclosure, including linear regression, random forests, decision tree learning, neural networks, deep neural networks, etc..

Each model develops a rule or algorithm over several epochs by varying the values of one or more variables affecting the inputs to more closely map to the desired result, but as the training dataset may be varied, and is preferably very large, perfect accuracy and precision may not be achievable. A number of epochs that make up a learning phase, therefore, may be set as a given number of trials or a fixed time/computing budget, or may be terminated before that number/budget is reached when the accuracy of a given model is high enough or low enough or an accuracy plateau has been reached. For example, if the training phase is designed to run n epochs and produce a model with at least <NUM>% accuracy, and such a model is produced before the nth epoch, the learning phase may end early and use the produced model satisfying the end-goal accuracy threshold. Similarly, if a given model is inaccurate enough to satisfy a random chance threshold (e.g., the model is only <NUM>% accurate in determining true/false outputs for given inputs), the learning phase for that model may be terminated early, although other models in the learning phase may continue training. Similarly, when a given model continues to provide similar accuracy or vacillate in its results across multiple epochs - having reached a performance plateau - the learning phase for the given model may terminate before the epoch number/computing budget is reached.

Once the learning phase is complete, the models are finalized. In some example embodiments, models that are finalized are evaluated against testing criteria. In a first example, a testing dataset that includes known target outputs for its inputs is fed into the finalized models to determine an accuracy of the model in handling data that has not been trained on. In a second example, a false positive rate or false negative rate may be used to evaluate the models after finalization. In a third example, a delineation between data clusters in each model is used to select a model that produces the clearest bounds for its clusters of data.

In some example embodiments, the DL model <NUM> is trained by a neural network model <NUM> (e.g., deep learning, deep convolutional, or recurrent neural network), which comprises a series of "neurons," such as Long Short Term Memory (LSTM) nodes, arranged into a network. A neuron is an architectural element used in data processing and artificial intelligence, particularly machine learning, that includes memory that may determine when to "remember" and when to "forget" values held in that memory based on the weights of inputs provided to the given neuron. Each of the neurons used herein is configured to accept a predefined number of inputs from other neurons in the network to provide relational and sub-relational outputs for the content of the frames being analyzed. Individual neurons may be chained together or organized into tree structures in various configurations of neural networks to provide interactions and relationship learning modeling for how each of the frames in an utterance is related to one another.

For example, an LSTM serving as a neuron includes several gates to handle input vectors (e.g., phonemes from an utterance), a memory cell, and an output vector (e.g., contextual representation). The input gate and output gate control the information flowing into and out of the memory cell, respectively, whereas forget gates optionally remove information from the memory cell based on the inputs from linked cells earlier in the neural network. Weights and bias vectors for the various gates are adjusted over the course of a training phase, and once the training phase is complete, those weights and biases are finalized for normal operation. One of skill in the art will appreciate that neurons and neural networks may be constructed programmatically (e.g., via software instructions) or via specialized hardware linking each neuron to form the neural network.

Neural networks utilize features for analyzing the data to generate assessments (e.g., recognize units of speech). A feature is an individual measurable property of a phenomenon being observed. The concept of the feature is related to that of an explanatory variable used in statistical techniques such as linear regression. Further, deep features represent the output of nodes in hidden layers of the deep neural network.

A neural network (e.g., the neural network model <NUM>), sometimes referred to as an artificial neural network or a neural network model, is a computing system based on consideration of biological neural networks of animal brains. Such systems progressively improve performance, which is referred to as learning, to perform tasks, typically without task-specific programming. For example, in image recognition, a neural network may be taught to identify images that contain an object by analyzing example images that have been tagged with a name for the object and, having learned the object and name, may use the analytic results to identify the object in untagged images. A neural network is based on a collection of connected units called neurons, where each connection between neurons, called a synapse, can transmit a unidirectional signal with an activating strength that varies with the strength of the connection. The receiving neuron can activate and propagate a signal to downstream neurons connected to it, typically based on whether the combined incoming signals, which are from potentially many transmitting neurons, are of sufficient strength, where strength is a parameter.

A DNN, also referred to as a CNN, is a stacked neural network, which is composed of multiple convolutional layers. The layers are composed of nodes, which are locations where computation occurs, loosely patterned on a neuron in the human brain, which fires when it encounters sufficient stimuli. A node combines input from the data with a set of coefficients, or weights, that either amplify or dampen that input, which assigns significance to inputs for the task the algorithm is trying to learn. These input-weight products are summed, and the sum is passed through what is called a node's activation function, to determine whether and to what extent that signal progresses further through the network to affect the ultimate outcome. A DNN uses a cascade of many layers of non-linear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Higher-level features are derived from lower-level features to form a hierarchical representation. The layers following the input layer may be convolution layers that produce feature maps that are filtering results of the inputs and are used by the next convolution layer.

In training of a DNN architecture, a regression, which is structured as a set of statistical processes for estimating the relationships among variables, can include minimization of a cost function. The cost function may be implemented as a function to return a number representing how well the neural network performed in mapping training examples to correct output. In training, if the cost function value is not within a predetermined range, based on the known training images, backpropagation is used, where backpropagation is a common method of training artificial neural networks that are used with an optimization method such as stochastic gradient descent (SGD) method.

Use of backpropagation can include propagation and weight updates. When an input is presented to the neural network, it is propagated forward through the neural network, layer by layer, until it reaches the output layer. The output of the neural network is then compared to the desired target output, using the cost function, and an error value is calculated for each of the nodes in the output layer. The error values are propagated backward, starting from the output, until each node has an associated error value that roughly represents its contribution to the original output. Backpropagation can use these error values to calculate the gradient of the cost function with respect to the weights in the neural network. The calculated gradient is fed to the selected optimization method to update the weights to attempt to minimize the cost function.

Even though the training architecture <NUM> is referred to as a deep learning architecture using a neural network model (and the model that is trained is referred to as a trained deep learning model, such as the trained DL models <NUM> and <NUM>), the disclosure is not limited in this regard and other types of machine learning training architectures may also be used for model training, using the techniques disclosed herein.

<FIG> is a diagram illustrating a system <NUM> for adjusting image resolution using the DLA of <FIG>, according to some example embodiments. Referring to <FIG>, the system <NUM> includes convolutional layers <NUM>, a residue generation module <NUM>, a bicubic upsampling module <NUM>, a pixel shifting module <NUM>, and an adder <NUM>. The convolutional layers <NUM> can be configured as a neural network model, such as the neural network model <NUM> of <FIG>.

The residue generation module <NUM> may comprise suitable circuitry, logic, interfaces, or code and is configured to convert the input LR image <NUM> from red-green-blue (RGB) color space to a Luma (Y), Chroma Blue Difference (Cb), and Chroma Red Difference (Cr) (or YCbCr) color space and obtain the Y-channel (brightness) LR residual image <NUM> (i.e., a grayscale image indicative of brightness) that corresponds to the input LR image <NUM>.

The bicubic upsampling module <NUM> may comprise suitable circuitry, logic, interfaces, or code and is configured to perform bicubic upsampling on the input LR image <NUM> to generate a base HR image <NUM>.

The convolutional layers <NUM> can be part of a neural network model (e.g., the neural network model <NUM>) and are configured to generate a plurality of HR residual sub-images <NUM> corresponding to an input LR image <NUM> based on an LR residual image (e.g., a grayscale version of the LR input image such as LR residual image <NUM>). Further details of the convolutional layers <NUM> are discussed hereinbelow in connection with <FIG>.

Pixel shifting module <NUM> may comprise suitable circuitry, logic, interfaces, or code and is configured to perform pixel shifting on the plurality of HR residual sub-images <NUM> to generate an HR residual image <NUM>.

In operation, a low-resolution (LR) input image <NUM> is processed (e.g., by the bicubic upsampling module <NUM> and the residue generation module <NUM>) to generate a base HR image <NUM> and an LR residual image <NUM> corresponding to the input LR image <NUM>. The convolutional layers <NUM> use multiple layers to involve the LR residual image <NUM> and generate a plurality of HR residual sub-images <NUM> corresponding to the input LR image <NUM>. In some aspects, the plurality of HR residual sub-images <NUM> includes four sub-images. The pixel shifting module <NUM> performs pixel shifting on the plurality of HR residual sub-images <NUM> to generate an HR residual image <NUM>. The adder <NUM> may comprise suitable circuitry, logic, interfaces, or code and is configured to add the base HR image <NUM> with the HR residual image <NUM> to generate an HR image <NUM> as an output image corresponding to the input LR image <NUM>.

By splitting the generation of the HR image <NUM> into generating the base HR image <NUM> and the HR residual image <NUM>, the amount of data that flows through the convolutional layers <NUM> is reduced, which increases the effective capacity of the model and its inference efficiency for reconstructing an HR image from an input LR image.

In some aspects, one or more functionalities performed by the bicubic upsampling module <NUM>, the residue generation module <NUM>, the pixel shifting module <NUM>, and the adder <NUM> can be performed by the convolutional layers <NUM>.

<FIG> is a diagram illustrating the configuration of convolutional layers <NUM> within the DLA as used by the system <NUM> of <FIG>, according to some example embodiments. Referring to <FIG>, the convolutional layers <NUM> include four layers, namely, convolutional layers <NUM>, <NUM>, <NUM>, and <NUM>, with layer depth (or a number of channels) <NUM> as indicated in <FIG>. The first convolutional layer <NUM> is configured to generate <NUM> channels of measurements (i.e., the layer has a depth of <NUM>), the second convolutional layer <NUM> is configured to generate <NUM> channels of measurements (i.e., the layer has a depth of <NUM>), and the third and fourth convolutional layers (<NUM> and <NUM> respectively) are each configured to generate <NUM> channels of measurements (i.e., the layers have a depth of <NUM>). Convolutional layers <NUM>-<NUM> are configured to generate their channel measurements using convolution kernels <NUM>, <NUM>, and <NUM>, respectively, of size 3x3 pixels.

The convolutional layers <NUM> can also include a one-channel input layer <NUM>, which corresponds to the image received as input to convolutional layers <NUM> - <NUM>. For example, the input layer <NUM> can be representative of the LR residual image <NUM> communicated as input to the convolutional layers <NUM> - <NUM>. The input layer <NUM> can include a kernel <NUM> (e.g., a 5x5 pixel kernel), which is used for communicating image data as input into the first convolutional layer <NUM>.

In operation, each of the convolutional layers <NUM> - <NUM> convolves the input received from a previous layer using multiple convolution kernels. For example, after the first convolutional layer <NUM> receives input data from the input layer <NUM> via the kernel <NUM>, the first convolutional layer <NUM> generates eight channels of measurements using the input from input layer <NUM> and eight different convolution kernels of size <NUM> x <NUM> pixels (such as convolutional kernel <NUM>). The measurements of each channel are communicated to the subsequent layer for additional convolutions. The fourth convolutional layer <NUM> outputs the plurality of HR residual sub-images <NUM> (e.g., each of the four channels of layer <NUM> outputs one HR residual sub-image for a total of four HR residual sub-images <NUM>).

Even though <FIG> illustrates convolutional layers <NUM> including four separate convolutional layers <NUM> - <NUM> with the indicated layer depth <NUM>, the disclosure is not limited in this regard and a different configuration of the convolutional layers <NUM> can also be used within the system <NUM> for performing UR-related functions.

<FIG> is a diagram <NUM> illustrating generation of training image pairs, which can be used for training of the DLA of <FIG>, according to some example embodiments. Referring to <FIG>, each training image pair includes a convolution layers input image <NUM> and a set of convolution layers target output images <NUM>. Both the convolutional layers input image <NUM> and the convolution layers target output images <NUM> are inputted into the convolutional layers <NUM> during the training phase, and the convolution layers target output images <NUM> do not comprise an actual output during the training phase. It should be understood that during an actual convolving operation, images to be processed are inputted into the convolutional layers <NUM> at <NUM> and the convolutional layers <NUM> will output a processed UR image(s) at <NUM>.

To generate the convolutional layers input image <NUM>, a low-pass filter (LPF) <NUM> is applied on an example training image I <NUM> to generate a filtered image <NUM>. The filtered image <NUM> is then downsampled (e.g., using downsampling (DS) module <NUM>) to generate a downsampled LR image ILR <NUM>. In some aspects, the filtered image <NUM> can be downsampled by a factor of <NUM>, represented by the following equation: ILR = I * N(<NUM>, σblur) ↓<NUM>, where * denotes a convolution operation, and N(<NUM>, σblur)denotes a Gaussian function with mean <NUM> and standard deviation of σblur.

In some aspects, the LPF <NUM> is used to remove the highfrequency signal and thereby avoid under-sample artifacts. In some aspects, downsampling by the DS module <NUM> is done by bicubic interpolation. The downsampled (e.g., half-sized) LR image <NUM> is then degraded by noise and blocking artifacts before it is fed it into the convolutional layers. More specifically, the noise addition module <NUM> is used to introduce noise into the downsampled LR image <NUM>. In some aspects, two kinds of noise are simulated, which are (<NUM>) photon noise and (<NUM>) Gaussian noise.

The photon noise is used to simulate the discrete characteristic of light and is simulated with a Poisson stochastic process, as follows: Ipho~ P(ILR · photons)/photons, where P(λ) denotes Poisson distribution with mean λ, and photons denotes the number of photons that produces the brightest color that the image can represent.

The Gaussian noise is used to simulate the stochastic noise from image sensors. It is calculated by adding a random image Igauss generated with a Gaussian stochastic process to the input image, as follows: Inoisy = Ipho + Igauss, with Igauss~N(<NUM>, σn), where σn stands for the targeted noise level.

Video compression and decompression usually cause blocking artifacts in the video frames. The artifacts are directly simulated by video compression and decompression. Training a neural network model (e.g., <NUM>) using convolutional layers <NUM> with degraded image quality on the input side will offer the neural network model noise reduction and artifact removal ability.

The noise addition module <NUM> introduces noise into the downsampled LR image <NUM> to generate a noisy LR image <NUM>. The noisy LR image <NUM> is further degraded by the artifacts addition module <NUM>, which introduces artifacts and generates an LR image <NUM> corresponding to the training image <NUM>. The LR image <NUM> is characterized by degraded image quality and is configured as a convolutional layers input image <NUM> into the convolutional layers <NUM>, for the training phase. The convolutional layers input image <NUM> will be used to train the convolutional layers <NUM> to generate the desired target output. Here, the desired output of the convolutional layers <NUM> is inputted as the convolutional layers target output images <NUM> during the training phase of the convolutional layers.

To generate the set of convolutional layers target output images <NUM> for the convolutional network, the training image I <NUM> is enhanced by the local contrast enhancement (LCE) module <NUM> that generates a contrast-enhanced image <NUM>. This processing contributes to teaching the convolutional layers <NUM> to produce images with high local contrast, which boosts the fine details recovered by super resolution. In some aspects, the LCE module <NUM> can perform the local contrast enhancement by applying an unbalanced unsharp mask, as follows: Ilc = I + min(I - I * N(<NUM>, σum), δmαx). where the difference between the training image and a Gaussian blurred version of it is truncated in the positive part with an upper bound δmαx (the truncation is beneficial in reducing halo artifacts that appear due to human eyes being sensitive to bright edges). The HR residual image <NUM> is then calculated by subtracting the upsampled LR image <NUM> of the downsampled LR image <NUM> from the contrast enhanced image <NUM> via the subtraction module <NUM>. The upsampled LR image <NUM> is generated by upsampling the downsampled LR image <NUM> by the upsampling module <NUM>. A pixel splitting operation <NUM> is applied to the HR residual image <NUM> so that a plurality of HR residual sub-images <NUM> are generated from the HR residual image <NUM> (e.g., by splitting 4x4 pixel blocks into the four sub-images). The plurality of HR residual sub-images <NUM> is configured as the convolutional layers target output images <NUM>, to be inputted into the convolutional layers <NUM> during the training phase.

In some aspects, multiple training pairs are generated before training the convolutional layers <NUM>, and the training pairs are saved in a data file (e.g., using a tfrecord format). During neural network model training (e.g., <NUM>), the training code can extract the training data and use such data to adjust the model parameters (e.g., of convolutional layers <NUM>) for performing UR processing. This speeds up the model training by avoiding complicated image processing on the fly and allows the model to be trained repeatedly with different parameters associated with the training data.

By using the neural network model training techniques discussed herein (e.g., degrading the image quality on the input side and enhancing the image quality on the output side), the convolutional layers <NUM> can generate images or video frames of superior visual perception during UR processing of LR images or video frames. Additionally, the use of a concise neural network model (e.g., convolutional layers <NUM> with a limited number of layers, such as four) that combines the functions of super-resolution, noise reduction, removal of blocking artifacts, and local contrast enhancement, the concise neural network model can be deployed on mobile devices (or other types of limited-resource devices, such as smart TVs, tablets, laptops, and other computing devices) for performing real-time UR processing.

In operational use, actual images or video frames to be processed will comprise input <NUM> to the convolutional layers <NUM>. The convolutional layers <NUM> will, as a result of the training, process the images or video frames to generate UR images or video frames at the output <NUM>. The outputted UR images or video frames can then be displayed, transmitted, stored, or otherwise used.

<FIG> is a flowchart of a method <NUM> of generation of training image pairs and using the pairs for DLA training and performing image resolution adjustment functionalities, according to some example embodiments. The method <NUM> includes operations <NUM> - <NUM>. By way of example and not limitation, the method <NUM> is described as being performed by the ultra-resolution management module <NUM>, which can be configured to execute within a mobile device such as device <NUM> illustrated in <FIG>.

Referring to <FIG>, the generation of a training image pair can start at operation <NUM> when a training image (e.g., <NUM>) is received. At operation <NUM>, the training image is filtered using a low-pass filter (e.g., <NUM>) to generate a filtered image (e.g., <NUM>). At operation <NUM>, the filtered image is downsampled (e.g., by the downsampling module <NUM>) to generate a downsampled LR image (e.g., <NUM>). At operation <NUM>, the image quality of the downsampled LR image is degraded by adding noise (e.g., by the noise addition module <NUM>) and artifacts (e.g., by the artifacts addition module <NUM>) to generate a low-resolution image corresponding to the training image and having degraded image quality (e.g., LR image <NUM>). At operation <NUM>, the LR image corresponding to the training image is configured as an input to the neural network model.

To generate the output image within the training pair, at operation <NUM>, an unbalanced unsharp mask is applied to the training image to generate a contrast-enhanced image (e.g., LCE module <NUM> generates the contrast-enhanced image <NUM>). At operation <NUM>, an upsampled version of the downsampled LR image is subtracted from the contrast-enhanced image to generate an HR residual image corresponding to the training image (e.g., HR residual image <NUM> is generated by the subtraction module <NUM> using the upsampled LR image <NUM> and the contrast-enhanced image <NUM>). At operation <NUM>, the HR residual image (corresponding to the training image) is split to generate a plurality of HR residual sub-images corresponding to the training image (e.g., a plurality of HR residual sub-images <NUM> are generated via a pixel splitting operation <NUM> performed on the HR residual image <NUM>). At operation <NUM>, the plurality of HR residual sub-images corresponding to the training image is configured as an output of the neural network model (e.g., the plurality of HR residual sub-images <NUM> is configured as a set of convolutional layer target output images <NUM> for training the convolutional layers <NUM>).

<FIG> is a flowchart of a method <NUM> for increasing image resolution of a digital image, according to some example embodiments. The method <NUM> includes operations <NUM>, <NUM>, <NUM>, and <NUM>. By way of example and not limitation, the method <NUM> is described as being performed by the ultra-resolution management module <NUM>, which can be configured to execute within a mobile device such as device <NUM> illustrated in <FIG>.

Referring to <FIG>, at operation <NUM>, bicubic upsampling is performed on a digital image to generate a base high-resolution (HR) image. For example, and in reference to <FIG>, a bicubic upsampling module generates the base HR image using the input LR image. At operation <NUM>, the digital image is converted from a red-green-blue (RGB) color space to a Luma (Y), Chroma Blue Difference (Cb), and Chroma Red Difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image. In some aspects, the RGB-to-YCrCb conversion can be performed by applying a plurality of convolutional layers of a neural network to the digital image (or by using a residue generation module) to generate the LR residual image.

At operation <NUM>, the LR residual image is converted (e.g., by using the plurality of convolutional layers) into a plurality of HR residual sub-images corresponding to the input LR image. For example, the convolutional layers use the LR residual image to generate a plurality of HR residual sub-images corresponding to the input LR image. At operation <NUM>, an HR image corresponding to the input LR image is generated using the base HR image and the plurality of HR residual sub-images. For example, the HR residual image is generated by pixel-shifting the plurality of HR residual sub-images. The HR image is generated using the base HR image and the HR residual image.

<FIG> is a block diagram illustrating a representative software architecture <NUM>, which may be used in conjunction with various device hardware described herein, according to some example embodiments. <FIG> is merely a non-limiting example of a software architecture <NUM> and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture <NUM> executes on hardware, such as device <NUM> of <FIG> that includes, among other things, processor <NUM>, memory <NUM>, storage <NUM> and/or <NUM>, and I/O interfaces <NUM> and <NUM>.

A representative hardware layer <NUM> is illustrated and can represent, for example, the device <NUM> of <FIG>. The representative hardware layer <NUM> comprises one or more processing units <NUM> having associated executable instructions <NUM>. Executable instructions <NUM> represent the executable instructions of the software architecture <NUM>, including implementation of the methods, modules and so forth of FIGS. <NUM>-<NUM>. Hardware layer <NUM> also includes memory or storage modules <NUM>, which also have executable instructions <NUM>. Hardware layer <NUM> may also comprise other hardware <NUM>, which represents any other hardware of the hardware layer <NUM>, such as the other hardware illustrated as part of device <NUM>.

In the example architecture of <FIG>, the software architecture <NUM> may be conceptualized as a stack of layers where each layer provides particular functionality. For example, the software architecture <NUM> may include layers such as an operating system <NUM>, libraries <NUM>, frameworks/middleware <NUM>, applications <NUM>, and presentation layer <NUM>. Operationally, the applications <NUM> or other components within the layers may invoke application programming interface (API) calls <NUM> through the software stack and receive a response, returned values, and so forth illustrated as messages <NUM> in response to the API calls <NUM>. The layers illustrated in <FIG> are representative in nature and not all software architectures <NUM> have all layers. For example, some mobile or special purpose operating systems may not provide frameworks/middleware <NUM>, while others may provide such a layer. Other software architectures may include additional or different layers.

The operating system <NUM> may manage hardware resources and provide common services. The operating system <NUM> may include, for example, a kernel <NUM>, services <NUM>, and drivers <NUM>. The kernel <NUM> may act as an abstraction layer between the hardware and the other software layers. For example, kernel <NUM> may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and so on. The drivers <NUM> may be responsible for controlling or interfacing with the underlying hardware. For instance, the drivers <NUM> may include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth, depending on the hardware configuration.

The libraries <NUM> may provide a common infrastructure that may be utilized by the applications <NUM> or other components or layers. The libraries <NUM> typically provide functionality that allows other software modules to perform tasks in an easier fashion than to interface directly with the underlying operating system <NUM> functionality (e.g., kernel <NUM>, services <NUM>, or drivers <NUM>). The libraries <NUM> may include system libraries <NUM> (e.g., C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, the libraries <NUM> may include API libraries <NUM> such as media libraries (e.g., libraries to support presentation and manipulation of various media format such as MPEG4, H. <NUM>, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., an OpenGL framework that may be used to render 2D and 3D in a graphic content on a display), database libraries (e.g., SQLite that may provide various relational database functions), web libraries (e.g., WebKit that may provide web browsing functionality), and the like. The libraries <NUM> may also include a wide variety of other libraries <NUM> to provide many other APIs to the applications <NUM> and other software components/modules.

The frameworks/middleware <NUM> (also sometimes referred to as middleware) may provide a higher-level common infrastructure that may be utilized by the applications <NUM> or other software components/modules. For example, the frameworks/middleware <NUM> may provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks/middleware <NUM> may provide a broad spectrum of other APIs that may be utilized by the applications <NUM> or other software components/modules, some of which may be specific to a particular operating system <NUM> or platform.

The applications <NUM> include built-in applications <NUM>, third-party applications <NUM>, and an ultra-resolution management module (URMM) <NUM>. In some aspects, the URMM <NUM> may comprise suitable circuitry, logic, interfaces or code and can be configured to perform one or more of the UR-related functions discussed in connection with <FIG>.

Examples of representative built-in applications <NUM> may include but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, or a game application. Third-party applications <NUM> may include any of the built-in applications <NUM> as well as a broad assortment of other applications. In a specific example, the third-party application <NUM> (e.g., an application developed using the Android™ or iOS™ software development kit (SDK) by an entity other than the vendor of the particular platform) may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or other mobile operating systems. In this example, the third-party application <NUM> may invoke the API calls <NUM> provided by the mobile operating system such as operating system <NUM> to facilitate functionality described herein.

The applications <NUM> may utilize built-in operating system functions (e.g., kernel <NUM>, services <NUM>, and drivers <NUM>), libraries (e.g., system libraries <NUM>, API libraries <NUM>, and other libraries <NUM>), and frameworks/middleware <NUM> to create user interfaces to interact with users of the system. Alternatively, or additionally, in some systems, interactions with a user may occur through a presentation layer, such as presentation layer <NUM>. In these systems, the application/module "logic" can be separated from the aspects of the application/module that interact with a user.

Some software architectures utilize virtual machines. In the example of <FIG>, this is illustrated by virtual machine <NUM>. A virtual machine creates a software environment where applications/modules can execute as if they were executing on a hardware machine (such as the device <NUM> of <FIG>, for example). A virtual machine <NUM> is hosted by a host operating system (e.g., operating system <NUM>) and typically, although not always, has a virtual machine monitor <NUM>, which manages the operation of the virtual machine <NUM> as well as the interface with the host operating system (i.e., operating system <NUM>). A software architecture <NUM> executes within the virtual machine <NUM> such as an operating system <NUM>, libraries <NUM>, frameworks/middleware <NUM>, applications <NUM>, or presentation layer <NUM>. These layers of software architecture executing within the virtual machine <NUM> can be the same as corresponding layers previously described or may be different.

<FIG> is a block diagram illustrating circuitry for a device that implements algorithms and performs methods, according to some example embodiments. All components need not be used in various embodiments. For example, clients, servers, and cloud-based network devices may each use a different set of components, or in the case of servers, for example, larger storage devices.

One example computing device in the form of a computer <NUM> (also referred to as computing device <NUM>, computer system <NUM>, or computer <NUM>) may include a processor <NUM>, memory <NUM>, removable storage <NUM>, non-removable storage <NUM>, input interface <NUM>, output interface <NUM>, and communication interface <NUM>, all connected by a bus <NUM>. Although the example computing device is illustrated and described as the computer <NUM>, the computing device may be in different forms in different embodiments.

The memory <NUM> may include volatile memory <NUM> and non-volatile memory <NUM> and may store a program <NUM>. The computing device <NUM> may include - or have access to a computing environment that includes - a variety of computer-readable media, such as the volatile memory <NUM>, the non-volatile memory <NUM>, the removable storage <NUM>, and the non-removable storage <NUM>. Computer storage includes random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

Computer-readable instructions stored on a computer-readable medium (e.g., the program <NUM> stored in the memory <NUM>) are executable by the processor <NUM> of the computing device <NUM>. A hard drive, CD-ROM, and RAM are some examples of articles including a non-transitory computer-readable medium such as a storage device. The terms "computer-readable medium" and "storage device" do not include carrier waves to the extent that carrier waves are deemed too transitory. "Computer-readable non-transitory media" includes all types of computer-readable media, including magnetic storage media, optical storage media, flash media, and solid-state storage media. It should be understood that software can be installed in and sold with a computer. Alternatively, the software can be obtained and loaded into the computer, including obtaining the software through a physical medium or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example. As used herein, the terms "computer-readable medium" and "machine-readable medium" are interchangeable.

The program <NUM> may utilize a customer preference structure using modules discussed herein, such as the URMM <NUM>, which may be the same as the URMM <NUM> of <FIG>.

Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine, an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any suitable combination thereof). Moreover, any two or more of these modules may be combined into a single module, and the functions described herein for a single module may be subdivided among multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

In some aspects, the URMM <NUM> as well as one or more other modules that are part of the program <NUM>, can be integrated as a single module, performing the corresponding functions of the integrated modules.

Although a few embodiments have been described in detail above, other modifications are possible within the scope of the following claims.

It should be further understood that software including one or more computer-executable instructions that facilitate processing and operations as described above with reference to any one or all of steps of the disclosure can be installed in and sold with one or more computing devices consistent with the disclosure. Alternatively, the software can be obtained and loaded into one or more computing devices, including obtaining the software through physical medium or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example.

Also, it will be understood by one skilled in the art that this disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the description or illustrated in the drawings, but are limited only by the scope of the appended claims.

The components of the illustrative devices, systems, and methods employed in accordance with the illustrated embodiments can be implemented, at least in part, in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. These components can be implemented, for example, as a computer program product such as a computer program, program code or computer instructions tangibly embodied in an information carrier, or in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus such as a programmable processor, a computer, or multiple computers.

A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other units suitable for use in a computing environment. Also, functional programs, codes, and code segments for accomplishing the techniques described herein can be easily construed as within the scope of the claims by programmers skilled in the art to which the techniques described herein pertain. Method steps associated with the illustrative embodiments can be performed by one or more programmable processors executing a computer program, code or instructions to perform functions (e.g., by operating on input data or generating an output). Method steps can also be performed by, and apparatus for performing the methods can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit), for example.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an ASIC, a FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.

Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The required elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example, semiconductor memory devices, e.g., electrically programmable read-only memory or ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, or data storage disks (e.g., magnetic disks, internal hard disks, or removable disks, magneto-optical disks, or CD-ROM and DVD-ROM disks). The processor and the memory can be supplemented by or incorporated in special purpose logic circuitry.

Those of skill in the art understand that information and signals may be represented using any of a variety of different technologies and techniques.

As used herein, "machine-readable medium" (or "computer-readable medium") comprises a device able to store instructions and data temporarily or permanently and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., Erasable Programmable Read-Only Memory (EEPROM)), or any suitable combination thereof. The term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store processor instructions. The term "machine-readable medium" shall also be taken to include any medium or a combination of multiple media, that is capable of storing instructions for execution by one or more processors <NUM>, such that the instructions, when executed by one or more processors <NUM>, cause the one or more processors <NUM> to perform any one or more of the methodologies described herein. Accordingly, a "machine-readable medium" refers to a single storage apparatus or device, as well as "cloud-based" storage systems or storage networks that include multiple storage apparatus or devices. The term "machine-readable medium" as used herein excludes signals per se.

In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the scope of the appended claims.

Claim 1:
A computer-implemented method for increasing image resolution of a digital image performed by one or more processors in communication with a memory, wherein the one or more processors and the memory are part of a system, wherein the method comprises the steps of:
• performing (step <NUM>) bicubic upsampling of the digital image to generate a base high-resolution, HR, image;
• converting (step <NUM>) the digital image from a red-green-blue, RGB, color space to a Luma, Y, Chroma Blue Difference, Cb, and Chroma Red Difference, Cr, YCbCr color space to generate a low-resolution, LR, residual image;
• converting (step <NUM>), using a plurality of convolutional layers of a neural network model, the LR residual image into a plurality of HR residual sub-images corresponding to the digital image,
o wherein the neural network model comprises an input layer, and the plurality of convolutional layers comprises four convolutional layers, wherein the input layer is configured to receive the digital image, and an output layer of the four convolutional layers is configured to output the plurality of HR residual sub-images, wherein:
▪ a first layer of the plurality of convolutional layers is configured with <NUM> x <NUM> pixel kernels and <NUM> channels;
▪ a second layer of the plurality of convolutional layers is configured with <NUM> x <NUM> pixel kernels and <NUM> channels;
▪ a third layer of the plurality of convolutional layers is configured with <NUM> x <NUM> pixel kernels and <NUM> channels; and
▪ a fourth layer of the plurality of convolutional layers is configured with <NUM> channels; and
• pixel shifting the plurality of HR residual sub-images to generate an HR residual image; and
• generating (step <NUM>) an HR image corresponding to the digital image, using the base HR image and the plurality of HR residual sub-images, wherein the generating of the HR image corresponding to the digital image comprises combining the HR residual image and the base HR image.