SUPER RESOLUTION USING CONVOLUTIONAL NEURAL NETWORK

An apparatus for super resolution imaging includes a convolutional neural network (104) to receive a low resolution frame (102) and generate a high resolution illuminance component frame. The apparatus also includes a hardware scaler (106) to receive the low resolution frame (102) and generate a second high resolution chrominance component frame. The apparatus further includes a combiner (108) to combine the high resolution illuminance component frame and the high resolution chrominance component frame to generate a high resolution frame (110).

BACKGROUND

Super-resolution imaging (SR) is a class of techniques that increase the resolution of images processed by an imaging system. For example, low resolution images may be converted into high resolution images with improved details using various SR techniques.

The same numbers are used throughout the disclosure and the figures to reference like components and features. Numbers in the100series refer to features originally found inFIG.1; numbers in the200series refer to features originally found inFIG.2; and so on.

DESCRIPTION OF THE EMBODIMENTS

Deep learning based super resolution may be used in restoring low resolution images and video frames to high resolution images and video frames. Currently, deep learning based methods may conduct training processes based on low and high resolution image pairs obtained by certain downsampling techniques. For example, a conventional super resolution technique using low resolution images downscaled with a bicubic filter may be used. For example, a conventional super resolution technique may use low resolution images downscaled by the bicubic filter. Some blind super resolution systems may further improve this downscaling process by combining bicubic filter with Gaussian smoothing using multiple kernels. This kind of training process may work for nature content. However, in screen or gaming content, severe overshoot and undershoot artifacts may be observed after the upscaling of sharp edges. As used herein, overshooting artifact are artifacts that appear as spurious bands or “ghosts” near edges. Overshooting artifacts may also be referred to as ringing artifacts. Nature content is video containing camera-captured video scenes. For example, nature content may contain fewer sharp edges. Screen content is video containing a significant portion of rendered graphics (excluding games), text, or animation rather than camera-captured video scenes. Gaming content is a significant portion of rendered game.

For deep learning based super resolution, two approaches are sometimes used to achieve higher quality output. For example, deep convolution networks may be used as a post-processing model of a traditional scaler to enhance details of the images and video resized by conventional methods such as bilinear, bicubic, Lanczos filters, etc. However, this may introduce a large computation workload to an inference device, especially when the input resolution of the images or videos is high. Another way to achieve higher quality output is to directly take a low resolution image or video frame as input, and then utilize a convolutional network to restore the details of high resolution images. For example, the convolutional network can be used to apply a series of neural network layers first to the low-resolution video frames to exact import feature maps used to restore high resolution details. After that, a dedicated neural network layer may upscale the low-resolution feature maps to a high-resolution output. In this way, part of a workload can be shifted to low resolution features. Shifting the workload in this manner may reduce the computation and bandwidth overhead compared with the previous way, as most of the compute may be conducted on the low-resolution instead of high-resolution.

Downsampling the ground truth high resolution training image to obtain a low resolution image is a straight forward and easy way to get training pairs for a neural network that may work for most nature content. However, for screen or gaming content, which may contain an extremely high frequency in the frequency domain, the high frequency information may be corrupted after the downsampling process. For example, a frame may be first transferred to the frequency domain by using certain kind of transformation. The transformation may be a discrete cosine transform, or a discrete Fourier transform. The main purpose of such transformation may be to use a linear combination of different bases to represent the image. The bases defined by each transform may contains various signals with different frequencies ranging from a very low frequency to a very high frequency. For sharp edges in the spatial or image domain, in order to represent this signal in the frequency domain, many high frequency bases may be used. Thus, sharp edges may usually contain much higher frequency components than the others. Moreover, downsampling using interpolation, such as via bilinear, bicubic, Lanczos, or other filters, may tend to corrupt such high frequency components. The neural network may never be able to learn how to process such high frequency input. Thus, when applied to real screen content cases, which in contrast to a training process may not suffer from any frequency corruption, artifacts may occur because that high frequency information is emphasized in an improper way.

In some examples, after a data augmentation tuning process, overshooting artifacts may almost be removed. However, the final high-resolution output may become blurry when compared with the results without using data augmentation, which may also cause some quality drop on other texture contents. The output becomes blurry compared with the result before tuning. Such overshooting issue may happen along black lines, and may be caused by using a rectified linear unit (ReLU) activation. Moreover, images or videos with repeated patterns may also display aliasing artifacts.

The present disclosure relates generally to techniques for super resolution using scalable neural networks. For example, the techniques include training methods and an example inference topology. First, in a data preparation stage, instead of traditional interpolation based downsampling process such as bilinear or bicubic downsampling, a nearest neighbor downsampling may be used for screen content for additional data augmentation. In the training stage, in addition to using an L1/L2 loss function, a self-similarity loss is used as part of the loss function to deal with aliasing artifacts. For the inference topology, the techniques also include a small scale network based on an enhanced deep super-resolution (EDSR) and replacing a ReLU activation with a parametric rectified linear unit (PReLU) activation to improve robustness of the network.

The techniques described herein thus enable elimination of overshoot, undershoot and aliasing problems in screen content without affecting the sharpness in restored image or video. The designed network can help users enable real time high quality super resolution with input videos of any resolution, such as with a resolutions of 1280×720 (720p), 1920×1080 (1080p), 2560×1440 (1440p), or more. For example, by only processing an illuminance channel via a convolutional neural network and using a hardware upscaler to process chrominance channels, the techniques may efficiently process video frames using less computational resources. In addition, the techniques described herein can eliminate artifacts in screen and gaming content with almost no side effects on the appearance of nature content. Thus, the techniques herein may be used to enhance the quality of images and video frames for nature, screen and gaming content.

FIG.1is a block diagram illustrating an example system for super resolution using a scalable neural network. The example system100can be implemented in the computing device1000inFIG.10using the methods600-900ofFIGS.6-9. For example, the system100can be trained using the methods600and700ofFIGS.6and7and executed using the methods800and900ofFIGS.8and9.

The example system100includes a low resolution frames102. The system100includes a convolutional neural network (CNN)104communicatively coupled to a source of the low resolution frames102. The system100further includes a hardware scaler106communicatively coupled to the source of the low resolution frames102. The system100also further includes a combiner108communicatively coupled to the convolutional neural network (CNN)104and the hardware scaler106.

The system100ofFIG.1illustrates an inference framework that directly takes low resolution video frames as input and utilizes a convolution neural network104to restore the details in an output high resolution frame110. In particular, the low resolution frame102may be fed into a CNN104and a hardware scaler106. In some examples, the low resolution frame102may in a YUV420format, where the size of the UV channels may be one fourth the size of the illuminance channel Y. The YUV format encodes a color image or video taking human perception into account, allowing reduced bandwidth for chrominance components. In some examples, a color conversion from RGB to YUV may be applied.

In various examples, the hardware scaler106may be an upsampler using a particular scaling factor. In some examples, the scaling factor is determined by the different sampling rates of high resolution and low resolution pairs of frames. For example, to convert 360p to 720p, the scaling factor is 2×. For example, the hardware scaler106can receive a low resolution image or video frame as input and upsamples the chrominance components image or video by two times in each direction. The output of the hardware scaler106may thus be high resolution images or video frames. For example, the high resolution images or video frames generated by the hardware scaler106may have a resolution of twice the input low resolution frames.

The CNN104may be any upscaling framework that takes low resolution frames102as input. The CNN104may be trained to learn a residual between the output of the neural network given a training pair including a low resolution input frame and a ground truth high resolution frame. For example, a number of weights of the neural network may be modified based on the calculated residual. In this manner, the CNN104may have been iteratively trained to output frames more closely resembling the ground truth of input low resolution frames in a training set of frames.

The combiner108combines the output high resolution frame of the CNN104with the high resolution frame from the hardware scaler106to generate a combined high resolution frame110. For example, the combined high resolution frame110may have improved detail as compared to the high resolution frame from the hardware scaler106. Moreover, in various examples, the system100may use a scalable CNN super resolution framework that includes a hardware scaler106and scalable CNN104, which can be extended as a quality requirement and computation capability increases. For example, the CNN104may be the scalable CNN200ofFIG.2.

The diagram ofFIG.1is not intended to indicate that the example system100is to include all of the components shown inFIG.1. Rather, the example system100can be implemented using fewer or additional components not illustrated inFIG.1(e.g., additional low resolution frames, high resolution frames, CNN networks, hardware scalers, etc.).

FIG.2is a block diagram illustrating an example scalable convolutional neural network for super resolution. The example scalable CNN200can be implemented in CNN104of the system100ofFIG.1, or the CNN1038of computing device1000inFIG.10using the methods600-900ofFIGS.6-9. For example, the scalable CNN200can be trained using the methods600and700ofFIGS.6and7and used to generate high resolution frames using the methods800and900ofFIGS.8and9.

The example scalable CNN200includes similarly numbered elements ofFIG.1. For example, the scalable CNN200is shown receiving a low resolution frame102and outputting a high resolution frame110. In some examples, in addition to YUV input frames102, the scalable CNN network200can be configured to support native RGB input frames102. In these examples, both training and inference may use images or video frames in RGB color space as input102, and no hardware scaler is used. The scalable CNN200further includes a first convolutional layer202A with PReLU activation. For example, the first convolutional layer202A may have parameter values of (K,1,3,N), where K is the convolutional kernel size, the first “1” value refers to the number of strides to apply the convolution, the “3” value refers to the number of input channels, and N means the number of output channels or feature maps. As another example, if the first convolutional layer202A is used in the CNN104, the parameter values may be (K,1,1,N), where the single channel may be the Y component of a YUV video frame. The scalable CNN200includes a residual block group204communicatively coupled to the first convolutional layer202A. The scalable CNN200further includes a second convolutional layer with PReLU activation202B communicatively coupled to the residual block group204. For example, the second convolutional layer202B may have parameter values of (K,1,N,N), where the first “N” is a number of input feature maps and the second “N” value is the number of feature maps in a new output set of feature maps. For example, the convolutional layer202B may have an input of N feature maps, and each feature map is a two-dimensional image patch. After the processing at the convolutional layer202B, the convolutional layer202B may output a new set of N feature maps, which are used to restore a high-resolution image or video. The system includes a combiner108communicatively coupled to the first convolutional layer202A and the second convolutional layer202B. The scalable CNN200also includes a transpose convolutional layer208with PReLU activation communicatively coupled to the combiner108. For example, the transpose convolutional layer208may have parameter values of (K,1,N,N). In various examples, the transpose convolution layer208upscales the input N feature maps by an integer factor. For example, the transpose convolution layer208may upscale the input features by a factor of 2 for 2× upscaling case. As one examples, if the size of each input feature map is p, then the transpose convolution layer208may output a new set of N feature maps, and the size of each feature map is 2p. The scalable CNN200further includes a third convolutional layer210communicatively coupled to the transpose convolutional layer208. For example, the third convolutional layer210may have (K,1,N,3) features. In some examples, such as if the scalable CNN200is used as the CNN104, then the parameter set for the third convolutional layer210may be (K, 1, N, 1), because only one channel may be output by the network. For example, the one channel may be the Y component channel. The residual block group204includes a number of residual blocks210. The use of a reduced number of residual blocks is210indicated by a dotted arrow. For example, the last residual block210of the residual block group204may not be used for operations with less computational complexity.

As shown in the example ofFIG.2, in various examples, a topology based on an enhanced deep super-resolution network (EDSR) structure may be deployed as a baseline framework for the scalable CNN200. The EDSR structure may be optimized by having unnecessary modules removed in comparison to conventional residual networks. In various examples, the internal weight and activation precision of the scalable CNN200may be reduced. For example, the scalable CNN200may use16-bit floating point representations instead of32-bit in the original EDSR structure. In addition, in various examples, the number of residual blocks and feature dimensions may be pruned in order to achieve real time performance with limited computation capability and memory bandwidth in mobile platforms. For example, the pruning of residual blocks210to use a lower number of residual blocks210is indicated by a dotted arrow. The number of feature maps N used in convolutional layers202A,202B,208, and210may also be reduced to reduce the feature dimensions of the scalable CNN200. By reducing this number, the total computational resources and memory bandwidth can be effectively reduced. In some examples, the network feature map size may also be adaptive to the input resolution. For example, the capability of CNN network can be further increased with computational growth. For example, by cascading more residual blocks or increasing the number of feature maps N, the system can be extended to a larger network and provide higher quality results. Similarly, to reduce computational intensity, the capability of the CNN network200may be decreased by either reducing the number of residual blocks or the number of feature maps N used in convolutional layers202A,202B,208, and210. In some examples, to improve cache locality, the size of the feature maps may also be adjusted. As used herein, the feature map size refers to a size of the image patches. For example, the size of the image patches may be (W/M)×(H/N). As one example, when M=N=1, the feature map size may be equal to the low resolution image width W and height H. In the inference stage, each low-resolution image may be divided into MxN image patches, whose size is (W/M)×(H/N). In some examples, an optimal feature map size may be used to improve the cache locality to achieve best system performance. For example, an optimal feature map size may be determined by running an inference multiple times using different feature map sizes to determine which feature map size has the best performance. In some examples, if more detailed information on the architecture of the computation devices is available, then a theoretical performance projection can be performed using different feature map sizes to determine an optimal feature size value.

In addition, the ReLU function of the EDSR structure may be replaced with a PReLU function. For example, the PReLU function may be the PReLU function ofFIG.4.

The diagram ofFIG.2is not intended to indicate that the example scalable CNN200is to include all of the components shown inFIG.2. Rather, the example scalable CNN200can be implemented using fewer or additional components not illustrated inFIG.2(e.g., additional low resolution frame, high resolution frames, convolutional layers, residual blocks, etc.).

FIG.3is a flow chart illustrating an example system for training a scalable convolutional neural network for super resolution. The example system300can be implemented to train the system100or the scalable CNN200ofFIGS.1and2, using the computing device1000ofFIG.10, or the computer readable media1100ofFIG.11.

The system300ofFIG.3includes a set of high resolution frames302. For example, the high resolution frames302may be a set of training frames. The system300includes a downscaler304. The system300includes low resolution frames306shown being output by the downscaler302. The system300includes a CNN-based super resolution unit308communicatively coupled to the downscaler304to receive downscaled low resolution frames306. The system300includes a set of reconstructed high resolution frames310shown being generated by the CNN-based super resolution unit308. For example, the CNN based super resolution network308may be implemented using the scalable convolutional neural network200ofFIG.2. The system300also further includes a loss calculator312communicatively coupled to the CNN-based super resolution unit308and shown receiving both the high resolution frames302and the reconstructed high resolution frames310.

The example system300for training a CNN-based super resolution unit308includes a first low resolution frame306and high resolution frame302pairs may be prepared before training. For example, a high resolution frame310may be captured by the device with higher sampling rate. In some examples, the high resolution frames310may be converted into YUV format from other image formats, such as RGB. In various examples, the downscaler304can generate low resolution frames306by downsampling high resolution frames302. In various examples, the high resolution frames302may be downscaled using a nearest neighbor downsampling method for purposes of data augmentation. For example, the training data set may be first generated in traditional manner, then screen and gaming content may be resized using a nearest neighbor method. In various examples, a proportion of nearest neighbor downsampled frames among the total training set may be controlled. By using nearest downsampled frames for training input, the resulting trained CNN based super resolution network308may successfully be prevented from generating overshoot artifacts on text and edges at inference. However, some distortion may be introduced on text areas if nearest downsampled frames are exclusively used for training input. For example, the text areas may appear to have a changed font style. In addition, some sharp details may also be removed along the lines. Thus, only training with neighbor downscaled data may degrade the high resolution output quality. Therefore, in some examples, the proportion of nearest neighbor training frames may be optimized and set to be used within 10% to 25% among the total training frames. In this way, the trained model for the CNN-based super resolution network308may not be over tuned.

In various examples, the CNN-based super resolution network308receives the downscaled low resolution frames306and generates reconstructed high resolution frames310. For example, the reconstructed high resolution frames310may match the resolution of the high resolution frames302.

The reconstructed high resolution frames310may be input with the original high resolution frames302into a loss calculator312to calculate a loss to be minimized. For example, the loss may be calculated using any suitable loss function. In various examples, the loss function used for training can be designed as L1/L2 of the output and ground truth, or any other suitable perceptual loss. In some examples, a gradient of the loss function with respect to weights of the CNN may be calculated using backpropagation. One or more weights of the CNN may be updated accordingly. By minimizing the loss function between the generated reconstructed high resolution frames310and their corresponding ground truth high resolution frames302, the CNN-based super resolution network308may finally converge to a certain degree. For example, the degree of convergence may be set as a predefined threshold.

In various examples, the resulting trained CNN-based super resolution network308may be used in an inference stage for improved super resolution imaging. For example, the trained CNN-based super resolution network308may be used as the system100ofFIG.1.

The diagram ofFIG.3is not intended to indicate that the example system300is to include all of the components shown inFIG.3. Rather, the example system300can be implemented using fewer or additional components not illustrated inFIG.3(e.g., additional high resolution frames, low resolution frames, reconstructed high resolution frames, downscalers, CNN based super resolution networks, losses, etc.). For example, the system300can also use the self-similarity loss and final loss ofFIG.5by introducing a CNN based downsampler.

FIG.4is a pair of graphs400showing a replacement of a ReLU activation402function with a PReLU activation function404. The vertical axes of the graphs indicate values for f(y) and the horizontal axes indicate values of y. In some examples, because the ReLU activation402may clamp the outputs y below zero to an f(y) value of zero, this may result in gradient vanishing during training. In particular, for y<0, the gradient of ReLU equals to 0, which means that gradient backpropagation may stop at this point, and all the layers before the ReLU may not be well optimized by the training. Thus, to improve training and resulting output quality, the ReLU activation402may be replaced with a PReLU activation404. For example, a quality improvement when using PReLU404may be particularly noticeable at inference with frames including sharp edges, such as text. Some types of content, such as screen content, may include sharp edges more often. Together with the data augmentation techniques described inFIG.3, a model trained using a PReLU activation404may remove overshoot artifacts on screen and gaming content, while preserving sharpness in nature content.

FIG.5is a block diagram illustrating an example system for training a scalable convolutional neural network for super resolution with a self-similarity loss. The example system500can be implemented to train the system100or the scalable CNN200ofFIGS.1and2, the computing device1000ofFIG.10, or the computer readable media1100ofFIG.11.

The system500ofFIG.5includes similarly numbered elements of the system300ofFIG.3. In addition, the system500includes a CNN based downscaler502communicatively coupled to the CNN based super resolution network308. In various examples, the CNN based downscaler502may have the same topology as the example scalable convolutional neural network200ofFIG.2, but with parameters configured differently to enable downscaling. In addition, the system500also includes a self-similarity loss calculator504communicatively coupled to the CNN-based downsampler502. The system500also further includes a final loss calculator506communicatively coupled to the self-similarity loss calculator504and the loss calculator312.

In the system500, the CNN based downscaler502can perform downsampling on the reconstructed high resolution frames310to generate downsampled reconstructed high resolution frames with a low resolution referred to herein as CNN based downsampled frames. For example, the CNN based downsampled frames may have a resolution similar to the low resolution frames306.

The self-similarity loss calculator504can calculate a self-similarity loss based on the low resolution frames306and the CNN based downsampled frames. In various examples, the self-similarity loss measures the similarity between the downscaled input frame and a downscaled copy of the reconstructed high resolution frame310. In various examples, the self-similarity loss can be used to regularize the CNN-network to suppress aliasing artifact via backpropagation.

The final loss calculator506can calculate a final loss based on the loss312and the self-similarity loss504. For example, the final loss may be calculated by the weighted average of loss312and self-similarity loss504. For example, the final loss may be calculated using the Equation:

where lossAis the loss calculated by loss calculator312, self_similarity_loss is the loss calculated by the self-similarity loss calculator504, and lambda is an empirically determined weighting parameter. Thus, the aliasing artifacts may be suppressed by using the final loss in the network optimization. Because the CNN based downsampler502is only used during training and not used during inference, the resulting system using the trained CNN based super resolution network308may be computationally very efficient at inference.

The diagram ofFIG.5is not intended to indicate that the example system500is to include all of the components shown inFIG.5. Rather, the example system500can be implemented using fewer or additional components not illustrated inFIG.5(e.g., additional high resolution ground truth frames, low resolution frames, high resolution frames, CNN networks, hardware scalers, etc.).

FIG.6is a process flow diagram illustrating a method600for training a scalable convolutional neural network for super resolution. The example method600can be implemented in the systems100and200ofFIGS.1and2, the computing device1000ofFIG.10, or the computer readable media1100ofFIG.11.

At block602, training frames are received. For example, the training frames may be high resolution frames used as ground truth frames. In various examples, the training frames may be frames in a YUV format.

At block604, the training frames are downscaled to generate low resolution training frames. For example, the training frames may be downscaled by a factor of two in each direction. Thus, each block of four pixels may be represented by one pixel in the low resolution training frames. In various examples, the training frames may be downscaled using nearest neighbor downscaling. In some examples, the training frames may include base part and an augmented part. The base part may be a low resolution frame generated by using bicubic interpolation. The augmented part may be a low resolution frame was generated by using nearest neighbor downscaling. In various examples, the percentage of augmented parts to the total sum of parts may be 10%-25%. May I know whether we need to emphasis these two parts here

At block606, the low resolution training frames are processed via the scalable convolutional neural network to generate reconstructed high resolution frames. For example, the reconstructed high resolution frames may have the same resolution as the high resolution training frames.

At block608, a loss is calculated based on a comparison of the training frames with the reconstructed high resolution frames. For example, the loss may be a L1/L2 loss or any other suitable perceptual loss.

At block610, the calculated loss is backpropagated. For example, one or more weights of the scalable convolutional neural network may be adjusted based on the calculated loss.

The process flow diagram ofFIG.6is not intended to indicate that the blocks of the example method600are to be executed in any particular order, or that all of the blocks are to be included in every case. Further, any number of additional blocks not shown may be included within the example method600, depending on the details of the specific implementation.

FIG.7is a process flow diagram illustrating a method700for training a scalable convolutional neural network for super resolution with a self-similarity loss. The example method700can be implemented in the systems100and200ofFIGS.1and2, the computing device1000ofFIG.10, or the computer readable media1100ofFIG.11.

At block702, training frames are received. For example, the training frames may be high resolution color frames or video frames. In various examples, the training frames may be video frames in a YUV format. For example, the convolutional neural network may be configured to receive the Y channel of the YUV format video frames. In some examples, the training frames may be in an RGB format. For example, the scalable convolutional neural network may be configured to support three channel input without the use of a scaler.

At block704, the training frames are downscaled to generate low resolution training frames. For example, the training frames may be downscaled by a factor of two in each direction. In various examples, the training frames may be downscaled using nearest neighbor downscaling.

At block706, the low resolution training frames are processed via the scalable convolutional neural network to generate reconstructed high resolution frames. For example, the reconstructed high resolution frames may have the same resolution as the high resolution training frames.

At block708, a first loss is calculated based on a comparison of the training frames with the reconstructed high resolution frames. For example, the loss may be a L1/L2 loss or any other suitable perceptual loss.

At block710, the reconstructed high resolution frames are processed to generate downsampled frames. For example, the reconstructed high resolution frames may be downsampled using a CNN based downsampler.

At block712, a self-similarity loss is calculated based on a comparison of the low resolution training frames with the downsampled frames. For example, the self-similarity loss may be calculated using a L1/L2 loss or any other suitable perceptual loss.

At block714, a final loss is calculated based on the self-similarity loss and the first loss. For example, the final loss may be calculated by combining the self-similarity loss with the first loss.

At block716, the final loss is backpropagated through the scalable convolutional neural network. For example, one or more weights of the scalable convolutional neural network may be adjusted based on the calculated loss.

The process flow diagram ofFIG.7is not intended to indicate that the blocks of the example method700are to be executed in any particular order, or that all of the blocks are to be included in every case. Further, any number of additional blocks not shown may be included within the example method700, depending on the details of the specific implementation.

FIG.8is a process flow diagram illustrating a method800for super resolution using a scalable neural network. The example method800can be implemented in the systems100and200ofFIGS.1and2, the computing device1000ofFIG.10, or the computer readable media1100ofFIG.11.

At block802, low resolution frames are received. For example, the low resolution video frames may be in a YUV video frame format. In some examples, the low resolution frames may be converted into the YUV frame format from an RGB format.

At block804, high resolution illuminance component frames are generated based on the low resolution frames via a convolutional neural network (CNN). For example, the high resolution illuminance component frames may be generated based on an illuminance component of the low resolution frames. In some examples, the CNN may be the scalable CNN ofFIG.2. In various examples, the CNN may be trained using nearest neighbor downsampling of high resolution ground truth training frames. In some examples, the CNN may be trained using a self-similarity loss function. For example, the CNN may be trained using the methods600or700ofFIGS.6and7. In some examples, the CNN may include a reduced residual block group. In various example, the CNN may also include a parametric rectified linear unit (PReLU) activation.

At block806, high resolution chrominance component frames are generated based on the low resolution frames via a hardware scaler. For example, the high resolution illuminance component frames may be generated based on chrominance components of the low resolution frames. In various examples, the hardware scaler may be an energy efficient hardware scaler.

At block808, the high resolution illuminance component frames are combined with the high resolution chrominance component frames to generate high resolution frames. For example, a high resolution illuminance component frame may be combined with a high resolution chrominance component frame to generate a high resolution YUV format video frame.

The process flow diagram ofFIG.8is not intended to indicate that the blocks of the example method800are to be executed in any particular order, or that all of the blocks are to be included in every case. Further, any number of additional blocks not shown may be included within the example method800, depending on the details of the specific implementation. For example, the method800may include adapting a feature map size of the convolutional neural network to a resolution of the low resolution frame. In some examples, the method800may include adjusting the number of feature maps in the convolutional neural network based on a memory bandwidth available to the processor.

FIG.9is a process flow diagram illustrating a method900for super resolution using a scalable neural network with PReLU activation. The example method900can be implemented in the systems100and200ofFIGS.1and2, the computing device1000ofFIG.10, or the computer readable media1100ofFIG.11.

At block902, low resolution frames are received. For example, the low resolution frames may be received in a YUV video frame format. In some examples, the frames may be received in an RGB format and converted into a YUV frame format.

At block904, high resolution frames are generated via a convolutional neural network (CNN) with a PReLU activation. For example, the high resolution illuminance component frames may be generated based on an illuminance component of the low resolution frames. In some examples, the CNN may be the scalable CNN ofFIG.2.

At block906, high resolution frames are generated via a hardware scaler with residual block group and PReLU activation. For example, the high resolution illuminance component frames may be generated based on chrominance components of the low resolution frames. In various examples, the hardware scaler may be an energy efficient hardware scaler.

At block908, the high resolution frames of the CNN are combined with the high resolution frames of the hardware scaler to generate combined high resolution frames. For example, a high resolution illuminance component frame may be combined with a high resolution chrominance component frame to generate a high resolution YUV format frame.

The process flow diagram ofFIG.9is not intended to indicate that the blocks of the example method900are to be executed in any particular order, or that all of the blocks are to be included in every case. Further, any number of additional blocks not shown may be included within the example method900, depending on the details of the specific implementation. For example, the method900may include adapting a feature map size of the convolutional neural network to a resolution of the low resolution frame, adjusting the number of feature maps in the convolutional neural network based on a memory bandwidth available to the processor, or both.

Referring now toFIG.10, a block diagram is shown illustrating an example computing device that can execute super resolution using a scalable neural network. The computing device1000may be, for example, a laptop computer, desktop computer, tablet computer, mobile device, or wearable device, among others. In some examples, the computing device1000may be an edge device in a cloud computing system. In various examples, the computing device1000may be a camera system. The computing device1000may include a central processing unit (CPU)1002that is configured to execute stored instructions, as well as a memory device1004that stores instructions that are executable by the CPU1002. The CPU1002may be coupled to the memory device1004by a bus1006. Additionally, the CPU1002can be a single core processor, a multi-core processor, a computing cluster, or any number of other configurations. Furthermore, the computing device1000may include more than one CPU1002. In some examples, the CPU1002may be a system-on-chip (SoC) with a multi-core processor architecture. In some examples, the CPU1002can be a specialized digital signal processor (DSP) used for video processing. The memory device1004can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory systems. For example, the memory device1004may include dynamic random access memory (DRAM).

The memory device1004can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory systems. For example, the memory device1004may include dynamic random access memory (DRAM).

The computing device1000may also include a graphics processing unit (GPU)1008. As shown, the CPU1002may be coupled through the bus1006to the GPU1008. The GPU1008may be configured to perform any number of graphics operations within the computing device1000. For example, the GPU1008may be configured to render or manipulate graphics images, graphics frames, videos, or the like, to be displayed to a user of the computing device1000.

The memory device1004can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory systems. For example, the memory device1004may include dynamic random access memory (DRAM). The memory device1004may include device drivers1010that are configured to execute the instructions for training multiple convolutional neural networks to perform sequence independent processing. The device drivers1010may be software, an application program, application code, or the like.

The CPU1002may also be connected through the bus1006to an input/output (I/O) device interface1012configured to connect the computing device1000to one or more I/O devices1014. The I/O devices1014may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touchpad or a touchscreen, among others. The I/O devices1014may be built-in components of the computing device1000, or may be devices that are externally connected to the computing device1000. In some examples, the memory1004may be communicatively coupled to I/O devices1014through direct memory access

The CPU1002may also be linked through the bus1006to a display interface1016configured to connect the computing device1000to a display device1018. The display device1018may include a display screen that is a built-in component of the computing device1000. The display device1018may also include a computer monitor, television, or projector, among others, that is internal to or externally connected to the computing device1000.

The computing device1000also includes a storage device1020. The storage device1020is a physical memory such as a hard drive, an optical drive, a thumbdrive, an array of drives, a solid-state drive, or any combinations thereof. The storage device1020may also include remote storage drives.

The computing device1000may also include a network interface controller (NIC)1022. The NIC1022may be configured to connect the computing device1000through the bus1006to a network1024. The network1024may be a wide area network (WAN), local area network (LAN), or the Internet, among others. In some examples, the device may communicate with other devices through a wireless technology. For example, the device may communicate with other devices via a wireless local area network connection. In some examples, the device may connect and communicate with other devices via Bluetooth® or similar technology.

The computing device1000further includes a camera1026. For example, the camera1026may include one or more imaging sensors. In some example, the camera1026may include a processor to generate video frames.

The computing device1000further includes a deep learning super resolution trainer1028. For example, the deep learning super resolution trainer1028can be used to train a neural network to perform super-resolution imaging. The deep learning super resolution trainer1028can include a downsampler1030, a loss calculator1032, and a backpropagator1034. In some examples, each of the components1030-1034of the deep learning super resolution trainer1028may be a microcontroller, embedded processor, or software module. The downsampler1030can downscale high resolution training frames to generate additional training pairs including base parts and augmented parts. For example, the downsampler1030can downscale training frames using a bicubic downsampling to generate a base part of each of the training frame pairs. In various examples, the downsampler1030can downscale training frames using nearest neighbor downsampling of high resolution ground truth training frames to generate an augmented part of each training frame pair. In various examples, the additional training pairs including the base parts and augmented parts may be 10 to 25 percent of the training dataset used. For example, the use of 10-25% of additional training pairs during training may regularize the network and improve the quality of the trained network. The loss calculator1032can calculate a loss based on a comparison of reconstructed high resolution frames and high resolution ground truth frames. For example, a convolutional neural network may be used to generate reconstructed high resolution frame from a low resolution training frame during training. In some examples, the loss calculator1032can calculate a self-similarity loss based on a comparison of a downsample reconstructed high resolution frame and a downsampled low resolution frame. For example, the loss calculator1032can calculate a self-similarity loss based on a CNN based downsampled frame generated from a reconstructed high resolution frame and a low resolution training frame generated by downscaling a high resolution ground truth frame. In various examples, the loss calculator1032can calculate a final loss based on the first loss and the self-similarity loss. For example, loss calculator1032can calculate a final loss by combining the first loss and the self-similarity loss. The backpropagator1034can backpropagate a loss to modify one or more weights of a CNN based super resolution network. In some examples, the backpropagator1034can backpropagate the final loss to modify one or more weights of a CNN based super resolution network.

The computing device also further includes a deep learning super resolution network1036. For example, the deep learning super resolution network1036may be a scalable convolutional neural network. The deep learning super resolution network1036can be used to execute super resolution on input frames to generate frames with higher resolution and detail. The deep learning super resolution network1036includes a convolutional neural network1038, a hardware scaler1040, and a combiner1042. The convolutional neural network1038can receive a low resolution frames and generate a high resolution illuminance component frames. For example, the convolutional neural network1038may be a small scale network based on enhanced deep super-resolution. In some examples, the convolutional neural network1038may include a parametric rectified linear unit (PReLU) activation. In various examples, the convolutional neural network1038may include a feature map size that is optimized to improve cache locality. The hardware scaler1040can receive the low resolution frames and generate a high resolution chrominance component frames. The combiner1042can combine the high resolution illuminance component frames and the high resolution chrominance component frames to generate high resolution frames. For example, the combined high resolution images may have improved detail in the illuminance component most noticeably by human vision.

The block diagram ofFIG.10is not intended to indicate that the computing device1000is to include all of the components shown inFIG.10. Rather, the computing device1000can include fewer or additional components not illustrated inFIG.10, such as additional buffers, additional processors, additional CNNs, and the like. The computing device1000may include any number of additional components not shown inFIG.10, depending on the details of the specific implementation. For example, the deep learning super resolution trainer1028may further include a CNN based downsampler to downsample a reconstructed high resolution frame for training the convolutional neural network. Furthermore, any of the functionalities of the nearest neighbor downsampler1030, the loss calculator1032, and the backpropagator1034, or the CNN1038, the hardware scaler1040, or combiner1042, may be partially, or entirely, implemented in hardware and/or in the processor1002. For example, the functionality may be implemented with an application specific integrated circuit, in logic implemented in the processor1002, or in any other device. In addition, any of the functionalities of the CPU1002may be partially, or entirely, implemented in hardware and/or in a processor. For example, the functionality of the deep learning super resolution trainer1028or the deep learning super resolution network1036may be implemented with an application specific integrated circuit, in logic implemented in a processor, in logic implemented in a specialized graphics processing unit such as the GPU1008, or in any other device.

FIG.11is a block diagram showing computer readable media1100that store code for performing super resolution using a scalable neural network. The computer readable media1100may be accessed by a processor1102over a computer bus1104. Furthermore, the computer readable medium1100may include code configured to direct the processor1102to perform the methods described herein. In some embodiments, the computer readable media1100may be non-transitory computer readable media. In some examples, the computer readable media1100may be storage media.

The various software components discussed herein may be stored on one or more computer readable media1100, as indicated inFIG.11. For example, a receiver module1106may be configured to receive low resolution frames. A trainer module1108may be configured to receiving high resolution ground truth training frames and train a CNN based on the training frames. For example, the trainer module1108may be configured to downscale the training frames to generate low resolution training frames to train the CNN. In some examples, the trainer module1108may be configured to downscale the training frames using nearest neighbor downsampling of the high resolution ground truth training frames. The trainer module1108may be configured to calculate a first loss based on a comparison of the training frames with reconstructed high resolution frames from the CNN. In some examples, the trainer module1108may be configured to process reconstructed high resolution frames to generate downsampled frames. In various examples, the trainer module1108may be configured to calculate a self-similarity loss based on a comparison of the low resolution training frames with the downsampled frames. For example, the trainer module1108may be configured to perform the methods600or700ofFIGS.6and7. A CNN module1110may be configured to generate high resolution illuminance component frames based on the low resolution frame. In some examples, the CNN module1110may be configured with a reduced residual block group. In various examples, the CNN module1110may be configured to include a parametric rectified linear unit (PReLU) activation. In some examples, the CNN module1110may be configured with an adaptable feature map size to match a resolution of the low resolution frame. In various examples, the CNN module1110may be configured with an adjustable number of feature maps based on a memory bandwidth available to a processor. A hardware scaler module1112may be configured to generate high resolution chrominance component frames based on the received low resolution frames. For example, the hardware scaler module1112may be configured to upscale the chrominance channels of the low resolution frames. A combiner module1114may be configured to combine the high resolution illuminance component frames of the CNN module1110with the high resolution chrominance component frames from the hardware scaler module1112to generate high resolution frames. For example, the combined high resolution frames may be YUV frames that include improved detail in the illuminance component.

The block diagram ofFIG.11is not intended to indicate that the computer readable media1100is to include all of the components shown inFIG.11. Further, the computer readable media1100may include any number of additional components not shown inFIG.11, depending on the details of the specific implementation.

EXAMPLES

Example 1 is an apparatus for super resolution imaging. The apparatus includes a convolutional neural network to receive a low resolution frame and generate a high resolution illuminance component frame. The apparatus also includes a hardware scaler to receive the low resolution frame and generate a high resolution chrominance component frame. The apparatus further includes a combiner to combine the high resolution illuminance component frame and the high resolution chrominance component frame to generate a high resolution frame.

Example 2 includes the apparatus of example 1, including or excluding optional features. In this example, the convolutional neural network is trained on additional training frame pairs generated including augmented parts generated using nearest neighbor downsampling of high resolution ground truth training frames.

Example 3 includes the apparatus of any one of examples 1 to 2, including or excluding optional features. In this example, the convolutional neural network is trained using a self-similarity loss function.

Example 4 includes the apparatus of any one of examples 1 to 3, including or excluding optional features. In this example, the convolutional neural network includes a small scale network based on enhanced deep super-resolution.

Example 5 includes the apparatus of any one of examples 1 to 4, including or excluding optional features. In this example, the convolutional neural network includes a parametric rectified linear unit (PReLU) activation.

Example 6 includes the apparatus of any one of examples 1 to 5, including or excluding optional features. In this example, the convolutional neural network is to generate reconstructed high resolution frame from a low resolution training frame during training. The reconstructed high resolution frame and a ground truth high resolution frame are used to calculate a loss used to train the convolutional neural network.

Example 7 includes the apparatus of any one of examples 1 to 6, including or excluding optional features. In this example, the apparatus includes a CNN based downsampler to downsample a reconstructed high resolution frame for training the convolutional neural network.

Example 8 includes the apparatus of any one of examples 1 to 7, including or excluding optional features. In this example, the apparatus includes a self-similarity loss calculator to calculate a self-similarity loss based on a CNN based downsampled frame generated from a reconstructed high resolution frame and a low resolution training frame generated by downscaling a high resolution ground truth frame.

Example 9 includes the apparatus of any one of examples 1 to 8, including or excluding optional features. In this example, the apparatus includes a final loss calculator to calculate a final loss based on a loss and a self-similarity loss, the final loss used to train the convolutional neural network during training.

Example 10 includes the apparatus of any one of examples 1 to 9, including or excluding optional features. In this example, a feature map size of the convolutional neural network is optimized to improve cache locality.

Example 11 is a method for super resolution imaging. The method includes receiving, via a processor, a low resolution frame. The method also includes generating, via a convolutional neural network (CNN), a high resolution illuminance component frame based on the low resolution frame. The method further includes generating, via a hardware scaler, a high resolution chrominance component frame based on the low resolution frame. The method also further includes combining, via the processor, the high resolution illuminance component frame and the high resolution chrominance component frame to generate a high resolution frame.

Example 12 includes the method of example 11, including or excluding optional features. In this example, the method includes training the convolutional neural network using nearest neighbor downsampling of high resolution ground truth training frames.

Example 13 includes the method of any one of examples 11 to 12, including or excluding optional features. In this example, the method includes training the convolutional neural network using a self-similarity loss function.

Example 14 includes the method of any one of examples 11 to 13, including or excluding optional features. In this example, generating the high resolution illuminance component frame includes using a CNN with a reduced residual block group.

Example 15 includes the method of any one of examples 11 to 14, including or excluding optional features. In this example, generating the high resolution illuminance component frame includes using a CNN with a parametric rectified linear unit (PReLU) activation.

Example 16 includes the method of any one of examples 11 to 15, including or excluding optional features. In this example, the method includes adapting a feature map size of the convolutional neural network to a resolution of the low resolution frame.

Example 17 includes the method of any one of examples 11 to 16, including or excluding optional features. In this example, the method includes adjusting a number of feature maps in the convolutional neural network based on a memory bandwidth available to the processor.

Example 18 includes the method of any one of examples 11 to 17, including or excluding optional features. In this example, the method includes training the convolutional neural network. Training the convolutional neural network includes receiving training frames. Training the convolutional neural network includes downscaling, via a downscaler, the training frames to generate low resolution training frames. Training the convolutional neural network also includes processing, via the convolutional neural network, the low resolution training frames to generate reconstructed high resolution frames. Training the convolutional neural network further includes calculating a loss based on a comparison of the training frames with the reconstructed high resolution frames. Training the convolutional neural network also further includes and backpropagating the calculated loss.

Example 19 includes the method of any one of examples 11 to 18, including or excluding optional features. In this example, the method includes training the convolutional neural network. Training the convolutional neural network includes processing, via a CNN based downsampler, reconstructed high resolution frames to generate downsampled frames. Training the convolutional neural network also includes calculating a self-similarity loss based on a comparison of low resolution training frames with the downsampled frames. Training the convolutional neural network further includes calculating a final loss based on the self-similarity loss and a loss calculated between high resolution training frames and the reconstructed high resolution frames. Training the convolutional neural network also further includes backpropagating the convolutional neural network based on the calculated final loss.

Example 20 includes the method of any one of examples 11 to 19, including or excluding optional features. In this example, the method includes receiving an RGB component frame and converting the RGB color frame into a YUV component frame.

Example 21 is at least one computer readable medium for super resolution imaging having instructions stored therein that direct the processor to receive a low resolution frame. The computer-readable medium also includes instructions that direct the processor to generate a high resolution illuminance component frame based on the low resolution frame. The computer-readable medium further includes instructions that direct the processor to generate a high resolution chrominance component frame based on the low resolution frame. The computer-readable medium also further includes instructions that direct the processor to and combine the high resolution illuminance component frame and the high resolution chrominance component frame to generate a high resolution frame.

Example 22 includes the computer-readable medium of example 21, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to train a convolutional neural network using nearest neighbor downsampling of high resolution ground truth training frames.

Example 23 includes the computer-readable medium of any one of examples 21 to 22, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to train a convolutional neural network using a self-similarity loss function.

Example 24 includes the computer-readable medium of any one of examples 21 to 23, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to generate the high resolution illuminance component frame using a convolutional neural network (CNN) with a reduced residual block group.

Example 25 includes the computer-readable medium of any one of examples 21 to 24, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to generate the high resolution illuminance component frame using a convolutional neural network (CNN) with a parametric rectified linear unit (PReLU) activation.

Example 26 includes the computer-readable medium of any one of examples 21 to 25, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to adapt a feature map size of the convolutional neural network to a resolution of the low resolution frame.

Example 27 includes the computer-readable medium of any one of examples 21 to 26, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to adjust a number of feature maps in the convolutional neural network based on a memory bandwidth available to the processor.

Example 28 includes the computer-readable medium of any one of examples 21 to 27, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to: receive training frames; downscale the training frames to generate low resolution training frames; process the low resolution training frames to generate reconstructed high resolution frames; calculate a loss based on a comparison of the training frames with the reconstructed high resolution frames; and backpropagate the calculated loss.

Example 29 includes the computer-readable medium of any one of examples 21 to 28, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to: process reconstructed high resolution frames to generate downsampled frames; calculate a self-similarity loss based on a comparison of low resolution training frames with the downsampled frames; calculate a final loss based on the self-similarity loss and a loss calculated between high resolution training frames and the reconstructed high resolution frames; and backpropagate the convolutional neural network based on the calculated final loss.

Example 30 includes the computer-readable medium of any one of examples 21 to 29, including or excluding optional features. In this example, the computer-readable medium includes instructions that cause the processor to receive an RGB component frame and convert the RGB color frame into a YUV component frame.

Example 31 is a system for super resolution imaging. The system includes a convolutional neural network to receive a low resolution frame and generate a high resolution illuminance component frame. The system also includes a hardware scaler to receive the low resolution frame and generate a high resolution chrominance component frame. The system further includes a combiner to combine the high resolution illuminance component frame and the high resolution chrominance component frame to generate a high resolution frame.

Example 32 includes the system of example 31, including or excluding optional features. In this example, the convolutional neural network is trained on additional training frame pairs generated including augmented parts generated using nearest neighbor downsampling of high resolution ground truth training frames.

Example 33 includes the system of any one of examples 31 to 32, including or excluding optional features. In this example, the convolutional neural network is trained using a self-similarity loss function.

Example 34 includes the system of any one of examples 31 to 33, including or excluding optional features. In this example, the convolutional neural network includes a small scale network based on enhanced deep super-resolution.

Example 35 includes the system of any one of examples 31 to 34, including or excluding optional features. In this example, the convolutional neural network includes a parametric rectified linear unit (PReLU) activation.

Example 36 includes the system of any one of examples 31 to 35, including or excluding optional features. In this example, the convolutional neural network is to generate reconstructed high resolution frame from a low resolution training frame during training. The reconstructed high resolution frame and a ground truth high resolution frame are used to calculate a loss used to train the convolutional neural network.

Example 37 includes the system of any one of examples 31 to 36, including or excluding optional features. In this example, the system includes a CNN based downsampler to downsample a reconstructed high resolution frame for training the convolutional neural network.

Example 38 includes the system of any one of examples 31 to 37, including or excluding optional features. In this example, the system includes a self-similarity loss calculator to calculate a self-similarity loss based on a CNN based downsampled frame generated from a reconstructed high resolution frame and a low resolution training frame generated by downscaling a high resolution ground truth frame.

Example 39 includes the system of any one of examples 31 to 38, including or excluding optional features. In this example, the system includes a final loss calculator to calculate a final loss based on a loss and a self-similarity loss, the final loss used to train the convolutional neural network during training.

Example 40 includes the system of any one of examples 31 to 39, including or excluding optional features. In this example, a feature map size of the convolutional neural network is optimized to improve cache locality.

Example 41 is a system for super resolution imaging. The system includes means for generating a high resolution illuminance component frame based on a received low resolution frame. The system also includes means for generating a high resolution chrominance component frame based on the received low resolution frame. The system further includes means for combining the high resolution illuminance component frame and the high resolution chrominance component frame to generate a high resolution frame.

Example 42 includes the system of example 41, including or excluding optional features. In this example, the means for generating the high resolution illuminance component frame is trained using nearest neighbor downsampling of high resolution ground truth training frames.

Example 43 includes the system of any one of examples 41 to 42, including or excluding optional features. In this example, the means for generating the high resolution illuminance component frame is trained using a self-similarity loss function.

Example 44 includes the system of any one of examples 41 to 43, including or excluding optional features. In this example, the means for generating the high resolution illuminance component frame includes a small scale network based on enhanced deep super-resolution.

Example 45 includes the system of any one of examples 41 to 44, including or excluding optional features. In this example, the means for generating the high resolution illuminance component frame includes a parametric rectified linear unit (PReLU) activation.

Example 46 includes the system of any one of examples 41 to 45, including or excluding optional features. In this example, the means for generating the high resolution illuminance component frame is to generate reconstructed high resolution frame from a low resolution training frame during training. The reconstructed high resolution frame and a ground truth high resolution frame are used to calculate a loss used to train the means for generating the high resolution illuminance component frame.

Example 47 includes the system of any one of examples 41 to 46, including or excluding optional features. In this example, the system includes means for downsampling a reconstructed high resolution frame for training the means for generating the high resolution illuminance component frame.

Example 48 includes the system of any one of examples 41 to 47, including or excluding optional features. In this example, the system includes means for calculating a self-similarity loss based on a CNN based downsampled frame generated from a reconstructed high resolution frame and a low resolution training frame generated by downscaling a high resolution ground truth frame.

Example 49 includes the system of any one of examples 41 to 48, including or excluding optional features. In this example, the system includes means for calculating a final loss based on a loss and a self-similarity loss, the final loss used to train the means for generating the high resolution illuminance component frame during training.

Example 50 includes the system of any one of examples 41 to 49, including or excluding optional features. In this example, a feature map size of the means for generating the high resolution illuminance component frame is optimized to improve cache locality.

It is to be noted that, although some aspects have been described in reference to particular implementations, other implementations are possible according to some aspects. Additionally, the arrangement and/or order of circuit elements or other features illustrated in the drawings and/or described herein need not be arranged in the particular way illustrated and described. Many other arrangements are possible according to some aspects.