Patent Publication Number: US-2023153946-A1

Title: System and Method for Image Super-Resolution

Description:
RELATED APPLICATION 
     This application claims the benefit of U.S. Provisional Application No. 63/264,058, filed on Nov. 15, 2021. The entire teachings of the above application are incorporated herein by reference. 
    
    
     BACKGROUND 
     Single-Image Super-Resolution (SISR) is a fundamental vision task to reconstruct a single, reliable high-resolution (HR) image from a single, low-resolution (LR) image. SISR has been utilized on various high-level tasks, such as face synthesis, medical imaging, surveillance imaging, and image generation for non-limiting examples. 
     SUMMARY 
     According to an example embodiment, a method for performing image super-resolution (SR) comprises performing image SR on a low-resolution (LR) representation of a high-resolution (HR) original image. The HR original image is at a higher resolution relative to a resolution of the LR representation. The image SR includes producing a reconstructed version of the HR original image based on element-unshuffled downsampling of the LR representation. The method further comprises outputting the reconstructed version produced. 
     The element-unshuffled downsampling may include performing an element-unshuffle operation. The element-unshuffle operation may include downsampling input features. The input features may include elements from a transformed version of the LR representation. The downsampling may include reducing a size of the input features by separating the input features into sub-features. 
     The separating may include selecting a subset of elements from an input feature of the input features and creating a sub-feature of the sub-features by grouping the subset of elements selected. 
     The image SR may further include performing the element-unshuffled downsampling. The element-unshuffled downsampling may produce a plurality of sub-features from input features. The input features may include elements from a transformed version of the LR representation. 
     The image SR may further include performing a max-pooling operation on the sub-features of the plurality of sub-features to produce a plurality of pooled sub-features. The image SR may further include convolving, using group convolution, pooled sub-features of the plurality of pooled sub-features. The convolving may include outputting low-frequency features. The image SR may further include upsampling the low-frequency features output from the convolving to produce up-sampled low-frequency features. The low-frequency features may be at a lower frequency relative to a frequency of the input features. The image SR may further include producing enhanced features by adding the up-sampled low-frequency features to the input features. 
     The image SR may further include producing the reconstructed version based on the enhanced features produced. 
     The element-unshuffled downsampling may include performing an element-unshuffle operation. The element-unshuffle operation may enable the element-unshuffled downsampling that yields higher performance relative to a performance based on downsampling via a different downsampling operation different from the element-unshuffled downsampling. The higher performance may include higher image quality. 
     The image SR may be performed in a non-recurrent, feed-forward manner. 
     According to another example embodiment, a system for performing image super-resolution (SR) comprises an element-unshuffled downsampler and an image SR module. The image SR module is configured to perform image SR on a low-resolution (LR) representation of a high-resolution (HR) original image. The HR original image is at a higher resolution relative to a resolution of the LR representation. The image SR module is further configured to produce a reconstructed version of the HR original image via the image SR performed. The image SR is based on element-unshuffled downsampling of the LR representation. The element-unshuffled downsampler is configured to perform the element-unshuffled downsampling. The image SR module is further configured to output the reconstructed version produced. 
     Alternative system embodiments parallel those described above in connection with the example method embodiment. 
     According to yet another example embodiment, a non-transitory computer-readable medium for performing image super-resolution (SR) has encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to perform image SR on a low-resolution (LR) representation of a high-resolution (HR) original image. The HR original image is at a higher resolution relative to resolution of the LR representation. The image SR includes producing a reconstructed version of the HR original image based on element-unshuffled downsampling of the LR representation. The sequence of instructions further causes the at least one processor to output the reconstructed version produced. 
     Alternative non-transitory computer-readable medium embodiments parallel those described above in connection with the example method embodiment. 
     It should be understood that example embodiments disclosed herein can be implemented in the form of a method, apparatus, system, or computer readable medium with program codes embodied thereon. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. 
       The foregoing will be apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments. 
         FIG.  1    is a block diagram of an example embodiment of an environment in which a mobile device performs image super-resolution (SR) optionally within an embodiment disclosed herein. 
         FIG.  2    is a block diagram of an example embodiment of a system for performing image SR. 
         FIG.  3    is a flow diagram of an example embodiment of a method for performing image SR. 
         FIG.  4    is a graph with an exemplification of peak signal-to-noise ratio (PSNR) vs. a number of Multi-Adds for different SR methods. 
         FIG.  5    is a block diagram of an example embodiment of an image SR module. 
         FIG.  6 A  is a block diagram of a standard residual block. 
         FIG.  6 B  is a block diagram of a depthwise separable convolution (DSC) residual block. 
         FIG.  6 C  is a block diagram of an example embodiment of Self-Residual DSC residual block. 
         FIG.  7 A  is a block diagram of an example embodiment of Element-Unshuffled Downsampling (EUD). 
         FIG.  7 B  is a block diagram of an example embodiment of an Element-Unshuffled Block (EUB). 
         FIG.  7 C  is a block diagram of an example embodiment of a Hybrid Element-Unshuffled Block (HEUB). 
         FIG.  8    is a block diagram of an example embodiment of a Hybrid Element-Unshuffled Network (HEUN) architecture. 
         FIG.  9    is a graph with an exemplification of a comparison among networks. 
         FIG.  10 A  is a bar chart of an exemplification of a comparison of different combinations of up- and down-scale operations. 
         FIG.  10 B  is a graph of an exemplification of results for different kernel settings. 
         FIGS.  11 A-F  are charts with exemplifications of a relationship between PSNR and Normalized Mean Error (NME). 
         FIG.  12 A- 1    is an exemplification of an image of a baby. 
         FIGS.  12 A- 2  through  12 A- 8    are exemplifications of NME visualizations for the image of  FIG.  12 A- 1   . 
         FIG.  12 B- 1    is an exemplification of an image of a butterfly. 
         FIGS.  12 B- 2  through  12 B- 8    are exemplifications of NME visualizations for the image of  FIG.  12 B- 1   . 
         FIG.  12 C- 1    is an exemplification of an image of a bird. 
         FIGS.  12 C- 2  through  12 C- 8    are exemplifications of NME visualizations for the image of  FIG.  12 C- 1   . 
         FIG.  12 D- 1    is an exemplification of an image of a head. 
         FIGS.  12 D- 2  through  12 D- 8    are exemplifications of NME visualizations for the image of  FIG.  12 D- 1   . 
         FIG.  12 E- 1    is an exemplification of an image of a woman. 
         FIGS.  12 E- 2  through  12 E- 8    are exemplifications of NME visualizations for the image of  FIG.  12 E- 1   . 
         FIG.  13 A- 1    is an exemplification of an image of an urban scene. 
         FIGS.  13 A- 2  through  13 A- 11    are exemplifications of visualization results for the image of  FIG.  13 A- 1    with different SR methods. 
         FIG.  13 B- 1    is an exemplification of an image of another urban scene. 
         FIGS.  13 B- 2  through  13 B- 11    are exemplifications of visualization results for the image of  FIG.  13 B- 1    with different SR methods. 
         FIG.  13 C- 1    is an exemplification of an image of yet another urban scene. 
         FIGS.  13 C- 2  through  13 C- 11    are exemplifications of visualization results for the image of  FIG.  13 C- 1    with different SR methods. 
         FIG.  13 D- 1    is an exemplification of an image of cartoon. 
         FIGS.  13 D- 2  through  13 D- 11    are exemplifications of visualization results for the image of  FIG.  13 D- 1    with different SR methods. 
         FIGS.  14 A-B  are a table with an exemplification of benchmark results with a bicubic (BI) degradation model. 
         FIG.  15    is a block diagram of an example internal structure of a computer optionally within an embodiment disclosed herein. 
     
    
    
     DETAILED DESCRIPTION 
     A description of example embodiments follows. 
     It should be understood that the terms “element-unshuffle” and “element-unshuffled” may be used interchangeably herein with the terms “pixel-unshuffle” and “pixel-unshuffled,” respectively, in an event the “element” of such terms is a picture element (pixel). It should be understood, however, that an image disclosed herein is not limited to a picture and, thus, an element thereof is not limited to a pixel. 
     Convolutional neural network (CNN) has achieved great success on image super-resolution (SR). However, most deep CNN-based SR models take massive computations to obtain high performance. Down-sampling features for multi-resolution fusion is an efficient and effective way to improve the performance of visual recognition. Still, it is counter-intuitive to downsample in the SR task, which needs to project a low-resolution input to high-resolution. An example embodiment disclosed herein includes a novel Hybrid Element-Unshuffled Network (HEUN) that introduces an efficient and effective downsampling module into the SR task. The network may include element-unshuffled downsampling and Self-Residual Depthwise Separable Convolutions. An example embodiment may utilize an element-unshuffle operation to downsample input features and use grouped convolution to reduce the channels. An example embodiment may further enhance a depthwise convolution&#39;s performance by adding the input features to its output. Experiments on benchmark datasets disclosed further below show that an example embodiment of HEUN disclosed herein achieves and surpasses the state-of-the-art (SOTA) reconstruction performance with fewer parameters and computation costs relative to conventional SR. An overview of SR is provided below. 
     1 Single Image Super-Resolution (SISR) Overview 
     Single Image Super-Resolution (SISR) is a fundamental vision task to reconstruct a faithful high-resolution (HR) image from a low-resolution (LR) image. SISR has been utilized on various high-level tasks, such as face synthesis (Yu Yin, et al., “Joint super-resolution and alignment of tiny faces. In AAAI, 2020, Yu Yin, et al., “Superfront: From low-resolution to high-resolution frontal face synthesis. In ACMMM, 2021), medical imaging (Wenzhe Shi, et al., “Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In MICCAI, 2013), surveillance imaging (Wilman W W Zou and Pong C Yuen. “Very low resolution face recognition problem,” TIP, 2012), and image generation (Karras, et al., “Progressive growing of gans for improved quality, stability, and variation. Submitted to ICLR 2018, 2017). Dong et al. (Chau, et al., “Learning a deep convolutional network for image super-resolution. In ECCV, 2014) first introduced CNN into SISR and achieved impressive performance in 2014. Afterwards, more deep CNN methods are proposed for the super-resolution tasks (Schulter, et al., “Fast and accurate image upscaling with super-resolution forests,” In CVPR, 2015, Huang et al., “Single image super-resolution from transformed self-exemplars. In CVPR, 2015, Kim, et al., “Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016, Kim, et al., “Deeply-recursive convolutional network for image super-resolution. In CVPR, 2016, Lim, et al., “Enhanced deep residual networks for single image super-resolution,” In CVPRW, 2017, Tong, et al., “Image super-resolution using dense skip connection. In ICCV, 2017, Tai, et al., “Memnet: A persistent memory network for image restoration,” In ICCV, 2017, Zhang et al., “Learning a single convolutional super-resolution network for multiple degradations,” Inc CVPR, 2018, Zhang et al., “Image super-resolution using very deep residual channel attention networks,” In ECCV, 2018). Among these, one of the most fundamental architectures is EDSR (Lim, et al., “Enhanced deep residual networks for single image super-resolution, In CVPRW, 2017). However, these networks need expensive computation resources, which is the main bottleneck for their deployment on mobile devices. 
     Manually designed lightweight structures have been proposed (Sifre, et al., “Rigid-motion scattering for image classification, PhD theses, Citeseet, 2014, Howard, et al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, abs/1704.04861, 2017, Chollet, “Xception: Deep Learning with depthwise separable convolutions,” In CVPR, 2017, Iandola, et al., “Squeezenet: Alexnet-level accuracy with 50×fewer parameters and 0.5 mb model size,” ICLR, 2017, Kim, et al., “Accurate image super-resolution using very deep convolutional networks,” In CVPR, 2016, Mark Sandler, et al., Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. CVPR, 2018, Xiangyu Zhang, et al., “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” In CVPR, 2018, Ningning Ma, et al., “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” In ECCV, 2018, Andrew Howard, et al., “Searching for mobilenetv3,” In ICCV, 2019, Tero Karras, et al., “Progressive growing of gans for improved quality, stability, and variation,” submitted to ICLR 2018, 2017, Kai Han, et al., “Ghostnet: More features from cheap operations,” In CVPR, 2020). Among these structures, the most fundamental one is the depthwise convolution layer (Laurent Sifre and P S Mallat. “Rigid-motion scattering for image classification,” PhD thesis, Citeseer, 2014), which processes the spatial information with a single convolution on each input feature. A 1×1 convolution layer named pointwise layer is usually deployed around the depthwise convolution layer for the communication among channels (Andrew G Howard, et al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, abs/1704.04861, 2017, Mark Sandler, et al., “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” CVPR, 2018, Andrew Howard, et al., “Searching for mobilenetv3,” In ICCV, 2019, Xiangyu Zhang, et al., “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” In CVPR, 2018, Ningning Ma, et al., “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” In ECCV, 2018). However, such structures are not popular in the SISR due to their significant performance loss. CARN (Namhyuk Ahn, et al., “Fast, accurate, and lightweight super-resolution with cascading residual network,” In ECCV, 2018) tried to use a similar structure as MobileNet (Andrew G Howard, et al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, abs/1704.04861, 2017) on SISR in 2018. They utilized the group convolution to reduce the parameters, but they had to introduce a complicated recurrent method to improve the performance. As shown in  FIG.  4   , disclosed further below, the computation costs and parameters of CARN are not satisfied. Therefore, it is still a main challenge to effectively implement depthwise convolution based lightweight structures to the image super-resolution (SR) task. 
     Besides using lightweight operations, the computation costs can be alleviated by reducing the size of feature maps (Mingxing Tan and Quoc Le. “Efficientnet: Rethinking model scaling for convolutional neural networks,” In ICML, 2019, Andrew G Howard, et al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, abs/1704.04861, 2017, Francois Chollet. “Xception: Deep learning with depthwise separable convolutions,” In CVPR, 2017, Forrest N Iandola, et al., “Squeezenet: Alexnet-level accuracy with 50×fewer parameters and 0.5 mb model size,” ICLR, 2017, Mark Sandler, et al., “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” CVPR, 2018, Xiangyu Zhang, et al., “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” In CVPR, 2018, Ningning Ma, et al., “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” In ECCV, 2018, Andrew Howard, et al., “Searching for mobilenetv3,” In ICCV, 2019, Mingxing JTan and Quoc V Le. “Mixnet: Mixed depthwise convolutional kernels,” In BMVC, 2019, Kai Han, et al., “Ghostnet: More features from cheap operations,” In CVPR, 2020). Meanwhile, size-reduced features can also improve high-level representations by merging with higher-resolution features in many tasks (Ke Sun, et al., “Deep high-resolution representation learning for human pose estimation,” In CVPR, 2019, Jingdong Wang, et al., “Deep high-resolution representation learning for visual recognition,” IEEE transactions on pattern analysis and machine intelligence, 43(10):3349-3364, 2020). However, it is counter-intuitive to apply downsampling modules in SISR, since SISR is an upsampling task that restores information of a low-resolution image. In contrast, the downsampling operation usually causes significant information loss. (Muhammad Haris, et al., “Deep back-projection networks for super-resolution,” In CVPR, 2018) proposed an iterative error-correcting feedback mechanism that calculates both up- and down-projection errors to guide the reconstruction. Furthermore, Li et al. (Zhen Li, et al., “Feedback network for image super-resolution,” In CVPR, 2019) also proposed a framework that introduced the downsampling module into SISR to generate high-level representations. Their success shows the possibility of getting pleasing high-resolution images through downsampling operations. However, they still had to utilize a recurrent method to resist the performance drop, which heavily increased the parameters and computation costs. An example embodiment disclosed herein enables image SR to generate a reliable (accurate) high-level representation with reduced parameters and computation cost relative to conventional SR enabling such image SR to be implemented, for non-limiting example, on a mobile device, such as disclosed below with regard to  FIG.  1   . 
       FIG.  1    is block diagram of an example embodiment of an environment  100  in which a mobile device, namely a first mobile device  101   a , performs image super-resolution (SR) optionally within an embodiment disclosed herein. In the environment  100  there are two users, namely a first user  102   a  and a second user  102   b . The first user  102   a  and second user  102   b  are communicating, electronically, over a wireless connection  103 , via the first mobile device  101   a  and a second mobile device  101   b , respectively. The second user  102   b  has captured a high-resolution (HR) original image  104  of a butterfly  105  via the second mobile device  101   b  and uses the second mobile device  101   b  to transmit a low-resolution (LR) representation  106  of the HR original image  104  to the first mobile device  101   a  of the first user  102   a . It should be understood that an example embodiment disclosed herein is not limited to being performed by a mobile device, that a subject of an image is not limited to a butterfly, and that a HR original image is not limited to a picture captured by a camera. 
     Continuing with reference to  FIG.  1   , the first mobile device  101   a  is configured to perform image SR on the LR representation  106  received by the first mobile device  101   a  and output a reconstructed version  108  of the HR original image  104  to a display screen  109  of the first mobile device  101   a , for the first user  102   a  to view. It should be understood that the HR original image  104  is not limited to being output to a display screen and may, for non-limiting example, be output to an electronic file. 
     In the example embodiment of  FIG.  1   , the first mobile device  101   a  is configured to produce, via the image SR performed, the reconstructed version  108  of the HR original image  104  based on element-unshuffled downsampling of the LR representation  106 . The reconstructed version  108  is a higher quality version of the HR original image  104  relative to the LR representation  106 . The element-unshuffled downsampling enables the first mobile device  101   a  to advantageously perform the image SR with fewer parameters and less computation cost relative to conventional image SR, thereby reducing an amount of resources and power of the first mobile device  101   a  used for such an application. According to an example embodiment, a system may perform image SR based on such element-unshuffled downsample, as disclosed below with regard to  FIG.  2   . 
       FIG.  2    is a block diagram of an example embodiment of a system  210  for performing image super-resolution (SR). The system  210  comprises an element-unshuffled downsampler  212  and an image SR module  214 . The image SR module  214  is configured to perform image SR on a low-resolution (LR) representation  206  of a high-resolution (HR) original image  204 . The HR original image  204  is at a higher resolution relative to a resolution of the LR representation  206 . The image SR module  214  is further configured to produce a reconstructed version  108  of the HR original image  204  via the image SR performed. The image SR is based on element-unshuffled downsampling of the LR representation  206 . The element-unshuffled downsampler  212  is configured to perform the element-unshuffled downsampling. The image SR module  214  is further configured to output the reconstructed version  108  produced. 
     To perform the element-unshuffled downsampling, the element-unshuffled downsampler  212  may be further configured to perform an element-unshuffle operation (not shown). The element-unshuffle operation may enable the element-unshuffled downsampling that yields higher performance relative to a performance based on downsampling via a different downsampling operation, different from the element-unshuffled downsampling. Such element-unshuffled downsampling is disclosed further below with regard to equations (3)-(5). The higher performance may include higher image quality as disclosed further below. An example embodiment of a method that may perform image SR in such manner is disclosed below with regard to  FIG.  3   . 
       FIG.  3    is a flow diagram of an example embodiment of a method  300  for performing image super-resolution (SR). The method begins ( 302 ) and comprises performing image SR on a low-resolution (LR) representation of a high-resolution (HR) original image ( 304 ). The HR original image is at a higher resolution relative to a resolution of the LR representation. The image SR includes producing a reconstructed version of the HR original image based on element-unshuffled downsampling of the LR representation. The method further comprises outputting the reconstructed version produced ( 306 ) and the method thereafter ends ( 308 ) in the example embodiment. Further technical details are disclosed below. 
     An example embodiment disclosed herein includes an effective way to design a lightweight network with depthwise convolutions and downsampling operations. An example embodiment disclosed herein may include an effective module referred to as Self-Residual Depthwise Separable Convolution to overcome the drawback in Depthwise Separable Convolution (DSC) (Andrew G Howard, et al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, abs/1704.04861, 2017) without any additional parameters. Previous explorations on downsampling features include (Ke Sun, et al., “Deep high-resolution representation learning for human pose estimation,” In CVPR, 2019, Muhammad Haris, et al., “Deep back-projection networks for super-resolution,” In CVPR, 2018, Zhen Li, et al., “Feedback network for image super-resolution,” In CVPR, 2019). In contrast, an example embodiment disclosed herein includes an element-unshuffled downsampler, such as the element-unshuffled downsampler  212  of  FIG.  2   , disclosed above, configured to perform an element-unshuffle operation and may include max-pooling and group convolution to further enhance the performance of DSC with similar computation costs as depthwise convolution. Such an example embodiment is disclosed further below with regard to  FIG.  5   . The element-unshuffle operation is the reverse (inverse) operation of pixel-shuffle (Wenzhe Shi, et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” In CVPR, 2016). Moreover, an example embodiment disclosed herein includes a practical, lightweight module referred to as an Element-Unshuffled Block (EUB) that may include the element-unshuffled downsampling and the Self-Residual DSC (SRDSC). An example embodiment disclosed further below may replace one Self-Residual DSC in the EUB with a standard convolution layer and construct a Hybrid Element-Unshuffled Network (HEUN) to achieve the state-of-the-art (SOTA) performance and slightly increase the performance number of the HEUN to beyond the SOTA performance. The overall comparison is shown in  FIG.  4   , disclosed below. 
       FIG.  4    is a graph  400  with an exemplification of peak signal-to-noise ratio (PSNR) vs. a number of Multi-Adds  422  for different SR methods. The graph  400  provides an illustration of the overall comparison on the Urban 100  dataset with ×4 scale. An example embodiment of HEUN-L, disclosed further below, achieves the best trade-off among the PSNR  420 , parameters  424 , and Multi-Adds  422 . In the graph  400 , the “+” symbol means the PSNR  420  results are generated with self-ensemble. Example embodiments are summarized below for non-limiting example and disclosed in detail, further below.
         an example embodiment of the Self-Residual DSC may be used to overcome the defects of the depthwise convolution in the SISR task with a simple and effective operation, which barely needs computation and additional parameters.   an example embodiment may employ a downsampling module with the element-unshuffle operation, which is useful to enhance the performance.   an example embodiment of a lightweight module, referred to interchangeably herein as EUB, may be employed with an example embodiment of a Self-Residual DSC and the element-unshuffled downsampler, which can provide reliable performance with fewer parameters and computation costs compared to conventional SR.   an example embodiment of the Hybrid Element-Unshuffled Block (EPUB) may include integrating the standard convolution into the EUB and constructing the effective and efficient HEUN to achieve a new SOTA performance with fewer parameters and Multi-Adds compared to the baselines.       

     Further, a relationship between PSNR and the Normalized Mean Error (NME) among the shallow features and deep features based on an ablation study is disclosed herein, which may be valuable in designing a network for SISR. Details regarding same are disclosed further below in Section 4.2. An overview of SR and deep lightweight structure for use in same is disclosed below. 
     2 Super Resolution and Deep Lightweight Structure 
     2.1 Super Resolution 
     Deep Super Resolution. An end-to-end mapping between the interpolated LR images and their HR counterparts was first established by SRCNN (Chao Dong, et al., “Learning a deep convolutional network for image super-resolution,” In ECCV, 2014). The SRCNN was further improved by its successors with advanced network architectures (Jiwon Kim, et al., “Accurate image super-resolution using very deep convolutional networks,” In CVPR, 2016, Kai Zhang, et al., “Learning deep cnn denoiser prior for image restoration,” In CVPR, 2017). As studied in (Chao Dong, et al. “Accelerating the super-resolution convolutional neural network,” In ECCV, 2016), computational costs are quadratically increased by this upsampling operation in data preprocessing. To solve the problem, an efficient sub-pixel convolution layer that upsampled the last LR feature maps to HR was introduced in ESPCN (Wenzhe Shi, et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” In CVPR, 2016). It was also adopted in residual-learning networks SRResNet (Christian Ledig, et al., “Photo-realistic single image super-resolution using a generative adversarial network,” In CVPR, 2017) and EDSR (Bee Lim, et al., “Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017). The performance of the SISR was then further improved by stacking more blocks with dense residuals (Ke Zhang, et al., “Residual networks of residual networks: Multilevel residual networks,” TCSVT, 2017, He Zhang and Vishal M Patel. “Densely connected pyramid dehazing network,” In CVPR, 2018, Yulun Zhang, et al., “Residual dense network for image super-resolution,” In CVPR, 2018, Yulun Zhang, et al., “Residual non-local attention networks for image restoration,”. In ICLR, 2019). Lightweight Super Resolution. LapSRN (Wei-Sheng Lai, et al., “Deep laplacian pyramid networks for fast and accurate super-resolution,” In CVPR, 2017) reduced the computation complexity by removing the bicubic interpolation before prediction. Inspired by Lap-SRN, a lot of works started to reconstruct the HR image from the origin LR input. Recursive learning was first introduced by DRCN (Jiwon Kim, et al., “Deeply-recursive convolutional network for image super-resolution,” In CVPR, 2016). Then it was widely used to reduce the parameters with weight sharing strategy (Ying Tai, et al., “Image super-resolution via deep recursive residual network,” In CVPR, 2017, Ying Tai, et al., “Memnet: A persistent memory network for image restoration,” In ICCV, 2017, Muhammad Haris, et al., “Deep back-projection networks for super-resolution,” In CVPR, 2018, Namhyuk Ahn, et al., “Fast, accurate, and lightweight super-resolution with cascading residual network,” In ECCV, 2018, Zhen Li, et al., “Feedback network for image super-resolution,” In CVPR, 2019). Besides the recurrent method, IDN (Zheng Hui, et al., “Fast and accurate single image super-resolution via information distillation network,”. In CVPR, 2018) and CARN (Namhyuk Ahn, et al., “Fast, accurate, and lightweight super-resolution with cascading residual network,” In ECCV, 2018) introduced the group convolution for the lightweight purpose. Further to the success of the residual operation in SISR, many works (Zheng Hui, et al., “Fast and accurate single image super-resolution via information distillation network,”. In CVPR, 2018, Zheng Hui, et al., “Lightweight image super-resolution with information multi-distillation network,” In ACMMM, 2019, Xiaotong Luo, et al., “Latticenet: Towards lightweight image super-resolution with lattice block,” In ECCV, 2020) adopted the residual into their lightweight design to keep the performance. A recent work named SMSR (Longguang Wang, et al., “Exploring sparsity in image super-resolution for efficient inference,” In CVPR, 2021) reduced the parameters and computation costs with pruning. Different from SMSR, an example embodiment disclosed herein may include a design of the lightweight network which can be further improved by pruning. 
     2.2 Deep Lightweight Structure 
     As the deep-learning models become deeper and larger, many researchers have been working on the lightweight networks. A faster activation function named rectified-linear activation function (ReLU) was proposed to accelerate the model in (Xavier Glorot, et al., “Deep sparse rectifier neural networks,” In AISTATS, 2011). A flattened CNN architecture that accelerated the feeding forward was presented in (Jonghoon Jin, et al., “Flattened convolutional neural networks for feedforward acceleration,” CoRR, 2014). Depthwise separable convolution was first proposed in (Laurent Sifre and P S Mallat. “Rigid-motion scattering for image classification,” PhD thesis, Citeseer, 2014) and was widely adopted in Inception models (Sergey Ioffe and Christian Szegedy. “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” In ICML, 2015), Xception net-work (Francois Chollet. “Xception: Deep learning with depthwise separable convolutions,” In CVPR, 2017), MobileNets (Andrew G Howard, et al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, abs/1704.04861, 2017, Mark Sandler, et al., “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” CVPR, 2018), ShuffleNets (Xiangyu Zhang, et al., “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” In CVPR, 2018, Ningning Ma, et al., “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” In ECCV, 2018) and CondenseNet (Gao Huang, et al., “Condensenet: An efficient densenet using learned group convolutions,” In CVPR, June 2018). Besides manually designed lightweight architectures, researchers proposed to use Neural Architecture Search (NAS) to find the optimal lightweight network (Hanxiao Liu, et al., “Darts: Differentiable architecture search,” In ICLR, 2019, Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. “Learning transferable architectures for scalable image recognition,” In CVPR, 2018, Han Cai, Ligeng Zhu, and Song Han. “Proxylessnas: Direct neural architecture search on target task and hardware,” In ICLR, 2019, Bichen Wu, et al., “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” In CVPR, 2019, Andrew Howard, et al., “Searching for mobilenetv3,” In ICCV, 2019, Mingxing JTan and Quoc V Le. “Mixnet: Mixed depthwise convolutional kernels,” In BMVC, 2019). All these networks are constructed based on the depthwise convolution as well. Thus, it is useful to explore an effective way to implement the depthwise convolution on SISR. An example embodiment disclosed herein includes a downsampling module which can significantly enhance the performance based on the depthwise convolution, such as the element-unshuffled downsampler  212 , disclosed above with regard to  FIG.  2   , and below with regard to  FIG.  5   . 
       FIG.  5    is a block diagram of an example embodiment of an image SR module  514 . With reference to  FIG.  2    and  FIG.  5   , to perform the element-unshuffled downsampling, the element-unshuffled downsampler  212  may be further configured to perform an element-unshuffle operation  512 . The element-unshuffle operation  512  may include downsampling input features  511 . The input features  511  may include elements from a transformed version (not shown) of the LR representation  206 . Such a transformed version may be produced via convolution, such as disclosed further below with regard to  FIG.  8   . 
     Continuing with reference to  FIGS.  2  and  5   , the downsampling may include reducing a size of the input features  511  by separating the input features  511  into sub-features  519 . To separate the input features  511  into the sub-features  519 , the element-unshuffled downsampler  212  may be further configured to select a subset of elements from an input feature  511 - 1  of the input features  511  and create a sub-feature (e.g.,  519 - 1 , . . . , or  519 - n ) of the sub-features  519  by grouping the subset of elements selected. 
     As such, the element-unshuffled downsampler  212  may be further configured to produce, via the element-unshuffled downsampling, a plurality of sub-features, namely the sub-features  519 - 1 , . . . , and  519 - n , from the input features  511 . For non-limiting example, to perform the image SR, the image SR module ( 214 ,  514 ) may be further configured to perform a max-pooling operation (not shown) on the sub-features  519  to produce a plurality of pooled sub-features (not shown). The image SR module ( 214 ,  514 ) may be further configured to convolve, using group convolution, pooled sub-features of the plurality of pooled sub-features. Such convolving may be performed via the convolution operation  516 . The convolving may output low-frequency features  521 . The low-frequency features  521  may be at a lower frequency relative to a frequency of the input features  511 . The image SR module ( 214 ,  514 ) may be further configured to upsample  518  the low-frequency features  521  output from the convolving to produce up-sampled low-frequency features  523 . The image SR module ( 214 ,  514 ) may be further configured to produce enhanced features  520  by adding, via an adder  525 , the up-sampled low-frequency features  523  to the input features  511 . With reference to  FIGS.  1 ,  2 , and  5   , the image SR module ( 214 ,  514 ) may be further configured to produce the reconstructed version  108  based on the enhanced features  520  produced. 
     3 Proposed Method—Example Embodiment 
     An example embodiment disclosed herein may include a lightweight structure called Hybrid Element-Unshuffled Block (HEUB) to replace the traditional Residual Convolution Block, which is shown in  FIG.  6 A . 
       FIG.  6 A  is a block diagram of a standard residual block  601 . The standard residual block  601  includes a first convolution (Conv) layer  632  followed by a rectified linear activation unit (ReLU) layer  634 , second Conv layer  632   a , and an adder  636  configured to produce a residual block output  637  of the standard residual block  601  by adding an input  631 , input to the first Conv Layer  632   a , to an output  635  from the second Conv layer  632   b.    
       FIG.  6 B  is a block diagram of a depthwise separable convolution (DSC) residual block  605  constructed by DSC. As such, the DSC residual block  605  includes a first depthwise convolution (D-Conv) layer  638   a  followed by a first pointwise convolution (P-Conv) layer  640   a , ReLU layer  644 , second D-Conv layer  638   b , second P-Conv layer  640 B, and an adder  646  configured to produce a residual block output  647  of the DSC residual block  605  by adding an input  651 , input to the first D-Conv layer  638   a , to an output  645  from the second P-Conv layer  640   b.    
       FIG.  6 C  is a block diagram of an example embodiment of a self-residual DSC residual block  607  that is based on an example embodiment of Self-Residual DSC disclosed herein. In contrast to conventional residual blocks, such as the standard residual block  601  and DSC residual block  605  of  FIG.  6 A  and  FIG.  6 B , respectively, the Self-Residual DSC residual bock  607  includes a first D-Conv layer  658   a  followed by a first adder  656   a  configured to add an input  651  to the first D-Conv layer  658   a  to produce a first sum  660   a  that is input to a first P-Conv layer  650   a . The first P-Conv layer  650   a  is followed by a ReLU layer  654 . A ReLU layer output  655  from the ReLU layer  654  is output to a second D-Conv layer  658   b  and a second adder  656   b . The second adder  656   b  is configured to produce a second sum  662   a  by adding the ReLU layer output  655  to a D-Conv output  657 , output from the second D-Conv layer  658   b . The second sum  662   a  may be input to a second P-Conv layer  650   b  and a third adder  656   c  may be configured to produce a residual block output  660  of the residual block  607  by adding an output  659  from the second P-Conv layer  650   b  to the input  651 , input to the first D-Conv layer  658   a.    
     An example embodiment of a proposed method disclosed herein may include three parts: a standard convolution layer, the proposed element-unshuffled downsampling, and the proposed EUB. The EUB may be an integration of the element-unshuffled downsampling and the Self-Residual DSC, disclosed above with regard to  FIG.  6 C . Therefore, this section is organized as follows: first, details of the Self-Residual DSC are introduced; the element-unshuffled downsampling will be introduced in the second sub-section; after which the details of the HEUN are presented. 
     3.1 Self-Residual DSC 
     DSC. Depthwise separable convolution (DSC) is composed by a depthwise layer and a pointwise layer as shown in  FIG.  6 B , disclosed above. The depthwise layer uses a single kernel for each input feature map. DSC is a popular lightweight module to reduce the redundant operations in the standard convolution. The conversion from the standard convolution to the DSC can be described as: 
         F   out   =C ( F   in )≈ P ( D ( F   in )),  (1)
 
     where F out  means the output features, C represents the standard convolution, F in  means the input features, D means the depthwise convolution, and P means the pointwise convolution. Depthwise convolution is the major part to process the spatial information of the input features, which needs far fewer parameters and computation costs than the standard convolution. 
     Self-Residual DSC. Self-Residual DSC may have a significant side effect on the performance of SISR since SISR needs to enrich the information. The side effect is disclosed further below in Section 4.2. To overcome the defects brought by the depthwise layer and to keep its ability to process the spatial information, an example embodiment includes a balanced trade-off design by simply adding the input before the depthwise layer to the output of the depthwise layer as shown in  FIG.  6 C , disclosed above. The whole structure is described as: 
         F   out   =P ( D ( F   in )+ F   in ).  (2)
 
     Comparing Equation (1) and Equation (2), one can easily figure out that the outputs of the Self-Residual DSC have more similarity to the inputs than the outputs of the DSC. An analysis of the importance of the similarity is provided in Section 4.2, further below. The self-residual does not introduce any additional parameters. Further, the additional computation costs of the addition operation can be ignored. 
     3.2 Element-Unshuffled Downsampling 
     Details regarding the element-unshuffled downsampling (EUD), which is shown in  FIG.  7 A , are disclosed in this section. 
       FIG.  7 A  is a block diagram of an example embodiment of Element-Unshuffled Downsampling (EUD)  770 . 
       FIG.  7 B  is a block diagram of an example embodiment of an Element-Unshuffled Block (EUB)  780 . 
       FIG.  7 C  is a block diagram of an example embodiment of a Hybrid Element-Unshuffled Block (HEUB)  790 . It should be noted that a group number of the depthwise convolution  716  in the EUD  770  is equal to the number of its inputs. 
     As disclosed in previous sections, low-frequency features can enhance the high-level representations (Ke Sun, et al., “Deep high-resolution representation learning for human pose estimation,” In CVPR, 2019, Jingdong Wang, et al., “Deep high-resolution representation learning for visual recognition,” IEEE transactions on pattern analysis and machine intelligence, 43(10):3349-3364, 2020). In the work (Jingdong Wang, et al., “Deep high-resolution representation learning for visual recognition,” IEEE transactions on pattern analysis and machine intelligence, 43(10):3349-3364, 2020), it is explored that repeating multi-resolution fusions can boost the high-resolution representations with the help of the low-resolution representations in image segmentation tasks. However, previous SR works (Muhammad Haris, et al., “Deep back-projection networks for super-resolution,” In CVPR, 2018, Zhen Li, et al., “Feedback network for image super-resolution,” In CVPR, 2019) took a lot of effort to use the low-frequency features in SISR with a heavy recurrent method. An example embodiment disclosed herein provides a more efficient way to utilize the low-frequency features with single forward inference for the SISR task. The proposed method is shown in  FIG.  5   , disclosed above, and in further detail below. 
     With reference to  FIG.  5   , an example embodiment of an element-unshuffled downsampler is disclosed. Note that the notation of the Cony Operation  516  in  FIG.  5    is a general operation. Disclosure with regard to the best operations is provided in the following sections. In this subsection, the element-unshuffle  512  operation is introduced. The focus then turns to exploring the most effective operations after the element-unshuffle  512  operation. 
     Element-unshuffle. The element-unshuffle  512  operation is a reverse (inverse) operation of pixel-shuffle (Wenzhe Shi, et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” In CVPR, 2016). As shown in  FIG.  5   , it divides (separates) a feature, such as the feature  517 , into several sub-features  519 , whose number in  FIG.  5    is four for non-limiting example. Four different colors/patterns are used to represent the different selections of sub-features  519  in  FIG.  5   . As shown in the  FIG.  5   , the sub-features  519  contain the complete information of the original features, that is, the input features  511 , but with lower resolution. Therefore, such sub-features  519  may be used to avoid information loss while reducing the size of the features, namely the input features  511 . 
     Element-Unshuffled Downsampling. With reference to  FIG.  5    and  FIG.  7 A , after the element-unshuffle ( 512 ,  712 ) operation, an efficient and effective structure may be used to process the low-resolution features. As shown in  FIG.  7 A , a max-pooling layer  713  may be employed after the element-unshuffle ( 512 ,  712 ). As such, a powerful non-linear operation is employed before the convolution  516  operation to extract better local features. Therefore, the max-pooling layer  713  is chosen instead of the average-pooling layer. The process may be described as: 
         F   out   =M ( F   in ), i∈{ 1,2,3,4}, j∈{ 1 . . .  n},   (3)
 
     After the non-linear operation, a group convolution, namely the D-Conv 716 layer, may be employed to reduce the channel of the input, which is actually a downsampling operation. The process can be described as: 
         F   out   =G ( F   in   ,F   in   ,F   in   ,F   in ), jε{ 1 . . .  n},   (4)
 
     To enhance the feature, an upsampler  718  may perform the upsample  518  operation that may be used to project the low-frequency features to high dimension, and an adder ( 520 ,  720 ) may be employed to add them to the original input features  511 . After that, a pointwise convolution  723  may be utilized for the communication among the channels. The process can be described as: 
         F   out   =P ( U ( F   in )+ L ),  (5)
 
     where U stands for the upsampling function, F in  means the input channels to the upsampler  718 , and L means the original input features  511 . An example embodiment may use a bi-linear upsampler. Experiments with regard to same are described in Section 4.2 further below. 
     3.3 Hybrid Element-Unshuffled Network 
     Element-Unshuffled Block. After the exploration of the Self-Residual DSC and the element-unshuffled downsampling, the lightweight Element-Unshuffled Block (EUB)  780  of  FIG.  7 B  is introduced. The EUB  780  is composed of the Self-Residual DSC  707 , disclosed above with regard to the Self-Residual DSC  607  of  FIG.  6 C , and the element-unshuffled downsampling  770  of  FIG.  7 A . The details of the EUB  780  are shown in  FIG.  7 B . The block can be represented as. 
         F   out   =P ( D (σ( EUD ( F   in )))+σ( EUD ( F   in )))+ F   in ,  (6)
 
     where the EUD denotes the whole procedure of the element-unshuffled downsampling, and a represents the ReLU (Xavier Glorot, et al., “Deep sparse rectifier neural networks,” In AISTATS, 2011) included as the ReLU  774  in the EUB  780 . 
     Hybrid Element-Unshuffled Block. To further improve the performance, an example embodiment integrates the standard convolution into the proposed EUB  780 , and a result of such integration may be referred to herein as a Hybrid Element-Unshuffled Block (HEUB). An example embodiment of HEUB  790  is shown in  FIG.  7 C . The kernel size of the standard convolution layer  792  is set to 3 for non-limiting example for the trade-off between the performance and the efficiency. The kernel setting of the rest modules is the same as for the EUB  780 , which will be demonstrated in Section 4.2, disclosed further below. 
     Hybrid Element-Unshuffled Network. The HEUB  790  may be used to construct an example embodiment of a Hybrid Element-Unshuffled Network (HEUN). The network is similar to EDSR (Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017). Since one HEUB has two residual blocks, an example embodiment may, for non-limiting example, construct the body parts with 8 HEUB to align the settings in EDSR. To further reduce the parameters, an example embodiment may use the tail of IMIDN (Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACMMM, 2019). An example embodiment of the architecture is shown in  FIG.  8   , disclosed below. 
       FIG.  8    is a block diagram of an example embodiment of a Hybrid Element-Unshuffled Network (HEUN) architecture  800 . The HEUN architecture  800  may be used to produce a reconstructed version  808  of a high-resolution (HR) original image (not shown) from a low-resolution (LR) representation  806  of the original image. The HEUN architecture  800  is based on EDSR (Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017), and the tail  833  is from IMDN (Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACMMM, 2019). The LR representation  806  may be transformed via convolution  843  to produce input features  811  from the transformed version. In the body  815  of the HEUN architecture  800 , a total number of HEUBs ( 890 - 1 ,  890 - 2 , . . . ,  890 -N) in the HEUN architecture  800  is no less than 8. The total number of HEUBs may be controlled for different sizes of the model. For example, HEUN-Medium (HEUN-M) has 8 HEUBs in total, and HEUN-Large (HEUN-L) has 12 HEUBs. The total number of HEUBs in HEUN-Small (HEUN-S) is 8, however, one HEUB is replaced with two EUBs. The upsampler for the final high-resolution output is the pixel-shuffle module (Wenzhe Shi, et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” In CVPR, 2016). 
     4 Experimental Results 
     4.1 Settings 
     Datasets and Metrics. Following (Song Han, et al., “Learning both weights and connections for efficient neural network,” In NeurIPS, 2015, Radu Timofte, et al., “Ntire 2017 challenge on single image super-resolution: Methods and results,” In CVPRW, 2017, Bee Lim, et al., “Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017, Kai Zhang, Wangmeng Zuo, and Lei Zhang. “Learning a single convolutional super resolution network for multiple degradations.” In CVPR, 2018), the dataset DIV2K (Radu Timofte, et al., “Ntire 2017 challenge on single image super-resolution: Methods and results,” In CVPRW, 2017) and Flickr2K (Bee Lim, et al., “Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017) was used as training data. Five standard benchmark datasets were used for testing: Set 5  (Marco Bevilacqua, et al., “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” In BMVC, 2012), Set 14  (Roman Zeyde, et al., “On single image scale-up using sparse-representations,” In Proc. 7th Int. Conf. Curves Surf., 2010), B 100  (David Martin, et al. “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” In ICCV, 2001), Urban 100  (Jia-Bin Huang, et al., “Single image super-resolution from transformed self-exemplars,” In CVPR, 2015), and Manga 109  (Yusuke Matsui, et al., “Sketch-based manga retrieval using manga 109  dataset,” Multimedia Tools and Applications, 2017). The SR results are evaluated with PSNR and SSIM (Zhou Wang, et al., “Image quality assessment: from error visibility to structural similarity,” TIP, 2004) on Y channel (i.e. luminance) of transformed YCbCr space. Following the work (Kai Zhang, Wangmeng Zuo, and Lei Zhang. “Learning a single convolutional super resolution network for multiple degradations.” In CVPR, 2018, Yulun Zhang, et al., “Residual non-local attention networks for image restoration,”. In ICLR, 2019), the degradation is bicubic downsampling by adopting the MATLAB® function imresize with the option bicubic (denote as BI for short). The BI model was used to simulate LR images with scaling factor 2, 3, and 4. In addition, a comparison of the parameters and Multi-Adds was made to evaluate the spatial and time complexity. 
     Training Setting. Following settings of (Bee Lim, et al., “Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017), in each training batch, 16 LR RGB patches were randomly extracted with the size of 48×48 as inputs. The patches were randomly augmented by flipping horizontally or vertically and rotating 90°. There are 14,200 iterations in one epoch. An example embodiment of HEUN was implemented with the PyTorch (Adam Paszke, et al., “Pytorch: An imperative style, high-performance deep learning library,” In NeurIPS, 2019) and updated with Adam optimizer (Adam Paszke, et al., “Pytorch: An imperative style, high-performance deep learning library,” In NeurIPS, 2019). The learning rate was initialized to 2×10-4 for all layers and follows the cosine scheduler with 250 epochs in each cycle. Some experiments used the step scheduler and will be emphasized in the caption for fair comparison. 
     4.2 Ablation Study 
     The effectiveness of the Self-Residual DSC is demonstrated first. Then, the enhancement of the element-unshuffled downsampling is shown. A set of experiments are implemented to figure out the best operation in the element-unshuffled downsampling. Further, the best setting of the kernel size in the EUB is explored. At last, the features are visualized and intuition disclosed. 
     
       
         
           
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                 Experiment results of different components in DSC generated  
               
               
                 with step schedule. Conv represents standard convolution. 
               
            
           
           
               
               
               
               
            
               
                   
                 PSNR 
                   
                   
               
               
                 Methods 
                 (Set5) 
                 Params 
                 Multi-Adds 
               
               
                   
               
               
                 Conv Res-Block 
                 38.05 
                 1343K 
                 309.9 G 
               
               
                 Conv + DSC 
                 37.88 
                  701K 
                 163.6 G 
               
               
                 Conv + P-Conv 
                 37.90 
                  690K 
                 161.5 G 
               
               
                 Conv + Self-Residual DSC 
                 37.91 
                  701K 
                 163.6 G 
               
               
                 DSC only 
                 37.65 
                  207K 
                  44.9 G 
               
               
                 P-Conv only 
                 37.36 
                  181K 
                  40.7 G 
               
               
                 Self-Residual DSC only 
                 37.73 
                  207K 
                  44.9 G 
               
               
                   
               
            
           
         
       
     
     Effectiveness of the Self-Residual DSC. From Table 1, disclosed above, it can be observed that the combination of standard convolution and DSC gets worse PSNR than the combination of standard convolution and the pointwise convolution. Therefore, a conclusion can be drawn that the depthwise convolution will obstruct the accuracy of the image reconstruction in the DSC. However, the depthwise convolution may not be abandoned for a design of the lightweight network without standard convolution. The results presented in Table 1 and  FIG.  9   , disclosed below, show that an example embodiment of Self-Residual DSC disclosed herein can overcome the defects of the depthwise convolution with a simple residual with barely no additional computation costs and parameters. 
       FIG.  9    is a graph  900  with an exemplification of a comparison among networks. The graph includes plots of PSNR  971  vs. epochs  973 . The comparison is among the networks constructed by pointwise convolution  969 , element-unshuffled downsampling (EUD)  970 , baseline DSC  975 , the Self-Residual DSC  907 , and the element-unshuffled block (EUB)  980 . 
     Effectiveness of the element-unshuffled downsampling. Six experiments were run to find the best combination of the pooling layer and the upsampler. The results are shown in Table 2, disclosed below. 
     
       
         
           
               
             
               
                 TABLE 2 
               
               
                   
               
               
                 The experiment results of different combination of  
               
               
                 pooling layers and upsamplers. The results are generated  
               
               
                 with cosine scheduler on Set5 with ×2 scale. 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
               
               
               
               
               
               
            
               
                 Nearest 
                 X 
                 ✓ 
                 ✓ 
                 X 
                 ✓ 
                 X 
                 X 
               
               
                 Bi-linear 
                 ✓ 
                 X 
                 X 
                 ✓ 
                 X 
                 ✓ 
                 ✓ 
               
               
                 Max-pooling 
                 X 
                 X 
                 ✓ 
                 ✓ 
                 X 
                 X 
                 X 
               
               
                 Avg-pooling 
                 ✓ 
                 ✓ 
                 X 
                 X 
                 X 
                 X 
                 ✓ 
               
               
                 PSNR 
                 37.83 
                 37.82 
                 37.85 
                 37.87 
                 37.82 
                 37.83 
                 37.83 
               
               
                   
               
            
           
         
       
     
     From Table 2, disclosed above, it can be observed that the model with max-pooling layer and bi-linear upsampling can achieve the best performance among all combinations. The performance of the element-unshuffled downsampling with other kinds of downsampling operations was also compared. The results are shown in  FIG.  10 A . 
       FIG.  10 A  is a bar chart  1000 A of an exemplification of a comparison of different combinations of up- and down-scale operations. The bar chart  1000 A includes PSNR  1071  values per up-sampler type  1072 . The results are from an ablation study of different combinations of up- and down-scale operations. The results were conducted with a step scheduler and should not be compared with other experiments disclosed herein. 
       FIG.  10 B  is a graph  1000 B of an exemplification of results for different kernel settings. Different settings on kernels of depthwise layer were employed in the EUB. The PSNR  1073  was calculated on Set 5  with ×2 scale. The graph  1000 B compares PSNR  1073  over epochs  1074  for the EUD module, depthwise convolution with 2 strides, max pooling, and average pooling, with both bi-linear and nearest upsampling. As shown in  FIG.  10 B , the element-unshuffled downsampling is far beyond others. 
     Further, a network was constructed with only element-unshuffled downsampling and its performance compared with the networks constructed by the baseline DSC, the Self-Residual DSC, and the EUB. The results are shown in  FIG.  9   , disclosed above. The network constructed with only element-unshuffled downsampling performs far worse than the baseline DSC. However, the network of EUB can perform better than the Self-Residual DSC. Specifically, the network constructed by EUB achieves 0.04 dB higher than the Self-Residual DSC only. The results show that the element-unshuffled downsampling can enhance the performance in combination with other spatial operations. 
     Ablation Study of the EUB. Some experiments were run to explore the impact brought by different settings of kernels. The results are shown in  FIG.  10 B , disclosed above. The performance was compared among the settings 1-3-1-3, 1-5-1-5, 3-3-3-3, 3-5-3-5, 1-3-3-3, and 1-3-3-5. From the  FIG.  10 B , it can be observed that the setting 1-3-3-5 performs superior to all other settings. Therefore, an example embodiment may use 1-3-3-5 as the kernel settings of the EUB. 
     Intuition. For the further exploration, heatmap features were generated using the Normalized Mean Error (NME) among their head features and body features. The NME can be described as NME=1/N∥F H −F B ∥ F , where N means the total number of the elements in the features, F H  means the output features from the head block, F B  means the output features from the body block, and ∥·∥F denotes the Frobenius norm. The relationship was plotted between the PSNR and NME for the network constructed with pointwise convolutions, the network with DSC, the network with Self-Residual DSC, the network with element-unshuffled downsampling, HEUN-S, HEUN-M, and HEUN-L. The results are presented in  FIGS.  11 A-F , disclosed below. Also shown is the result of single image besides the mean results. 
       FIGS.  11 A-F  are charts ( 1100 A-F) with exemplifications of a relationship between PSNR  1181  and the NME  1183 . The chart  1100 A of  FIG.  11 A  was generated on the whole Set 5  dataset with ×2 scale. The results of  FIGS.  11 B-F  were generated for the image of a baby, butterfly, bird, head, and woman in the Set 5 , as shown in the charts  100 B,  1100 C,  1100 D,  1100 E, and  1100 F, respectively. 
     From  FIGS.  11 A-F , it can be observed that the NME of the pointwise structured network is much lower than the DSC baseline network, which means the main change of the features is brought by the depthwise convolution. The PSNR results of the pointwise structured network and element-unshuffled downsampler structured network show that limited changes will decrease the performance. However, the DSC network has large NME but limited PSNR, which means the larger NME does not represent the better performance. Although the chart generated from the bird, namely the chart  1100 D of  FIG.  11 D , indicates that there exists some variance, the conclusion still stands. Also visualized is the NME result among the head features and body features of these networks, respectively. The visualization results are shown in  FIGS.  12 A- 1  through  12 E- 8   , disclosed below. 
       FIG.  12 A- 1    is an exemplification of an image  1200 A- 1  of a baby. 
       FIGS.  12 A- 2  through  12 A- 8    are exemplifications of NME visualizations for the image  1200 A- 1  of  FIG.  12 A- 1   .  FIGS.  12 A- 2  through  12 A- 8    are exemplifications of the NME visualizations of Pointwise Convolution, DSC, Self-Residual DSC (SRDSC), element-unshuffled downsampling, HEUN-S, HEUN-M, and HEUN-L, respectively. The brighter color means a higher NME value. The features are binarized for better visualization. 
       FIG.  12 B- 1    is an exemplification of an image  1200 B- 1  of a butterfly. 
       FIGS.  12 B- 2  through  12 B- 8    are exemplifications of NME visualizations for the image  1200 B- 1  of  FIG.  12 B- 1   .  FIGS.  12 B- 2  through  12 B- 8    are exemplifications of the NME visualizations of Pointwise Convolution, DSC, SRDSC, element-unshuffled downsampling, HEUN-S, HEUN-M, and HEUN-L, respectively. The brighter color means a higher NME value. The features are binarized for better visualization. 
       FIG.  12 C- 1    is an exemplification of an image  1200 C- 1  of a bird. 
       FIGS.  12 C- 2  through  12 C- 8    are exemplifications of NME visualizations for the image  1200 C- 1  of  FIG.  12 C- 1   .  FIGS.  12 C- 2  through  12 C- 8    are exemplifications of the NME visualizations of Pointwise Convolution, DSC, SRDSC, element-unshuffled downsampling, HEUN-S, HEUN-M, and HEUN-L, respectively. The brighter color means a higher NME value. The features are binarized for better visualization. 
       FIG.  12 D- 1    is an exemplification of an image  1200 D- 1  of a head. 
       FIGS.  12 D- 2  through  12 D- 8    are exemplifications of NME visualizations for the image  1200 D- 1  of  FIG.  12 D- 1   .  FIGS.  12 D- 2  through  12 D- 8    are exemplifications of the NME visualizations of Pointwise Convolution, DSC, SRDSC, element-unshuffled downsampling, HEUN-S, HEUN-M, and HEUN-L, respectively. The brighter color means a higher NME value. The features are binarized for better visualization. 
       FIG.  12 E- 1    is an exemplification of an image  1200 E- 1  of a woman. 
       FIGS.  12 E- 2  through  12 E- 8    are exemplifications of NME visualizations for the image  1200 E- 1  of  FIG.  12 E- 1   .  FIGS.  12 E- 2  through  12 E- 8    are exemplifications of the NME visualizations of Pointwise Convolution, DSC, SRDSC, element-unshuffled downsampling, HEUN-S, HEUN-M, and HEUN-L, respectively. The brighter color means a higher NME value. The features are binarized for better visualization. 
     Thus, it can be concluded that the EUD can significantly reduce the NME among the shallow features and deep features. Adjusting the number of the modules will improve the performance of the architecture. Further, it can be observed that the NME among the head and body features gets smaller with an integration of the standard convolution into the EUB, comparing the heatmap features of HEUN-S and HEUN-M. Considering the NME of the pointwise structured network, it is natural to think about the communication among the features can also help to learn the similarity among the features. 
     From the scatter figure of mean results, it was noticed that the performance increases explosively with the increase of the NME at first. Then the performance starts to drop after the NME surpasses the value around 0.007. Further, the NME gets smaller by increasing the number of element-unshuffled downsamplers. Therefore, it can be concluded that there may exist an optimal NME value, and increasing element-unshuffled downsamplers or adding residuals will reduce the NME of the network towards the optimal. Such intuition can help in a design of the network structure or in applying the pruning strategy on SISR. However, it was also noticed that the optimal NME is variable with the inputs. More experiments are needed to validate the conclusion for different tail structures and datasets in the future. 
     4.3 Comparison Results 
     Simulating a LR image with a BI degradation model is widely used in image SR settings. For the BI degradation model, an example embodiment of a HEUN network was compared with 12 state-of-the-art SR methods: SRCNN (Chao Dong, et al., “Image super-resolution using deep convolutional networks,” TPAMI, 2016), VDSR (Jiwon Kim, et al., “Accurate image super-resolution using very deep convolutional networks,” In CVPR, 2016), DRCN (Jiwon Kim, et al., “Deeply-recursive convolutional network for image super-resolution,” In CVPR, 2016), DRRN (Ying Tai, et al., “Image super-resolution via deep recursive residual network,” In CVPR, 2017), LapSRN (Wei-Sheng Lai, et al., “Deep laplacian pyramid networks for fast and accurate super-resolution,” In CVPR, 2017), MemNet (Ying Tai, et al., “Memnet: A persistent memory network for image restoration,” In ICCV, 2017), CARN (Namhyuk Ahn, et al., “Fast, accurate, and lightweight super-resolution with cascading residual network,” In ECCV, 2018), IDN (Zheng Hui, et al., “Fast and accurate single image super-resolution via information distillation network,”. In CVPR, 2018), SRFBN-S (Zhen Li, et al., “Feedback network for image super-resolution,” In CVPR, 2019), IMDN (Zheng Hui, et al., “Lightweight image super-resolution with information multi-distillation network,” In ACMMM, 2019), LatticeNet (Xiaotong Luo, et al., “Latticenet: Towards lightweight image super-resolution with lattice block,” In ECCV, 2020), and SMSR (Longguang Wang, et al., “Exploring sparsity in image super-resolution for efficient inference,” In CVPR, 2021). All of them are popular lightweight SR methods. 
     Visualization Results. Visualization results are shown in  FIGS.  13 A- 2  through  13 A- 11   ,  FIGS.  13 B- 2  through  13 B- 11   ,  FIGS.  13 C- 2  through  13 C- 11   , and  FIGS.  13 D- 2  through  13 D- 11   . Such visualization results provide a visual comparison with lightweight SR networks on the Urban 100  and Manga 109  datasets, as disclosed below. The results are generated with a ×4 scale on the Urban 100  and Manga 109  datasets. 
       FIG.  13 A- 1    is an exemplification of an image  1300 A- 1 , namely the img. 044  image from the Urban 100  dataset, with a ×4 scale. 
       FIGS.  13 A- 2  through  13 A- 11    are exemplifications of visualization results for the image  1300 A- 1  of  FIG.  13 A- 1    with different SR methods. Specifically,  FIG.  13 A- 2    is an exemplification of a visualization result, namely the HQ  1300 A- 2  image, that is based on HQ while  FIGS.  13 A- 3 ,  13 A- 4 ,  13 A- 5 ,  13 A- 6 ,  13 A- 7 ,  13 A- 8 ,  13 A- 9 ,  13 A- 10 , and  13 A- 11    are exemplifications of visualization results, namely the Bicubic  1300 A- 3  image, SRCNN  1300 A- 4  image, VDSR  1300 A- 5  image, LapSRN  1300 A- 6  image, MemNet  1300 A- 7  image, CARN  1300 A- 8  image, IMDN  1300 A- 9  image, SMSR  1300 A- 10  image, and HEUN-L  1300 A- 11  image, that are based on Bicubic, SRCNN, VDSR, LapSRN, MemNet, CARN, IMDN, SMSR, and HEUN-L, respectively. 
       FIG.  13 B- 1    is an exemplification of an image  1300 B- 1 , namely the img. 089  image from the Urban 100  dataset, with a ×4 scale. 
       FIGS.  13 B- 2  through  13 B- 11    are exemplifications of visualization results for the image  1300 B- 1  of  FIG.  13 B- 1    with different SR methods. Specifically,  FIG.  13 B- 2    is an exemplification of a visualization result, namely the HQ  1300 B- 2  image, that is based on HQ while  FIGS.  13 B- 3 ,  13 B- 4 ,  13 B- 5 ,  13 B- 6 ,  13 B- 7 ,  13 B- 8 ,  13 B- 9 ,  13 B- 10 , and  13 B- 11    are exemplifications of visualization results, namely the Bicubic  1300 B- 3  image, SRCNN  1300 B- 4  image, VDSR  1300 B- 5  image, LapSRN  1300 B- 6  image, MemNet  1300 B- 7  image, CARN  1300 B- 8  image, IMDN  1300 B- 9  image, SMSR  1300 B- 10  image, and HEUN-L  1300 B- 11  image, that are based on Bicubic, SRCNN, VDSR, LapSRN, MemNet, CARN, IMDN, SMSR, and HEUN-L, respectively. 
       FIG.  13 C- 1    is an exemplification of an image  1300 C- 1 , namely the img. 092  image from the Urban 100  dataset, with a ×4 scale. 
       FIGS.  13 C- 2  through  13 C- 11    are exemplifications of visualization results for the image  1300 C- 1  of  FIG.  13 C- 1    with different SR methods. Specifically,  FIG.  13 C- 2    is an exemplification of a visualization result, namely the HQ  1300 C- 2  image, that is based on HQ while  FIGS.  13 C- 3 ,  13 C- 4 ,  13 C- 5 ,  13 C- 6 ,  13 C- 7 ,  13 C- 8 ,  13 C- 9 ,  13 C- 10 , and  13 C- 11    are exemplifications of visualization results, namely the Bicubic  1300 C- 3  image, SRCNN  1300 C- 4  image, VDSR  1300 C- 5  image, LapSRN  1300 C- 6  image, MemNet  1300 C- 7  image, CARN  1300 C- 8  image, IMDN  1300 C- 9  image, SMSR  1300 C- 10  image, and HEUN-L  1300 C- 11  image, that are based on Bicubic, SRCNN, VDSR, LapSRN, MemNet, CARN, IMDN, SMSR, and HEUN-L, respectively. 
       FIG.  13 D- 1    is an exemplification of an image  1300 D- 1 , namely the YumeiroCook. image from the Manga 109  dataset, with a ×4 scale. 
       FIGS.  13 D- 2  through  13 D- 11    are exemplifications of visualization results for the image  1300 D- 1  of  FIG.  13 D- 1    with different SR methods. Specifically,  FIG.  13 D- 2    is an exemplification of a visualization result, namely the HQ  1300 D- 2  image, that is based on HQ while  FIGS.  13 D- 3 ,  13 D- 4 ,  13 D- 5 ,  13 D- 6 ,  13 D- 7 ,  13 D- 8 ,  13 D- 9 ,  13 D- 10 , and  13 D- 11    are exemplifications of visualization results, namely the Bicubic  1300 D- 3  image, SRCNN  1300 D- 4  image, VDSR  1300 D- 5  image, LapSRN  1300 D- 6  image, MemNet  1300 D- 7  image, CARN  1300 D- 8  image, IMDN  1300 D- 9  image, SMSR  1300 D- 10  image, and HEUN-L  1300 D- 11  image, that are based on Bicubic, SRCNN, VDSR, LapSRN, MemNet, CARN, IMDN, SMSR, and HEUN-L, respectively. 
     As should be appreciated from the visualization results, compared with other methods, an example embodiment of a HEUN-L network disclosed herein generates better reconstruction results, especially on Manga 109 . An example embodiment of a HEUN-L network has fewer artifacts than other methods. 
     Quantitative Results. Quantitative results are shown in Table 3 of  FIGS.  14 A and  14 B , disclosed below, for ×2, ×3, and ×4 SR. 
       FIGS.  14 A-B  are a table, namely Table 3, with an exemplification of Benchmark results for different methods  1401  with the BI degradation model. Table 3 shows average PSNR and structural similarity index (SSIM) values for a scaling factor  1403  of ×2, ×3, and ×4. In Table 3, the top-2 least parameters (params) of the parameters  1405 , Multi-Adds  1407 , and performance on each of the datasets, namely the Set 5   1409 , Set 14   1411 , B 100   1413 , Urban 100   1415 , and Manga 100   1417  datasets, are highlighted with bold (e.g., in red) (best) and underlined (e.g., in blue) (second best), respectively. In Table 3, the “+” symbol denotes the results are generated with self-ensemble. 
     As shown in Table 3, among all methods, an example embodiment of HEUN-L achieves the new SOTA performance on every dataset with the scale of ×3 and ×4. When the scale is ×2, an example embodiment of an HEUN-L network still achieves the best performance on Set 14 , B 100 , and Manga 109 . Its PSNR on Urban 100  is a little lower than LatticeNet, but its SSIM is higher. Although HEUN-L cannot catch up with the LatticeNet (Xiaotong Luo, et al., “Latticenet: Towards lightweight image super-resolution with lattice block,” In ECCV, 2020), on the Set 5  dataset with the scale of ×2, its computation costs and parameters are smaller. The results of HEUN-L+ show that performance can be further improved with a self-ensemble technique. 
     An example embodiment of the HEUN-M network can achieve top-3 performance on Set 14  and B 100  for each scale. Further, compared with other competitive methods, such as IMDN (Zheng Hui, et al., “Lightweight image super-resolution with information multi-distillation network,” In ACMMM, 2019), LatticeNet (Xiaotong Luo, et al., “Latticenet: Towards lightweight image super-resolution with lattice block,” In ECCV, 2020), and SMSR (Longguang Wang, et al., “Exploring sparsity in image super-resolution for efficient inference,” In CVPR, 2021), it only has two thirds or even fewer parameters and Multi-Adds. Moreover, it can achieve top-5 performance on each dataset with any scale. Furthermore, the HEUN-M network can be significantly improved using the self-ensemble technique. An example embodiment of HEUN-S achieves comparable performance with the second least parameters among all the methods. When the scale is set to ×3 and ×4, it takes the minor computation costs among all the methods. 
     An example embodiment of the HEUN-S network was compared with the SRFBN-S (Zhen Li, et al., “Feedback network for image super-resolution,” In CVPR, 2019) and CARN (Namhyuk Ahn, et al., “Fast, accurate, and lightweight super-resolution with cascading residual network,” In ECCV, 2018), since SRFBN-S also uses the low-frequency features to enhance the inference features and CARN implements group convolutions for the lightweight purpose as well. As shown in the table, an example embodiment of the HEUN-S network can achieve around 0.08 dB than SRFBN-S on the PSNR of Set 5  on average. In the meanwhile, the parameters and Multi-Adds of the HEUN-S network are 64.4% and 4.2%, respectively. The comparisons show that an example embodiment of the proposed module can significantly improve the SR performance using low-resolution features without any complicated operations. Compared with CARN, an example embodiment of a model disclosed herein achieves better PSNR when the scale is ×2 and ×3 with 14.8% and 17.9% of its size and Multi-Adds. 
     Inference Time. Real-world results are disclosed in addition to the theoretical evaluation. The real-world results are shown in Table 4, disclosed below. 
     
       
         
           
               
             
               
                 TABLE 4 
               
             
            
               
                   
               
               
                 A comparison of the storage size (KB) and inference  
               
               
                 time (ms) for real-world test. The results are generated  
               
               
                 on Set 5 with scale x2. 
               
            
           
           
               
               
               
               
               
            
               
                   
                   
                 PSNR 
                   
                 Inference 
               
               
                   
                 Methods 
                 (Set5) 
                 Size 
                 Time 
               
               
                   
                   
               
               
                   
                 EDSR Baseline 
                 38.05 
                 5366 KB 
                 460 ms 
               
               
                   
                 HEUN-L 
                 38.09 
                 2858 KB 
                 240 ms 
               
               
                   
                 HEUN-M 
                 38.03 
                 1902 KB 
                 176 ms 
               
               
                   
                 HEUN-S 
                 37.93 
                  904 KB 
                 162 ms 
               
               
                   
                   
               
            
           
         
       
     
     From Table 4, disclosed above, it can be observed that the speed of HEUN-S is not much faster as the theoretical calculation comparing with the HEUN-M. This may be caused by the better optimization of standard convolution on CUDA. It is understood that HEUN-S can perform faster in an environment specified for edge devices. 
     In summary and for non-limiting example, as disclosed above, a lightweight network referred to herein as Hybrid Element-Unshuffled Network (HEUN) may be employed for image SR. An example embodiment may include the Self-Residual Depth-wise Separable Convolution to overcome the defects of the depthwise convolution, and the element-unshuffled downsampling may be employed to enhance the performance with low-frequency representations. Both proposed modules take limited computation costs and parameters. With the two proposed modules, an example embodiment of a lightweight block, referred to herein as a Hybrid Element-Unshuffled Block, may be designed with a standard convolution layer and an Element-Unshuffled Block. Further, as disclosed above, an example embodiment of HEUN can achieve new SOTA performance with limited parameters and Multi-Adds. In addition, disclosure with regard to a discovery of a relationship between the PSNR and the NME among the shallow features and deep features is provided. It is understood that the phenomenon should be general, and can be taken advantage of for network design. 
       FIG.  15    is a block diagram of an example of the internal structure of a computer  1500  in which various embodiments of the present disclosure may be implemented. The computer  1500  contains a system bus  1552 , where a bus is a set of hardware lines used for data transfer among the components of a computer or digital processing system. The system bus  1552  is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, input/output ports, network ports, etc.) that enables the transfer of information between the elements. Coupled to the system bus  1552  is an I/O device interface  1554  for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer  1500 . A network interface  1556  allows the computer  1500  to connect to various other devices attached to a network (e.g., global computer network, wide area network, local area network, etc.). Memory  1558  provides volatile or non-volatile storage for computer software instructions  1560  and data  1562  that may be used to implement embodiments (e.g., method  300 ) of the present disclosure, where the volatile and non-volatile memories are examples of non-transitory media. Disk storage  1564  provides non-volatile storage for computer software instructions  1560  and data  1562  that may be used to implement embodiments (e.g., method  300 ) of the present disclosure. A central processor unit  1566  is also coupled to the system bus  1552  and provides for the execution of computer instructions. 
     As used herein, the term “module” may refer to any hardware, software, firmware, electronic control component, processing logic, and/or processor device, individually or in any combination, including without limitation: an application specific integrated circuit (ASIC), a field-programmable gate-array (FPGA), an electronic circuit, a processor and memory that executes one or more software or firmware programs, and/or other suitable components that provide the described functionality. 
     Example embodiments disclosed herein may be configured using a computer program product; for example, controls may be programmed in software for implementing example embodiments. Further example embodiments may include a non-transitory computer-readable medium that contains instructions that may be executed by a processor, and, when loaded and executed, cause the processor to complete methods described herein. It should be understood that elements of the block and flow diagrams may be implemented in software or hardware, such as via one or more arrangements of circuitry of  FIG.  15   , disclosed above, or equivalents thereof, firmware, a combination thereof, or other similar implementation determined in the future. 
     In addition, the elements of the block and flow diagrams described herein may be combined or divided in any manner in software, hardware, or firmware. If implemented in software, the software may be written in any language that can support the example embodiments disclosed herein. The software may be stored in any form of computer readable medium, such as random-access memory (RAM), read-only memory (ROM), compact disk read-only memory (CD-ROM), and so forth. In operation, a general purpose or application-specific processor or processing core loads and executes software in a manner well understood in the art. It should be understood further that the block and flow diagrams may include more or fewer elements, be arranged or oriented differently, or be represented differently. It should be understood that implementation may dictate the block, flow, and/or network diagrams and the number of block and flow diagrams illustrating the execution of embodiments disclosed herein. 
     The teachings of all patents, published applications, and references cited herein are incorporated by reference in their entirety. 
     While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments encompassed by the appended claims.