Patch orientation‐specified network for learning‐based image super‐resolution

Electronics LettersVolume 55, Issue 23 p. 1233-1235 Image and vision processing and display technologyFree Access Patch orientation-specified network for learning-based image super-resolution S.B. Yoo, Corresponding Author S.B. Yoo sbyoo@etri.re.kr Intelligent Convergence Research Laboratory, Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of KoreaSearch for more papers by this authorM. Han, M. Han Intelligent Convergence Research Laboratory, Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of KoreaSearch for more papers by this author S.B. Yoo, Corresponding Author S.B. Yoo sbyoo@etri.re.kr Intelligent Convergence Research Laboratory, Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of KoreaSearch for more papers by this authorM. Han, M. Han Intelligent Convergence Research Laboratory, Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of KoreaSearch for more papers by this author First published: 01 November 2019 https://doi.org/10.1049/el.2019.1219Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Learning-based image super-resolution is considered as a promising solution to reconstruct a high-resolution image from a low-resolution image. To improve the super-resolution performance dramatically, this Letter focuses on the effect of training dataset on the performance and proposes an image super-resolution scheme based on patch orientation-specified network. In particular, a deep neural network is trained using patches with a specific orientation and angular transformation is combined with the neural network to cope with various orientations in input patches. Experimental results show the suggested network model is superior to existing state-of-the-art super-resolution alternatives. Introduction Since patch orientation can serve as an important clue for recovering the missing high-frequency (HF) components due to image downscaling, it is useful in electronic displays for accurate image super-resolution (SR). In general, existing learning-based image SR techniques [1-7] tend to utilise public training datasets consisting of image patches with various orientations to keep the robustness of the SR performance to the arbitrary direction of input low-resolution (LR) patches. In contrast to this trend, in this Letter, we notice that building a trained model based on patch datasets with a specific orientation can provide a significant improvement on input LR patches with the specific orientation. As shown in Fig. 1, to confirm the effect of patch orientation in the training dataset on the SR performance, we perform an investigation by applying an existing state-of-the-art SR algorithm [2] to a LR image having only unidirectional edges. In the investigation, the network1 model is obtained by training the neural network parameters using general patch datasets with various orientations. Unlike this, the network2 model is obtained using training patches with a specific orientation only, such as vertical edges. Given a LR image with vertical edges only, we perform the upscaling with a scale factor 3 by using the two trained models, the network1 and network2, respectively. The upscaled images in Fig. 1 show that the orientation-specified network (or network1) outperforms the general non-specified network (or network2) for the LR image by significantly improving a peak signal-to-noise ratio (PSNR) value by 3.77 dB. Based on the investigation, we consider that the network specialised to the input orientation is needed for the SR performance increase. Meanwhile, by re-training model parameters in recent image SR algorithms [1-6] based on convolutional neural network for each orientation separately, it may be possible to achieve the SR performance comparable to that expected from our investigation. However, as expected, it requires a high computational complexity for the training procedure as well as a huge amount of memories for many models for all orientations. To address these problems, in this Letter, we suggest an orientation matching based on angular transformation in the same direction as the dataset and combine it with a neural network particularly designed for accurate image SR. Fig 1Open in figure viewerPowerPoint Effect of patch orientation in training dataset on the SR accuracy Patch orientation-specified network Based on the investigation described above, we define an image resolution degradation process as (1)where x and y denote high-resolution (HR) and LR images, respectively, and n denotes an additive noise. In addition, rθ and rθT denote clockwise and counter-clockwise rotation operators by angle θ, respectively. Here, B and D denote blurring and downsampling operators, respectively. To reflect the defined image degradation process in the neural network properly, we present a patch orientation-specified network (POSNet) considering both unidirectional and multi-directional input LR patches, as illustrated in Fig. 2. The proposed POSNet consists of two parallel networks, specified high-directivity (HD) network for upscaling of unidirectional edge patches and non-specified low-directivity (LD) network for that of multi-directional texture patches. Unlike existing algorithms [1-7], we adaptively apply the two neural networks according to the input patch orientation. As a result, the dual network model via the adaptive network selection enables to maximise the SR performance on input HD patches while keeping the performance on input LD patches. Fig 2Open in figure viewerPowerPoint Architecture of the POSNet For the network selection, we upscale the input LR image via bicubic interpolation and compute the gradient magnitude g and orientation θ in every pixel position of the interpolated patch of M × M as follows: (2)To investigate the directivity strength of the interpolated patch, we first obtain an orientation histogram h via the following procedure: (3)where R denotes the rounding operator and δθ denotes the bin size in the histogram. To address the noise problem, G is used as a threshold to determine meaningful gradients. By using the orientation histogram h, we find the largest and the second largest histogram values, v1 and v2. We then classify the patch as the HD patch if v2 < min(v1/4, M). Otherwise, the patch is classified as the LD patch. Examples of the classification results are given in Fig. 2. Given the determined HD patch, we apply the specified HD network trained using HD patches with near zero angle satisfying the condition h(0) = v1. Considering the radial expansion due to the rotation, zero padding in the patch boundary is done in advance so that the padded patch size becomes sqrt(2) × M. The padded patch is then rotated around the position of (mc, nc) = (M/sqrt(2), M/sqrt(2)) so that the rotated patch has a zero angle. For the rotated patch, iterative architectures are applied to obtain the sharp HR patch. They consist of 20-layer architectures and each layer architecture has three operations: convolution, batch normalisation (BN), and rectified linear units (ReLU). Firstly, we perform the convolution using K pre-trained filter parameters aL with a size of N × N. The BN and ReLU then follow the convolution for stable convergence. After the iteration, a fully-connected layer is applied for reshaping of output tensors. By re-rotating the output HR patch to the original angle and removing the outline, the HD network output can be acquired. For the LD patch, we apply another parallel network, non-specified LD network, trained using LD patches only. The HR version of the LD patch is obtained by applying the same iterative architectures as in the HD network to the patch with pre-trained parameters βL instead of aL. All of the above steps are also described in Algorithm 1. Algorithm 1: Image SR based on the POSNet Input: LR image y and trained parameters Output: HR image x 1: Obtain by interpolating y to HR. 2: Calculate using a patch as in (2) and (3). 3: if v2 < min(v2/4, V) do 4: Obtain yp by inserting zeros to outsides of . 5: Rotate yp by degree of , by . 6: for L = 1,…,20 do 7: Apply convolution with trained parameters by . 8: Apply BN and ReLU by max(BN , 0). 9: end for 10: Obtain yFC by applying a fully-connected layer to reshape output tensors. 11: Re-rotate it by degree of , by . 12: Obtain for the HD patch by crop . 13: else do 14: Obtain for the LD patch by running steps 6–10 above with instead of . 15: end if 16: Obtain a full-resolution result via integration of patch-wise results x by running steps 2–15 above for all pixel position (x, y) in ycubic. Experiments To train our two parallel networks, specified HD network and non-specified LD network, we use 800 images from the DIV2K dataset used in [3, 4] and 400 images from the Berkeley segmentation dataset (BSDS500). For a given ground truth (GT) image, we synthesise three LR images with scale factors 2, 3, and 4, and train our networks for each scale factor separately. For training the specified HD network, we select patches whose gradient orientation θ is nearly 0 from HD patches in the datasets and then obtain the network parameters aL using the selected patches only. In addition, the non-specified LD network parameters βL are obtained via the training with all LD patches in the datasets. In our network selection stage, M, δθ, and G are set to 17, π/12, and 5, respectively. Additionally, the number of filters K is set to 64 and the filter size N is set to 3 in every layer architecture. Given a set of GT HR images and their corresponding interpolated LR images, we adopt the mean squared error as a loss function. The network training for minimising the loss function is performed on NVIDIA 1080 GPU, under MATLAB with MatConvNet package for about 24 h. To evaluate the SR performance of the trained network, we utilise five images from the Set5 dataset and 14 images from the Set14 dataset, which have been commonly used in [1-6]. We also compare the performance with existing state-of-the-art SR algorithms [1-3]. Table 1 summarises average PSNR and structure similarity (SSIM) [8] values computed from upscaled results of different algorithms on the two datasets. It can be noticed in the table that the proposed POSNet provides the best quantitative quality for every case by successfully recovering GT pixel values and structures. Figs. 3 and 4 also show the comparison of the subjective visual quality on the Set5 and Set14 images, respectively, for a scale factor 4. We note in the figures that the proposed POSNet outperforms the existing algorithms in many edge regions such as the wing pattern, the eyelashes, the branch, the hat, the cloth, the alphabet, and the toy, by increasing the sharpness significantly. The orientation matching process based on angular transformation, which is enforced in the orientation-specified neural network, mainly achieves this remarkable visual improvement. Table 1. Comparison of quantitative quality in terms of average PSNR (dB) and SSIM Dataset Scale factor Metric Bicubic VDSR [1] DnCNN [2] SRMD [3] POSNet Set5 (5 images) × 2 PSNR 33.64 37.56 37.58 37.53 37.75 SSIM 0.929 0.959 0.959 0.959 0.961 × 3 PSNR 30.39 33.67 33.75 33.86 34.24 SSIM 0.868 0.922 0.922 0.923 0.927 × 4 PSNR 28.42 31.35 31.40 31.59 32.06 SSIM 0.810 0.885 0.885 0.887 0.890 Set14 (14 images) × 2 PSNR 30.22 33.02 33.03 33.12 33.28 SSIM 0.868 0.913 0.913 0.914 0.915 × 3 PSNR 27.53 29.77 29.81 29.84 30.27 SSIM 0.774 0.832 0.832 0.833 0.841 × 4 PSNR 25.99 27.99 28.04 28.15 28.51 SSIM 0.702 0.766 0.767 0.772 0.779 Bold values indicate the best scores. Fig 3Open in figure viewerPowerPoint Comparison of the subjective visual quality on four images from the Set5 dataset, Butterfly, Baby, Bird, and Woman Fig 4Open in figure viewerPowerPoint Comparison of the subjective visual quality on three images from the Set14 dataset, Comic, Ppt2, and Barbara Conclusion Despite patch orientation is an important clue for restoring HF information in the SR field, existing learning-based SR approaches have limitations in utilising this clue. In this Letter, we propose a dual network model that is suitable for both unidirectional and multi-directional input patches, based on the adaptive selection according to the directivity strength. Specifically, to apply the specialised network trained for a zero angle on the arbitrary input, we design the angular transformation and combine it with iterative architectures. The superior performance of the proposed network on two test datasets is also verified in terms of quantitative and subjective qualities. Acknowledgment This work was supported by 'The Cross-Ministry Giga KOREA Project' grant funded by the Korea government (MSIT) (no. GK19P0600, Development and Demonstration of Smart City Service over 5G Network). References 1Kim, J., Lee, J.K., Lee, K.M.: ' Accurate image super-resolution using very deep convolutional networks'. IEEE Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, June 2016, pp. 1646– 1654 2Zhang, K., Zuo, W., Chen, Y. et. al.,: 'Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising', Trans. Image Process., 2017, 26, (7), pp. 3142– 3155 (https://doi/org/10.1109/TIP.2017.2662206) 3Zhang, K., Zuo, W., Zhang, L.: ' Learning a single convolutional super-resolution network for multiple degradations'. IEEE Computer Vision Pattern Recognition, Salt Lake City, UT, USA, June 2018, pp. 3262– 3271 4Zhang, Y., Sun, L., Yan, C. et. al.,: 'Adaptive residual networks for high-quality image restoration', Trans. Image Process., 2018, 27, (7), pp. 3150– 3163 (https://doi/org/10.1109/TIP.2018.2812081) 5Zhang, Y., Li, K., Li, K. et. al.,: ' Image super-resolution using very deep residual channel attention networks'. IEEE European Conf. on Computer Vision, Munich, Germany, September 2018, pp. 286– 301 6Zhang, Y., Tian, Y., Kong, Y. et. al.,: ' Residual dense network for image super-resolution'. IEEE Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June 2018, pp. 2472– 2481 7Chen, H., Xue, J., Zhang, S. et. al.,: 'Image super-resolution based on adaptive cosparse regularisation', Electron. Lett., 2014, 50, (24), pp. 1834– 1836 (https://doi/org/10.1049/el.2014.1429) 8Wang, Z., Bovik, A.C., Sheikh, H.R. et. al.,: 'Image quality assessment: from error visibility to structural similarity', Trans. Image Process., 2014, 13, (4), pp. 600– 612 (https://doi/org/10.1109/TIP.2003.819861) Citing Literature Volume55, Issue23November 2019Pages 1233-1235 FiguresReferencesRelatedInformation