# Rendered Image Superresolution Reconstruction with Multichannel Feature Network

Authors: Zhipeng Ren; Jianping Zhao; Chunyi Chen; Yan Lou; Xiaocong Ma; Pengyu Tao
Journal: Scientific Programming
 (2022)

---

## Abstract

In the process of film and television production, clear images can give the audience a real sensory experience, but high-resolution images require a massive amount of production time and highly specialized imaging equipment, which is not a cost-effective solution at the moment. To achieve a better cost efficiency during video production, we propose a multichannel featured superresolution network model that utilizes rendered low-resolution images according to their characteristics. This model includes a feature extraction layer, a series of subnetworks, and a reconstruction module. Inside the network model, a series of subnetworks are cascaded to improve the information flow from coarse to fine, which helps to fully extract the depth, normal vector, edge, and texture features from low-resolution rendered images to reconstruct the high-resolution image. Additionally, residual learning is introduced at each stage to further improve the reconstruction performance. We experiment with the model on the classic Disney Monte Carlo datasets and compare it with several related algorithms. The results show that our algorithm is able to reconstruct the image with clearer details and texture. Thus, our research not only helps to maintain the audience’s sensory experience but also increases the efficiency of film and television production, which also brings considerable economic benefits.


---

## Body

## 1. Introduction

Image resolution is a set of performance parameters used to evaluate the richness of detailed information contained in an image. This includes temporal resolution, spatial resolution, and color scale resolution, which all together indicate the ability of an imaging system that can reflect the detailed information of an object. Compared with low-resolution (LR) images, high-resolution (HR) images usually contain higher pixel density, richer texture details, and higher fidelity. Modern film and television productions have high demands for the visual effect on picture qualities as technologies advance. However, to achieve a high-definition picture quality, it inevitably requires a long image rendering time, and the production cost also increases accordingly. How to reduce rendering time and production cost without affecting the visual experience of the audience is a vital research direction. More practically, it is important to maintain an ideal balance between the rendering time and the visual quality of an image such that the cost of the production can be kept minimum, and thus, how to improve the image’s resolution efficiently and quickly is the key to solving the above problems.Physically, solving low-resolution problems of images is often too costly. Constrained by many factors such as processing power of the hardware, format type of the image, network bandwidth, and the image degradation model itself, it is hard to obtain an ideal high-resolution image without applying edge sharpening and block deblurring. In many cases, animations and effect maps are drawn by cartographic software to restore details. Similarly, it is entirely possible that the details of high-resolution (HR) images can be recovered from low-resolution (LR) images by using various types of algorithms. Early methods include interpolation, such as double edge interpolation and Lanczos resampling [1]. More powerful methods using statistical image prior [2, 3] or internal patch recursion [4], probabilistic graphic model [5], neighborhood embedding [6, 7], sparse encoding [8], and linear or nonlinear regression are also used to convert LR images back into HR images [9–11].The method of using deep learning to improve image resolution has made rapid development in recent years. Various superresolution network algorithms continue to be invented for better efficiency of restoring. In 2014, Dong et al. [12] first applied a convolutional neural network to an image superresolution task and proposed a superresolution convolutional neural network model to predict the nonlinear relationship between interpolated images and high-resolution images (SRCNN). The model carried out end-to-end image pair training, which greatly improved the reconstruction effect at that time and thus proved the excellent performance of deep learning in the field of image superresolution. On top of that, Dong et al. [13] proposed to enlarge the size of the deconvolution layer and improve the SRCNN model to reconstruct a high-resolution image (FSRCNN). In 2016, Kim et al. proposed two very deep convolution neural networks, VDSR [14] and DRCN [15] models, to deeply extract image features and to accelerate the network’s convergence speed by introducing the residual idea. It has been proved that the deep network can extract more features and achieve a better reconstruction effect. Tai et al. [16] constructed a persistent memory network for image restoration (MemNet), accompanied by memory blocks with a tight connection structure. In 2021, Hu et al. [17] designed a network to extract more detailed image features and cross-fused HR images (MSICF).Although these models have made significant progress, they are mainly aimed at the restoration of natural images. However, there is a lack of research focused on the superresolution reconstruction of rendered images with multichannel features in the field of video production, and this has brought up drawbacks due to some bottlenecks. For example, the single-channel network cannot extract the main features such as depth and the normal vector of rendered images. Furthermore, the utilization rate of image information obtained from different channels is not sufficient, and this can result in insufficient details of reconstructed images. Given the above situation, we propose a superresolution reconstruction method for rendered images and produce more realistic results. The main contributions of this paper are summarized as follows:(1)
We propose a multichannel featured superresolution reconstruction algorithm of the rendered image, which is able to obtain multidimensional information from different channels such as depth, normal vector, edges, and texture in the rendered image. After the multichannel featured information passes through the network model, it is fused into three RGB color channels. At the same time, global residual and local residual learning are introduced to prevent the disappearance or explosion of the gradient, accelerate the convergence speed, and obtain a considerable reconstruction effect.(2)
The network model includes a feature extraction layer, a series of subnetworks, and a reconstruction module. The extraction layer adopts double filters, which can retain high-frequency information such as image edge and texture details. The subnetworks cascade to improve the information flow from coarse to fine, and the reconstruction layer adopts weighting to improve the reconstruction accuracy. Compared with several other advanced algorithms, experiments show that the algorithm in this paper is more effective in image detail restoration.(3)
In the process of video production, our innovative method firstly renders the low-resolution image and then applies the proposed superresolution network algorithm to reconstruct the high-resolution image. In this way, we not only maintain an excellent level of audience’s sensory experience but also reduce the film and television production time and cost for achieving increased economic benefits. As a result, this research has brought distinct economic and practical values to the video industry.


## 2. The Proposed Method

We propose a multichannel featured superresolution reconstruction algorithm for rendered images, as shown in Figure1. It includes a feature extraction layer, a series of subnetworks, and a reconstruction module. The image input enters the subnetwork after convolution in the first layer, and the information is output to the reconstruction layer after passing through each subnetwork. At the same time, the overall residual learning is introduced. Finally, the reconstruction layer carries the weighted fusion to reconstruct the three-channel high-resolution image.Figure 1
The architecture of the proposed network.


### 2.1. The Proposed Network

The extraction layer extracts the features of the original low-resolution image. The extraction layer of our design is composed of 64 filters with a convolution kernel size of3∗3. We defined convolution layer expression as S0=f0x, where x is the input low-resolution image, f0 is the feature extraction function, and S0 is the output after feature extraction. In the subnetwork shown in Figure 2, according to the multichannel features of the rendered image, we design a cross-module with multifeatures to extract the multichannel, fuse the feature information, and then stack several submodules to learn the residual information between the input and output features. More specifically, the feature information of 10 channels enters the network, and the output of the feature extracted by convolution is used as the input of the next subnetwork. The convolution operation is carried out in a turn-taking sequence. Once the information has passed through all subnetworks, all types of features such as texture, depth, and normal vector can be fully extracted. At the same time, local residual learning is introduced to provide supplemental image feature data, which improves the accuracy and efficiency of image feature learning process. The first convolution layer adopts 64 filters, which are superimposed and fused after entering the subnetwork; it is then followed by layers that adopt 128, 192, and 256 filters; all come with a size of 3∗3, respectively.Figure 2
The architecture of each subnetwork.In Figure2, it can be represented by equation (1). We define fq⋅ as the feature extraction function and Sq−1 as the input of the qth subnetwork. M is the number of subnetworks. The expression of subnetwork is GqM⋅1M−1. We introduce local residual learning between G⋅ and GLast⋅ for the subnetwork to obtain different image features. We define S0 as the expression of the first subnetwork, ξqq=1,2,3,…,S as the expression of the qth subnetwork, and n as the number of subnetwork modules. This gives (1) where Sn is the subnetwork.(1)Sq=ξqSq−1=GqlastGqM−1fqSq−1+Sq−1,=GqlastGqM−1⋯Gq1fqSq−1⋯+Sq−1,(2)Sn=ξnξn−1⋯ξ1S0⋯.In the training process, to make the prediction closer to the real data, all prediction data are managed in a cascade way from the subnetwork to the final output of the whole network. We defineRq⋅ as the expression of the reconstruction layer, Sq as the output of the qth subnetwork, and y^ as the output of the whole network. Concurrently, we use global residual learning to improve the reconstruction accuracy. In this way, all intermediate predictions are deeply convoluted, and the final output is weighted at the reconstruction layer. Equation (2) can be expressed as follows:(3)yq^=RqSq+x.


### 2.2. Preliminary Work

For the preliminary preparation of data training, we select 200 HR images from the classic Disney Monte Carlo rendered image datasets which contain ten-channel information respectively, including “R,” “G,” “B,” “normal.R,” “normal.G,” “normal.B,” “albedo.R,” “albedo.G,” “albedo.B,” and “depth.Z.” The letter “R,” “G,” and “B” represent red, green, and blue, “normal.R,” “normal.G,” and “normal.B” represent three vectors perpendicular to the surface, and “albedo.R,” “albedo.G,” and “albedo.B” represent different albedo textures. The “depth. Z″ represents the depth of the image. We expand the training data by performing horizontal, vertical, and flip operations on these images, reduce the sampling by 2, 3, and 4 times, and then restore them to their original size by applying a bicubic interpolation algorithm. Finally, the interpolated LR images and the original HR images are divided into64∗64 fragments, which are stored in pairs in a training data file.


### 2.3. Residual Learning

In previous neural network models, each layer learns a mapping ofy=Fx, where x is the input, y is the output, and Fx is the mapping function. One can think in the way that the error of y happens to accumulate with the deepening of the layers, and the gradient becomes more divergent in the process of backpropagation as the error becomes larger. Meanwhile, we add the original input at the end of each layer, the output becomes Fx+x, and the mapping relationship of each layer then becomes y=Fx+x. It is obvious that the value of Fx tends to be irrelevant and negligible compared to the size of x In this way, even if the number of layers continues to increase, the error Fx is still controlled at a very small value, as shown in Figure 3. Generally speaking, it is noteworthy that the residual learning includes global residual learning and local residual learning, which has been widely used in deep learning [18–20]. More particularly in this paper, our designed network model introduces global residual learning into the overall network and local residual learning into the subnetwork to ensure stable training can be carried out. Moreover, we also prove the effectiveness of residual learning by conducting subsequent training.Figure 3
Schematic diagram of the residual network.The depth of the whole model can be calculated by (3). On the right side inside the brackets, multiplication of 3 means that there are three convolution layers during one extraction operation, and the addition of 1 means that each subnetwork contains one convolution layer. In addition, the added value of 1 at the end of the right side represents the last convolution layer in the reconstructed network.(4)Depth=M×3+1×n+1.


## 2.1. The Proposed Network

The extraction layer extracts the features of the original low-resolution image. The extraction layer of our design is composed of 64 filters with a convolution kernel size of3∗3. We defined convolution layer expression as S0=f0x, where x is the input low-resolution image, f0 is the feature extraction function, and S0 is the output after feature extraction. In the subnetwork shown in Figure 2, according to the multichannel features of the rendered image, we design a cross-module with multifeatures to extract the multichannel, fuse the feature information, and then stack several submodules to learn the residual information between the input and output features. More specifically, the feature information of 10 channels enters the network, and the output of the feature extracted by convolution is used as the input of the next subnetwork. The convolution operation is carried out in a turn-taking sequence. Once the information has passed through all subnetworks, all types of features such as texture, depth, and normal vector can be fully extracted. At the same time, local residual learning is introduced to provide supplemental image feature data, which improves the accuracy and efficiency of image feature learning process. The first convolution layer adopts 64 filters, which are superimposed and fused after entering the subnetwork; it is then followed by layers that adopt 128, 192, and 256 filters; all come with a size of 3∗3, respectively.Figure 2
The architecture of each subnetwork.In Figure2, it can be represented by equation (1). We define fq⋅ as the feature extraction function and Sq−1 as the input of the qth subnetwork. M is the number of subnetworks. The expression of subnetwork is GqM⋅1M−1. We introduce local residual learning between G⋅ and GLast⋅ for the subnetwork to obtain different image features. We define S0 as the expression of the first subnetwork, ξqq=1,2,3,…,S as the expression of the qth subnetwork, and n as the number of subnetwork modules. This gives (1) where Sn is the subnetwork.(1)Sq=ξqSq−1=GqlastGqM−1fqSq−1+Sq−1,=GqlastGqM−1⋯Gq1fqSq−1⋯+Sq−1,(2)Sn=ξnξn−1⋯ξ1S0⋯.In the training process, to make the prediction closer to the real data, all prediction data are managed in a cascade way from the subnetwork to the final output of the whole network. We defineRq⋅ as the expression of the reconstruction layer, Sq as the output of the qth subnetwork, and y^ as the output of the whole network. Concurrently, we use global residual learning to improve the reconstruction accuracy. In this way, all intermediate predictions are deeply convoluted, and the final output is weighted at the reconstruction layer. Equation (2) can be expressed as follows:(3)yq^=RqSq+x.


## 2.2. Preliminary Work

For the preliminary preparation of data training, we select 200 HR images from the classic Disney Monte Carlo rendered image datasets which contain ten-channel information respectively, including “R,” “G,” “B,” “normal.R,” “normal.G,” “normal.B,” “albedo.R,” “albedo.G,” “albedo.B,” and “depth.Z.” The letter “R,” “G,” and “B” represent red, green, and blue, “normal.R,” “normal.G,” and “normal.B” represent three vectors perpendicular to the surface, and “albedo.R,” “albedo.G,” and “albedo.B” represent different albedo textures. The “depth. Z″ represents the depth of the image. We expand the training data by performing horizontal, vertical, and flip operations on these images, reduce the sampling by 2, 3, and 4 times, and then restore them to their original size by applying a bicubic interpolation algorithm. Finally, the interpolated LR images and the original HR images are divided into64∗64 fragments, which are stored in pairs in a training data file.


## 2.3. Residual Learning

In previous neural network models, each layer learns a mapping ofy=Fx, where x is the input, y is the output, and Fx is the mapping function. One can think in the way that the error of y happens to accumulate with the deepening of the layers, and the gradient becomes more divergent in the process of backpropagation as the error becomes larger. Meanwhile, we add the original input at the end of each layer, the output becomes Fx+x, and the mapping relationship of each layer then becomes y=Fx+x. It is obvious that the value of Fx tends to be irrelevant and negligible compared to the size of x In this way, even if the number of layers continues to increase, the error Fx is still controlled at a very small value, as shown in Figure 3. Generally speaking, it is noteworthy that the residual learning includes global residual learning and local residual learning, which has been widely used in deep learning [18–20]. More particularly in this paper, our designed network model introduces global residual learning into the overall network and local residual learning into the subnetwork to ensure stable training can be carried out. Moreover, we also prove the effectiveness of residual learning by conducting subsequent training.Figure 3
Schematic diagram of the residual network.The depth of the whole model can be calculated by (3). On the right side inside the brackets, multiplication of 3 means that there are three convolution layers during one extraction operation, and the addition of 1 means that each subnetwork contains one convolution layer. In addition, the added value of 1 at the end of the right side represents the last convolution layer in the reconstructed network.(4)Depth=M×3+1×n+1.


## 3. Model Training

Every 64 groups of the stored training image data are divided into a batch, and each batch contains64∗64 segmented images that are categorized by 10 different channels. Once these images are prepared, they are then sent to the network as fuzzy input for training. After learning the deep convolution network and extracting the data features of each channel, we generate a 3-channel target image. By calculating the loss function between the target image and the real image and then updating the weight parameters through backpropagation, we can get an optimized network model. Through training, we discover that the deepening of the convolution layer of the subnetwork leads to a disappearance of the gradient, so we choose to set 10 convolution layers for each subnetwork. Tentatively, when the number of subnetworks is 3, the loss function can be reduced to an ideal effect, and this concludes that the depth of the whole network is also ideal when set to 31.We compared the training-with-residual-learning results against those ones without it and found that the network with residual learning can accelerate the convergence and thus improve the accuracy. As shown in Table1, we go through iteration 100 times, and the loss function converges to a very observable degree at the 40th epoch.Table 1
Residual networks rapidly approach their convergence within 10 epochs.
ScaleEpoch10204080100(a) Initial learning rate 0.12xResidual36.0636.8537.2537.9837.95Nonresidual26.7033.8236.4137.3137.07Difference9.363.030.840.670.88(b) Initial learning rate 0.01.2xResidual35.8936.5437.0237.5337.27Nonresidual29.4133.2936.3636.9436.50Difference6.483.250.660.590.77(c) Initial learning rate 0.001.2xResidual36.1336.5837.1737.5637.34Nonresidual33.7135.2236.3237.1436.48Difference2.421.360.850.420.86The bold values indicate the network with residual learning converges faster comparing without residual learning.Our network is very deep and uses a64∗64 training map, which is larger than the receptive field information of general algorithms. Therefore, we used a stochastic gradient descent (SGD) solver and filled zero before convolution to keep the sizes of all feature maps the same for pixels near the image boundary [21]. We also train our network model at different scales, using 2x, 3x, and 4x images, and combine all LR and HR pairs of different scales into our training datasets. We set the momentum parameter to 0.9 and the weight attenuation to 10−4 [22, 23]. We also set the initial learning rate to 0.1, and then gradually reduced it by 10 times for every 25 epochs. To deal with the gradient problem of disappearance and explosion, we set the gradient clip on the gradient within a designed range −θ,θ, where θ is set to 0.4 [24–26]. Each convolution layer is followed by standard regularization and activation operations, and the negative slope of the activation layer is 0.2 [27, 28]. The training code of our network is shown in Table 2. After the training of 100 generations, we select the 83rd generation model as the follow-up experiment for analysis and evaluation according to the effect of the loss function.Table 2
The training code of our network.
Algorithm: Rendered image superresolution reconstruction with multifeature networkInput: HR/LR training dataOutput: Reconstruction imageInitialization: batch size = 64, learning rate = 0.1, momentum = 0.9weight-decay = 0.0001, nEpochs = 100Step 1: DataLoader (HR/LR)Step 2: Model = MFSR ()Step 3: selected SGD () select MSEloss ()Step 4: Loop StartStep 4.1: Fori = 1, 2, ..., 100Step 4.2: calculateSq by equation (1)Step 4.3: calculateDq by equation (2)Step 4.4: calculateY′ by equation (3)Step 4.5: Lossfunction loss.backword ()Step 4.6: update the parameter optimizer.step ()Step 4.7: save network modelStep 4.8: End forStep 5: Loop End


## 4. Experiments and Comparisons

### 4.1. Environment and Data

We use a computer with Intel Core I7-3770 CPU (3.40 GHz), NVIDIA GeForce GTX 1060 GPU, 16 GB memory, 64-bit Linux operating system, and 1 TB hard disk. The software includes MatlabR2016a, CUDA 8 tool kit, cuDNN 5.0, PyCharm, and PyTorch.We then select 15, 25, 50, and 100 rendered images from the classic Disney Monte Carlo datasets as our test data. The test data are downsampled using Python program code by 2, 3, and 4 times and then magnified to the original size by applying bicubic interpolation method. Finally, the images are classified and stored in four folders named car, house, classroom, and bathroom, and the preparation of experimental data is completed.


### 4.2. Evaluation and Comparison

We take the peak signal-to-noise ratio (PSNR) [29], structural similarity (SSIM) [30], and execution time (ET) as the evaluation indexes of this experiment. The shorter the execution time (ET) is [31], the faster the algorithm runs. The mathematical expression of PSNR is shown in (5) and (6), where Xori and Xres present the original image and the reconstructed image, respectively, the MSE refers to the mean squared error between Xori and Xres, and m and n represent the number of rows and columns of the image respectively. The unit of the PSNR is dB. It is known that the larger the value of the PSNR is, the closer the reconstructed image is to the ground truth image.(5)PSNR=10log102552MSE,(6)MSE=1mn∑i=1m∑j=1mXorii,j−Xresi,j2.The mathematical expression of the SSIM is shown in (7). The SSIM evaluates the image quality by comparing the similarity of the structural information between images. If the value of the SSIM is close to 1, it means that the reconstructed image is close to the original image.(7)SSIM=∑i=1m∑j=1mXorii,j×Xresi,j∑i=1m∑j=1mXorii,j2×∑i=1m∑j=1mXresi,j2.We select this training model for experimenting with car, house, classroom, and bathroom test datasets. After inputting the experimental images into our network model and outputting the three-channel RGB superresolution images, the color conversion is carried out, and the brightness values of the images are extracted as the reference values for calculating the evaluation indexes. The car datasets amplified by 2, 3, and 4 times produce an average result of the PSNR with values at 37.99 dB, 34.97 dB, and 32.83 dB and an average result of the SSIM with values of 0.9762, 0.9421, and 0.9179. To verify the effectiveness of the proposed method, the above experimental data are also input into other six advanced superresolution algorithms, the SRCNN, the FSRCNN, the VDSR, the DRCN, the MenNet, and the MSICF, for direct results comparison.As shown in Table3, the PSNR value of the algorithm proposed in this paper exceeds the above reconstruction algorithms when using different combinations of magnifications and test sets. The experimental results show that our algorithm is better than the other six advanced superresolution algorithms. On the house test set amplified twice, the PSNR value produced by the algorithm in this paper is 1.21 dB higher than the one produced by the MSICF and 1.87 dB higher than the one produced by the MemNet. On the bathroom test set, the PSNR value produced by the algorithm in this paper is 2.11 dB higher than the one produced by the VDSR and 3.28 dB higher than the one produced by the SRCNN. It is obvious in Table 4 that the SSIM values produced by the algorithm in this paper are higher than those produced by the other six advanced superresolution algorithms when different magnifications and test sets are applied. With the classroom test set magnified by 4 times, the SSIM value produced by the algorithm in this paper is 0.0318 higher than the one produced by the MSICF and 0.0830 higher than the one produced by the MemNet. On the car test set, the SSIM value produced by the algorithm in this paper is 0.0321 higher than the one produced by the VDSR and 0.0505 higher than the one produced by the SRCNN. Figure 4 shows the PSNR and the SSIM values when applying different algorithms using the house and classroom test sets, and these are direct indications for demonstrating the superiority of our algorithm over the others.Table 3
Average PSNR for different algorithms at car, house, classroom, and bathroom.
DatasetScaleBicubicSRCNN [12]FSRCNN [13]VDSR [14]DRCN [15]MemNet [16]MSICF [17]OursCarx232.0235.0135.4936.2936.3536.5536.7737.99Housex227.8630.5530.9831.7431.8332.0132.6733.88Classroomx224.4226.2826.4727.1527.1627.0828.7529.72Bathroomx227.3930.0630.3531.2331.3231.4632.5833.34Carx328.6031.5932.0732.9133.1633.3433.9234.97Housex323.5125.6225.7126.8326.7426.9627.3528.51Classroomx322.3124.0924.1825.1325.1025.0626.3727.65Bathroomx323.0525.1125.2026.5426.4326.3726.8428.01Carx426.8029.8130.2331.1731.2531.4631.8832.83Housex421.7223.3423.4524.5724.5324.6625.1726.48Classroomx420.0521.7721.9222.8422.8122.8423.7224.91Bathroomx421.4823.0323.1224.2624.1824.3524.8826.03The bold values indicate the best performance.Table 4
Average SSIM for different algorithms at car, house, classroom, and bathroom.
DatasetScaleBicubicSRCNN [12]FSRCNN [13]VDSR [14]DRCN [15]MemNet [16]MSICF [17]OursCarx20.89780.93480.94450.95190.95310.96030.96100.9762Housex20.85030.91310.91270.91150.92100.92980.93490.9457Classroomx20.73410.79980.80850.82920.82750.82830.86490.8916Bathroomx20.84730.89710.90120.91010.90740.91130.93370.9406Carx30.81240.88690.89850.91900.92050.92430.92840.9421Housex30.65830.75790.76110.80350.79980.80730.82290.8544Classroomx30.61730.72250.72440.75130.75290.75310.79170.8382Bathroomx30.62880.74320.74350.79660.79680.78510.80150.8293Carx40.71270.86740.88520.88580.88630.88710.88980.9179Housex40.59030.65510.65720.67790.67840.67950.74110.7729Classroomx40.51380.59140.60150.66710.66790.71340.76460.7964Bathroomx40.57150.64780.64990.66850.66740.66770.72360.7612The bold values indicate the best performance.Figure 4
Comparison of PSNR and SSIM results with 2x, 3x, and 4x scale factors on house and classroom datasets.
(a)(b)(c)(d)(e)(f)We also record the execution time of these current algorithms, as shown in Figure5. With the car experimental test set magnified twice, the execution time of our algorithm is 0.008 seconds, the execution time of the MSICF algorithm is 0.02 seconds, and the execution time of the MemNet algorithm is 1.87 seconds. Comparing these algorithms mentioned above, our algorithm runs relatively fast and has a clear lead in producing the best efficiency.Figure 5
Comparison of running time and PSNR performance with a 2x scale factor on the car dataset.


### 4.3. Experimental Analysis

We analyze the above algorithms in terms of sensory effects. As shown in Figures6–9, we compare our algorithm with the current six advanced algorithms. Our method can clearly reconstruct the edges, lines, and textures of the image and thus improve the quality of the image by doing so. The experimental results on the “60272340–08192spp” picture in the bathroom test set enlarged by 3 times are shown in Figure 6. The image restored by our method shows a better visual experience with improved clarity of the background. The gloss and smoothness of the two cylinders have also been significantly improved, while other algorithms still have various degrees of blur or distortion.Figure 6
Comparison of reconstructed results on bathroom by different superresolution models (×3). (a) HR. (b) Bicubic. (c) SRCNN. (d) FSRCNN. (e) VDSR. (f) DRCN. (g) MemNet. (h) MSICF. (i) Ours.
(a)(b)(c)(d)(e)(f)(g)(h)(i)The experimental results on the “16532963–08192spp” picture in the car test set enlarged by 3 times are shown in Figure7. The image recovered by our method is closer to the real image. The outline of the car rearview mirror is relatively complete and smooth, and the lines, edges, and corners of the car frame are clearly visible. The recovery effect of the VDSR algorithm is obviously blurred, and the images recovered by the MemNet and the MSICF algorithms are too stiff in terms of contour and lines.Figure 7
Comparison of reconstructed results on car by different superresolution models (×3). (a) HR. (b) Bicubic. (c) SRCNN. (d) FSRCNN. (e) VDSR. (f) DRCN. (g) MemNet. (h) MSICF. (i) Ours.
(a)(b)(c)(d)(e)(f)(g)(h)(i)The effect of the experiment on the “98036695–08192spp” picture in the 4x enlarged classroom test set is shown in Figure8. The SRCNN and the DRCN algorithms blur and distort the restoration of window frame lines, and the MemNet, the MSICF, and other algorithms distort the edge of the flag. Our method can restore clear lines and visible edge of the flag and make vertical lines and corners of the wall behind the flag to look more intuitively at the same time.Figure 8
Comparison of reconstructed results on classroom by different superresolution models (×4). (a) HR. (b) Bicubic. (c) SRCNN. (d) FSRCNN. (e) VDSR. (f) DRCN. (g) MemNet. (h) MSICF. (i) Ours.
(a)(b)(c)(d)(e)(f)(g)(h)(i)The effect of the experiment on the “32481698–08192spp” picture in the 4x enlarged house test set is shown in Figure9. Most algorithms such as the FSRCNN and the DRCN do not restore enough clarity to the lines and gaps of the fence, resulting in distortion and deformation. Our method can give a clear view on the number and gap of fences, and thus, our restoration on edges, corners, and cylinders is obviously closer to the real image.Figure 9
Comparison of reconstructed results on house by different superresolution models (×4). (a) HR. (b) Bicubic. (c) SRCNN. (d) FSRCNN. (e) VDSR. (f) DRCN. (g) MemNet. (h) MSICF. (i) Ours.
(a)(b)(c)(d)(e)(f)(g)(h)(i)


## 4.1. Environment and Data

We use a computer with Intel Core I7-3770 CPU (3.40 GHz), NVIDIA GeForce GTX 1060 GPU, 16 GB memory, 64-bit Linux operating system, and 1 TB hard disk. The software includes MatlabR2016a, CUDA 8 tool kit, cuDNN 5.0, PyCharm, and PyTorch.We then select 15, 25, 50, and 100 rendered images from the classic Disney Monte Carlo datasets as our test data. The test data are downsampled using Python program code by 2, 3, and 4 times and then magnified to the original size by applying bicubic interpolation method. Finally, the images are classified and stored in four folders named car, house, classroom, and bathroom, and the preparation of experimental data is completed.


## 4.2. Evaluation and Comparison

We take the peak signal-to-noise ratio (PSNR) [29], structural similarity (SSIM) [30], and execution time (ET) as the evaluation indexes of this experiment. The shorter the execution time (ET) is [31], the faster the algorithm runs. The mathematical expression of PSNR is shown in (5) and (6), where Xori and Xres present the original image and the reconstructed image, respectively, the MSE refers to the mean squared error between Xori and Xres, and m and n represent the number of rows and columns of the image respectively. The unit of the PSNR is dB. It is known that the larger the value of the PSNR is, the closer the reconstructed image is to the ground truth image.(5)PSNR=10log102552MSE,(6)MSE=1mn∑i=1m∑j=1mXorii,j−Xresi,j2.The mathematical expression of the SSIM is shown in (7). The SSIM evaluates the image quality by comparing the similarity of the structural information between images. If the value of the SSIM is close to 1, it means that the reconstructed image is close to the original image.(7)SSIM=∑i=1m∑j=1mXorii,j×Xresi,j∑i=1m∑j=1mXorii,j2×∑i=1m∑j=1mXresi,j2.We select this training model for experimenting with car, house, classroom, and bathroom test datasets. After inputting the experimental images into our network model and outputting the three-channel RGB superresolution images, the color conversion is carried out, and the brightness values of the images are extracted as the reference values for calculating the evaluation indexes. The car datasets amplified by 2, 3, and 4 times produce an average result of the PSNR with values at 37.99 dB, 34.97 dB, and 32.83 dB and an average result of the SSIM with values of 0.9762, 0.9421, and 0.9179. To verify the effectiveness of the proposed method, the above experimental data are also input into other six advanced superresolution algorithms, the SRCNN, the FSRCNN, the VDSR, the DRCN, the MenNet, and the MSICF, for direct results comparison.As shown in Table3, the PSNR value of the algorithm proposed in this paper exceeds the above reconstruction algorithms when using different combinations of magnifications and test sets. The experimental results show that our algorithm is better than the other six advanced superresolution algorithms. On the house test set amplified twice, the PSNR value produced by the algorithm in this paper is 1.21 dB higher than the one produced by the MSICF and 1.87 dB higher than the one produced by the MemNet. On the bathroom test set, the PSNR value produced by the algorithm in this paper is 2.11 dB higher than the one produced by the VDSR and 3.28 dB higher than the one produced by the SRCNN. It is obvious in Table 4 that the SSIM values produced by the algorithm in this paper are higher than those produced by the other six advanced superresolution algorithms when different magnifications and test sets are applied. With the classroom test set magnified by 4 times, the SSIM value produced by the algorithm in this paper is 0.0318 higher than the one produced by the MSICF and 0.0830 higher than the one produced by the MemNet. On the car test set, the SSIM value produced by the algorithm in this paper is 0.0321 higher than the one produced by the VDSR and 0.0505 higher than the one produced by the SRCNN. Figure 4 shows the PSNR and the SSIM values when applying different algorithms using the house and classroom test sets, and these are direct indications for demonstrating the superiority of our algorithm over the others.Table 3
Average PSNR for different algorithms at car, house, classroom, and bathroom.
DatasetScaleBicubicSRCNN [12]FSRCNN [13]VDSR [14]DRCN [15]MemNet [16]MSICF [17]OursCarx232.0235.0135.4936.2936.3536.5536.7737.99Housex227.8630.5530.9831.7431.8332.0132.6733.88Classroomx224.4226.2826.4727.1527.1627.0828.7529.72Bathroomx227.3930.0630.3531.2331.3231.4632.5833.34Carx328.6031.5932.0732.9133.1633.3433.9234.97Housex323.5125.6225.7126.8326.7426.9627.3528.51Classroomx322.3124.0924.1825.1325.1025.0626.3727.65Bathroomx323.0525.1125.2026.5426.4326.3726.8428.01Carx426.8029.8130.2331.1731.2531.4631.8832.83Housex421.7223.3423.4524.5724.5324.6625.1726.48Classroomx420.0521.7721.9222.8422.8122.8423.7224.91Bathroomx421.4823.0323.1224.2624.1824.3524.8826.03The bold values indicate the best performance.Table 4
Average SSIM for different algorithms at car, house, classroom, and bathroom.
DatasetScaleBicubicSRCNN [12]FSRCNN [13]VDSR [14]DRCN [15]MemNet [16]MSICF [17]OursCarx20.89780.93480.94450.95190.95310.96030.96100.9762Housex20.85030.91310.91270.91150.92100.92980.93490.9457Classroomx20.73410.79980.80850.82920.82750.82830.86490.8916Bathroomx20.84730.89710.90120.91010.90740.91130.93370.9406Carx30.81240.88690.89850.91900.92050.92430.92840.9421Housex30.65830.75790.76110.80350.79980.80730.82290.8544Classroomx30.61730.72250.72440.75130.75290.75310.79170.8382Bathroomx30.62880.74320.74350.79660.79680.78510.80150.8293Carx40.71270.86740.88520.88580.88630.88710.88980.9179Housex40.59030.65510.65720.67790.67840.67950.74110.7729Classroomx40.51380.59140.60150.66710.66790.71340.76460.7964Bathroomx40.57150.64780.64990.66850.66740.66770.72360.7612The bold values indicate the best performance.Figure 4
Comparison of PSNR and SSIM results with 2x, 3x, and 4x scale factors on house and classroom datasets.
(a)(b)(c)(d)(e)(f)We also record the execution time of these current algorithms, as shown in Figure5. With the car experimental test set magnified twice, the execution time of our algorithm is 0.008 seconds, the execution time of the MSICF algorithm is 0.02 seconds, and the execution time of the MemNet algorithm is 1.87 seconds. Comparing these algorithms mentioned above, our algorithm runs relatively fast and has a clear lead in producing the best efficiency.Figure 5
Comparison of running time and PSNR performance with a 2x scale factor on the car dataset.


## 4.3. Experimental Analysis

We analyze the above algorithms in terms of sensory effects. As shown in Figures6–9, we compare our algorithm with the current six advanced algorithms. Our method can clearly reconstruct the edges, lines, and textures of the image and thus improve the quality of the image by doing so. The experimental results on the “60272340–08192spp” picture in the bathroom test set enlarged by 3 times are shown in Figure 6. The image restored by our method shows a better visual experience with improved clarity of the background. The gloss and smoothness of the two cylinders have also been significantly improved, while other algorithms still have various degrees of blur or distortion.Figure 6
Comparison of reconstructed results on bathroom by different superresolution models (×3). (a) HR. (b) Bicubic. (c) SRCNN. (d) FSRCNN. (e) VDSR. (f) DRCN. (g) MemNet. (h) MSICF. (i) Ours.
(a)(b)(c)(d)(e)(f)(g)(h)(i)The experimental results on the “16532963–08192spp” picture in the car test set enlarged by 3 times are shown in Figure7. The image recovered by our method is closer to the real image. The outline of the car rearview mirror is relatively complete and smooth, and the lines, edges, and corners of the car frame are clearly visible. The recovery effect of the VDSR algorithm is obviously blurred, and the images recovered by the MemNet and the MSICF algorithms are too stiff in terms of contour and lines.Figure 7
Comparison of reconstructed results on car by different superresolution models (×3). (a) HR. (b) Bicubic. (c) SRCNN. (d) FSRCNN. (e) VDSR. (f) DRCN. (g) MemNet. (h) MSICF. (i) Ours.
(a)(b)(c)(d)(e)(f)(g)(h)(i)The effect of the experiment on the “98036695–08192spp” picture in the 4x enlarged classroom test set is shown in Figure8. The SRCNN and the DRCN algorithms blur and distort the restoration of window frame lines, and the MemNet, the MSICF, and other algorithms distort the edge of the flag. Our method can restore clear lines and visible edge of the flag and make vertical lines and corners of the wall behind the flag to look more intuitively at the same time.Figure 8
Comparison of reconstructed results on classroom by different superresolution models (×4). (a) HR. (b) Bicubic. (c) SRCNN. (d) FSRCNN. (e) VDSR. (f) DRCN. (g) MemNet. (h) MSICF. (i) Ours.
(a)(b)(c)(d)(e)(f)(g)(h)(i)The effect of the experiment on the “32481698–08192spp” picture in the 4x enlarged house test set is shown in Figure9. Most algorithms such as the FSRCNN and the DRCN do not restore enough clarity to the lines and gaps of the fence, resulting in distortion and deformation. Our method can give a clear view on the number and gap of fences, and thus, our restoration on edges, corners, and cylinders is obviously closer to the real image.Figure 9
Comparison of reconstructed results on house by different superresolution models (×4). (a) HR. (b) Bicubic. (c) SRCNN. (d) FSRCNN. (e) VDSR. (f) DRCN. (g) MemNet. (h) MSICF. (i) Ours.
(a)(b)(c)(d)(e)(f)(g)(h)(i)


## 5. Conclusion

In this paper, the main contributions are as follows: (1) Through the above experiments, it is proved that our method is able to extract more features of the image and reconstruct the image with clearer details and texture than other related algorithms. (2) Film and television producers can save time on rendering images and then utilize our proposed model to superresolution the images later. This provides an excellent cost-effective solution that may increase the profitability to the current video industry. (3) Our method does not affect the audience’s sensory experience when LR images come to play. This is a major advantage when data size is limited. The research content of this paper provides an initial solution to the stated problem with promising results. More specifically, we have achieved the expected research objectives, have reached the goal of creating a cost-effective solution to the film and television production without affecting the audience’s sensory experience, and thus have brought distinct economic values to the industry. In the future, we will continue to optimize the parameters of the network model and to introduce the attention mechanism into the model to improve the efficiency and recovery effect of the algorithm. At the same time, we will also apply our research results to the field of medical CT imaging and infrared imaging with remote sensing.



---

*Source: 9393589-2022-08-29.xml*