# An Old Photo Image Restoration Processing Based on Deep Neural Network Structure

Authors: Ruoyan Wang
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Old photos retain precious historical image information, but today’s existing old photos often have varying degrees of damage. Although these old photos can be digitally processed and then restored, the restoration of old photos involves multiple areas of image restoration and has various types of degradation. Currently, there is no unified model for repairing multiple types of degradation of old photos. Photo restoration technology still has a lot of developments. Traditional image restoration technology mainly repairs the missing areas of the image based on mathematical formulas or thermal diffusion. This technology can only repair images with simple structures and small damaged areas and is difficult to apply in people’s daily lives. The emergence of deep learning technology has accelerated the pace of research on image restoration. This article will discuss the methods of repairing old photos based on deep neural networks. It is aimed at proposing an image restoration method based on deep neural network to enhance the effect of image restoration of old photos and provide more possibilities for restoration of old photos. This article discusses the background significance of image restoration methods, designs an image restoration model based on deep neural networks, and introduces the structure, principle, and loss function of the model. Finally, this article did a comparative experiment to compare the model in this article with other models and draw the conclusion: in the blur repair experiment, the algorithm in this paper is better than other algorithms for the peak signal-to-noise ratio and structure similarity of the repaired image; in the damage repair experiment, the value of the algorithm’ s peak signal-to-noise ratio is 32.34, and the structure similarity under different damage average levels is 0.767, which is also higher than other algorithms. Therefore, the model in this paper has the best effect on image restoration.


---

## Body

## 1. Introduction

As a medium of information transmission, photos record people’s meaningful time and carry many people’s beautiful memories. But decades ago, people could only save the few paper-based photos in photo frames and photo albums. With the passage of time, many old photos have turned yellow due to oxidation, and they are even easily torn apart by accident. With the advent of image restoration technology, photos that have been yellowed or even damaged can be virtually restored on a computer to preserve people’s memories of the past. Deep learning is a branch of machine learning. It is an algorithm that uses artificial neural networks as the architecture to characterize and learn data. In recent years, deep learning has been widely used in computer vision tasks due to its powerful representation of image data. Some researchers have tried to apply deep learning to image tasks to improve the high-quality content structure and structure of image restoration algorithms. Texture capabilities, especially deep convolutional neural networks and generative adversarial networks, perform well in feature extraction and image generation and have good image restoration effects on old photos. Therefore, more and more people use deep neural networks for image restoration of old photos.Traditionally, image restoration can use diffusion-based methods to deal with missing scene information and perceptual information. This method spreads the local structure to the location part or constructs one pixel of the missing part every time, while maintaining surrounding pixels’ consistency. Although traditional algorithms can solve some of the problems of hole repair and crease repair in old photos, these methods only focus on defect filling and are having difficulty solving the fading and blur problems of old photos. Therefore, an additional component is needed to provide reasonable imagination (the illusion from the machine). This additional information may be provided by high-level models of natural images, such as deep neural network calculations and other neural network methods, which mainly rely on the illusion of a pretrained neural network to fill in the missing parts of the image. Repairing technology based on deep neural network saves time and effort [1, 2]. As one of the branches of image processing technology, this technology can be used to repair damaged photo paintings and remove redundant texts.The innovation of this paper is (1) propose a new image repair model; the design of this model is based on the convolutional neural network in the deep neural network and the generative adversarial network; the model is the result of the two improvements. (2) Two experiments are designed, which are, namely, blur repair and damage repair. The algorithm in this paper is compared with other algorithms, and it is verified that the algorithm in this paper is superior to other algorithms in image blur repair and damage repair.


## 2. Relating Work

Image restoration is to process a damaged image to make it a perfect image. This technology is of great significance to many fields, so many scholars have conducted research on it. Duan and Wang proposed a neural dynamics image restoration method. The Gang method uses Echo State Network (ESN) to estimate the original image. Duan and Wang compare the results obtained with the improved ESN with the results obtained with several state-of-the-art methods. Experiments have proved that the proposed neural dynamics method can make different fuzzy or noisy instances get better image recovery [3]. Xu et al. proposed a two-level domain decomposition method to directly solve the total variational minimization problem. The method is composed of overlapping domain decomposition technology and coarse grid correction, and an iterative algorithm forms a small sized and better condition system. Experiments show that this method is fast and robust, especially suitable for large-size images, but this method has some flaws in detail design [4]. Peng and Cosman proposed an underwater scene depth estimation method based on image blur and light absorption, which can be used to restore and enhance underwater images in image formation models. This method can better accurately estimate the depth of underwater scenes. The experimental results of restoration of real and synthetic underwater images show that this method is superior to other underwater image restoration methods based on IFM, but the disadvantage of this experiment is that the dimensions of the experimental test are somewhat less [5]. Nikonorov et al. proved that the combination of high-precision lens design and postcapture calculation and reconstruction can achieve higher image quality. In the reconstruction process, Nikonorov et al. used a series of color correction, deconvolution, and feed-forward deep learning neural networks and found that improvements in lens manufacturing and image processing may contribute to the emergence of ultralight imaging systems. The shortcomings of this research are the lack of data display [6]. Ono proposes a new plug-and-play image restoration method based on primal-dual splitting. This method can use the properties of primal-dual segmentation to make up for the shortcomings of traditional methods and is a very flexible plug-and-play image restoration method. Experimental results show that this method is more efficient than traditional algorithms [7]. Liu et al. introduced a nonlocal (NL) extension to TV regularization, which uses pixel-level content adaptive distribution to model the sparsity of image gradients. Liu et al. also use the NL similarity of natural images to achieve a more accurate estimation of the gradient. The experimental results show that the proposed method can produce better objective and subjective image quality, but the experimental method is more complicated and has less operability [8]. Dong et al. introduced a general image restoration model based on wavelet frames, called “general model,” which includes most of the existing wavelet frame-based models as special cases. Dong et al. also performed an asymptotic analysis of the general model when the image resolution reached infinity, thereby establishing a connection between the general model in the discrete environment and the new variable model in the continuum environment. The flaw of this study is that the established connection is not convincing enough [9].


## 3. Image Restoration Processing of Old Photos Based on Deep Neural Network

### 3.1. Deep Neural Network Theory on Image Restoration

Deep learning is a branch of machine learning, which is commonly used to learn the underlying laws and distributions in data. Deep learning models are usually composed of neural networks. And commonly used deep learning models include Convolutional neural networks (CNN) and generative adversarial networks (GAN). More and more researchers try to apply CNN and GAN to image restoration tasks and propose some image restoration models based on CNN and GAN. Comparing with traditional image restoration modeling, the deep learning-based method takes advantage of big data [10–12]. It learns the deep features of the image from training to obtain the representation ability and significantly improves the quality of image restoration [13].


#### 3.1.1. Convolutional Neural Network

Convolutional neural network (CNN) is a neural network specifically designed for data with a grid-like structure. CNN can use the spatial structure relationship to reduce the number of parameters that need to be learned, thereby improving the training efficiency of the backpropagation algorithm. Convolutional layer, pooling layer, and nonlinear activation layer are the basic components of CNN, which will be described in details.(1)
Convolutional layerThe convolutional layer of CNN is composed of many neurons, and the shared weights among the three network connection methods are used between every two convolutional layers. In image processing, the convolution operation is generally composed of several convolution kernels, which are used to extract different features in the image, such as horizontal, vertical, and diagonal edges, and the deeper the convolutional layer, the higher the level of features extracted. CNN extracts features by automatically learning the weights of the convolution kernels during training, and the features extracted by different convolution kernels will be combined into the input of the next layer, so more advanced features can be obtained [14–16]. The calculation process of convolution is shown in Figure 1.Figure 1
Convolution calculation process.In recent years, with the emergence of different visual tasks, many variants of convolution have been proposed, such as deconvolution, subpixel convolution, and hole convolution. Deconvolution can solve the problem of semantic pixel-level reconstruction. Subpixel convolution can get information from the direct original image. Hole convolution can expand the field of perception without using pooling operations.Assume that the pixel value corresponding to the point (x,y) of the subimage of the input image is pxy, and the pixel value corresponding to the point (m,n) of the convolution kernel matrix is kmn. The convolution kernel acts on a certain position of the image, and the calculation formula is shown.
(1)f∑m−1s∑n−1skmn∙px+m−1,y+n−1+c.F∙ is the activation function, and c is the bias term.
(2)
Pooling layerThe pooling layer is one of the commonly used components in the current convolutional neural network. It imitates the human visual system to reduce the dimensionality of the data and uses more advanced features to represent the image. Pooling has the effect of reducing information redundancy, improving the scale invariance of the model, and preventing overfitting. The pooling operation is shown in Figure2.
(3)
Nonlinear activation layerFigure 2
Pooling operation.In order to solve the problem of inseparable nonlinearity, the convolutional neural network usually introduces an activation function layer, so that the network has the learning ability of nonlinear mapping. Common activation functions include Sigmoid function, ELU function, and ReLU function [17–19]. The formulas are as follows:
(2)Sigmoidx=11+exp−x,ELUx=maxαexpx−1,xα>0,ReLUx=max0,x.Figure3 is the image of the Sigmoid function, ELU function, and ReLU function, respectively. Among them, the Sigmoid function is smooth and easy to derive, but the activation function is computationally expensive. ELU does not have the problem of neuron death, which can shorten the training time and improve accuracy in the neural network. The ReLU function can more efficiently perform gradient descent, and reverse propagation avoids the problems of gradient explosion and gradient disappearance.Figure 3
Function image.


#### 3.1.2. Generative Confrontation Network

Generative confrontation network is a generative model developed in recent years, and it has its presence in many applications, such as image generation and text generation. The model has two main characteristics. One is that the model does not require any prior assumptions; the other is that the model relies on the forward propagation of its generator to generate real-like samples, which is simple.GAN is divided into two parts: generation network and discriminant network. The steps are as follows: first, the generation network generates a sample that is as similar to the original sample as possible according to the input information, then judges the generated sample and the original sample through the discriminant network, then continues to repeat this process, and finally reaches a balance [20]. The schematic diagram is shown in Figure 4.Figure 4
Schematic diagram of the steps to generate a confrontation network.In Figure4, Z represents the input information, Generator represents the generation network, Xfake represents the samples generated by the generation network, Xreal represents the real samples, and Discriminator represents the discrimination network which is mainly to distinguish between Xfake and Xreal. The optimized objective function is as follows:
(3)minmaxG,D=minmaxEx−pdataxlogDx+Ez−pzlog1−DGz.Among them,DGz represents the probability of the generated sample passing the discrimination. In its objective function, in order to ensure that VG,D can achieve the maximum value, generally, the discriminator is trained once, and then, the generator is trained. When the generator is fixed, the optimal discriminator can be obtained by deriving VD,G; the formula is
(4)D′x=PgxPgx+Pdatax.There are two parts to the generative adversarial network, namely, the generator and the discriminator. In the whole process, the generator makes the image itself generate more in line with the real image as much as possible to deceive the discriminator; while the discriminator is just the opposite; it tries its best to distinguish the true and false of the image. The constant game between the two eventually reached a dynamic equilibrium [21]. At this time, the image generated by the generator can be false and true, but the discriminator cannot determine the authenticity of the image, and the true or false judgment of the image is roughly 0.5.


### 3.2. Design of an Old Photo Image Restoration Model Based on Deep Neural Network

(1)
Self-attention mechanism designThe principle of self-attention is similar to that of the human brain. When watching pictures or listening to speech, people tend to focus on the location where the information is dizzy, ignoring some unimportant details, because the attention mechanism balances the relationship between modeling ability and computational efficiency and makes up for the lack of convolution. When there are multiple levels of dependencies between different regions of the image, the attention mechanism can be used to capture this deep level. Global connection and coordination of the details of each pixel in the image are observed, so the attention mechanism can be applied to the residual module of the generator to better construct some texture details in the image superresolution task [22, 23], and its specific structure is shown in Figure 5.Figure 5
Self-attention mechanism structure.After two convolution operations, the attention score can be obtained, which is calculated as follows:(5)aj,i=expsij∑i=1nexpsij,wheresij=fxitgxj.aj,i represents the attention score, which represents the degree of attention of the network to i when the synthesis area is j, and the weighted feature of the attention layer is expressed as o=o1,o2,⋯,on to obtain
(6)oi=∑i=1naj,ihxi,hxi=Whxi.Multiply the attention-weighted feature output with a balance parameter, and then, reinput it into the attention feature map, as shown in the following formula:(7)yi=γoi+xi.γ represents the balance parameter. Its function is to adjust the weight of the decoder features and the resulting attention features. yi represents the final output result of the attention layer.
(2)
Network structure designIn order to reduce the impact of the generated data and reduce the spatial distribution difference between the synthesized photo and the real photo, the old photo restoration is described here as an image conversion problem. Regarding clear and complete photos and old photos as images from different spaces, in order to learn the mapping between them, this article transforms the images in three spaces [24], as shown in Figure 6.Figure 6
Model network structure.The key to this method is that real old photo data and synthetic photo data can be encoded into the same hidden space. For this reason, we can use a variational auto encoder (variational auto encoder is an important type of generative model. Since its proposal, it has shown strong unsupervised learning ability, with fast training speed, low training cost, strong robustness, and high quality of reconstructed images) to encode the image and then use the generative adversarial network to adjust the network according to the domain gap. In the first stage, two variational auto encoder (VAE) networks are used to do spatial mapping. When the input object isi, the expression of the objective function is as follows:
(8)LVAEi=KLER,XZi,iN0,1+αEZi−ER,XZi,iGR,XiR⟶R∣Zi−i1+LVAE,GANi.ER,X represents the encoder, and ER,XZi,i represents the prior probability distribution of Zi obeyed by ER,X when input is i.The second stage is a mapping network that uses synthetic image pairs {x,y} to guide image restoration through their hidden space mapping. In simple terms, the role of the mapping network is to restore degraded photos to original photos [25]. At this stage, the loss function of the mapping network μ is
(9)Lμx,y=λ1Lμ,l1+Lμ,GAN+λ2LFM.Lμ,l1 represents the loss of L1, and Lμ,GAN represents the generative adversarial network, which is used to synthesize photos that look real.
(3)
Degradation repair designIn some old photos, for the damage of some structural defects, it may often be necessary to search for the information of the entire image and then select effective information to fill in to maintain the consistency of the global structure. Therefore, it is necessary to design a support for local search and global search which is the network of two mechanisms [26]. In this chapter, the scratch mask of the original old photo can be used as input to prevent the network from using the pixels of the damaged area to repair the damaged area. Let C, H, and W be the number of channels, height, and width, respectively. Let FϵRC×HW be the mapping feature of the middle layer; mϵ0,1HW means that the binary mask is scaled to the same size; m is 1 to indicate that the area is damaged and 0 to indicate that the area is intact. The relationship between position i and position j in the feature map F is represented by sij, and sijϵFϵRHW×HW represents the relationship between each pixel, as follows:
(10)sij=fij1−mj∑∀k1−mkfi,k,fij=expθFiT∗ϕFj.BothFi and Fj are vectors of C∗1, and θ (∙) and ϕ (∙) are functions that make F map to the Gaussian distribution.Therefore, when the input mask area is identified as a damaged area, the global information can be used to repair, otherwise the local feature information can be used, so the global branch and the local branch can be merged under the guidance of the input mask. The formula is as follows:(11)Ffuse=1−m⨀ρlocalF+m⨀ρglobalo.In the formula,⨀ represents the Hadamard product of the matrix, and ρlocal and ρglobal both represent the nonlinear transformation of the branch residual block, which can deal with the structural defects of old photos. The specific network structure is shown in Table 1.
(4)
Network parameter setting of generator and discriminatorTable 1
Network structure convolution design.
ModuleLayerKernel sizeOutput sizeEncoderEConv7∗7/1256∗256∗64Conv4∗4/2128∗128∗64Conv4∗4/264∗64∗64GeneratorGResblock∗43∗3/164∗64∗64Deconv4∗4/2128∗128∗64Conv7∗7/1256∗256∗3MappingτConv3∗3/164∗64∗128Resblock∗63∗3/164∗64∗512Partial nonlocal1∗1/164∗64∗512The generative confrontation network is mainly composed of a generator and a discriminator. The generator is responsible for generating a high-resolution image similar to the real image. The discriminator is used to determine whether the generated image comes from real training set data or a fake one. The generator in this article consists of three parts. The first part is the downsampling module; you can see three convolution kernels of different sizes. The second part is the self-attention residual module, which is composed of 16 identical self-attention residual blocks, and each small module is composed of a self-attention layer and a convolutional layer. The last part is the upsampling image reconstruction module. This part is mainly two subpixel convolutional layers for pixel amplification. Comparing with ordinary deconvolution, the subpixel convolution can reduce the risk caused by artificial factors and make the reconstructed image quality higher.The task of the discriminator is to determine whether the input image is from the sample space or a fake image generated by the generator and to learn the difference between the two. In this chapter, the discriminator network uses a deep convolutional neural network, a batch normalization layer is added to the discriminator network, and the LeakyRelu activation function (α=0.2) is used [27]. The specific method is to input the generated image into the network through multilayer convolution to obtain the feature map and then input the feature map to the fully connected layer. Finally, the Sigmoid activation function is used to perform two classifications of true and false to determine whether it is a real picture or a fake picture. So far, the design of the generator and the discriminator is completed, and the two generate high-resolution images through confrontational optimization to realize the high-definition processing of old photos.


### 3.3. Loss Function Design

The loss function is a function proposed to evaluate the difference between the generated image and the real image in the image restoration model. This function guides the learning of the image restoration model through the evaluation results. The smaller the value of the loss function, the better the robustness of the model. In the image superresolution reconstruction, the loss function is used to determine the difference between the original image and the generated image as an optimization function of the model. The loss function in this paper is composed of three parts: appearance matching loss, KL divergence loss, and confrontation loss. Appearance matching loss is used to constrain the generation of image structure, KL divergence loss narrows the high-dimensional spatial distribution between the generated image and the real image, and the confrontation loss is used to reduce the difference between the generated image and the real image [28, 29].
(1)
KL lossThe KL loss makes the generated path close to the standard normal distribution of the high-dimensional space of the reconstructed path. This article has two KL loss functions.(12)lKLr=−KLqz∣IcNm0,σ2nI,lKLb=−KLqz∣Icpϕz∣Im.In the formula,lKLr represents the KL loss of the reconstructed path, lKLb represents the KL loss of the generated path, z represents the latent vector, q∙∣∙ represents the importance sampling function, Nm0,σ2nI represents the normal distribution, and pϕ∙∣∙ represents the conditional prior [30].
(2)
Appearance matching lossIn order to increase the ability of the generator to generate images, the image features extracted in the encoding stage are input into the decoder to generate a reconstructed image. In this paper, the appearance matching loss is used to constrain the generation of the image structure. The loss function of the appearance matching loss is shown in the following formula:(13)lappr=Irec−Ig1,lappb=M∗Igen−Ig1.In the formula,lappr represents the appearance matching loss of the reconstructed path, lappb represents the appearance matching loss of the generated path, Irec represents the reconstructed image, Igen represents the reconstructed image, Ig represents the real image, and M represents the binary mask of visible pixels.
(3)
Fight against lossThis part of the loss is composed of generator loss and discriminator loss [31]. The calculation formula of generator loss is shown in the following formula:
(14)LG=EDGz.Z represents the random noise input to the generating network, Gz represents the sample generated by the generating network, and DGz represents the output result when the input of the network is judged to be the generated sample.The loss function of the discriminator is as follows:(15)Li=EDx−EDGz+Lgp,Lgp=λE∣∇Dαx−1−αGz∣−12,LD=Li+Lgp.(4)
Total loss functionThe total loss function of the network consists of three parts: KL loss, appearance matching loss, and counter loss. The calculation formula of the total loss function is shown in the following formula:(16)L=αKLLKLr+LKLb+αapplappr+lappb+αadLadr+Ladb.In the formula,αKL, αapp, and αad all represent hyperparameters, which are used to adjust the proportions of KL loss, appearance matching loss, and counter loss in the total loss function.


## 3.1. Deep Neural Network Theory on Image Restoration

Deep learning is a branch of machine learning, which is commonly used to learn the underlying laws and distributions in data. Deep learning models are usually composed of neural networks. And commonly used deep learning models include Convolutional neural networks (CNN) and generative adversarial networks (GAN). More and more researchers try to apply CNN and GAN to image restoration tasks and propose some image restoration models based on CNN and GAN. Comparing with traditional image restoration modeling, the deep learning-based method takes advantage of big data [10–12]. It learns the deep features of the image from training to obtain the representation ability and significantly improves the quality of image restoration [13].


### 3.1.1. Convolutional Neural Network

Convolutional neural network (CNN) is a neural network specifically designed for data with a grid-like structure. CNN can use the spatial structure relationship to reduce the number of parameters that need to be learned, thereby improving the training efficiency of the backpropagation algorithm. Convolutional layer, pooling layer, and nonlinear activation layer are the basic components of CNN, which will be described in details.(1)
Convolutional layerThe convolutional layer of CNN is composed of many neurons, and the shared weights among the three network connection methods are used between every two convolutional layers. In image processing, the convolution operation is generally composed of several convolution kernels, which are used to extract different features in the image, such as horizontal, vertical, and diagonal edges, and the deeper the convolutional layer, the higher the level of features extracted. CNN extracts features by automatically learning the weights of the convolution kernels during training, and the features extracted by different convolution kernels will be combined into the input of the next layer, so more advanced features can be obtained [14–16]. The calculation process of convolution is shown in Figure 1.Figure 1
Convolution calculation process.In recent years, with the emergence of different visual tasks, many variants of convolution have been proposed, such as deconvolution, subpixel convolution, and hole convolution. Deconvolution can solve the problem of semantic pixel-level reconstruction. Subpixel convolution can get information from the direct original image. Hole convolution can expand the field of perception without using pooling operations.Assume that the pixel value corresponding to the point (x,y) of the subimage of the input image is pxy, and the pixel value corresponding to the point (m,n) of the convolution kernel matrix is kmn. The convolution kernel acts on a certain position of the image, and the calculation formula is shown.
(1)f∑m−1s∑n−1skmn∙px+m−1,y+n−1+c.F∙ is the activation function, and c is the bias term.
(2)
Pooling layerThe pooling layer is one of the commonly used components in the current convolutional neural network. It imitates the human visual system to reduce the dimensionality of the data and uses more advanced features to represent the image. Pooling has the effect of reducing information redundancy, improving the scale invariance of the model, and preventing overfitting. The pooling operation is shown in Figure2.
(3)
Nonlinear activation layerFigure 2
Pooling operation.In order to solve the problem of inseparable nonlinearity, the convolutional neural network usually introduces an activation function layer, so that the network has the learning ability of nonlinear mapping. Common activation functions include Sigmoid function, ELU function, and ReLU function [17–19]. The formulas are as follows:
(2)Sigmoidx=11+exp−x,ELUx=maxαexpx−1,xα>0,ReLUx=max0,x.Figure3 is the image of the Sigmoid function, ELU function, and ReLU function, respectively. Among them, the Sigmoid function is smooth and easy to derive, but the activation function is computationally expensive. ELU does not have the problem of neuron death, which can shorten the training time and improve accuracy in the neural network. The ReLU function can more efficiently perform gradient descent, and reverse propagation avoids the problems of gradient explosion and gradient disappearance.Figure 3
Function image.


### 3.1.2. Generative Confrontation Network

Generative confrontation network is a generative model developed in recent years, and it has its presence in many applications, such as image generation and text generation. The model has two main characteristics. One is that the model does not require any prior assumptions; the other is that the model relies on the forward propagation of its generator to generate real-like samples, which is simple.GAN is divided into two parts: generation network and discriminant network. The steps are as follows: first, the generation network generates a sample that is as similar to the original sample as possible according to the input information, then judges the generated sample and the original sample through the discriminant network, then continues to repeat this process, and finally reaches a balance [20]. The schematic diagram is shown in Figure 4.Figure 4
Schematic diagram of the steps to generate a confrontation network.In Figure4, Z represents the input information, Generator represents the generation network, Xfake represents the samples generated by the generation network, Xreal represents the real samples, and Discriminator represents the discrimination network which is mainly to distinguish between Xfake and Xreal. The optimized objective function is as follows:
(3)minmaxG,D=minmaxEx−pdataxlogDx+Ez−pzlog1−DGz.Among them,DGz represents the probability of the generated sample passing the discrimination. In its objective function, in order to ensure that VG,D can achieve the maximum value, generally, the discriminator is trained once, and then, the generator is trained. When the generator is fixed, the optimal discriminator can be obtained by deriving VD,G; the formula is
(4)D′x=PgxPgx+Pdatax.There are two parts to the generative adversarial network, namely, the generator and the discriminator. In the whole process, the generator makes the image itself generate more in line with the real image as much as possible to deceive the discriminator; while the discriminator is just the opposite; it tries its best to distinguish the true and false of the image. The constant game between the two eventually reached a dynamic equilibrium [21]. At this time, the image generated by the generator can be false and true, but the discriminator cannot determine the authenticity of the image, and the true or false judgment of the image is roughly 0.5.


## 3.1.1. Convolutional Neural Network

Convolutional neural network (CNN) is a neural network specifically designed for data with a grid-like structure. CNN can use the spatial structure relationship to reduce the number of parameters that need to be learned, thereby improving the training efficiency of the backpropagation algorithm. Convolutional layer, pooling layer, and nonlinear activation layer are the basic components of CNN, which will be described in details.(1)
Convolutional layerThe convolutional layer of CNN is composed of many neurons, and the shared weights among the three network connection methods are used between every two convolutional layers. In image processing, the convolution operation is generally composed of several convolution kernels, which are used to extract different features in the image, such as horizontal, vertical, and diagonal edges, and the deeper the convolutional layer, the higher the level of features extracted. CNN extracts features by automatically learning the weights of the convolution kernels during training, and the features extracted by different convolution kernels will be combined into the input of the next layer, so more advanced features can be obtained [14–16]. The calculation process of convolution is shown in Figure 1.Figure 1
Convolution calculation process.In recent years, with the emergence of different visual tasks, many variants of convolution have been proposed, such as deconvolution, subpixel convolution, and hole convolution. Deconvolution can solve the problem of semantic pixel-level reconstruction. Subpixel convolution can get information from the direct original image. Hole convolution can expand the field of perception without using pooling operations.Assume that the pixel value corresponding to the point (x,y) of the subimage of the input image is pxy, and the pixel value corresponding to the point (m,n) of the convolution kernel matrix is kmn. The convolution kernel acts on a certain position of the image, and the calculation formula is shown.
(1)f∑m−1s∑n−1skmn∙px+m−1,y+n−1+c.F∙ is the activation function, and c is the bias term.
(2)
Pooling layerThe pooling layer is one of the commonly used components in the current convolutional neural network. It imitates the human visual system to reduce the dimensionality of the data and uses more advanced features to represent the image. Pooling has the effect of reducing information redundancy, improving the scale invariance of the model, and preventing overfitting. The pooling operation is shown in Figure2.
(3)
Nonlinear activation layerFigure 2
Pooling operation.In order to solve the problem of inseparable nonlinearity, the convolutional neural network usually introduces an activation function layer, so that the network has the learning ability of nonlinear mapping. Common activation functions include Sigmoid function, ELU function, and ReLU function [17–19]. The formulas are as follows:
(2)Sigmoidx=11+exp−x,ELUx=maxαexpx−1,xα>0,ReLUx=max0,x.Figure3 is the image of the Sigmoid function, ELU function, and ReLU function, respectively. Among them, the Sigmoid function is smooth and easy to derive, but the activation function is computationally expensive. ELU does not have the problem of neuron death, which can shorten the training time and improve accuracy in the neural network. The ReLU function can more efficiently perform gradient descent, and reverse propagation avoids the problems of gradient explosion and gradient disappearance.Figure 3
Function image.


## 3.1.2. Generative Confrontation Network

Generative confrontation network is a generative model developed in recent years, and it has its presence in many applications, such as image generation and text generation. The model has two main characteristics. One is that the model does not require any prior assumptions; the other is that the model relies on the forward propagation of its generator to generate real-like samples, which is simple.GAN is divided into two parts: generation network and discriminant network. The steps are as follows: first, the generation network generates a sample that is as similar to the original sample as possible according to the input information, then judges the generated sample and the original sample through the discriminant network, then continues to repeat this process, and finally reaches a balance [20]. The schematic diagram is shown in Figure 4.Figure 4
Schematic diagram of the steps to generate a confrontation network.In Figure4, Z represents the input information, Generator represents the generation network, Xfake represents the samples generated by the generation network, Xreal represents the real samples, and Discriminator represents the discrimination network which is mainly to distinguish between Xfake and Xreal. The optimized objective function is as follows:
(3)minmaxG,D=minmaxEx−pdataxlogDx+Ez−pzlog1−DGz.Among them,DGz represents the probability of the generated sample passing the discrimination. In its objective function, in order to ensure that VG,D can achieve the maximum value, generally, the discriminator is trained once, and then, the generator is trained. When the generator is fixed, the optimal discriminator can be obtained by deriving VD,G; the formula is
(4)D′x=PgxPgx+Pdatax.There are two parts to the generative adversarial network, namely, the generator and the discriminator. In the whole process, the generator makes the image itself generate more in line with the real image as much as possible to deceive the discriminator; while the discriminator is just the opposite; it tries its best to distinguish the true and false of the image. The constant game between the two eventually reached a dynamic equilibrium [21]. At this time, the image generated by the generator can be false and true, but the discriminator cannot determine the authenticity of the image, and the true or false judgment of the image is roughly 0.5.


## 3.2. Design of an Old Photo Image Restoration Model Based on Deep Neural Network

(1)
Self-attention mechanism designThe principle of self-attention is similar to that of the human brain. When watching pictures or listening to speech, people tend to focus on the location where the information is dizzy, ignoring some unimportant details, because the attention mechanism balances the relationship between modeling ability and computational efficiency and makes up for the lack of convolution. When there are multiple levels of dependencies between different regions of the image, the attention mechanism can be used to capture this deep level. Global connection and coordination of the details of each pixel in the image are observed, so the attention mechanism can be applied to the residual module of the generator to better construct some texture details in the image superresolution task [22, 23], and its specific structure is shown in Figure 5.Figure 5
Self-attention mechanism structure.After two convolution operations, the attention score can be obtained, which is calculated as follows:(5)aj,i=expsij∑i=1nexpsij,wheresij=fxitgxj.aj,i represents the attention score, which represents the degree of attention of the network to i when the synthesis area is j, and the weighted feature of the attention layer is expressed as o=o1,o2,⋯,on to obtain
(6)oi=∑i=1naj,ihxi,hxi=Whxi.Multiply the attention-weighted feature output with a balance parameter, and then, reinput it into the attention feature map, as shown in the following formula:(7)yi=γoi+xi.γ represents the balance parameter. Its function is to adjust the weight of the decoder features and the resulting attention features. yi represents the final output result of the attention layer.
(2)
Network structure designIn order to reduce the impact of the generated data and reduce the spatial distribution difference between the synthesized photo and the real photo, the old photo restoration is described here as an image conversion problem. Regarding clear and complete photos and old photos as images from different spaces, in order to learn the mapping between them, this article transforms the images in three spaces [24], as shown in Figure 6.Figure 6
Model network structure.The key to this method is that real old photo data and synthetic photo data can be encoded into the same hidden space. For this reason, we can use a variational auto encoder (variational auto encoder is an important type of generative model. Since its proposal, it has shown strong unsupervised learning ability, with fast training speed, low training cost, strong robustness, and high quality of reconstructed images) to encode the image and then use the generative adversarial network to adjust the network according to the domain gap. In the first stage, two variational auto encoder (VAE) networks are used to do spatial mapping. When the input object isi, the expression of the objective function is as follows:
(8)LVAEi=KLER,XZi,iN0,1+αEZi−ER,XZi,iGR,XiR⟶R∣Zi−i1+LVAE,GANi.ER,X represents the encoder, and ER,XZi,i represents the prior probability distribution of Zi obeyed by ER,X when input is i.The second stage is a mapping network that uses synthetic image pairs {x,y} to guide image restoration through their hidden space mapping. In simple terms, the role of the mapping network is to restore degraded photos to original photos [25]. At this stage, the loss function of the mapping network μ is
(9)Lμx,y=λ1Lμ,l1+Lμ,GAN+λ2LFM.Lμ,l1 represents the loss of L1, and Lμ,GAN represents the generative adversarial network, which is used to synthesize photos that look real.
(3)
Degradation repair designIn some old photos, for the damage of some structural defects, it may often be necessary to search for the information of the entire image and then select effective information to fill in to maintain the consistency of the global structure. Therefore, it is necessary to design a support for local search and global search which is the network of two mechanisms [26]. In this chapter, the scratch mask of the original old photo can be used as input to prevent the network from using the pixels of the damaged area to repair the damaged area. Let C, H, and W be the number of channels, height, and width, respectively. Let FϵRC×HW be the mapping feature of the middle layer; mϵ0,1HW means that the binary mask is scaled to the same size; m is 1 to indicate that the area is damaged and 0 to indicate that the area is intact. The relationship between position i and position j in the feature map F is represented by sij, and sijϵFϵRHW×HW represents the relationship between each pixel, as follows:
(10)sij=fij1−mj∑∀k1−mkfi,k,fij=expθFiT∗ϕFj.BothFi and Fj are vectors of C∗1, and θ (∙) and ϕ (∙) are functions that make F map to the Gaussian distribution.Therefore, when the input mask area is identified as a damaged area, the global information can be used to repair, otherwise the local feature information can be used, so the global branch and the local branch can be merged under the guidance of the input mask. The formula is as follows:(11)Ffuse=1−m⨀ρlocalF+m⨀ρglobalo.In the formula,⨀ represents the Hadamard product of the matrix, and ρlocal and ρglobal both represent the nonlinear transformation of the branch residual block, which can deal with the structural defects of old photos. The specific network structure is shown in Table 1.
(4)
Network parameter setting of generator and discriminatorTable 1
Network structure convolution design.
ModuleLayerKernel sizeOutput sizeEncoderEConv7∗7/1256∗256∗64Conv4∗4/2128∗128∗64Conv4∗4/264∗64∗64GeneratorGResblock∗43∗3/164∗64∗64Deconv4∗4/2128∗128∗64Conv7∗7/1256∗256∗3MappingτConv3∗3/164∗64∗128Resblock∗63∗3/164∗64∗512Partial nonlocal1∗1/164∗64∗512The generative confrontation network is mainly composed of a generator and a discriminator. The generator is responsible for generating a high-resolution image similar to the real image. The discriminator is used to determine whether the generated image comes from real training set data or a fake one. The generator in this article consists of three parts. The first part is the downsampling module; you can see three convolution kernels of different sizes. The second part is the self-attention residual module, which is composed of 16 identical self-attention residual blocks, and each small module is composed of a self-attention layer and a convolutional layer. The last part is the upsampling image reconstruction module. This part is mainly two subpixel convolutional layers for pixel amplification. Comparing with ordinary deconvolution, the subpixel convolution can reduce the risk caused by artificial factors and make the reconstructed image quality higher.The task of the discriminator is to determine whether the input image is from the sample space or a fake image generated by the generator and to learn the difference between the two. In this chapter, the discriminator network uses a deep convolutional neural network, a batch normalization layer is added to the discriminator network, and the LeakyRelu activation function (α=0.2) is used [27]. The specific method is to input the generated image into the network through multilayer convolution to obtain the feature map and then input the feature map to the fully connected layer. Finally, the Sigmoid activation function is used to perform two classifications of true and false to determine whether it is a real picture or a fake picture. So far, the design of the generator and the discriminator is completed, and the two generate high-resolution images through confrontational optimization to realize the high-definition processing of old photos.


## 3.3. Loss Function Design

The loss function is a function proposed to evaluate the difference between the generated image and the real image in the image restoration model. This function guides the learning of the image restoration model through the evaluation results. The smaller the value of the loss function, the better the robustness of the model. In the image superresolution reconstruction, the loss function is used to determine the difference between the original image and the generated image as an optimization function of the model. The loss function in this paper is composed of three parts: appearance matching loss, KL divergence loss, and confrontation loss. Appearance matching loss is used to constrain the generation of image structure, KL divergence loss narrows the high-dimensional spatial distribution between the generated image and the real image, and the confrontation loss is used to reduce the difference between the generated image and the real image [28, 29].
(1)
KL lossThe KL loss makes the generated path close to the standard normal distribution of the high-dimensional space of the reconstructed path. This article has two KL loss functions.(12)lKLr=−KLqz∣IcNm0,σ2nI,lKLb=−KLqz∣Icpϕz∣Im.In the formula,lKLr represents the KL loss of the reconstructed path, lKLb represents the KL loss of the generated path, z represents the latent vector, q∙∣∙ represents the importance sampling function, Nm0,σ2nI represents the normal distribution, and pϕ∙∣∙ represents the conditional prior [30].
(2)
Appearance matching lossIn order to increase the ability of the generator to generate images, the image features extracted in the encoding stage are input into the decoder to generate a reconstructed image. In this paper, the appearance matching loss is used to constrain the generation of the image structure. The loss function of the appearance matching loss is shown in the following formula:(13)lappr=Irec−Ig1,lappb=M∗Igen−Ig1.In the formula,lappr represents the appearance matching loss of the reconstructed path, lappb represents the appearance matching loss of the generated path, Irec represents the reconstructed image, Igen represents the reconstructed image, Ig represents the real image, and M represents the binary mask of visible pixels.
(3)
Fight against lossThis part of the loss is composed of generator loss and discriminator loss [31]. The calculation formula of generator loss is shown in the following formula:
(14)LG=EDGz.Z represents the random noise input to the generating network, Gz represents the sample generated by the generating network, and DGz represents the output result when the input of the network is judged to be the generated sample.The loss function of the discriminator is as follows:(15)Li=EDx−EDGz+Lgp,Lgp=λE∣∇Dαx−1−αGz∣−12,LD=Li+Lgp.(4)
Total loss functionThe total loss function of the network consists of three parts: KL loss, appearance matching loss, and counter loss. The calculation formula of the total loss function is shown in the following formula:(16)L=αKLLKLr+LKLb+αapplappr+lappb+αadLadr+Ladb.In the formula,αKL, αapp, and αad all represent hyperparameters, which are used to adjust the proportions of KL loss, appearance matching loss, and counter loss in the total loss function.


## 4. Image Restoration and Processing Experiment of Old Photos Based on Deep Neural Network Structure

### 4.1. Experimental Design

This experiment is divided into two parts. One is the experiment of image blur restoration. Sharpness is a very important property of photos, so this part of the experiment explores the effect of the algorithm in this article on the blur restoration of old photos. The other part is the experiment of image damage repair. Because old photos are often prone to scratches and breakage, this part explores the effect of the algorithm in this paper on the damage repair of old photos.(1)
Source of experimental dataBecause old photos are often precious memories of a family and have important historical significance, most old photo holders think that most of the old photos that are damaged do not have the meaning of uploading to the Internet. Some old photo data sets on the Internet are mostly better saved old photos, resulting in fewer damaged old photos on the Internet. Since the real old photo data is limited, the data set used in this experiment is the data set used for image restoration. This experiment selects two data sets, namely, the CelebA-HQ data set and the Places2 data set; these two data sets are human face data set and natural landscape data set. This experiment deliberately selects these two data sets with different image categories to verify whether the model has a better effect on the restoration of faces and scene images.(2)
Experimental environment designThis experiment chooses to build a training environment on a cloud server, and the neural network training framework uses PyTorch 1.2.0. PyTorch uses dynamic graphs and code thinking like PyTorch, allowing users to edit easily and quickly and adjust the structure during deployment. It supports faster and dynamic training and integrates with other PyTorch ecosystems more conveniently. The operating system is Urbanu16.04, and Table2 is a partial dependency package of Python.Table 2
Python partial dependency packages.
PackageVersionOpenCV-Python4.5.18SciPy1.5.4Torch1.2.0Wheel0.33.1TensorboardX2.1In this experiment, the weight values in the loss function are designed as shown in Table3.
(3)
Experimental evaluation parameter designTable 3
Loss function weight value design.
ParameterWeight valueKL loss weight coefficient20Weight coefficient of appearance matching loss20Counter loss weight coefficient1The evaluation indicators of this experiment are peak signal-to-noise ratio PSNR and structural similarity SSIM. PSNR is an objective standard for evaluating images. The value range of structural similarity is [0,1]. The closer the SSIM is to 1, the better the repair effect of the model. Since both PSNR and SSIM are often used as the measurement of reconstruction quality in the image field, they are very suitable for the evaluation of image restoration of old photos in this article. The larger the two values, the better the image restoration effect.(1)
Peak signal-to-noise ratioThe peak signal-to-noise ratio measure evaluates the generated image by measuring the degree of distortion or noise level of the image, and the evaluation result is expressed by the ratio between credible information and noise. The higher the PSNR value, the smaller the distortion of the image conversion process, and the more realistic the generated image. PSNR is the most used objective evaluation method when evaluating generated images. The calculation formula of PSNR is shown.(17)MSE=∑i=0M−1∑j=0N−1I0i,j−Ii,j2M×N,PSNR=10logGf2MSE.MSE stands for mean square error,i,j stands for the position of the real image, I0i,j stands for the pixel value of the real image at that position, Ii,j stands for the pixel value at i,j in the restored image, and M×N stands for the area size of the restored image.
(2)
Structural similarityThis method completes the evaluation of the image by comparing the similarity of the two images. The larger the SSIM value, the higher the similarity between the restored image and the real image, and the maximum SSIM value is 1. Because the information connection between digital images is close and the correlation between pixels is strong, the combined structure information of the image has nothing to do with its contrast and brightness. SSIM refers to the brightness, contrast, and structure of the image when modeling the degree of distortion of the image, so as to give a comprehensive evaluation result that reflects the properties of the object structure in the scene modeling. Among them, the brightness and contrast are represented by the average and standard deviation of the images, and the structural similarity between the images is represented by the covariance of the images. SSIM pays more attention to the similarity of two images, and it reflects the attributes of the object structure when modeling the scene. The calculation formula for structural similarity is shown below.(18)SSIMx,y=2μxμy+C12σxy+C2μx2+μy2+C1σx2+σy2+C2.Bothx and y represent the input image, μ represents the average value of x, σx2 represents the variance of x, μy represents the average value of y, σy2 represents the variance of y, and σxy represents the covariance of x,y.
(1)
Comparison algorithm selectionThe comparison algorithms chosen in this article are SRCNN, SRGAN, and Criminisi. SRCNN is an image restoration algorithm based on convolutional neural network (CNN), SRCNN is an image restoration algorithm based on generative adversarial network (GAN), and Criminisi is a traditional image restoration algorithm based on texture synthesis. These image restoration algorithms include both traditional algorithms and algorithms based on neural networks in deep learning. This experiment compares these algorithms with the algorithms proposed in this article, which are more convincing.


### 4.2. Image Restoration Experiment Based on Blur Restoration

This experiment selects 1000 photos from each of the CelebA-HQ data set and Places2 data set for experiment and uses each algorithm for image restoration and explores the peak signal-to-noise ratio and structural similarity of each algorithm for image blur restoration.(1)
Analysis of peak signal-to-noise ratio resultsIn this section, several algorithms perform image restoration on 1000 photos in the CelebA-HQ data set and the Places2 data set. The PSNR results of each algorithm are shown in Figure7.Figure 7
Peak signal-to-noise ratio results.It can be seen from Figure7 that whether it is on the CelebA-HQ data set or the Places2 data set, the value of the peak signal-to-noise ratio of the algorithm in this paper is the highest. On the CelebA-HQ data set, the value of the peak signal-to-noise ratio of the algorithm in this paper is 32.34. On the Places2 data set, the value of the peak signal-to-noise ratio of the algorithm in this paper is 30.82. And the peak signal-to-noise ratio of the traditional algorithm Criminisi is the lowest on the two data sets. Comparing the PSNR results of the two data sets, it can be seen that the PSNR value of each algorithm on the CelebA-HQ data set is always slightly higher than the Places2 data set, which proves that the repair effect of each algorithm on the face is higher than the repair effect of the scene. To sum up, the peak signal-to-noise ratio of the algorithm in this paper is the highest in any data set, which proves that the image restoration effect of the algorithm in this paper is the most satisfactory, and the algorithm in this paper has a better restoration effect on face images than scene images.
(2)
Analysis of structural similarity resultsIn this experiment, several algorithms were used to repair 1000 photos in the CelebA-HQ data set and the Places2 data set, and the SSIM value of each algorithm was explored. The results are shown in Figure8.Figure 8
Structural similarity results.As can be seen from Figure8, in the SSIM dimension, the value of this algorithm is always greater than other algorithms. The SSIM value of this algorithm on the CelebA-HQ data set is 0.885. The SSIM value on the Places2 data set is 0.879. Like PSNR, the algorithm with the smallest SIMM value is Criminisi. The SSIM values of this algorithm in the CelebA-HQ data set and Places2 data set are 0.849 and 0.831, respectively. Therefore, it can be seen that the algorithm in this paper is better than other algorithms for the structural similarity of the repaired image.


### 4.3. Image Restoration Experiment Based on Damage Degree

In this experiment, we select 100 pictures from the CelebA-HQ data set and process these pictures with photoshop, using tools in photoshop to simulate damage on the photos and dividing the 100 photos into 5 groups, each with 20. Five groups’ damaged area of the image accounts for 5%, 10%, 15%, 20%, and 25% of the entire image, respectively.(1)
Analysis of peak signal-to-noise ratio resultsThe four algorithms are used to repair 100 pictures in the 5 groups, respectively, and the PSNR results obtained are shown in Figure9.Figure 9
Peak signal-to-noise ratio results for different levels of damage.It can be seen from Figure9 that the peak signal-to-noise ratios of the four algorithms all decrease as the degree of picture damage increases. It can be seen that the greater the degree of damage, the more difficult it is to restore the photo. Comparing these four algorithms, it can be seen that the peak signal-to-noise ratio of the algorithm in this paper is the highest regardless of the degree of damage. After calculation, the average value of the peak signal-to-noise ratio of the algorithm in this paper is 19.11 under different degrees of damage. While the PSNR values of several other algorithms are close, it is impossible to know which algorithm has the lowest PSNR value.
(2)
Analysis of structural similarity resultsThe four algorithms are used to repair 100 pictures in these 5 groups, and the SSIM results obtained are shown in Figure10.Figure 10
Structural similarity results for different damage levels.It can be seen from Figure10 that each algorithm has a different repair effect on damaged images. However, no matter what the degree of damage, the SSIM value of the algorithm in this paper is always greater than that of other algorithms, and the SSIM value of the algorithm in this paper has a downward trend which is smaller than other algorithms’ trend. After calculation, the average SSIM of the algorithm in this paper is 0.767 under different damage levels. Therefore, the algorithm in this paper has the best repair effect on damaged images.It can be seen from the above two parts of experiments that the PSNR and SSIM values of the algorithm in this paper are higher than those of other algorithms, whether it is for blur repair or damage repair. Therefore, the algorithm in this paper performs best for both blur repair and damage repair of old photos.


## 4.1. Experimental Design

This experiment is divided into two parts. One is the experiment of image blur restoration. Sharpness is a very important property of photos, so this part of the experiment explores the effect of the algorithm in this article on the blur restoration of old photos. The other part is the experiment of image damage repair. Because old photos are often prone to scratches and breakage, this part explores the effect of the algorithm in this paper on the damage repair of old photos.(1)
Source of experimental dataBecause old photos are often precious memories of a family and have important historical significance, most old photo holders think that most of the old photos that are damaged do not have the meaning of uploading to the Internet. Some old photo data sets on the Internet are mostly better saved old photos, resulting in fewer damaged old photos on the Internet. Since the real old photo data is limited, the data set used in this experiment is the data set used for image restoration. This experiment selects two data sets, namely, the CelebA-HQ data set and the Places2 data set; these two data sets are human face data set and natural landscape data set. This experiment deliberately selects these two data sets with different image categories to verify whether the model has a better effect on the restoration of faces and scene images.(2)
Experimental environment designThis experiment chooses to build a training environment on a cloud server, and the neural network training framework uses PyTorch 1.2.0. PyTorch uses dynamic graphs and code thinking like PyTorch, allowing users to edit easily and quickly and adjust the structure during deployment. It supports faster and dynamic training and integrates with other PyTorch ecosystems more conveniently. The operating system is Urbanu16.04, and Table2 is a partial dependency package of Python.Table 2
Python partial dependency packages.
PackageVersionOpenCV-Python4.5.18SciPy1.5.4Torch1.2.0Wheel0.33.1TensorboardX2.1In this experiment, the weight values in the loss function are designed as shown in Table3.
(3)
Experimental evaluation parameter designTable 3
Loss function weight value design.
ParameterWeight valueKL loss weight coefficient20Weight coefficient of appearance matching loss20Counter loss weight coefficient1The evaluation indicators of this experiment are peak signal-to-noise ratio PSNR and structural similarity SSIM. PSNR is an objective standard for evaluating images. The value range of structural similarity is [0,1]. The closer the SSIM is to 1, the better the repair effect of the model. Since both PSNR and SSIM are often used as the measurement of reconstruction quality in the image field, they are very suitable for the evaluation of image restoration of old photos in this article. The larger the two values, the better the image restoration effect.(1)
Peak signal-to-noise ratioThe peak signal-to-noise ratio measure evaluates the generated image by measuring the degree of distortion or noise level of the image, and the evaluation result is expressed by the ratio between credible information and noise. The higher the PSNR value, the smaller the distortion of the image conversion process, and the more realistic the generated image. PSNR is the most used objective evaluation method when evaluating generated images. The calculation formula of PSNR is shown.(17)MSE=∑i=0M−1∑j=0N−1I0i,j−Ii,j2M×N,PSNR=10logGf2MSE.MSE stands for mean square error,i,j stands for the position of the real image, I0i,j stands for the pixel value of the real image at that position, Ii,j stands for the pixel value at i,j in the restored image, and M×N stands for the area size of the restored image.
(2)
Structural similarityThis method completes the evaluation of the image by comparing the similarity of the two images. The larger the SSIM value, the higher the similarity between the restored image and the real image, and the maximum SSIM value is 1. Because the information connection between digital images is close and the correlation between pixels is strong, the combined structure information of the image has nothing to do with its contrast and brightness. SSIM refers to the brightness, contrast, and structure of the image when modeling the degree of distortion of the image, so as to give a comprehensive evaluation result that reflects the properties of the object structure in the scene modeling. Among them, the brightness and contrast are represented by the average and standard deviation of the images, and the structural similarity between the images is represented by the covariance of the images. SSIM pays more attention to the similarity of two images, and it reflects the attributes of the object structure when modeling the scene. The calculation formula for structural similarity is shown below.(18)SSIMx,y=2μxμy+C12σxy+C2μx2+μy2+C1σx2+σy2+C2.Bothx and y represent the input image, μ represents the average value of x, σx2 represents the variance of x, μy represents the average value of y, σy2 represents the variance of y, and σxy represents the covariance of x,y.
(1)
Comparison algorithm selectionThe comparison algorithms chosen in this article are SRCNN, SRGAN, and Criminisi. SRCNN is an image restoration algorithm based on convolutional neural network (CNN), SRCNN is an image restoration algorithm based on generative adversarial network (GAN), and Criminisi is a traditional image restoration algorithm based on texture synthesis. These image restoration algorithms include both traditional algorithms and algorithms based on neural networks in deep learning. This experiment compares these algorithms with the algorithms proposed in this article, which are more convincing.


## 4.2. Image Restoration Experiment Based on Blur Restoration

This experiment selects 1000 photos from each of the CelebA-HQ data set and Places2 data set for experiment and uses each algorithm for image restoration and explores the peak signal-to-noise ratio and structural similarity of each algorithm for image blur restoration.(1)
Analysis of peak signal-to-noise ratio resultsIn this section, several algorithms perform image restoration on 1000 photos in the CelebA-HQ data set and the Places2 data set. The PSNR results of each algorithm are shown in Figure7.Figure 7
Peak signal-to-noise ratio results.It can be seen from Figure7 that whether it is on the CelebA-HQ data set or the Places2 data set, the value of the peak signal-to-noise ratio of the algorithm in this paper is the highest. On the CelebA-HQ data set, the value of the peak signal-to-noise ratio of the algorithm in this paper is 32.34. On the Places2 data set, the value of the peak signal-to-noise ratio of the algorithm in this paper is 30.82. And the peak signal-to-noise ratio of the traditional algorithm Criminisi is the lowest on the two data sets. Comparing the PSNR results of the two data sets, it can be seen that the PSNR value of each algorithm on the CelebA-HQ data set is always slightly higher than the Places2 data set, which proves that the repair effect of each algorithm on the face is higher than the repair effect of the scene. To sum up, the peak signal-to-noise ratio of the algorithm in this paper is the highest in any data set, which proves that the image restoration effect of the algorithm in this paper is the most satisfactory, and the algorithm in this paper has a better restoration effect on face images than scene images.
(2)
Analysis of structural similarity resultsIn this experiment, several algorithms were used to repair 1000 photos in the CelebA-HQ data set and the Places2 data set, and the SSIM value of each algorithm was explored. The results are shown in Figure8.Figure 8
Structural similarity results.As can be seen from Figure8, in the SSIM dimension, the value of this algorithm is always greater than other algorithms. The SSIM value of this algorithm on the CelebA-HQ data set is 0.885. The SSIM value on the Places2 data set is 0.879. Like PSNR, the algorithm with the smallest SIMM value is Criminisi. The SSIM values of this algorithm in the CelebA-HQ data set and Places2 data set are 0.849 and 0.831, respectively. Therefore, it can be seen that the algorithm in this paper is better than other algorithms for the structural similarity of the repaired image.


## 4.3. Image Restoration Experiment Based on Damage Degree

In this experiment, we select 100 pictures from the CelebA-HQ data set and process these pictures with photoshop, using tools in photoshop to simulate damage on the photos and dividing the 100 photos into 5 groups, each with 20. Five groups’ damaged area of the image accounts for 5%, 10%, 15%, 20%, and 25% of the entire image, respectively.(1)
Analysis of peak signal-to-noise ratio resultsThe four algorithms are used to repair 100 pictures in the 5 groups, respectively, and the PSNR results obtained are shown in Figure9.Figure 9
Peak signal-to-noise ratio results for different levels of damage.It can be seen from Figure9 that the peak signal-to-noise ratios of the four algorithms all decrease as the degree of picture damage increases. It can be seen that the greater the degree of damage, the more difficult it is to restore the photo. Comparing these four algorithms, it can be seen that the peak signal-to-noise ratio of the algorithm in this paper is the highest regardless of the degree of damage. After calculation, the average value of the peak signal-to-noise ratio of the algorithm in this paper is 19.11 under different degrees of damage. While the PSNR values of several other algorithms are close, it is impossible to know which algorithm has the lowest PSNR value.
(2)
Analysis of structural similarity resultsThe four algorithms are used to repair 100 pictures in these 5 groups, and the SSIM results obtained are shown in Figure10.Figure 10
Structural similarity results for different damage levels.It can be seen from Figure10 that each algorithm has a different repair effect on damaged images. However, no matter what the degree of damage, the SSIM value of the algorithm in this paper is always greater than that of other algorithms, and the SSIM value of the algorithm in this paper has a downward trend which is smaller than other algorithms’ trend. After calculation, the average SSIM of the algorithm in this paper is 0.767 under different damage levels. Therefore, the algorithm in this paper has the best repair effect on damaged images.It can be seen from the above two parts of experiments that the PSNR and SSIM values of the algorithm in this paper are higher than those of other algorithms, whether it is for blur repair or damage repair. Therefore, the algorithm in this paper performs best for both blur repair and damage repair of old photos.


## 5. Discussion

With the continuous development of computer vision technology, images have become an important medium for transmitting information. How to obtain high-quality images that people need from massive image data is particularly important. As a key technology in image processing, image restoration has become one of the important research topics in the field of computer vision. Old photos often save precious historical memories, so it is of great significance to use image restoration technology in the restoration of old photos. There are many image restoration methods, including traditional methods, such as diffusion-based methods, texture synthesis-based methods, and data-driven methods. This year, due to the development of artificial intelligence and deep learning, people have used deep neural networks for image restoration, and various image restoration models have been proposed, and good results have been achieved. This article is performed to improve the deep neural network and propose a new image restoration method.


## 6. Conclusion

Image restoration technology is the core technology for restoring old photos and images. Old photos are often blurred and damaged, so it is necessary to propose a suitable restoration method to reproduce the scene of the old photos. Based on the deep neural network, this article proposes a new method for repairing old photos. This article first introduces the background and significance of the topic. Then the general convolutional neural network and the generative confrontation network are introduced, and its structure and principle are explained. Then, this paper designs a new image restoration method, which is based on the convolutional neural network and the generative adversarial network in the deep neural network, and the loss function is designed. Finally, this paper does an experiment to test the performance of this algorithm in image restoration. The results show that the peak signal-to-noise ratio and structural similarity of the algorithm in this paper are higher than those of other algorithms. Therefore, compared with other algorithms, the algorithm in this paper not only has better performance in blur repair but also has better performance in damage repair. The model in this article is more suitable for the field of restoration of old photos. From the experimental point of view, the restoration effect is good, but there is still improvement in the restoration effect, especially the restoration of damaged photos which is not ideal enough, so the focus of future work will be put on repairing broken photos.



---

*Source: 7415342-2022-03-21.xml*