# Application of Deep Learning Intelligent Laser Scanning Technology in Mural Digitization

Authors: Zhiming Dong
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Ancient Chinese murals have a long history and a large number of types. They are witnesses to the development of ancient Chinese civilization. They still have important historical, artistic, and scientific values, as well as other values such as cultural relics, economy, and missionary education. Yet, the clear and smooth images of earlier have been damaged and covered by fog after the ancient murals were destroyed by human being and the nature. Therefore, the ancient murals that exist till now have been bothered by different damages and sickness. Protection and prevention are needed the most. In recent years, the application of scientific and technological means has played a role in the protection of ancient murals, making the work methods of cultural relics protection more scientific and diverse. In the context of the increasingly rich digital protection of cultural relics, the protection of murals requires more innovative work. However, at present, the resolution of ancient murals is low and the texture details are ambiguous, which leads to the problems of insufficient viewing and low research value. This paper focuses on ancient Chinese murals and conducts exploration on the phenomenon of mural damage and blurred colors. The deep learning intelligent laser scanning technology is used to extract the damaged ancient mural images. In this thesis, the images of murals have been restored using the new-super resolution technology to achieve the optimal mural images so that the mural images are more beautiful and artistic, which provides new ideas for the protection of murals.


---

## Body

## 1. Introduction

Murals are the textured paintings on the wall made of materials such as clay, woods, bricks, and stones. The pigment employed in the murals shows a great diversity, and so do the greasepaint and pigment of traditional drawings. The form and theme of the murals are decided by the style and content of the buildings and also should be in line with the space environment of buildings.After the long history, there have been artistic classics that are still astonishing like the Dunhuang murals [1]. They still have important historical, artistic, and scientific values, as well as other values such as cultural relics, economy, and missionary education [2].Therefore, the protection of mural images is urgent. The digital protection of cultural heritage is not only a major national demand for the protection and promotion of cultural heritage but also a frontier research topic for multidisciplinary integration [3]. The technologies including digital museum collections and digital communication are used by the digital preservation for the culture heritage, which can change and restore the nonmaterial cultural heritage into the digital form featured by sharing and reproduction.Each iteration of the image propagates that the pixel information of the image goes in the line of the iso-illuminance to the missing area [4]. Yizhen Chen said a super-resolution algorithm gets rooted in generative adversarial network. However, this method has poor adaptability, and the processing scene is relatively simple, and there are often problems of edge blurring and loss of high-frequency details [5]. The emergence of 3D laser scanning technology can solve this problem very effectively. 3D laser scanning technology has the characteristics of high efficiency, speed, and accuracy. It can effectively reconstruct image elements in murals [6].This paper improves the previous image restoration method through the deep learning laser scanning technology, completes the initial restoration of the mural image, and then further conducts laser scanning on the mural image [7]. Through 3D laser scanning, we can directly obtain the image information in the murals, instead of traditional manual drawing, which will save a lot of manpower, material resources, and time [8].


## 2. Related Theories

### 2.1. Deep Learning

The deep learning can accomplish matrix counting with large scale and trigger the development of image recognition. This field has a great potential in application and development. Neural network structure contains multiple hidden layers, which can describe the characteristics of the target object more abstractly and at a deeper level [9].In the process of supervised learning, the relationship of the characteristics of the data, which is trained, and the tags will be discovered [10]. The training data of unsupervised learning are not labeled, a deep belief network (DBN), etc. [11]. The best model is trained by the supervision learning training, and the experiment employs the model to reflect all the information into a certain output. The judgment should be made for the output for the purpose of classification. The learning is the imitation of human beings no matter it is a supervision learning or a machine learning.The generative modelG can present the features of the data [12]. The discriminative model D is a classifier that estimates the possibility that an input sample maybe from the training one [13]. Where X represents the real image information, Z represents the noise image information, XP represents the probability distribution of the real image information, ZP (z) represents the probability distribution of the generated data, G (z) represents the reconstructed image data, D (x) represents the reconstructed image data, and D (g (z)) represents the possibility of judging whether the reconstructed image information is real through the discrimination network. Log D (x) represents the judgment of the discrimination network on the real image data D (g (z)) represents the judgment on the reconstructed image information [14]. It is shown in Figure 1:(1)minGmaxDVD,G=EX∼PxlogDX+EZ∼Pzlog1−DGz.Figure 1
Generative adversarial network structure diagram.


### 2.2. The Application of DL

The traditional recognition algorithm cannot meet the high recognition requirements of remote sensing image processing because it relies heavily on artificial feature extraction. Deep learning technology can fully utilize the spatial structure information of 3D laser scanning technology [15]. The combination of 3D laser scanning technology and deep learning technology can effectively extract useful features of images [16].


### 2.3. Application of 3D LST

Since the 2D laser radar and the PTZ device are to be upgraded, therefore, in order to make the acquisition process easier and the acquisition results more precise, a 3D laser scanner acquisition device is designed [17]. After many experiments, it has been proved that the system can be well adapted to complex 3D scenes, and the point cloud technology can be used to obtain better model reconstruction effect [18].


### 2.4. Image Super-Resolution Reconstruction

In view of the problem that the aesthetic value of mural images decreases due to the rough details in the mural image processing, an ultra-high-resolution mural image optimization method is provided. Using deep learning ideas and a phased approach to network models better mural image optimization is achieved. Through different methods, learning the improvement ideas of different methods will provide new inspiration for super-resolution reproduction of mural images [19]. Jianfang came up with an enhanced super-resolution way, which is stable and can solve problems in ancient murals with low resolution and blurred texture details. Based on GAN, the method is improved to extract images using dense residual blocks. Finally, it is found that the method is better than the other related algorithms [20].Deep learning-based MRI super-resolution reconstruction techniques have attracted attention due to their resolution over traditional super-resolution reconstruction techniques [21].


## 2.1. Deep Learning

The deep learning can accomplish matrix counting with large scale and trigger the development of image recognition. This field has a great potential in application and development. Neural network structure contains multiple hidden layers, which can describe the characteristics of the target object more abstractly and at a deeper level [9].In the process of supervised learning, the relationship of the characteristics of the data, which is trained, and the tags will be discovered [10]. The training data of unsupervised learning are not labeled, a deep belief network (DBN), etc. [11]. The best model is trained by the supervision learning training, and the experiment employs the model to reflect all the information into a certain output. The judgment should be made for the output for the purpose of classification. The learning is the imitation of human beings no matter it is a supervision learning or a machine learning.The generative modelG can present the features of the data [12]. The discriminative model D is a classifier that estimates the possibility that an input sample maybe from the training one [13]. Where X represents the real image information, Z represents the noise image information, XP represents the probability distribution of the real image information, ZP (z) represents the probability distribution of the generated data, G (z) represents the reconstructed image data, D (x) represents the reconstructed image data, and D (g (z)) represents the possibility of judging whether the reconstructed image information is real through the discrimination network. Log D (x) represents the judgment of the discrimination network on the real image data D (g (z)) represents the judgment on the reconstructed image information [14]. It is shown in Figure 1:(1)minGmaxDVD,G=EX∼PxlogDX+EZ∼Pzlog1−DGz.Figure 1
Generative adversarial network structure diagram.


## 2.2. The Application of DL

The traditional recognition algorithm cannot meet the high recognition requirements of remote sensing image processing because it relies heavily on artificial feature extraction. Deep learning technology can fully utilize the spatial structure information of 3D laser scanning technology [15]. The combination of 3D laser scanning technology and deep learning technology can effectively extract useful features of images [16].


## 2.3. Application of 3D LST

Since the 2D laser radar and the PTZ device are to be upgraded, therefore, in order to make the acquisition process easier and the acquisition results more precise, a 3D laser scanner acquisition device is designed [17]. After many experiments, it has been proved that the system can be well adapted to complex 3D scenes, and the point cloud technology can be used to obtain better model reconstruction effect [18].


## 2.4. Image Super-Resolution Reconstruction

In view of the problem that the aesthetic value of mural images decreases due to the rough details in the mural image processing, an ultra-high-resolution mural image optimization method is provided. Using deep learning ideas and a phased approach to network models better mural image optimization is achieved. Through different methods, learning the improvement ideas of different methods will provide new inspiration for super-resolution reproduction of mural images [19]. Jianfang came up with an enhanced super-resolution way, which is stable and can solve problems in ancient murals with low resolution and blurred texture details. Based on GAN, the method is improved to extract images using dense residual blocks. Finally, it is found that the method is better than the other related algorithms [20].Deep learning-based MRI super-resolution reconstruction techniques have attracted attention due to their resolution over traditional super-resolution reconstruction techniques [21].


## 3. Technology

### 3.1. 3D Laser Scanning Technology

In the implementation process, a variety of high-tech surveying and mapping instruments are integrated. On this basis, based on the noncontact high-speed laser measurement method, calculate the distance between the target and the center by recording the time difference, as shown in the following equation:(2)d=c×Δt2,where D is the measured distance; C is the light speed; t is the variation. Laser scanner system using laser scanning technology is mainly composed of hardware system and software system, as shown in Figure 2. The computer system records the emission time t1 and the optical scanning system. Location and angle information (X): when the optical receiving system receives the light signal reflected by the target, the computer system records the receiving time t2 and controls the laser to emit light from different angles through the scanning control system.Figure 2
3D Laser scanning system.The coordinate center defined by the laser scanner is the original part, andX and Y are perpendicular to each other. The results of measurements are shown in Figure 3.Figure 3
Schematic diagram of 3D laser measurement.During the 3D laser scanning process, the ranging observation value S is mainly obtained by the 3D laser scanner through the data acquisition system. When using the ground laser scanning 3D measurement technology in 3D laser scanning for soil and water conservation monitoring, the coordinate system usually used is the internal coordinate system of the instrument; among them, theX axis is in the lateral scanning plane, and the Y axis and Z axis are, respectively, in the lateral scanning plane. Inner and X-axis and the horizontal one are vertical to each other, and formula (3) for the coordinates of the 3D laser foot point is obtained as follows:(3)X=Dcosβcosα,Y=Dcosβsinα,Z=Dsinβ.In this formula, the three-dimensional coordinates of the measured object are represented by the variables “X, Y, and Z”; the distance between the measured object and the instrument is represented by D; the horizontal and vertical scanning angles are represented by α and β, respectively.In addition to the object displacement Δ = , the measurement sensitivity is also related to several other parameters such as ax,β, f, and l. A is the angle between the incident laser beam and the surface normal of the measured object, that is, the incident angle, and we define this parameter as the working angle. 1 is the distance between the spot where the laser irradiates the surface of the object to be measured and the main plane of the receiving lens, and we define this parameter as the working distance of the system. We select the optimal incident angle of the system laser projection by observing the changing trend of the working angle a and the measurement sensitivity δ at the positions of 30 mm, 60 mm, 90 mm, and 120 mm, respectively, as shown in Figure 4.Figure 4
Relationship between working anglea and measurement sensitivity δ.When the measurement range and working angle of the system are determined, the relationship between the working distance I and the measurement sensitivityδ is known. For angles of 10°, 20°, 30°, 40°, 50°, 60°, 70°, and 80°, the changing trends of the working distance l and the measurement sensitivity δ are shown in Figure 5.Figure 5
Relationship between working distance l1 and measurement sensitivityδ.


### 3.2. Network Structure Design

#### 3.2.1. Generative Network Design

The generation system adopts the FCN structure and the decoder decoder structure. The result of the mural image is set as only the image information of the mask area, while the image information of other areas remains unchanged. The result of generating a network is usually a size of 128×128 preprocessed mural pictures. When an image having 24 to 36 pixel values is randomly obtained and the mask is completed, and the pixel value of the mask area is set to zero in the input image, the mask mural image is obtained. The FCN network obtains the image feature signal through the convolution layer, restores the feature map signal to the size image of the original pixel through the deconvolution layer calculation, and outputs the image value of the image restoration non mask area, and saves the restored mask area image information, and outputs the final restored image value. The generated network details are shown in Table1.Table 1
Generate network details information table.
TypeKernelDilationStrideOutputsConv5 × 611 × 264Conv3 × 212 × 132Conv3 × 111 × 1128Conv3 × 112 × 2256Conv3 × 111 × 2128Dilated conv3 × 321 × 1256Dilated conv3 × 331 × 3256Dilated conv3 × 161 × 364Conv3 × 111 × 1256Conv4 × 111/2 × 2128Deconv3 × 111 × 2128Conv4 × 312 × 1/264Deconv3 × 311 × 232Output3 × 312 × 13In order to better generate close to the real repair effect, this paper proposes to improve the model by using the MSE and adversarial loss function. The MSE and the adversarial are optimized simultaneously with the different network, and the GNM and the DNM are optimized.The function is used to measure the quality of their built image. The final loss function calculation formula (4) can be expressed as follows, in which the coefficients employed to balance different losses are symbolized by input 1, 2.(4)lG=lMSESR+λ1lVGGSR+λ2ladv.The pixel-level MSE loss calculation formula (5) is as follows:(5)lMSESR=1r2WH∑x=1rW∑y=1rHIx,yHR−GθGILRx,y2.W and H are the width and height, respectively, and represent Euclidean distance between real image pixels and the generated pixels.The MSE loss can make the reconstructed graphics have extremely high peak signal-to-noise ratios, but usually lack high-frequency content. We calculate the MSE by summing all squared losses for each sample and the formula is shown as(6)MSE=1N∑x,y∈Dy−predictionx2,whereN represents the total number of samples; (x, y) represents data, wherein represents the attribute set in the exercise data; and y represents the true value of the exercise data. The prediction (x) is the expected value of the exercise data X.Based on the same principle, the main purpose of the judgment system is to make the output of the power generation system approximately zero, while the output of the judgment system of the real data is approximately one. The resistance loss formula is shown as follows:(7)minGmaxDVD,G=EX∼PdataxlogDX+EZ∼Pzlog1−DGz.Here,x refers to the real data, while z indicates the other data, and data(x)P is the probability of real data, ( )zP z represents the probability distribution of synthetic data. Such combined data result represents the data result combined by the generation network, and D (x) represents the discrimination method, and represents the probability of judging the statistics of the true number to be true. D (g (z)) is a probability that the computer network determines whether the synthesized data generated by the computer network is actual. Log D (x) represents the determination of the real data by the discriminating computer network. D (g (z)) represents the determination of the comprehensive information.Loss function of WGAN (8) is like this(8)LD=Ex∼PrDx−Ex∼PgDx.Gradients of the discriminatorD(x) are not greater than a finite constant K by adding the Lipschitz limit, as shown in(9)∇XDx≤K.Generally,K is equal to 1, and adversarial loss calculation formula (10) may be obtained by weighted combination with the original discriminator loss as follows:(10)ladv=−Ex∼PrDx+Ex∼PgDx+λEx∼Px^∇xDxp−12,rp shows different data in the model. Xp is the distribution obtained by randomly sampling the real and generated data.


#### 3.2.2. Discriminant Network Design

The network structure basically adopts the CNN design, that is, the image is obtained through the convolution layer to obtain the local signal of the image, then the signal is filtered and filtered by the pool layer, and then linked to the full connection layer to integrate the local image signal into the feature vector, and finally the image is classified. The two discriminant networks respectively output feature vectors of the same dimension, and then connect the two feature vectors to output the feature vectors. The basic structure of the discrimination network is shown in Figure6.Figure 6
Discriminant network structure diagram.The global discriminant network can take all the images as input, including the real mural images and the synthesized images in the generation network. Each convolution layer can pass through 2 × 2 to deepen the feature map so as to obtain more abundant graphic signals in the graph. The most detailed global feature discrimination network structure is shown in Table2.Table 2
Global discriminant network.
TypeKernelStrideOutputsConv5 × 52 × 264Conv5 × 52 × 2128Conv5 × 52 × 2156Conv5 × 52 × 2512Conv5 × 52 × 2512FC--1024


### 3.3. Super-Resolution Reconstruction to Get the Mural More Artistic

In order to reduce the aesthetic value of the painting effect, a super-resolution reconstruction method is invented to improve the painting effect. Super-resolution reconstruction is a kind of image processing method that computer processes the pixel sequence of low resolution image to restore the high-resolution image.This method is based on the generated network. First, the reconstructed high-resolution image is output to the generated network, and then it is judged as a real high-resolution network. Then, through the transfer learning method, the high batch uniformity is gradually removed, and the network model is established in stages, so as to achieve further high mural image optimization.The super-resolution reconstruction method of the enhanced art mural pictures is based on the computer network and follows the basic construction method of generating the confrontation network system. The discrimination system judges whether the output image in the generation system and the real image are true or false. The network structure of the super-resolution reconstruction method for art painting images is shown in Figure7.Figure 7
Network structure diagram of super-resolution reconstruction ways for enhanced Chinese mural images.


## 3.1. 3D Laser Scanning Technology

In the implementation process, a variety of high-tech surveying and mapping instruments are integrated. On this basis, based on the noncontact high-speed laser measurement method, calculate the distance between the target and the center by recording the time difference, as shown in the following equation:(2)d=c×Δt2,where D is the measured distance; C is the light speed; t is the variation. Laser scanner system using laser scanning technology is mainly composed of hardware system and software system, as shown in Figure 2. The computer system records the emission time t1 and the optical scanning system. Location and angle information (X): when the optical receiving system receives the light signal reflected by the target, the computer system records the receiving time t2 and controls the laser to emit light from different angles through the scanning control system.Figure 2
3D Laser scanning system.The coordinate center defined by the laser scanner is the original part, andX and Y are perpendicular to each other. The results of measurements are shown in Figure 3.Figure 3
Schematic diagram of 3D laser measurement.During the 3D laser scanning process, the ranging observation value S is mainly obtained by the 3D laser scanner through the data acquisition system. When using the ground laser scanning 3D measurement technology in 3D laser scanning for soil and water conservation monitoring, the coordinate system usually used is the internal coordinate system of the instrument; among them, theX axis is in the lateral scanning plane, and the Y axis and Z axis are, respectively, in the lateral scanning plane. Inner and X-axis and the horizontal one are vertical to each other, and formula (3) for the coordinates of the 3D laser foot point is obtained as follows:(3)X=Dcosβcosα,Y=Dcosβsinα,Z=Dsinβ.In this formula, the three-dimensional coordinates of the measured object are represented by the variables “X, Y, and Z”; the distance between the measured object and the instrument is represented by D; the horizontal and vertical scanning angles are represented by α and β, respectively.In addition to the object displacement Δ = , the measurement sensitivity is also related to several other parameters such as ax,β, f, and l. A is the angle between the incident laser beam and the surface normal of the measured object, that is, the incident angle, and we define this parameter as the working angle. 1 is the distance between the spot where the laser irradiates the surface of the object to be measured and the main plane of the receiving lens, and we define this parameter as the working distance of the system. We select the optimal incident angle of the system laser projection by observing the changing trend of the working angle a and the measurement sensitivity δ at the positions of 30 mm, 60 mm, 90 mm, and 120 mm, respectively, as shown in Figure 4.Figure 4
Relationship between working anglea and measurement sensitivity δ.When the measurement range and working angle of the system are determined, the relationship between the working distance I and the measurement sensitivityδ is known. For angles of 10°, 20°, 30°, 40°, 50°, 60°, 70°, and 80°, the changing trends of the working distance l and the measurement sensitivity δ are shown in Figure 5.Figure 5
Relationship between working distance l1 and measurement sensitivityδ.


## 3.2. Network Structure Design

### 3.2.1. Generative Network Design

The generation system adopts the FCN structure and the decoder decoder structure. The result of the mural image is set as only the image information of the mask area, while the image information of other areas remains unchanged. The result of generating a network is usually a size of 128×128 preprocessed mural pictures. When an image having 24 to 36 pixel values is randomly obtained and the mask is completed, and the pixel value of the mask area is set to zero in the input image, the mask mural image is obtained. The FCN network obtains the image feature signal through the convolution layer, restores the feature map signal to the size image of the original pixel through the deconvolution layer calculation, and outputs the image value of the image restoration non mask area, and saves the restored mask area image information, and outputs the final restored image value. The generated network details are shown in Table1.Table 1
Generate network details information table.
TypeKernelDilationStrideOutputsConv5 × 611 × 264Conv3 × 212 × 132Conv3 × 111 × 1128Conv3 × 112 × 2256Conv3 × 111 × 2128Dilated conv3 × 321 × 1256Dilated conv3 × 331 × 3256Dilated conv3 × 161 × 364Conv3 × 111 × 1256Conv4 × 111/2 × 2128Deconv3 × 111 × 2128Conv4 × 312 × 1/264Deconv3 × 311 × 232Output3 × 312 × 13In order to better generate close to the real repair effect, this paper proposes to improve the model by using the MSE and adversarial loss function. The MSE and the adversarial are optimized simultaneously with the different network, and the GNM and the DNM are optimized.The function is used to measure the quality of their built image. The final loss function calculation formula (4) can be expressed as follows, in which the coefficients employed to balance different losses are symbolized by input 1, 2.(4)lG=lMSESR+λ1lVGGSR+λ2ladv.The pixel-level MSE loss calculation formula (5) is as follows:(5)lMSESR=1r2WH∑x=1rW∑y=1rHIx,yHR−GθGILRx,y2.W and H are the width and height, respectively, and represent Euclidean distance between real image pixels and the generated pixels.The MSE loss can make the reconstructed graphics have extremely high peak signal-to-noise ratios, but usually lack high-frequency content. We calculate the MSE by summing all squared losses for each sample and the formula is shown as(6)MSE=1N∑x,y∈Dy−predictionx2,whereN represents the total number of samples; (x, y) represents data, wherein represents the attribute set in the exercise data; and y represents the true value of the exercise data. The prediction (x) is the expected value of the exercise data X.Based on the same principle, the main purpose of the judgment system is to make the output of the power generation system approximately zero, while the output of the judgment system of the real data is approximately one. The resistance loss formula is shown as follows:(7)minGmaxDVD,G=EX∼PdataxlogDX+EZ∼Pzlog1−DGz.Here,x refers to the real data, while z indicates the other data, and data(x)P is the probability of real data, ( )zP z represents the probability distribution of synthetic data. Such combined data result represents the data result combined by the generation network, and D (x) represents the discrimination method, and represents the probability of judging the statistics of the true number to be true. D (g (z)) is a probability that the computer network determines whether the synthesized data generated by the computer network is actual. Log D (x) represents the determination of the real data by the discriminating computer network. D (g (z)) represents the determination of the comprehensive information.Loss function of WGAN (8) is like this(8)LD=Ex∼PrDx−Ex∼PgDx.Gradients of the discriminatorD(x) are not greater than a finite constant K by adding the Lipschitz limit, as shown in(9)∇XDx≤K.Generally,K is equal to 1, and adversarial loss calculation formula (10) may be obtained by weighted combination with the original discriminator loss as follows:(10)ladv=−Ex∼PrDx+Ex∼PgDx+λEx∼Px^∇xDxp−12,rp shows different data in the model. Xp is the distribution obtained by randomly sampling the real and generated data.


### 3.2.2. Discriminant Network Design

The network structure basically adopts the CNN design, that is, the image is obtained through the convolution layer to obtain the local signal of the image, then the signal is filtered and filtered by the pool layer, and then linked to the full connection layer to integrate the local image signal into the feature vector, and finally the image is classified. The two discriminant networks respectively output feature vectors of the same dimension, and then connect the two feature vectors to output the feature vectors. The basic structure of the discrimination network is shown in Figure6.Figure 6
Discriminant network structure diagram.The global discriminant network can take all the images as input, including the real mural images and the synthesized images in the generation network. Each convolution layer can pass through 2 × 2 to deepen the feature map so as to obtain more abundant graphic signals in the graph. The most detailed global feature discrimination network structure is shown in Table2.Table 2
Global discriminant network.
TypeKernelStrideOutputsConv5 × 52 × 264Conv5 × 52 × 2128Conv5 × 52 × 2156Conv5 × 52 × 2512Conv5 × 52 × 2512FC--1024


## 3.2.1. Generative Network Design

The generation system adopts the FCN structure and the decoder decoder structure. The result of the mural image is set as only the image information of the mask area, while the image information of other areas remains unchanged. The result of generating a network is usually a size of 128×128 preprocessed mural pictures. When an image having 24 to 36 pixel values is randomly obtained and the mask is completed, and the pixel value of the mask area is set to zero in the input image, the mask mural image is obtained. The FCN network obtains the image feature signal through the convolution layer, restores the feature map signal to the size image of the original pixel through the deconvolution layer calculation, and outputs the image value of the image restoration non mask area, and saves the restored mask area image information, and outputs the final restored image value. The generated network details are shown in Table1.Table 1
Generate network details information table.
TypeKernelDilationStrideOutputsConv5 × 611 × 264Conv3 × 212 × 132Conv3 × 111 × 1128Conv3 × 112 × 2256Conv3 × 111 × 2128Dilated conv3 × 321 × 1256Dilated conv3 × 331 × 3256Dilated conv3 × 161 × 364Conv3 × 111 × 1256Conv4 × 111/2 × 2128Deconv3 × 111 × 2128Conv4 × 312 × 1/264Deconv3 × 311 × 232Output3 × 312 × 13In order to better generate close to the real repair effect, this paper proposes to improve the model by using the MSE and adversarial loss function. The MSE and the adversarial are optimized simultaneously with the different network, and the GNM and the DNM are optimized.The function is used to measure the quality of their built image. The final loss function calculation formula (4) can be expressed as follows, in which the coefficients employed to balance different losses are symbolized by input 1, 2.(4)lG=lMSESR+λ1lVGGSR+λ2ladv.The pixel-level MSE loss calculation formula (5) is as follows:(5)lMSESR=1r2WH∑x=1rW∑y=1rHIx,yHR−GθGILRx,y2.W and H are the width and height, respectively, and represent Euclidean distance between real image pixels and the generated pixels.The MSE loss can make the reconstructed graphics have extremely high peak signal-to-noise ratios, but usually lack high-frequency content. We calculate the MSE by summing all squared losses for each sample and the formula is shown as(6)MSE=1N∑x,y∈Dy−predictionx2,whereN represents the total number of samples; (x, y) represents data, wherein represents the attribute set in the exercise data; and y represents the true value of the exercise data. The prediction (x) is the expected value of the exercise data X.Based on the same principle, the main purpose of the judgment system is to make the output of the power generation system approximately zero, while the output of the judgment system of the real data is approximately one. The resistance loss formula is shown as follows:(7)minGmaxDVD,G=EX∼PdataxlogDX+EZ∼Pzlog1−DGz.Here,x refers to the real data, while z indicates the other data, and data(x)P is the probability of real data, ( )zP z represents the probability distribution of synthetic data. Such combined data result represents the data result combined by the generation network, and D (x) represents the discrimination method, and represents the probability of judging the statistics of the true number to be true. D (g (z)) is a probability that the computer network determines whether the synthesized data generated by the computer network is actual. Log D (x) represents the determination of the real data by the discriminating computer network. D (g (z)) represents the determination of the comprehensive information.Loss function of WGAN (8) is like this(8)LD=Ex∼PrDx−Ex∼PgDx.Gradients of the discriminatorD(x) are not greater than a finite constant K by adding the Lipschitz limit, as shown in(9)∇XDx≤K.Generally,K is equal to 1, and adversarial loss calculation formula (10) may be obtained by weighted combination with the original discriminator loss as follows:(10)ladv=−Ex∼PrDx+Ex∼PgDx+λEx∼Px^∇xDxp−12,rp shows different data in the model. Xp is the distribution obtained by randomly sampling the real and generated data.


## 3.2.2. Discriminant Network Design

The network structure basically adopts the CNN design, that is, the image is obtained through the convolution layer to obtain the local signal of the image, then the signal is filtered and filtered by the pool layer, and then linked to the full connection layer to integrate the local image signal into the feature vector, and finally the image is classified. The two discriminant networks respectively output feature vectors of the same dimension, and then connect the two feature vectors to output the feature vectors. The basic structure of the discrimination network is shown in Figure6.Figure 6
Discriminant network structure diagram.The global discriminant network can take all the images as input, including the real mural images and the synthesized images in the generation network. Each convolution layer can pass through 2 × 2 to deepen the feature map so as to obtain more abundant graphic signals in the graph. The most detailed global feature discrimination network structure is shown in Table2.Table 2
Global discriminant network.
TypeKernelStrideOutputsConv5 × 52 × 264Conv5 × 52 × 2128Conv5 × 52 × 2156Conv5 × 52 × 2512Conv5 × 52 × 2512FC--1024


## 3.3. Super-Resolution Reconstruction to Get the Mural More Artistic

In order to reduce the aesthetic value of the painting effect, a super-resolution reconstruction method is invented to improve the painting effect. Super-resolution reconstruction is a kind of image processing method that computer processes the pixel sequence of low resolution image to restore the high-resolution image.This method is based on the generated network. First, the reconstructed high-resolution image is output to the generated network, and then it is judged as a real high-resolution network. Then, through the transfer learning method, the high batch uniformity is gradually removed, and the network model is established in stages, so as to achieve further high mural image optimization.The super-resolution reconstruction method of the enhanced art mural pictures is based on the computer network and follows the basic construction method of generating the confrontation network system. The discrimination system judges whether the output image in the generation system and the real image are true or false. The network structure of the super-resolution reconstruction method for art painting images is shown in Figure7.Figure 7
Network structure diagram of super-resolution reconstruction ways for enhanced Chinese mural images.


## 4. Results and Analysis

### 4.1. Experimental Environment and Experimental Design

The hardware environment built in this experiment: CPU is Inter Core i7-7700K, memory is 16 GB, graphics card is NVIDIA Ge Force GTX 1080Ti, has 3D laser scanner, CCD camera, and GPS receiver; the software environment built: CUDA version is 9.0, cu DNN version is 7.0, and operating system is Windows 10; it uses Python 3.6, using the PyTorch framework to write test experiments. The compiled software used is the device. The training set of the data used in this experiment has 800 DIV2K images, 2650 Flickr2K images, and 90 murals. The test dataset has 30 murals. The factor for the high-resolution and low-resolution images in this paper is 4. Finally, the mural dataset is sent to the network to continue training, and the generator and discriminator are renewed alternately. The model is converged, and finally the following experimental analysis is carried out on the obtained results.


### 4.2. Evaluation Index of 3D Laser Scanning Results

Using two commonly used nonsubjective assessment indexes: the two indicators judge quality of image reconstruction. The higher the PSNR number of the two images, the less the contortion between scanned images and high-resolution images, that is, the more effective the scanned results; the way judges reconstructed image. The more similar the points of the two images, the closer the SSIM value is to 1, which is more consistent with the public’s visual sensory effect. The specific calculation expression of PSNR is shown in formula (11), and the specific calculation expression of SSIM is shown in (12).(11)PSNR=10×log102252×W×H∑i=1W∑j=1Hx¯i,j−xi,j2.(12)SSIMX,Y=2μXμY+C12σXY+C2μX2+μY2+C1σX2+σY2+C2.W and H represent the width and height in the model, respectively, x(i, j)−x(i, j) indicates the Euclidean distance between pixels of the two images; μx and μy indicate the context of the images X and Y, respectively, σx and σy represent the standard deviation of the images X and Y, respectively, σXY represents the variables of the images X and Y, which are linearly transformed, and C1 and C2 are constant XY numbers.It can be concluded that the response curve changes sharply at the pixel values ​​near both ends, and the data fit is poor. In order to better restore the response curve, a weight function is introduced to increase the proportion of exposure data located in the middle part of the response function. As shown in (13), it is the response curve closest to the actual response curve, and the fitting process is shown in Figure 8:(13)wz=z−zmin,z≤12zmax+zmin,zmax−z,z>12zmax+zmin.Figure 8
Schematic diagram of the response curve solution process. (a) Collected pixels; (b) response curve.
(a)(b)The horizontal axis in Figure8(a) corresponds to the pixel value Z of the image, and the vertical axis is the logarithm of the exposure 1nX. Select 3 pixels in 5 images with different exposures to restore the response curve g, the 3 symbols in the figure represent 3 pixels at different positions of an image, and 3 points are selected for 5 images corresponding to 3-curve fragment. According to the constraint condition g(Zmi) = 0, the three curve segments are combined into one by moving up and down, thus expressing the corresponding curve between exposure and pixel value, as shown in Figure 8(b).


### 4.3. Analysis of Data

In order to prove the effect of laser scanning technology in rebuilding old frescoes, 4 images are chosen in the rebuilt 4x high-resolution frescoes at random and compared with the corresponding low-resolution frescoes, and the PSNR and SSIM of the two groups of images were mainly compared. The exploration consequences are clearly presented in Table3. It will be known from the consequences in Table 3 that reconstructed murals also get good PSNR and SSIM values.Table 3
SSIM and PSNR after reconstruction of murals.
ImageabcdPSNR25.7130.0125.0430.11SSIM0.6760.7720.8160.718In order to further study the characteristics of laser scanning technology from image texture information, overall brightness, and clarity, 50 professional evaluators were selected to subjectively score the reconstruction results of conventional images of murals of different styles and mural maps of 3D laser scanning technology, as shown in Figure9.Figure 9
Subjective scoring map of the reconstructed murals.After statistics, the overall consistency and structural continuity scores of the three different algorithms are averaged, respectively. To visually compare the specific information of satisfaction evaluation, a histogram is made as shown in Figure10. By comparison, the score, which is consistent with the previous ones and consecutive in structure, is 3.3–3.8 under the first and second algorithm, and the mentioned algorithms are 3.8 and 4.1.Figure 10
Comparison of subjective scores under different algorithms.10 test mural images were randomly selected as samples through experiments, and the objective comparison of the restoration effect of artificially destroyed mural images was carried out by two methods. The comparison of PSER under different algorithms is shown in Table4. It analyzes the similar aspects from three perspectives including the brightness, comparison, and the structure. The SSIM values ​​under the different algorithms are shown in Table 5.Table 4
PSER values under different algorithms.
SampleDocument 1Document 2Algorithm138.6337.6940.53236.9438.3442.67336.5838.2141.95442.6741.1845.36537.9338.0642.67638.2838.0943.62739.9438.3342.88841.2342.5845.95942.6342.3945.231038.1538.9741.36Table 5
SSIM values under different algorithms.
SampleDocument 1Document 2Algorithm10.8530.8750.85320.7760.8120.93130.8650.8500.87140.7660.7900.83150.8480.8270.90160.7730.8310.88970.7320.8380.85180.7580.7630.86290.8160.7940.847100.7980.8280.863We calculated the average value of PSNR and SSIM corresponding to each image under the three algorithms. Compared with the literature [1] algorithm, the algorithm in this paper has obvious improvement in visual continuity because the algorithm in this paper adopts the idea of ​​confrontation generation. Repairing murals has better repair effect for larger damaged areas, and the distortion compared to the original image is smaller, and the final PSNR value is increased by 3∼5 dB on average. The use of two discriminant networks has played a role in promoting the optimization of the generative network model; at the same time, the local consistency of the image has been greatly improved, which makes the SSIM value increase by 0.05∼0.07 on average.


## 4.1. Experimental Environment and Experimental Design

The hardware environment built in this experiment: CPU is Inter Core i7-7700K, memory is 16 GB, graphics card is NVIDIA Ge Force GTX 1080Ti, has 3D laser scanner, CCD camera, and GPS receiver; the software environment built: CUDA version is 9.0, cu DNN version is 7.0, and operating system is Windows 10; it uses Python 3.6, using the PyTorch framework to write test experiments. The compiled software used is the device. The training set of the data used in this experiment has 800 DIV2K images, 2650 Flickr2K images, and 90 murals. The test dataset has 30 murals. The factor for the high-resolution and low-resolution images in this paper is 4. Finally, the mural dataset is sent to the network to continue training, and the generator and discriminator are renewed alternately. The model is converged, and finally the following experimental analysis is carried out on the obtained results.


## 4.2. Evaluation Index of 3D Laser Scanning Results

Using two commonly used nonsubjective assessment indexes: the two indicators judge quality of image reconstruction. The higher the PSNR number of the two images, the less the contortion between scanned images and high-resolution images, that is, the more effective the scanned results; the way judges reconstructed image. The more similar the points of the two images, the closer the SSIM value is to 1, which is more consistent with the public’s visual sensory effect. The specific calculation expression of PSNR is shown in formula (11), and the specific calculation expression of SSIM is shown in (12).(11)PSNR=10×log102252×W×H∑i=1W∑j=1Hx¯i,j−xi,j2.(12)SSIMX,Y=2μXμY+C12σXY+C2μX2+μY2+C1σX2+σY2+C2.W and H represent the width and height in the model, respectively, x(i, j)−x(i, j) indicates the Euclidean distance between pixels of the two images; μx and μy indicate the context of the images X and Y, respectively, σx and σy represent the standard deviation of the images X and Y, respectively, σXY represents the variables of the images X and Y, which are linearly transformed, and C1 and C2 are constant XY numbers.It can be concluded that the response curve changes sharply at the pixel values ​​near both ends, and the data fit is poor. In order to better restore the response curve, a weight function is introduced to increase the proportion of exposure data located in the middle part of the response function. As shown in (13), it is the response curve closest to the actual response curve, and the fitting process is shown in Figure 8:(13)wz=z−zmin,z≤12zmax+zmin,zmax−z,z>12zmax+zmin.Figure 8
Schematic diagram of the response curve solution process. (a) Collected pixels; (b) response curve.
(a)(b)The horizontal axis in Figure8(a) corresponds to the pixel value Z of the image, and the vertical axis is the logarithm of the exposure 1nX. Select 3 pixels in 5 images with different exposures to restore the response curve g, the 3 symbols in the figure represent 3 pixels at different positions of an image, and 3 points are selected for 5 images corresponding to 3-curve fragment. According to the constraint condition g(Zmi) = 0, the three curve segments are combined into one by moving up and down, thus expressing the corresponding curve between exposure and pixel value, as shown in Figure 8(b).


## 4.3. Analysis of Data

In order to prove the effect of laser scanning technology in rebuilding old frescoes, 4 images are chosen in the rebuilt 4x high-resolution frescoes at random and compared with the corresponding low-resolution frescoes, and the PSNR and SSIM of the two groups of images were mainly compared. The exploration consequences are clearly presented in Table3. It will be known from the consequences in Table 3 that reconstructed murals also get good PSNR and SSIM values.Table 3
SSIM and PSNR after reconstruction of murals.
ImageabcdPSNR25.7130.0125.0430.11SSIM0.6760.7720.8160.718In order to further study the characteristics of laser scanning technology from image texture information, overall brightness, and clarity, 50 professional evaluators were selected to subjectively score the reconstruction results of conventional images of murals of different styles and mural maps of 3D laser scanning technology, as shown in Figure9.Figure 9
Subjective scoring map of the reconstructed murals.After statistics, the overall consistency and structural continuity scores of the three different algorithms are averaged, respectively. To visually compare the specific information of satisfaction evaluation, a histogram is made as shown in Figure10. By comparison, the score, which is consistent with the previous ones and consecutive in structure, is 3.3–3.8 under the first and second algorithm, and the mentioned algorithms are 3.8 and 4.1.Figure 10
Comparison of subjective scores under different algorithms.10 test mural images were randomly selected as samples through experiments, and the objective comparison of the restoration effect of artificially destroyed mural images was carried out by two methods. The comparison of PSER under different algorithms is shown in Table4. It analyzes the similar aspects from three perspectives including the brightness, comparison, and the structure. The SSIM values ​​under the different algorithms are shown in Table 5.Table 4
PSER values under different algorithms.
SampleDocument 1Document 2Algorithm138.6337.6940.53236.9438.3442.67336.5838.2141.95442.6741.1845.36537.9338.0642.67638.2838.0943.62739.9438.3342.88841.2342.5845.95942.6342.3945.231038.1538.9741.36Table 5
SSIM values under different algorithms.
SampleDocument 1Document 2Algorithm10.8530.8750.85320.7760.8120.93130.8650.8500.87140.7660.7900.83150.8480.8270.90160.7730.8310.88970.7320.8380.85180.7580.7630.86290.8160.7940.847100.7980.8280.863We calculated the average value of PSNR and SSIM corresponding to each image under the three algorithms. Compared with the literature [1] algorithm, the algorithm in this paper has obvious improvement in visual continuity because the algorithm in this paper adopts the idea of ​​confrontation generation. Repairing murals has better repair effect for larger damaged areas, and the distortion compared to the original image is smaller, and the final PSNR value is increased by 3∼5 dB on average. The use of two discriminant networks has played a role in promoting the optimization of the generative network model; at the same time, the local consistency of the image has been greatly improved, which makes the SSIM value increase by 0.05∼0.07 on average.


## 5. Conclusion

The ancient murals are damaged badly due to the history and environment factors. The application of digital images technology in the protection for the ancient culture relics is considered. For the problem, the thesis has put forward an adversarial network based on the deep learning with 3D technology to restore the resolution of the murals. After the reconstruction, the high-resolution murals are clear and bright and retain rich textured details. Moreover, the ornamental value and research value of murals have been improved to a certain extent, and the loss of ancient murals has been further protected [14].



---

*Source: 8439616-2022-08-19.xml*