# Ancient Stone Inscription Image Denoising and Inpainting Methods Based on Deep Neural Networks

Authors: Haoming Zhang; Yue Qi; Xiaoting Xue; Yahui Nan
Journal: Discrete Dynamics in Nature and Society
 (2021)

---

## Abstract

Chinese ancient stone inscriptions contain Chinese traditional calligraphy culture and art information. However, due to the long history of the ancient stone inscriptions, natural erosion, and poor early protection measures, there are a lot of noise in the existing ancient stone inscriptions, which has adverse effects on reading these stone inscriptions and their aesthetic appreciation. At present, digital technologies have played important roles in the protection of cultural relics. For ancient stone inscriptions, we should obtain more perfect digital results without multiple types of noise, while there are few deep learning methods designed for processing stone inscription images. Therefore, we propose a basic framework for image denoising and inpainting of stone inscriptions based on deep learning methods. Firstly, we collect as many images of stone inscriptions as possible and preprocess these images to establish an inscriptions image dataset for image denoising and inpainting. In addition, an improved GAN with a denoiser is used for generating more virtual stone inscription images to expand the dataset. On the basis of these collected and generated images, we designed a stone inscription image denoising model based on multiscale feature fusion and introduced Charbonnier loss function to improve this image denoising model. To further improve the denoising results, an image inpainting model with the coherent semantic attention mechanism is introduced to recover some effective information removed by the former denoising model as much as possible. The experimental results show that our image denoising model achieves better results on PSNR, SSIM, and CEI. The final results have obvious visual improvement compared with the original stone inscription images.


---

## Body

## 1. Introduction

The establishment of digital cultural relic library is not only conducive to the scientific researches and meticulous management of each cultural relic, but also enables the cultural relics to go out of the geographical space constraints of the exhibition halls, so that the public can appreciate the exquisite cultural relics from different angles. Chinese ancient stone inscriptions contain rich historical and cultural information and have important historical value. Therefore, it is urgent to apply advanced technologies for the effective digital preservation of these stone inscriptions, which is of great significance to the inheritance and development of national culture.The stone inscriptions contains various types of noise, as shown in Figure1. There may be lots of spots, areas of corrosion, scratches, and so on. These types of noise are complex and diverse, which results in that it is difficult for traditional denoising models to deal with these stone inscriptions images.Figure 1
Different types of noise in ancient stone inscription images.In recent years, the rapid development of computer vision technologies provides a new way for the digitalization of stone inscriptions and the improvements of their image quality. At present, most of stone inscriptions image processing methods are proposed based on different types of image filters, which cannot deal with these diverse noise well. However, there are several difficulties for designing the deep learning-based model to process these stone inscriptions images. For example, there is no public image datasets of the stone inscriptions at present, and it is difficult to obtain enough images of the stone inscriptions. In addition, due to the long history of these ancient stone inscriptions, natural erosion, and poor early protections, most of ancient stone inscriptions are seriously damaged. Moreover, image denoising models may make the strokes in the stone inscriptions too smooth and incomplete in the images after denoising.In order to solve these problems and get perfect digital image data of stone inscriptions, advanced image processing technologies are needed to process these images. At present, deep learning methods have been widely applied to image processing, and image denoising and inpainting are two common computer vision tasks, while there are few deep learning methods designed for processing stone inscription images. Therefore, this paper design some deep learning-based methods to deal with the complex noise of the stone inscription images, which are not common in ordinary images. Our work would have benefits for cultural relics protection and the inheritance of Chinese calligraphy culture.The main points of this paper are as follows:(1)
The building and expansion of image dataset of stone inscriptions: we collect the images from Forest of Stone Steles Museum using cameras, image preprocessing, and classification carried out to form the original image dataset. Moreover, we designed an improved GAN model with a denoiser for image generation to expand the image dataset.(2)
A stone inscriptions image denoising and inpainting framework is proposed based on deep learning networks. An image denoising network based on multiscale feature fusion is designed for dealing with multiple types of noise in stone inscriptions images, and Charbonnier loss function is introduced to improve image denoising model. Then, to solve the problem of missing strokes in the denoising results, based on a cascade semantic segmentation network architecture, an image inpainting model with the coherent semantic attention mechanism is proposed to recover some effective information.


## 2. Related Works

### 2.1. Data Augmentation

Data augmentation is mainly used to expand the training dataset, so that the trained models have stronger generalization ability [1]. Traditional data augmentation methods mainly include horizontal/vertical flip, rotation, scaling, clipping, and translation. Although these operations can increase the amount of related image data, they do not produce any new images in essence. Generative adversarial networks [2] (GAN) are generative models, which have been widely used in related visual tasks [3]. GAN mainly consists of two parts: generator (G) and discriminator (D). In the training process, the goal of G is to generate virtual images, while the goal of D is to separate the images generated by G from the real images. After this continuous “game” training, the models can generate the final images similar to the original images.


### 2.2. Image Denoising

Image denoising has been paid extensive attention for a long time. For stone inscription image denoising, image filters are commonly used in image denoising methods. Zhang et al. [4] protect the strokes of calligraphy characters, but it is difficult to achieve satisfactory results for the complex noise. Shi et al. [5] use Butterworth low-pass filter to process the high and low frequency components of stone inscription images; this method can effectively remove the burr noise of strokes. However, these traditional image denoising methods can achieve good denoising results, but there still exist many problems, such as relying on prior knowledge, manually setting parameters, and removing few types of noise.In recent years, researchers put forward many deep learning-based denoising models. For example, Xie et al. [6] proposed an image denoising method to remove a single type of noise effectively. Priyanka and Wang [7] proposed a deep fully symmetric convolution neural network structure to generate clean images, and this method is an end-to-end network without using prior information. Burger et al. [8] used multilayer perceptron for image denoising, but the method only can deal with one single type of noise. Zhang and Wu [9] introduced residual learning strategy to separate noise from the images. Zhao et al. [10] proposed an image denoising method without knowing the types of noise, while this method requires paired images (a noisy image and its corresponding clear image) as the input data. Zhao et al. [11] regarded the denoising task as a special case of noise transfer task and then proposed a deep unsupervised image denoising model by learning the transformation of noise. In addition, the input of the model is the noise level graphs [12].


### 2.3. Image Inpainting

The core challenge of image inpainting is to maintain the global semantic structure and generate real texture details for the missing image regions. Moreover, the inpainting images should maintain the global consistency and local pixel continuity of the images. Barnes et al. [13] proposed a block search algorithm to search the optimal image fitting from the boundaries of the missing regions to fill the missing parts. Then, Wilczkowiak et al. [14] improved Barnes’ algorithm to further determine the search areas and find the best image blocks. The methods based on block search lack the understanding of high-level semantic features, which results in that it is difficult to reconstruct the details of local structures.In recent years, the methods based on convolutional neural networks [15, 16] use the data distribution to capture the semantic information of the images. These methods are more reasonable to repair the missing image areas, but cannot effectively utilize the context information. While the effective utilization of the context information would benefit for achieving good results, the related methods can be divided into two types: the first type of methods use the spatial attention mechanism to recover the missing image areas with the reference of the surrounding image features [17, 18]. These methods can ensure that the generated image areas are consistent with the context semantic information, but they are only effective for rectangular missing in the images, and the results have pixel discontinuity at the edges of the images. The second type of methods directly predicts the missing image areas from the effective information of original images [19, 20]. These methods can deal with any missing regions in the images.Above all, there are only a few researches on image denoising algorithm of stone inscription images, while there are few works on the automatic generation and image inpainting of stone inscriptions images. The existing denoising methods often result in the loss of effective information, and the small amount of fixed stone inscription images is also a natural challenge. Therefore, this paper mainly focuses on the automatic generation, multitype noise removal, and image inpainting methods of Chinese ancient stone inscriptions images, so as to obtain the images with good visual effects.


## 2.1. Data Augmentation

Data augmentation is mainly used to expand the training dataset, so that the trained models have stronger generalization ability [1]. Traditional data augmentation methods mainly include horizontal/vertical flip, rotation, scaling, clipping, and translation. Although these operations can increase the amount of related image data, they do not produce any new images in essence. Generative adversarial networks [2] (GAN) are generative models, which have been widely used in related visual tasks [3]. GAN mainly consists of two parts: generator (G) and discriminator (D). In the training process, the goal of G is to generate virtual images, while the goal of D is to separate the images generated by G from the real images. After this continuous “game” training, the models can generate the final images similar to the original images.


## 2.2. Image Denoising

Image denoising has been paid extensive attention for a long time. For stone inscription image denoising, image filters are commonly used in image denoising methods. Zhang et al. [4] protect the strokes of calligraphy characters, but it is difficult to achieve satisfactory results for the complex noise. Shi et al. [5] use Butterworth low-pass filter to process the high and low frequency components of stone inscription images; this method can effectively remove the burr noise of strokes. However, these traditional image denoising methods can achieve good denoising results, but there still exist many problems, such as relying on prior knowledge, manually setting parameters, and removing few types of noise.In recent years, researchers put forward many deep learning-based denoising models. For example, Xie et al. [6] proposed an image denoising method to remove a single type of noise effectively. Priyanka and Wang [7] proposed a deep fully symmetric convolution neural network structure to generate clean images, and this method is an end-to-end network without using prior information. Burger et al. [8] used multilayer perceptron for image denoising, but the method only can deal with one single type of noise. Zhang and Wu [9] introduced residual learning strategy to separate noise from the images. Zhao et al. [10] proposed an image denoising method without knowing the types of noise, while this method requires paired images (a noisy image and its corresponding clear image) as the input data. Zhao et al. [11] regarded the denoising task as a special case of noise transfer task and then proposed a deep unsupervised image denoising model by learning the transformation of noise. In addition, the input of the model is the noise level graphs [12].


## 2.3. Image Inpainting

The core challenge of image inpainting is to maintain the global semantic structure and generate real texture details for the missing image regions. Moreover, the inpainting images should maintain the global consistency and local pixel continuity of the images. Barnes et al. [13] proposed a block search algorithm to search the optimal image fitting from the boundaries of the missing regions to fill the missing parts. Then, Wilczkowiak et al. [14] improved Barnes’ algorithm to further determine the search areas and find the best image blocks. The methods based on block search lack the understanding of high-level semantic features, which results in that it is difficult to reconstruct the details of local structures.In recent years, the methods based on convolutional neural networks [15, 16] use the data distribution to capture the semantic information of the images. These methods are more reasonable to repair the missing image areas, but cannot effectively utilize the context information. While the effective utilization of the context information would benefit for achieving good results, the related methods can be divided into two types: the first type of methods use the spatial attention mechanism to recover the missing image areas with the reference of the surrounding image features [17, 18]. These methods can ensure that the generated image areas are consistent with the context semantic information, but they are only effective for rectangular missing in the images, and the results have pixel discontinuity at the edges of the images. The second type of methods directly predicts the missing image areas from the effective information of original images [19, 20]. These methods can deal with any missing regions in the images.Above all, there are only a few researches on image denoising algorithm of stone inscription images, while there are few works on the automatic generation and image inpainting of stone inscriptions images. The existing denoising methods often result in the loss of effective information, and the small amount of fixed stone inscription images is also a natural challenge. Therefore, this paper mainly focuses on the automatic generation, multitype noise removal, and image inpainting methods of Chinese ancient stone inscriptions images, so as to obtain the images with good visual effects.


## 3. Methods

### 3.1. Establishment of the Image Dataset

Forest of Stone Steles Museum is one of the earliest and most famous museums in collecting ancient Chinese steles, and we collect the images from Forest of Stone Steles Museum for many times. For each common stele, six images were taken from different angles. While for a larger one, we divided it into several smaller parts, and six images were taken from different angles for each part. Finally, we obtained a total of 1,663 images.Image denoising models based on deep learning require pairs of images (noisy images and their corresponding clear images). Therefore, we constructed 827 pairs of images with single Chinese character and 500 pairs of images with multicharacters based on the collected images.


### 3.2. Stone Inscription Image Generation Based on Improved GAN

The amount of stone inscriptions images is not enough for image denoising models, so we design an improved GAN to generate more stone inscriptions images. Then, we can obtain a large number of noisy images with different levels of noise paired with clear images. These generated images can further expand the diversities of the noise and improve the performance of the denoising models. Figure2 shows the overall structure of stone inscription image generation based on improved GAN.Figure 2
The model of stone inscription image generation based on improved GAN.We assume that the noisy image isY and the clean image is X. Instead of learning X directly from Y, we learn the joint distribution of X and Y [21]. The methods based on Bayesian can predict the probability of an event happening using limited information. The following formula is a general description of Bayesian formula:(1)pBi|A=pBiPA|Bi∑j=1npBjpA|Bj.A noise reducerR is used to learn the distribution of a clean image X from a noisy image Y; that is px|y≈pRx|y. Then, we can obtain pairs of images x˜,y by the following two formulas:(2)y∼pyx˜=Ry⇒x˜,y,PRx,y=pRx|ypy.The generated distributionpRx|y is much similar to the real distribution px|y, and x˜ is very close to the clear image y.Then, a potential variableZ of the standard normal distribution is introduced to generate the noisy image Y from the clear image X, which is represented by conditional probability distribution py|x,z. To this end, a generator G is used to make the learned distribution pGy|x,z approximate to py|x,z. By calculating the marginal probability distribution of Z, pGy|x,z can be expressed as follows:(3)pGx,y=∫zpGy|x,zpxpzdz≈1L∑iLpGy|x,zipx.Then, the output of this generator is sampled to obtain a pair of imagesx,y˜.(4)z∼pzx∼pxy˜=Gx,z⇒x,y˜.The generated distributionpGy|x,z is much similar to the real distribution py|x,z, and y˜ is very close to the noise image.To make sure whether the generated pairs of images are consistent with the real image pairs, a discriminatorD is used to learn x,y from x,y˜ and x˜,y.Finally, the loss function is(5)minR,GmaxDLganR,G,D=Ex,yDx,y−αEx˜,yDx˜,y−1−αEx,y˜Dx,y˜.


### 3.3. Image Denoising and Inpainting for Stone Inscription Images

We propose a deep learning-based image denoising and inpainting framework for the stone inscriptions images. This framework mainly consists of two parts: image denoising network based on multiscale feature fusion and cascade image inpainting network with coherent semantic attention mechanism. We will describe this image denoising and inpainting framework in detail.


#### 3.3.1. Image Denoising Network Based on Multiscale Feature Fusion

The types of noise in stone inscription images are complex and diverse, which requires the denoising model to be able to deal with multiple types of noise simultaneously. In this paper, we utilize the image denoising network based on multiscale feature fusion (IDN-MSFF) to remove multiple types of noise in stone inscriptions images, which consist of four stages: initial visual feature extraction, rough-grained feature fusion, fine-grained feature fusion, and reconstruction. The main structure is shown in Figure3.Figure 3
Image denoising network based on multiscale feature fusion.As shown in Figure4, through the analysis of the noise in stone inscription images, we find out that the patterns of the noise are very similar to the rain streaks. Therefore, we can deal with the noise in stone inscription images by referring to the methods of removing the rain streaks [22]. Although the deep learning-based denoising models for natural images have achieved good performance, it is difficult to capture the intrinsic correlations of noise in a single feature scale. The noise in stone inscription images span different scales.Figure 4
The noise in a stone inscription image.In our image denoising network, the Gaussian kernel function is first used to sample the original noisy image into several images with different scales, and these images are input into the network. Then, several convolution layers in this network are used to extract the features of these images.Rough-grained feature fusion: multiple parallel residual convolution units are used to learn the global texture features, so that the network can use the similarity among global features to represent the noise patterns and fuse these multiscale visual features.Fine-grained feature fusion: the channel attention mechanism is used to increase the discrimination between the noise and the clear images, which makes the fusion representation of features more effective for image denoising. In order to reduce the computational burden of the network, step convolution is used to reduce the spatial dimension of features.Reconstruction module: the low-level multiscale features obtained by rough-grained feature fusion and the high-level multiscale features obtained by fine-grained feature fusion are further fused. Finally, the deconvolution is used to restore a denoised image.Loss function: the least square error is a common used loss function, but it is easy to produce smooth visual effect for the denoised images. Therefore, we use Charbonnier loss function in our network. This loss function has higher tolerance for small errors and better robustness.


#### 3.3.2. Cascaded Image Inpainting Network with Coherent Semantic Attention Mechanism

Many ancient stone inscriptions are damaged seriously, which results in that the results obtained by denoising models lose some parts of strokes. Figure5 shows the original image, the ground truth, and a denoising result image.Figure 5
The problem of image denoising. (a) The original image. (b) Ground truth. (c) The denoising result image.
(a)(b)(c)When the cultural relic protection workers repair the incomplete cultural relics, they first observe the overall structure; then, they conceive the contents of missing parts; finally, they gradually repair smaller parts. The whole process needs to ensure the global consistency and local continuity. Inspired by that, we introduce the coherent semantic attention mechanism into a cascaded image inpainting network. We first initialize the unknown regions with the most similar features in the known regions; then, we gradually optimize the filling contents for the unknown regions to ensure the internal continuity and global consistency of the missing blocks.U-net [23] can extract semantic level features efficiently. We divide U-net into two parts: U-net1 and U-net2. U-net1 tries to learn the rough representation of the missing regions, while U-net2 would achieve fine-tuning of rough representation by introducing coherent semantic attention mechanism [24]. The input of U-net1 is pairs of images: the denoised images Iin and the ground truth Igt, and the output is Ip. U-net2 takes Iin and Ip as the input, and the output is the final inpainting result Ir.(1)
Rough image inpainting by U-net1: as shown in Figure6, several convolution layers (4 × 4) are used for feature extraction, skipping are used for feature fusion, and upsampling is used for feature recovery. The input and output of U-net are symmetrical and have the same size. Due to the multilevel feature fusion by skip operations, U-net1 can achieve better results.(2)
Fine image inpainting by U-net2: as shown in Figure7, coherent semantic attention mechanism is introduced in the fourth layer of encoder stage. Encoder stage uses 3 × 3 ordinary convolutions and 4 × 4 detailed convolutions. U-net2 has better abilities to extract effective visual features on the premise of losing a small amount of information. The output of U-net1 is further restored by U-net2, and the final results should have both the overall consistency of images and the coherence of local regions.Figure 6
The structure of U-net1.Figure 7
The structure of U-net2.Coherence semantic attention mechanism: assume thatM is the missing regions of the image, and M¯ is the other known regions in the same image. For each missing pixel, the coherent semantic attention mechanism module searches for the most similar pixel blocks from M¯ and uses the blocks, which are generated by previous missing regions, as the secondary parts to update the next missing region blocks. Two formulas are used to measure the similarity:(6)Dmaxi=mi,m¯imi.m¯i,Dadi=mi,mi−1mi.mi−1,where Dmaxi represents the similarity between the missing region mi and the known region m¯i. Dadi is the similarity between the two adjacent image regions. Dmaxi is used to ensure the global consistency of the missing image regions, and Dadi is used to ensure the coherence of the generated image regions.Loss function: the loss function is designed based on the feature space of real sample image extracted by resnet-50 [25] and the image feature space generated by coherent semantic attention layers. Therefore, the final loss function is(7)LC=∑y∈MCSAIipy−δnIgty22+CSAdIipy−δnIgty22.


## 3.1. Establishment of the Image Dataset

Forest of Stone Steles Museum is one of the earliest and most famous museums in collecting ancient Chinese steles, and we collect the images from Forest of Stone Steles Museum for many times. For each common stele, six images were taken from different angles. While for a larger one, we divided it into several smaller parts, and six images were taken from different angles for each part. Finally, we obtained a total of 1,663 images.Image denoising models based on deep learning require pairs of images (noisy images and their corresponding clear images). Therefore, we constructed 827 pairs of images with single Chinese character and 500 pairs of images with multicharacters based on the collected images.


## 3.2. Stone Inscription Image Generation Based on Improved GAN

The amount of stone inscriptions images is not enough for image denoising models, so we design an improved GAN to generate more stone inscriptions images. Then, we can obtain a large number of noisy images with different levels of noise paired with clear images. These generated images can further expand the diversities of the noise and improve the performance of the denoising models. Figure2 shows the overall structure of stone inscription image generation based on improved GAN.Figure 2
The model of stone inscription image generation based on improved GAN.We assume that the noisy image isY and the clean image is X. Instead of learning X directly from Y, we learn the joint distribution of X and Y [21]. The methods based on Bayesian can predict the probability of an event happening using limited information. The following formula is a general description of Bayesian formula:(1)pBi|A=pBiPA|Bi∑j=1npBjpA|Bj.A noise reducerR is used to learn the distribution of a clean image X from a noisy image Y; that is px|y≈pRx|y. Then, we can obtain pairs of images x˜,y by the following two formulas:(2)y∼pyx˜=Ry⇒x˜,y,PRx,y=pRx|ypy.The generated distributionpRx|y is much similar to the real distribution px|y, and x˜ is very close to the clear image y.Then, a potential variableZ of the standard normal distribution is introduced to generate the noisy image Y from the clear image X, which is represented by conditional probability distribution py|x,z. To this end, a generator G is used to make the learned distribution pGy|x,z approximate to py|x,z. By calculating the marginal probability distribution of Z, pGy|x,z can be expressed as follows:(3)pGx,y=∫zpGy|x,zpxpzdz≈1L∑iLpGy|x,zipx.Then, the output of this generator is sampled to obtain a pair of imagesx,y˜.(4)z∼pzx∼pxy˜=Gx,z⇒x,y˜.The generated distributionpGy|x,z is much similar to the real distribution py|x,z, and y˜ is very close to the noise image.To make sure whether the generated pairs of images are consistent with the real image pairs, a discriminatorD is used to learn x,y from x,y˜ and x˜,y.Finally, the loss function is(5)minR,GmaxDLganR,G,D=Ex,yDx,y−αEx˜,yDx˜,y−1−αEx,y˜Dx,y˜.


## 3.3. Image Denoising and Inpainting for Stone Inscription Images

We propose a deep learning-based image denoising and inpainting framework for the stone inscriptions images. This framework mainly consists of two parts: image denoising network based on multiscale feature fusion and cascade image inpainting network with coherent semantic attention mechanism. We will describe this image denoising and inpainting framework in detail.


### 3.3.1. Image Denoising Network Based on Multiscale Feature Fusion

The types of noise in stone inscription images are complex and diverse, which requires the denoising model to be able to deal with multiple types of noise simultaneously. In this paper, we utilize the image denoising network based on multiscale feature fusion (IDN-MSFF) to remove multiple types of noise in stone inscriptions images, which consist of four stages: initial visual feature extraction, rough-grained feature fusion, fine-grained feature fusion, and reconstruction. The main structure is shown in Figure3.Figure 3
Image denoising network based on multiscale feature fusion.As shown in Figure4, through the analysis of the noise in stone inscription images, we find out that the patterns of the noise are very similar to the rain streaks. Therefore, we can deal with the noise in stone inscription images by referring to the methods of removing the rain streaks [22]. Although the deep learning-based denoising models for natural images have achieved good performance, it is difficult to capture the intrinsic correlations of noise in a single feature scale. The noise in stone inscription images span different scales.Figure 4
The noise in a stone inscription image.In our image denoising network, the Gaussian kernel function is first used to sample the original noisy image into several images with different scales, and these images are input into the network. Then, several convolution layers in this network are used to extract the features of these images.Rough-grained feature fusion: multiple parallel residual convolution units are used to learn the global texture features, so that the network can use the similarity among global features to represent the noise patterns and fuse these multiscale visual features.Fine-grained feature fusion: the channel attention mechanism is used to increase the discrimination between the noise and the clear images, which makes the fusion representation of features more effective for image denoising. In order to reduce the computational burden of the network, step convolution is used to reduce the spatial dimension of features.Reconstruction module: the low-level multiscale features obtained by rough-grained feature fusion and the high-level multiscale features obtained by fine-grained feature fusion are further fused. Finally, the deconvolution is used to restore a denoised image.Loss function: the least square error is a common used loss function, but it is easy to produce smooth visual effect for the denoised images. Therefore, we use Charbonnier loss function in our network. This loss function has higher tolerance for small errors and better robustness.


### 3.3.2. Cascaded Image Inpainting Network with Coherent Semantic Attention Mechanism

Many ancient stone inscriptions are damaged seriously, which results in that the results obtained by denoising models lose some parts of strokes. Figure5 shows the original image, the ground truth, and a denoising result image.Figure 5
The problem of image denoising. (a) The original image. (b) Ground truth. (c) The denoising result image.
(a)(b)(c)When the cultural relic protection workers repair the incomplete cultural relics, they first observe the overall structure; then, they conceive the contents of missing parts; finally, they gradually repair smaller parts. The whole process needs to ensure the global consistency and local continuity. Inspired by that, we introduce the coherent semantic attention mechanism into a cascaded image inpainting network. We first initialize the unknown regions with the most similar features in the known regions; then, we gradually optimize the filling contents for the unknown regions to ensure the internal continuity and global consistency of the missing blocks.U-net [23] can extract semantic level features efficiently. We divide U-net into two parts: U-net1 and U-net2. U-net1 tries to learn the rough representation of the missing regions, while U-net2 would achieve fine-tuning of rough representation by introducing coherent semantic attention mechanism [24]. The input of U-net1 is pairs of images: the denoised images Iin and the ground truth Igt, and the output is Ip. U-net2 takes Iin and Ip as the input, and the output is the final inpainting result Ir.(1)
Rough image inpainting by U-net1: as shown in Figure6, several convolution layers (4 × 4) are used for feature extraction, skipping are used for feature fusion, and upsampling is used for feature recovery. The input and output of U-net are symmetrical and have the same size. Due to the multilevel feature fusion by skip operations, U-net1 can achieve better results.(2)
Fine image inpainting by U-net2: as shown in Figure7, coherent semantic attention mechanism is introduced in the fourth layer of encoder stage. Encoder stage uses 3 × 3 ordinary convolutions and 4 × 4 detailed convolutions. U-net2 has better abilities to extract effective visual features on the premise of losing a small amount of information. The output of U-net1 is further restored by U-net2, and the final results should have both the overall consistency of images and the coherence of local regions.Figure 6
The structure of U-net1.Figure 7
The structure of U-net2.Coherence semantic attention mechanism: assume thatM is the missing regions of the image, and M¯ is the other known regions in the same image. For each missing pixel, the coherent semantic attention mechanism module searches for the most similar pixel blocks from M¯ and uses the blocks, which are generated by previous missing regions, as the secondary parts to update the next missing region blocks. Two formulas are used to measure the similarity:(6)Dmaxi=mi,m¯imi.m¯i,Dadi=mi,mi−1mi.mi−1,where Dmaxi represents the similarity between the missing region mi and the known region m¯i. Dadi is the similarity between the two adjacent image regions. Dmaxi is used to ensure the global consistency of the missing image regions, and Dadi is used to ensure the coherence of the generated image regions.Loss function: the loss function is designed based on the feature space of real sample image extracted by resnet-50 [25] and the image feature space generated by coherent semantic attention layers. Therefore, the final loss function is(7)LC=∑y∈MCSAIipy−δnIgty22+CSAdIipy−δnIgty22.


## 3.3.1. Image Denoising Network Based on Multiscale Feature Fusion

The types of noise in stone inscription images are complex and diverse, which requires the denoising model to be able to deal with multiple types of noise simultaneously. In this paper, we utilize the image denoising network based on multiscale feature fusion (IDN-MSFF) to remove multiple types of noise in stone inscriptions images, which consist of four stages: initial visual feature extraction, rough-grained feature fusion, fine-grained feature fusion, and reconstruction. The main structure is shown in Figure3.Figure 3
Image denoising network based on multiscale feature fusion.As shown in Figure4, through the analysis of the noise in stone inscription images, we find out that the patterns of the noise are very similar to the rain streaks. Therefore, we can deal with the noise in stone inscription images by referring to the methods of removing the rain streaks [22]. Although the deep learning-based denoising models for natural images have achieved good performance, it is difficult to capture the intrinsic correlations of noise in a single feature scale. The noise in stone inscription images span different scales.Figure 4
The noise in a stone inscription image.In our image denoising network, the Gaussian kernel function is first used to sample the original noisy image into several images with different scales, and these images are input into the network. Then, several convolution layers in this network are used to extract the features of these images.Rough-grained feature fusion: multiple parallel residual convolution units are used to learn the global texture features, so that the network can use the similarity among global features to represent the noise patterns and fuse these multiscale visual features.Fine-grained feature fusion: the channel attention mechanism is used to increase the discrimination between the noise and the clear images, which makes the fusion representation of features more effective for image denoising. In order to reduce the computational burden of the network, step convolution is used to reduce the spatial dimension of features.Reconstruction module: the low-level multiscale features obtained by rough-grained feature fusion and the high-level multiscale features obtained by fine-grained feature fusion are further fused. Finally, the deconvolution is used to restore a denoised image.Loss function: the least square error is a common used loss function, but it is easy to produce smooth visual effect for the denoised images. Therefore, we use Charbonnier loss function in our network. This loss function has higher tolerance for small errors and better robustness.


## 3.3.2. Cascaded Image Inpainting Network with Coherent Semantic Attention Mechanism

Many ancient stone inscriptions are damaged seriously, which results in that the results obtained by denoising models lose some parts of strokes. Figure5 shows the original image, the ground truth, and a denoising result image.Figure 5
The problem of image denoising. (a) The original image. (b) Ground truth. (c) The denoising result image.
(a)(b)(c)When the cultural relic protection workers repair the incomplete cultural relics, they first observe the overall structure; then, they conceive the contents of missing parts; finally, they gradually repair smaller parts. The whole process needs to ensure the global consistency and local continuity. Inspired by that, we introduce the coherent semantic attention mechanism into a cascaded image inpainting network. We first initialize the unknown regions with the most similar features in the known regions; then, we gradually optimize the filling contents for the unknown regions to ensure the internal continuity and global consistency of the missing blocks.U-net [23] can extract semantic level features efficiently. We divide U-net into two parts: U-net1 and U-net2. U-net1 tries to learn the rough representation of the missing regions, while U-net2 would achieve fine-tuning of rough representation by introducing coherent semantic attention mechanism [24]. The input of U-net1 is pairs of images: the denoised images Iin and the ground truth Igt, and the output is Ip. U-net2 takes Iin and Ip as the input, and the output is the final inpainting result Ir.(1)
Rough image inpainting by U-net1: as shown in Figure6, several convolution layers (4 × 4) are used for feature extraction, skipping are used for feature fusion, and upsampling is used for feature recovery. The input and output of U-net are symmetrical and have the same size. Due to the multilevel feature fusion by skip operations, U-net1 can achieve better results.(2)
Fine image inpainting by U-net2: as shown in Figure7, coherent semantic attention mechanism is introduced in the fourth layer of encoder stage. Encoder stage uses 3 × 3 ordinary convolutions and 4 × 4 detailed convolutions. U-net2 has better abilities to extract effective visual features on the premise of losing a small amount of information. The output of U-net1 is further restored by U-net2, and the final results should have both the overall consistency of images and the coherence of local regions.Figure 6
The structure of U-net1.Figure 7
The structure of U-net2.Coherence semantic attention mechanism: assume thatM is the missing regions of the image, and M¯ is the other known regions in the same image. For each missing pixel, the coherent semantic attention mechanism module searches for the most similar pixel blocks from M¯ and uses the blocks, which are generated by previous missing regions, as the secondary parts to update the next missing region blocks. Two formulas are used to measure the similarity:(6)Dmaxi=mi,m¯imi.m¯i,Dadi=mi,mi−1mi.mi−1,where Dmaxi represents the similarity between the missing region mi and the known region m¯i. Dadi is the similarity between the two adjacent image regions. Dmaxi is used to ensure the global consistency of the missing image regions, and Dadi is used to ensure the coherence of the generated image regions.Loss function: the loss function is designed based on the feature space of real sample image extracted by resnet-50 [25] and the image feature space generated by coherent semantic attention layers. Therefore, the final loss function is(7)LC=∑y∈MCSAIipy−δnIgty22+CSAdIipy−δnIgty22.


## 4. Results

### 4.1. Experimental Setups

The models in this paper are run in Ubuntu 16.04LT-S and carried out on CPU Intel Xeon E5-2650 v4, GPU Ge-Force GTX 1080, RAM 64G.


### 4.2. Stone Inscription Image Expansion

The parameters of this experiment are set as follows: the input images have the size of 64 × 64, and batch_ size is 32. The learning rate is 2e−4, the regularization coefficient is 0.5, and the number of convolution kernels of generator and discriminator is 64. The training image dataset has 827 pairs of original images and their corresponding clear images; after 1000 epochs, a total of useful 3000 noisy images are generated (as shown in Figure 8). From these images, we can see that the same Chinese characters in the images have different patterns, and different images also have different types of noise. These generated images can well expand the original stone inscription image dataset and provide reliable image data support for image denoising and inpainting in the later stages.Figure 8
Partial generated images.


### 4.3. Stone Inscription Image Denoising

In this experiment, the parameters are set as follows: the number of pyramid layers is set to 3 for image sampling, and 1/2 sampling and 1/4 sampling of the original image are taken respectively. In the rough-gained feature fusion module, the numbers of filters in three blocks are set to 32, 64, and 128 respectively. The depths of the fine-gained fusion module and reconstruction module are set to 10 and 3 respectively, and the input images are pairs of images with the size of 96 × 96 × 3. The optimization algorithm is Adam algorithm; batch_ size is 8. The learning rate is 2e−4, and epoch = 30. In this part, the expanded image dataset is used: some of them are used as test images, and other images are used as training images. Then, we analyze the performance of the methods qualitatively and quantitatively.(1)
Qualitative analysis: Figure9 is an intuitive display of the image denoising results, and we can see that our method can accurately remove the noise from the stone inscription images and achieve good denoising results, while there also are some residual noise, and some parts of Chinese character strokes are removed.(2)
Quantitative analysis: we first use PSNR and structural similarity (SSIM) to evaluate the performance of different methods.(8)PSNR=10×log2552MSE.Figure 9
The image denoising results. (a) The noisy images. (b) The denoised results. (c) The ground-truth.
(a)(b)(c)SSIM is to calculate the similarity of the contrast, brightness, and structure between images.(9)lX,Y=2μXμY+C1μX2+μY2+C1,cX,Y=2σXσY+C2σX2+σY2+C2,sX,Y=σXY+C3σX+σY+C3,SSIMX,Y=lX,Y×cX,Y×sX,Y,where μX andμY represent the mean value of the images X and Y, σX and σY are the variance between X and Y, and σXY is the covariance.In addition, a comprehensive evaluation index (CEI) [26] is also used in this paper.(10)CEI=4σxyx¯y¯σx2+σy2x¯2+y¯2.Then, several image denoising methods and stone inscription image denoising methods are used as comparison methods. For example, Zheng et al. [27] completed the font extraction from calligraphy images by k nearest neighbor algorithm and guided filter and achieved good results. L0 smooth [28] and BM3D [29] can also be used in stone inscription image denoising. S-RMD [30] is a deep learning-based image denoising model, which has achieved pretty good denoising effects on a number of commonly used natural image datasets. The average values of PSNR, SSIM, and CEI (APSNR, ASSIM and ACEI) are used for evaluating different image denoising methods, and the final results are shown in Table 1.Table 1
Comparison of different denoising methods (APSNR, ASSIM, and ACEI).
MethodsAPSNRASSIMACEIZheng [27]28.3640.95520.6839S-RMD [30]28.1150.97370.6514L0 smooth [28]28.2190.97210.6684BM3D [29]28.150.97260.6509IDN-MSFF43.2570.99320.7023IDN-MSFF [+]45.3230.99410.7382IDN-MSFF [+] represents the results obtained by IDN-MSFF after image expansion. From the results, we can see that IDN-MSFF can get better image denoising results in three evaluation indexes, and IDN-MSFF [+] can further improve the denoising performance.


### 4.4. Stone Inscription Image Inpainting

In image inpainting model, learning rate = 2e−4, β = 0.5, batch size = 1, and epoch = 120. In the experiment, 458 images are used as the training images, and 196 images are used as test images. In model training, a mask with a fixed size is used to partially block a part in the image. Adam algorithm is used to optimize the image inpainting model.Figure10 shows several test results. Each output image contains three parts, Figure 10(a) represents the input images with masks, Figure 10(b) shows the inpainting results, and Figure 10(c) shows the ground truth. It can be seen that the masked parts of the input images have been well restored, and the results of the stone inscription image inpainting is very similar to the ground truth in visual effect.Figure 10
Stone inscription image inpainting results. (a) The input images with masks. (b) The inpainting results. (c) The ground-truth.
(a)(b)(c)


## 4.1. Experimental Setups

The models in this paper are run in Ubuntu 16.04LT-S and carried out on CPU Intel Xeon E5-2650 v4, GPU Ge-Force GTX 1080, RAM 64G.


## 4.2. Stone Inscription Image Expansion

The parameters of this experiment are set as follows: the input images have the size of 64 × 64, and batch_ size is 32. The learning rate is 2e−4, the regularization coefficient is 0.5, and the number of convolution kernels of generator and discriminator is 64. The training image dataset has 827 pairs of original images and their corresponding clear images; after 1000 epochs, a total of useful 3000 noisy images are generated (as shown in Figure 8). From these images, we can see that the same Chinese characters in the images have different patterns, and different images also have different types of noise. These generated images can well expand the original stone inscription image dataset and provide reliable image data support for image denoising and inpainting in the later stages.Figure 8
Partial generated images.


## 4.3. Stone Inscription Image Denoising

In this experiment, the parameters are set as follows: the number of pyramid layers is set to 3 for image sampling, and 1/2 sampling and 1/4 sampling of the original image are taken respectively. In the rough-gained feature fusion module, the numbers of filters in three blocks are set to 32, 64, and 128 respectively. The depths of the fine-gained fusion module and reconstruction module are set to 10 and 3 respectively, and the input images are pairs of images with the size of 96 × 96 × 3. The optimization algorithm is Adam algorithm; batch_ size is 8. The learning rate is 2e−4, and epoch = 30. In this part, the expanded image dataset is used: some of them are used as test images, and other images are used as training images. Then, we analyze the performance of the methods qualitatively and quantitatively.(1)
Qualitative analysis: Figure9 is an intuitive display of the image denoising results, and we can see that our method can accurately remove the noise from the stone inscription images and achieve good denoising results, while there also are some residual noise, and some parts of Chinese character strokes are removed.(2)
Quantitative analysis: we first use PSNR and structural similarity (SSIM) to evaluate the performance of different methods.(8)PSNR=10×log2552MSE.Figure 9
The image denoising results. (a) The noisy images. (b) The denoised results. (c) The ground-truth.
(a)(b)(c)SSIM is to calculate the similarity of the contrast, brightness, and structure between images.(9)lX,Y=2μXμY+C1μX2+μY2+C1,cX,Y=2σXσY+C2σX2+σY2+C2,sX,Y=σXY+C3σX+σY+C3,SSIMX,Y=lX,Y×cX,Y×sX,Y,where μX andμY represent the mean value of the images X and Y, σX and σY are the variance between X and Y, and σXY is the covariance.In addition, a comprehensive evaluation index (CEI) [26] is also used in this paper.(10)CEI=4σxyx¯y¯σx2+σy2x¯2+y¯2.Then, several image denoising methods and stone inscription image denoising methods are used as comparison methods. For example, Zheng et al. [27] completed the font extraction from calligraphy images by k nearest neighbor algorithm and guided filter and achieved good results. L0 smooth [28] and BM3D [29] can also be used in stone inscription image denoising. S-RMD [30] is a deep learning-based image denoising model, which has achieved pretty good denoising effects on a number of commonly used natural image datasets. The average values of PSNR, SSIM, and CEI (APSNR, ASSIM and ACEI) are used for evaluating different image denoising methods, and the final results are shown in Table 1.Table 1
Comparison of different denoising methods (APSNR, ASSIM, and ACEI).
MethodsAPSNRASSIMACEIZheng [27]28.3640.95520.6839S-RMD [30]28.1150.97370.6514L0 smooth [28]28.2190.97210.6684BM3D [29]28.150.97260.6509IDN-MSFF43.2570.99320.7023IDN-MSFF [+]45.3230.99410.7382IDN-MSFF [+] represents the results obtained by IDN-MSFF after image expansion. From the results, we can see that IDN-MSFF can get better image denoising results in three evaluation indexes, and IDN-MSFF [+] can further improve the denoising performance.


## 4.4. Stone Inscription Image Inpainting

In image inpainting model, learning rate = 2e−4, β = 0.5, batch size = 1, and epoch = 120. In the experiment, 458 images are used as the training images, and 196 images are used as test images. In model training, a mask with a fixed size is used to partially block a part in the image. Adam algorithm is used to optimize the image inpainting model.Figure10 shows several test results. Each output image contains three parts, Figure 10(a) represents the input images with masks, Figure 10(b) shows the inpainting results, and Figure 10(c) shows the ground truth. It can be seen that the masked parts of the input images have been well restored, and the results of the stone inscription image inpainting is very similar to the ground truth in visual effect.Figure 10
Stone inscription image inpainting results. (a) The input images with masks. (b) The inpainting results. (c) The ground-truth.
(a)(b)(c)


## 5. Conclusion

This paper aims to solve the problems of multitype noise removal and image inpainting of stone inscriptions images. To this end, we have established a dataset of stone inscriptions images. Then, based on the existing related models, we introduce multiscale feature fusion and coherent semantic attention mechanism to design the denoising model and image inpainting model for the stone inscription images. The final experimental results have shown the advantages of the methods used in this paper, while we will still have many works to improve in the future. For example, we need more stone inscriptions images for model training, and larger missing image regions need to be recovered in some stone inscriptions images.



---

*Source: 7675611-2021-12-20.xml*