Coarse-to-Fine Generative Model for Oracle Bone Inscriptions Inpainting (2024)

Abstract: Due to ancient origin, there are many incomplete characters in the unearthed Oracle Bone Inscriptions(OBI), which brings the great challenges to recognition and research. In recent years, image inpainting techniques have made remarkable progress. However, these models are unable to adapt to the unique font shape and complex text background of OBI. To meet these aforementioned challenges, we propose a two-stage method for restoring damaged OBI using Generative Adversarial Networks (GAN), which incorporates a dual discriminator structure to capture both global and local image information. In order to accurately restore the image structure and details, the spatial attention mechanism and a novel loss function are proposed. By feeding clear copies of existing OBI and various types of masks into the network, it learns to generate content for the missing regions. Experimental results demonstrate the effectiveness of our proposed method in completing OBI compared to several stateof-the-art techniques.

heading: 1 Introduction
text: Since the earliest discovery of Oracle Bone Inscriptions(OBI), over 5,000 distinct character forms have been identified, which have significantly advanced our comprehension of many characters’ meanings. These deciphered OBIs provide invaluable historical information crucial for understanding various aspects of ancient Chinese politics, society, religion, and more.
Recognizing and interpreting are important topics in the field of OBI research. Due to the lack of physical objects, the images of rubbings in the recorded books are the main carriers of research. However, some OBIs have suffered varying degrees of residual erosion and damage on their surface, resulting in a large number of incomplete fonts in
*Corresponding author: wangshibin@htu.edu.cn
the inscriptions and rubbings seen today. With the rapid development of image generation technology, many image restoration problems difficult to solve in traditional methods have found new research avenues. The comprehensive application of artificial intelligence and other technologies has become a new research direction in the restoration of OBIs.
Zeng et al. (2019) proposed the Pyramid Context Encoder Network (PEN). It is based on the U-Net structure and encodes and decodes contextual semantics to ensure visual and semantic consistency. Li et al. (2020) developed the Recurrent Feature Reasoning (RFR) network, featuring a plug-andplay RFR module and a Knowledge Consistent Attention (KCA) module. They infer the hole boundaries and capture the distant feature information. Wu et al. (2021) introduced a two-stage (coarse-tofine) model. It combines a Local Binary Pattern (LBP) Waller et al. (2013) network and incorporates a new spatial attention mechanism. These methods have enhanced image processing. However, they only grasp limited connections between textures and edges. They fail to fully comprehend image semantics and complex structures. Additionally, they overlook the interplay between global and local features. Given the complexities behind incomplete fonts and unique font features, existing image restoration models struggle to effectively complete OBI image inpainting tasks.
To meet these challenges, we propose a twostage (coarse-to-fine) font inpainting network. Our network incorporates a dual discriminator structure to capture both global and local image information. Specifically, we employ a global discriminator to focus on the spatial correlation between damaged and undamaged regions. The local discriminator concentrates on the local patch information. To effectively understand the intrinsic features of the image, we introduce a novel loss function to accurately restore the structure and details. Through extensive comparisons, our framework demonstrates
107
state-of-the-art performance in OBI image inpainting tasks.

heading: 2 Method

heading: 2.1 Network Architecture
text: The network is a two-stage deep generative model. Both stages consist of encoder-decoder pipeline and follow an adversarial model Goodfellow et al. (2014). The network architecture is shown as Figure. 1. The damaged image consists of the missing regions filled with white pixels, represented as Iin. Lin denotes the LBP Waller et al. (2013) structural information extracted from the damaged oracle Iin in the grayscale channel. M represents a binary mask, where 1 indicates the missing regions and 0 indicates the known regions.
In the first stage, the generator G1 includes seven feature extraction blocks and feature restoration modules. Each feature extraction block consists of LeakyReLU Xu et al. (2015), a convolutional layer, and InstanceNorm2d Ulyanov et al. (2016). The decoder generates the content of the missing region through seven feature restoration modules, which consist of ReLU Nair and Hinton (2010), transposed convolution, and InstanceNorm2d Ulyanov et al. (2016). Finally, G1 and D1 generate the completed LBP structural information Lout and Lo.
In the second stage, an additional spatial attention layer is added to the fifth layer of the encoder. This layer builds the correlations not only within the known region but also among the missing regions.
Due to a single discriminator judging the image authenticity solely from a global perspective and being unable to handle the details, artifacts and structural inconsistencies may arise in the restoration results. The dual discriminator, on the other hand, judges the image from both global and local perspectives. They compete with each other to learn more effective weights.

heading: 2.2 Dual Discriminator
text: The structure of Dual PatchGAN Isola et al. (2017) Discriminator (DP) is as shown in Figure. 2. The left branch is a global discriminator that focuses on the spatial correlation between damaged and undamaged regions. Its input consists of an image and a mask, and output is a 3D feature. Each CSL block consists of a 5⇥5 convolution, SpectralNorm Miyato et al. (2018) and LeakyReLU with ↵= 0.2. In the first two CSL blocks, the number of convo-
lutional output channels is 64 and 128, while in the others it is 256. The right branch is a local discriminator with five 4⇥4 convolutions, which focuses on the local patch. The first four layers use the LeakyReLU with ↵= 0.2, the Sigmoid for the last layer and the BatchNorm2d for normalization in the middle three layers. The local discriminator can be formulated as:
⌧adv2 = min G2 max D2
EIg ⇥ log D2(Ig) ⇤
+EIin [log (1 D2 (G2 (Iin , M)))] (1)
Our objective function for the global discriminator can be formulated as:
⌧adv3 = EIin⇠PIin (Iin) [D3 (G2 (Iin ))] (2)
⌧D3 = EIg⇠Pdata(Ig) [ReLU (1 D3 (Ig))] +EIin⇠PIin (Iin) [ReLU (1 + D3 (G2 (Iin )))]
(3) where G2 represents the second stage generator, D2 and D3 represent the right and left branches of the dual discriminator, respectively.

heading: 2.3 Multi-level Fusion Loss Function
text: We reduce the difference between the original image and the inpainting image by using a multi-level fusion loss function(MLFLF) to enhance the stability of training.
The reconstruction loss is defined as:
Lr = kLo Lgk2 (4)
Lo = Lin (1 M) + Lout M (5) The adversarial loss Yan et al. (2018) is defined
as:
⌧adv1 = min G1 max D1
ELg ⇥ log D1(Lg) ⇤
+ELin [log (1 D1 (G1 (Lin , M)))] (6)
The pixel-level reconstruction loss is responsible for directly comparing each pixel of the generated image with the target image:
Lvalid
= 1
Sum(1 M) k(Lout Lg) (1 M)k1 (7)
Lhole = 1
Sum(M) k(Lout Lg) Mk1 (8)
The Total Variation (TV) Liu et al. (2018) loss reduces noise and discontinuities, resulting in a smoother and more continuous appearance:
Ltv = kLo(i, j + 1) Lo(i, j)k1 + kLo(i + 1, j) Lo(i, j)k1
(9)
The multi-scale loss compares the differences between ground truth images and mapping results
of different scales:
⌧m = X
h2d k h (Io) h (Ig)k2 (10)
Io = Iin (1 M) + Iout M (11) We apply the perceptual loss Johnson et al. (2016) and style loss Gatys et al. (2016) defined on the VGG-16 Simonyan and Zisserman (2014) (pretrained on ImageNet Deng et al. (2009)) to enhance the recovery of structural and textual information.
Iper = X
i
k i (Io) i (Ig)k1 (12)
Istyle = X
i
k i (Io) i (Ig)k1 (13)
where i is the feature map of i th layer in ImageNet-pretrained VGG-16 network, i(·) = i(·) i(·)T is from (Buades et al., 2005).

heading: 3 Experiments

heading: 3.1 Dateset
text: We select 2000 OBI images for training and 100 for testing from the oracle bone images produced by the Key Laboratory of Oracle Information Processing of the Ministry of Education in Henan Province. To better validate the results of the experiment, we use the masks to simulate the broken regions of
OBIs. These masks are divided into irregular and regular types. The irregular masks are obtained from the NVIDIA dataset Liu et al. (2018), while the regular masks are square masks of fixed size (25% of the total image pixels) placed in the center of the image.

heading: 3.2 Qualitative Comparisons
text: In this section, we conduct the experimental comparisons with other image restoration models.
For the restoration of OBI with irregular masks, the visualization results are shown in Figure. 3. The input image (b) shows the damaged OBI images. (c) demonstrates the results of using the PEN network Zeng et al. (2019) with a mode collapse. (d) using the RFR network Li et al. (2020) fails to accomplish the complementation task effectively. Note in particular the comparison between (e) the LBPLSA network Quan et al. (2022) and (f) our network. Our network evidently produces more realistic completion results from the smoother strokes in the first row of Figure. 3. And fewer or no artifacts appear at the end of the strokes in the rest of the lines. In contrast, the LBPLSA network exhibits severe artifacting and discontinuities in
strokes. It fails to adequately complete the objectives. The presence of artifacts indicates that the network did not accurately understand the missing content in the image. As a result, it fills in unrealistic textures and structures.
We also explore the classic center mask completion scenario in image inpainting. Given that most of the OBI content lies in the center, it is challenging for the network to infer the main content of the characters from just one stroke at the boundary. The generated results are depicted in Figure. 4. We can see that the generated results of the PEN network (c) collapse again and the RFR network (d) fails to meet the target requirement. Focus on the comparison between LBPLSA (e) and our method (f) again, LBPLSA generates the images with more artifacts and doesn’t effectively learn the semantic information of the OBIs. For instance, in the second row of Figure. 4, the strokes generated by the LBPLSA are opposite to the ground truth. More artifacts are present in rows 5 and 6. Under the same experimental configuration, our network achieves results that are closer to the ground truth.

heading: 3.3 Quantitative Comparisons
text: In terms of evaluation metrics, we follow the structural similarity index (SSIM) Wang et al. (2004) and peak signal-to-noise ratio (PSNR) as outlined in references Ren et al. (2019). The evaluation results are presented in Table 1.
Compared with other methods, the scores of each indicator in our model have been improved. The DP structure can effectively capture both the global and local image information. Additionally, the loss function component introduced MLFLF optimizes semantic plausibility and structural consistency. The integration of DP structure and MLFLF component produces the images with reduced pixellevel differences and leads to significant improvements across SSIM, PSNR, and L1 distance metrics, which indicates the high accuracy and effec-
tiveness in image inpainting tasks.

heading: 3.4 Ablation Studies
text: The ablation studies are conducted under mask rates ranging from 20% to 30%. We evaluate the effectiveness of our proposed method by contrasting three different experimental settings, including the LBPLSA method, the SN method only with DP component and the complete method. The generated results are depicted in Figure. 5. Part (a) represents the ground truth OBIs. The input images with various degrees of damage are generated by masks, shown in (b). The completion results of LBPLSA (c), SN (d), and ours (e) are sequentially displayed.
Compared with the LBPLSA method, the SN method shows some improvement with the introduction of the DP structure. The incorporation
of the DP structure enables the model to better capture both global and local image information, which improves the restoration results to a certain extent. However, the SN method lacks the further optimization from the MLFLF component. It still has certain limitations and fails to fully exploit the intrinsic features of the images.
Furthermore, our complete method achieves further improvements across all metrics within the experimental scope. By leveraging the dual advantages of DP and MLFLF, our method can more accurately restore the structure and details of the images. This makes the restoration results closer to the original images. Compared to the methods only with DP, the addition of the MLFLF compo-
nent further enhances the clarity and quality of the restored images. This leads to better performance across metrics, such as SSIM, PSNR, and L1 distance, as demonstrated in the ablation study metrics presented in Table 2.
Through the ablation studies, we validate the crucial roles of DP and MLFLF in image restoration tasks. The DP structure enhances the model’s understanding of images, while the MLFLF module further optimizes detail and texture restoration. This showcases significant advantages across all metrics. These experimental results validate our method’s effectiveness. They emphasize the importance of leveraging the dual advantages of DP and MLFLF in image inpainting tasks.

heading: 4 Conclusion
text: We propose the two-stage (coarse-to-fine) network for efficient OBI image inpainting. This new framework consists of an enhanced LBP network and integrated DP and MLFLF components. Specifically, we design a novel dual discriminator network. The first stage LBP learning network adopts a U-Net architecture, aimed at accurately predicting structural information in missing regions. This guides the second image inpainting network in better filling missing pixels. In the second stage image generation network, we employ dual discriminators to complete the masked regions. Compared to several state-of-the-art methods, experimental results demonstrate the effectiveness of DP and MLFLF components in the proposed method in completing OBI image inpainting tasks.
In the future, we plan to further develop our network to achieve more powerful functions, such as increasing the speed, realizing editing functions, and improving the efficiency of paleographers. Our goal is to solve the problem of more complex noise or higher mask coverage. We believe that our two-stage (coarse-to-fine) generation model can be extended to very high-resolution coloring applications by improving the first-stage generation results.
Acknowledgments This study was funded by the scientific and technological project in Henan Province in 2022 (Grant No. 222102210187), the Key Research Project for Higher Education Institutions in Henan Province (Grant No. 24A520018).

heading: Future Work
text: In the future, we plan to further develop our network to achieve more powerful functions, such as increasing the speed, realizing editing functions, and improving the efficiency of paleographers. Our goal is to solve the problem of more complex noise or higher mask coverage. We believe that our two-stage (coarse-to-fine) generation model can be extended to very high-resolution coloring applications by improving the first-stage generation results. Acknowledgments This study was funded by the scientific and technological project in Henan Province in 2022 (Grant No. 222102210187), the Key Research Project for Higher Education Institutions in Henan Province (Grant No. 24A520018)