# Recovering Microscopic Images in Material Science Documents by Image Inpainting
## 2 Related Works and Backgrounds
### Image Inpainting Methods  
Image inpainting methods in the field of computer vision are mainly classified into two categories: non-learning approaches and deep learning approaches [15]. Moreover, non-learning approaches are further divided into patch-based types and diffusion-based types, and deep learning approaches are further divided into convolutional neural network (CNN)-based types and generative adversarial network (GAN)-based types. The list of these approaches from the past to the current day, in terms of image inpainting methods, is presented in Table 1.  
Among the two image inpainting approaches, the non-learning-based approach is known as the more traditional method; this method is often used to copy the most similar pixel information that is adjacent to the target areas and that correspond to either damaged or missing regions in the image [16; 17; 18; 19; 20; 21; 22]. On the other hand, the patch-based method in the non-learning approach is designed to locate the patches that are most similar to the target area by using random sampling and propagation instead of thoroughly searching the entire image. For instance, [PERSON] et al. proposed the PatchMatch algorithm, which operates by densely matching the patches between two images [19]. In addition, it is suitable for use with various images, as it is able to handle the desired texture and color changes. Diffusion-based methods in the non-learning approach operate by reconstructing target regions, which is achieved by analyzing the entire image and then gradually diffusing the \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Category** & **Type** & **Method** & **Dataset** \\ \hline \multirow{8}{*}{Non-learning approach} & \multirow{4}{*}{Patch-based} & [PERSON] et al. [16] & - \\ \cline{3-4}  & & & [PERSON] et al. [17] & - \\ \cline{3-4}  & & [PERSON] et al. [18] & - \\ \cline{3-4}  & & PatchMatch algorithm [19] & - \\ \cline{3-4}  & & [PERSON] et al. [20] & - \\ \cline{3-4}  & & [PERSON] et al. [21] & - \\ \cline{3-4}  & & [PERSON] et al. [22] & - \\ \hline \multirow{8}{*}{Deep-learning approach} & \multirow{4}{*}{GAN-based} & \multirow{2}{*}{Shift-Net [23]} & Places [27], \\  & & & Paris street View [28] \\ \cline{3-4}  & & & \\ \cline{3-4}  & & Global and Local [24] & Places2 [27], \\ \cline{3-4}  & & & ImageNet [29], \\ \cline{3-4}  & & & CMP FAcade [30] \\ \cline{3-4}  & & & Places2 [27], \\ \cline{3-4}  & & & ImageNet [29], \\ \cline{3-4}  & & & CelebA [31], \\ \cline{3-4}  & & & CelebA-HQ [32], \\ \cline{3-4}  & & & DTD [33] \\ \cline{3-4}  & & & Gated convolution [26] & Places2 [27] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: List of various image inpainting methods in the field of computer vision. Figure 1: Examples of microscopic images in material science documents: (**a**) represents the STEM image of Au nanoparticles. reprinted with permission from Ref. [12]. Copyright (2020) Elsevier; (**b**) represents the TEM image of Co\({}_{8}\)Fe\({}_{2}\)-MOF. reprinted with permission from Ref. [13]. Copyright (2021) Elsevier; and (**c**) represents the SEM image of S/PCMSs composites. reprinted with permission from Ref. [14]. Copyright (2019) Elsevier. information (e.g., color, texture, and shape of the image) of pixels that are closest to the target region. For instance, [PERSON] et al. proposed a diffusion-based method that is based on partial differential equations (PDE), which allows the propagation of gradient information from the known regions to the target regions for the purposes of filling in the missing pixels [20]. In addition, it is also suitable for large-scale image inpainting tasks, as it is easy to implement and can be easily parallelized. However, this approach has two limitations: (1) it is useless for dealing with complex pixel information around large target areas, and (2) it requires a high computational cost. In contrast to the non-learning approach, the deep learning approach can learn from and extract the complex features of large target areas in images with a more accurate and faster prediction.