# Recovering Microscopic Images in Material Science Documents by Image Inpainting
## 3 Results
### Comparisons of Masks and Models  
In the first experiment, we confirmed the effects of two masks with respect to the statistical mask and the threshold-based mask. We tested whether the input that is used by the statistical masks via the use of shift-net was the most popular deep learning method. The additives of the microscopic images in the material science documents appeared mainly in the four corners (i.e., top-left, bottom-left, top-right, and bottom-right), but they often appear in other areas (i.e., the center). Since the statistical mask has fixed four corner areas, it is difficult to become flexible masking. In Figure 5b, in the case of the statistical mask, we observed good results in all the additive regions that were top-left and bottom-left, but the center region was not restored at all. However, in the case of the threshold-based mask, Figure 5c shows that we observed better results with respect to restoring all the additives inside the image. In particular, the results in the area around the C label in Figure 5b,c look different. The statistical mask used a fixed square mask that includes the area around the C label as well as the C label, but the threshold-based mask contains only pixels belonging to the C label. It means that the statistical mask is relatively larger than the threshold-based masks in the same situation. Then, the statistical mask should restore the area around the C label, and the restoring noise on the area can be inserted. Therefore, the result in the area around the additives by using the threshold-based masks can be better than the result in the area around the C label by using the statistical masks.  
In the second experiment, we confirmed the effects of the four latest deep learning methods (shift-net, global and local, contextual attention, and gated convolution) for the purposes of image inpainting in our datasets of microscopy images. Through the first experiment, we tested the input used by the threshold-based masks via the use of the four latest deep learning methods. Then, we conducted qualitative evaluations of their results. Figure 6 shows three examples of the inputs and outputs that were obtained with the four different methods for qualitative comparison. It was found that the results of shift-net, global and local, and contextual attention were unsatisfactory because of blurriness in relation to the position of the characters and scale bar, as shown in Figure 6b-d. However, the gated convolution achieved better results than the other methods for all the examples, as shown in Figure 6e.  
Figure 5: Comparison of the results, which was achieved by using a statistical mask and a threshold mask from the example of microscopic image (reprinted with permission from Ref. [41]. Copyright (2019) Elsevier): (**a**) represents input image; (**b**) represents output image using a statistical mask; and (**c**) represents the output image when using a threshold-based mask.  
Next, we conducted quantitative evaluations of their results. Furthermore, we used three evaluation metrics, the structural similarity index measure (SSIM) and \(\ell_{1}/\ell_{2}\) errors [26,42]. Then, we tested the images in order to measure the three metrics for the four methods and then calculated the averages and standard deviations of each method. Then, the additive's area in the entire figure is usually small. Since all the pixels except restoring area in the outputs are the same as the ones in the originals, the metrics of the whole images with respect to none or using methods are nearly high score and their difference could be significantly trivial. To show the difference in the results between with and without the methods, we calculated the metrics of local regions where the characters and scale bars are closely packed. In Table 2, we observed that a gate convolution possessed the best performance measures of SSIM and \(\ell_{1}/\ell_{2}\) errors.