# A Review of Image Inpainting Methods Based on Deep Learning
## 2 Datasets and Evaluation Metrics
### Evaluation Metrics  
Evaluation metrics reflect the performance of an algorithm. Newly proposed algorithms can only be recognized if they achieve good results on widely accepted evaluation metrics. Existing commonly used evaluation metrics are divided into two categories: subjective evaluation and objective evaluation [49]. In the field of image generation, subjective evaluations, with humans as the observers, are more indicative of the effectiveness of the generated images. Image inpainting is also similar in this respect. However, due to limitations like human resources and personal biases, the evaluations may be unfair. Commonly used objective evaluation metrics include mean squared error (MSE), the peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM), and the Frechet inception distance (FID). Mean squared error (MSE) calculates the similarity of images by taking the expected value of the squared differences between the pixel points of two images. A smaller value indicates greater similarity between images. During model training, the L2 reconstruction loss is commonly used. The formula for calculating MSE is as follows:  
\[\text{MSE}=\frac{1}{m}\sum_{i=1}^{m}\left(X_{i}-Y_{i}\right)^{2} \tag{1}\]  
where represents the pixel variable of the image and represents the pixel points of the image. The peak signal-to-noise ratio (PSNR) is based on the error between corresponding pixels and is used to evaluate the quality of an image in comparison to the true image. The value typically ranges between 20 and 40. A higher value indicates lower distortion and better image quality. The formula for calculating PSNR is as follows:  
\[\text{SNR}=20\cdot\log_{10}\left(\frac{\text{MAX}_{I}}{\sqrt{\text{MSE}}}\right) \tag{2}\]  
where MAX stands for the maximum pixel value of the image and MSE is the mean squared error. The structural similarity index (SSIM) measures the structural similarity of two images, simulating human visual perception. A larger value suggests less image distortion. When two images are identical, the SSIM value is 1. The theory behind SSIM suggests that natural images possess a highly structured feature, meaning pixels have a strong correlation that carries important information about the structure of visual scenes. In its application, images are often divided into blocks, and the SSIM is calculated for each block before taking the average. Given two images (x and y), the formula to compute SSIM is:  
\[\text{SSIM}(X,Y)=\frac{(2\varphi_{X}\varphi_{Y}+C_{1})(2\delta_{XY}+C_{2})}{( \varphi_{X}^{2}+\varphi_{Y}^{2}+C_{1})(\delta_{X}^{2}+\delta_{Y}^{2}+C_{2})} \tag{3}\]  
where and represent the pixel average of the images, denotes the standard deviation of the pixels of the images, denotes the covariance between the images, and are constants. The Frechet inception distance (FID) is a significant metric for GAN networks, assessing the quality and diversity of generated images. It has also been widely used in image inpainting techniques. A smaller FID value suggests that the two data distributions are closer, resulting in better inpainting effects. The formula to compute FID is:  
\[\text{FID}(X,Y)=\left\|\mu_{X}-\mu_{Y}\right\|_{2}^{2}+T_{r}(\Sigma_{X}+\Sigma_{ Y}-2(\Sigma_{X}\Sigma_{Y})^{\frac{1}{2}}) \tag{4}\]  
where and denote the means of the images, represents the trace of a matrix, and indicates the covariance. Although the current objective evaluation metrics provide valuable reference points, each metric still has its issues. Researchers often use pixel-level similarity metrics like SSIM, and PSNR or directly adopt metrics from the image generation domain like FID, KID, etc. However, FID tends to favor blurry generated images, while KID overlooks the pairing relationships of images. PSNR does not accurately reflect human visual perception. Typically, after image compression, the output image differs somewhat from the original. Due to the variability in human visual features, the evaluation often does not align with human perception, although PSNR remains a valuable metric.