# Image Error Concealment Based on Deep Neural Network
## 2 Problem Formulation  
Similar to the other SEC methods, we reconstructed the missing pixels using the correctly received ones. More specifically, we used the local information to predict the missing pixels and non-local information to correct the predictions in this paper.  
Let \(\mathbf{O}\) be the original image, \(\mathbf{X}\) be the corresponding corrupted image, and \(\mathbf{X}\) can be divided into an available part, \(\mathbf{S}\), and an unavailable part, \(\mathbf{U}\); that is, \(\mathbf{X}=\mathbf{U}\cup\mathbf{S}\). Therefore, the EC problem can be considered to reconstruct the unavailable part \(\mathbf{U}\) through utilizing the information from the available part \(\mathbf{S}\). Without a loss of generality, we suppose that only one pixel is reconstructed at a time. Define \(\mathbf{y}_{i}\) as a pixel group that contains pixel \(i\), which is located on the contour of \(\mathbf{U}\), as shown in Figure 1. Note that \(\mathbf{y}_{i}\) can be regarded as a combination of \(\mathbf{y}_{i}^{\mathrm{s}}\) and \(\mathbf{y}_{i}^{\mathrm{u}}\). \(\mathbf{y}_{i}^{\mathrm{u}}\) is the pixel set that contains the missing pixels, and here, it is actually the missing pixel \(i\), since we reconstruct only one pixel at a time. \(\mathbf{y}_{i}^{\mathrm{s}}\) is a pixel set formed by the adjacent and available neighbors of \(\mathbf{y}_{i}^{\mathrm{u}}\). We call \(\mathbf{y}_{i}^{\mathrm{s}}\) a support area of \(\mathbf{y}_{i}^{\mathrm{u}}\), since \(\mathbf{y}_{i}^{\mathrm{s}}\) can be regarded as the spatial context of \(\mathbf{y}_{i}^{\mathrm{u}}\). Obviously, we can estimate \(\mathbf{y}_{i}^{\mathrm{u}}\) from \(\mathbf{y}_{i}^{\mathrm{s}}\) by utilizing the local correlation of natural images. Considering that natural images have non-local self-similarity properties, in our method, the non-local information is also used in the image reconstruction. We use the non-local information to correct the predictions. The reconstruction of missing pixel \(\mathbf{y}_{i}^{\mathrm{u}}\) can be defined as:  
\[\hat{\mathbf{y}}_{i}^{\mathrm{u}}=\boldsymbol{\phi}(\mathbf{y}_{i}^{\mathrm{ s}})+\lambda\mathbf{F}(\mathbf{S}_{i}) \tag{1}\]  
where \(\mathbf{y}_{i}^{\mathrm{s}}\) is the support area of \(\mathbf{y}_{i}^{\mathrm{u}}\) and \(\mathbf{S}_{i}\) is a set of samples, which are similar to \(\mathbf{y}_{i}^{\mathrm{s}}\). Model \(\phi(\cdot)\) is used to predict the missing pixels through the corresponding support areas. Function \(F(\cdot)\) uses the similar sample set \(\mathbf{S}_{i}\) to correct the predictions of \(\boldsymbol{\phi}(\cdot)\). \(\lambda\) is a factor that balances the corrections and predictions, \(\hat{\mathbf{y}}_{i}^{\mathrm{u}}\) is the final reconstruction that will be used to replace the missing pixel.  
Figure 1: Structure of the corrupted image. Each square stands for one pixel. The black pixels are unavailable ones. The white pixels are correctly received ones. The pixel being predicted is marked in red, and the pixels marked blue are the pixels corresponding to the current pixel \(\mathbf{y}_{i}^{\mathrm{u}}\) of the similar samples.  
In addition, under the sequential framework, the corrupted images are reconstructed pixel by pixel in sequence. The sequence, which is often called a scan order, is very important to the reconstruction performance, since it determines the available context of each missing pixel. After each pixel is reconstructed, we update the available part **S** and the unavailable part **U**. The EC task is accomplished when the unavailable part **U** becomes an empty pixel set.  
Therefore, the focus of our work is to build a model \(\mathbf{\phi}(\cdot)\) for predicting the missing pixels, find an approach to search for useful non-local information, and use it to correct the predictions. Moreover, it is also an important issue to determine an appropriate scan order in our work.