# Image Error Concealment Based on Deep Neural Network
## 3 Our Proposal  
In this work, we propose an EC method that take into account both the local and non-local information in the reconstruction of the corrupted images. More specifically, we exploit the local information to predict the missing pixels and the non-local information to correct the predictions. In our method, a deep neural network is designed to serve as the prediction model \(\mathbf{\phi}(\cdot)\) to achieve the EC purpose. The designed neural network can be divided into two parts: the prediction part and the auto-encoder (AE) part. The prediction part utilizes the local correlation among pixels to predict the missing ones. The AE part is used to extract the image features. Through those extracted features, we collected similar samples from the whole image. Since the designed neural network can extract image features through the AE part and predict missing pixels using the prediction part, we call it an AE-P network.  
As illustrated in Figure 2, given a corrupted image **X**, we first collect all the possible samples from the available area to serve as the training set \(\textbf{T}=[\text{t}_{1},\text{t}_{2},\dots,\text{t}_{n}]\), and each sample \(\text{t}\) in set **T** can be regarded as a combination of an available part \(\text{t}^{\text{s}}\) and a missing part \(\text{t}^{\text{u}}\). Here, it should be pointed out that the true value of \(\text{t}^{\text{u}}\) in training set **T** is known. Therefore, the training set **T** can be divided into two subsets \(\textbf{T}^{\text{s}}\) and \(\textbf{T}^{\text{u}}\), which are composed of \(\text{t}^{\text{s}}\) and \(\text{t}^{\text{u}}\), respectively. Then, the collected training set **T** is used to train the designed AE-P network. More specifically, we train the AE part network through unsupervised learning and the prediction part network through supervised learning. Let \(\textbf{y}_{i}^{\text{u}}\) be the current missing pixel on the contour of the unavailable region, and \(\textbf{y}_{i}^{\text{s}}\) be the corresponding support area. When we input \(\textbf{y}_{i}^{\text{s}}\) into the trained AE-P network, on the one hand, we can get the prediction \(\mathbf{\phi}(\textbf{y}_{i}^{\text{s}})\) of the missing pixel through the prediction part network. On the other hand, the AE part network extracts the features of \(\textbf{y}_{i}^{\text{s}}\) by comparing the similarities in the feature domain and the pixel domain. Thus, we searched for samples similar to \(\textbf{y}_{i}^{\text{s}}\) from the whole image, which served as a similar sample set, **S**, as illustrated in the blue box in Figure 2. Then, we used these similar samples to correct the prediction \(\mathbf{\phi}(\textbf{y}_{i}^{\text{s}})\) based on the non-local similarity of the natural images. Therefore, the final reconstruction \(\hat{\textbf{y}}_{i}^{\text{u}}\) is determined through combining the prediction and correction, as shown in Formula (1). In this way, both the local and non-local information are taken into account for reconstruction. In addition, to alleviate the error propagation problem, we proposed an adaptive scan order based on the joint credibility of the support area and the reconstruction.  
Figure 2: Block diagram of the proposed method. The missing pixel \(\textbf{y}_{i}^{\text{u}}\) is under process. The square patches \(\text{t}_{i}(i=1,2,\dots,n)\) are the collected samples for training. The square patches \(\textbf{s}_{i}(i=1,2,\dots,m)\) are the samples that are similar to the current support area \(\textbf{y}_{i}^{\text{s}}\).