# Image Error Concealment Based on Deep Neural Network
## 3 Our Proposal
### Training Data Collection  
For training the designed AE-P network, we need a large number of samples. In order to collect appropriate training data, a template matching scheme is utilized to match and collect samples from the whole image. A challenging problem of the scheme is how to design the template shape to get the maximum available information. Similar to many SEC methods, templates with a square shape, as shown in Figure 5a, are widely used in training data collecting. The corresponding context of the missing pixel is shown in the blue dotted box in Figure 5b. It can be observed that only the available pixels in area A, which is part of the context, are used to reconstruct the current missing pixel. However, it is true that the pixels in areas B and C also have high correlation with the missing pixel. Thus, the available pixels in areas B and C should also be taken into account in image reconstruction. Since more available information is considered, the reconstruction of the missing pixel will be more reliable. Figure 4: Structure of the designed auto-encoder prediction (AE-P) network. In order to get more available information from the context, eight templates with different shapes are designed in our method, as shown in Figure 6. These eight templates can be embedded around the missing pixels to augment the information collected from the context. Unlike square templates, which can only match one support area from the context for each missing pixel, we can collect multiple support areas. As illustrated in Figure 7, four support areas, \(\mathbf{y}_{i}^{\text{s}_{1}}\), \(\mathbf{y}_{i}^{\text{s}_{2}}\), \(\mathbf{y}_{i}^{\text{s}_{3}}\), and \(\mathbf{y}_{i}^{\text{s}_{4}}\) around the current missing pixel \(\mathbf{y}_{i}^{\text{u}}\) can be collected through matching the eight templates in the context. The available information obtained by these four support areas is shown in the blue box in Figure 7. For each support area, a reconstruction will be generated through the trained AE-P network. By combining the reconstructions of these support areas, the final reconstruction is therefore determined through all the available pixels of these support areas. The final reconstruction of the missing pixel \(\mathbf{y}_{i}^{\text{u}}\) is determined as:  
\[\hat{\mathbf{y}}_{i}^{\text{u}}=\frac{1}{N}{\sum_{j=1}^{N}\hat{\mathbf{y}}_{i }^{\text{u}_{j}}} \tag{4}\]  
where \(\hat{\mathbf{y}}_{i}^{\text{u}_{j}}\) is the reconstruction corresponding to the support area \(\hat{\mathbf{y}}_{i}^{\text{s}_{j}}\), \(N\) is the number of matched support areas, and \(\hat{\mathbf{y}}_{i}^{\text{u}}\) is the final reconstruction of the missing pixel. This formula shows that the final reconstruction of the missing pixel is the average of the reconstructions corresponding to all of the collected support areas. Figure 5: The red square is the current missing pixel, the gray squares are the available pixels, and the black squares are the unavailable pixels. (**a**) The template with square shape. (**b**) The context of the current missing pixel through matching the square template. Figure 6: The eight different templates used in our method. The black squares stand for the missing pixels, and the white squares are the successfully received ones. Eight templates with different shapes are utilized to collect training data for training the AE-P network in the proposed method. However, the relative location of the missing pixels and the corresponding support area are different among the collected samples. In order to train the AE-P network, we normalized all the collected training samples into a standard shape through rotating and flipping. As shown in Figure 6, we defined the template shape _down-left_ as the standard shape and transformed all the collected training samples into the standard shape. For example, the procedure of transforming the shape from _up-left_ into the standard shape is shown in Figure 8.  
In our method, we collected the training data from the available regions of the input-corrupted images.