## 2 Materials and Methods
### Virtual View Optimization  
When the depth information of objects in the scene is relatively complex, the inpainting model can be applied several times for processing until the occluded area is completely processed. In Section 2.3, the color and depth of the occluded area are processed, respectively. In order to further improve the quality of the virtual view, an optimized GAN was added into the inpainting network to establish the connection between color and depth through the regional consistency relationship. As an image understanding method, the semantic segmentation defines a unique label for each pixel in the input image. Semantic segmentation images contain various types of objects whose edges generate depth jumps and whose internal depths are mostly continuous. The spatial geometric relation information of objects in the image can be acquired from depth images. After combining the above features and matching RGB images as the input of the virtual view optimization network, more accurate virtual view images are acquired through the C64-C128-C256-C512 network architecture proposed in [53], where C is the Convolution-BatchNorm-ReLU block, and 64, 128, 256 and 512 are the numbers of filters. In depth estimation, the method adopted in this paper is to acquire the depth image through semantic segmentation, so it can also be used to acquire the input image in the optimization network. The virtual view optimization network framework is shown in Figure 6.  
Figure 6: The framework of virtual view optimization network.  
In Figure 6, \(G_{d}^{\prime}\) is the depth generator, \(G_{c}^{\prime}\) is the color generator, \(D_{d}^{\prime}\) is the depth discriminator, \(D_{c}^{\prime}\) is the color discriminator and \(D^{\prime}\) is the shared discriminator. The color information and the depth information of the image are made consistent by the shared discriminator \(D^{\prime}\). The shared discriminator \(D^{\prime}\) receives a set of RGB-D as input, and differentiates the input content, then corresponds to the true RGB-D. The depth generator \(G_{d}^{\prime}\) and color generator \(G_{c}^{\prime}\) receive feedback from their respective discriminators to achieve optimization. The optimization loss function is defined as \(L_{opt}\), and the expression is as follows:  
\[L_{opt}=E_{x,y}\big{[}\log D^{\prime}(y_{c},y_{d})\big{]}+E_{x}\big{[}\log \big{(}1-D^{\prime}\big{(}G_{c}^{\prime}(x_{c}),G_{d}^{\prime}(x_{d})\big{)} \big{)}\big{]} \tag{12}\]  
where \(x_{c}\) is the color image that has not been optimized, \(x_{d}\) is the depth image that has not been optimized, \(y_{c}\) is the color image that has been optimized and \(y_{d}\) is the depth image that has been optimized. The final optimized color image and depth image are represented by \(\hat{\vec{G}}_{c}\) and \(\hat{\vec{G}}_{d}\), respectively, whose expressions are as follows:  
\[\hat{\vec{G}}_{c}=\min_{G_{c}^{\prime}}\max_{D_{RGB}^{\prime}}+\lambda^{\prime }\min_{G_{c}^{\prime}}\max_{D^{\prime}}L_{opt}+\lambda_{L_{2}}L_{2}\big{(}y_{ c},G_{c}^{\prime}(x_{c})\big{)} \tag{13}\]  
\[\hat{\vec{G}}_{d}=\min_{G_{d}^{\prime}}\max_{D_{d}^{\prime}}+\lambda^{\prime} \min_{G_{d}^{\prime}}\max_{D^{\prime}}L_{opt}+\lambda_{L_{2}}L_{2}\big{(}y_{ d},G_{d}^{\prime}(x_{d})\big{)} \tag{14}\]  
where \(\lambda^{\prime}=0.8\) and \(\lambda_{L_{2}}=80\). \(L_{2}\) loss is chosen because it is closer to the true output.