# Efficient Video Frame Interpolation Using Generative Adversarial Networks
## 3 The Proposed Method
### Loss Functions  
The training process of GAN-based networks relies on the loss function that measures the discrepancy between the generated model (or data) and the true model (or data). In this paper, the similarity of a generated frame is judged by a combination of two types of losses, which include the adversarial loss and the reconstruction loss. The overall loss function is expressed as follows,  
\[L\Big{(}X,\ \hat{X}\Big{)}=\ \lambda_{1}\times L_{\text{adv}}\Big{(}X,\ \hat{X}\Big{)}+\ \lambda_{2}\times L_{2}\Big{(}X,\ \hat{X}\Big{)}+\ \lambda_{3}\times L_{\text{MS\_SSIM}}\Big{(}X,\ \hat{X}\Big{)}+\ \lambda_{4}\times L_{\text{GDL}}\Big{(}X,\ \hat{X}\Big{)}. \tag{3}\] \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layers** & **Details** & **Output Size** \\ \hline
1 & Conv2d(3, 128, stride = 1) BatchNorm2d LeakyReLU & \(128\times 64\times 64\) \\
2 & Conv2d(4, 128, stride = 2) BatchNorm2d LeakyReLU & \(128\times 32\times 32\) \\
3 & Conv2d(4, 256, stride = 2) BatchNorm2d LeakyReLU & \(256\times 16\times 16\) \\
4 & Conv2d(4, 512, stride = 2) BatchNorm2d LeakyReLU & \(512\times 8\times 8\) \\
5 & TransposedConv2d(4, 256, stride = 2) BatchNorm2d LeakyReLU & \(256\times 16\times 16\) \\
6 & TransposedConv2d (4, 128, stride = 2) BatchNorm2d LeakyReLU & \(128\times 32\times 32\) \\
7 & TransposedConv2d (4, 64, stride = 2) BatchNorm2d LeakyReLU & \(64\times 64\times 64\) \\
8 & TransposedConv2d (3, 3, stride = 1) tanh & \(3\times 64\times 64\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Proposed generator network’s structure. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layers** & **Details** & **Output Size** \\ \hline
1 & Conv2d(4, 64, stride = 2) LeakyReLU & \(64\times 32\times 32\) \\
2 & Conv2d(4, 128, stride = 2) BatchNorm2d LeakyReLU & \(128\times 16\times 16\) \\
3 & Conv2d(4, 256, stride = 2) BatchNorm2d LeakyReLU & \(256\times 8\times 8\) \\
4 & Conv2d(4, 512, stride = 2) BatchNorm2d LeakyReLU & \(512\times 4\times 4\) \\
5 & Conv2d(4, 1, stride = 1) Sigmoid & \(1\times 512\times 512\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Proposed discriminator network’s structure. Figure 2: The model of the proposed generator network. The network takes two \(3\times 64\times 64\) matrices and uses the previous and the future frames as the input. The details of the parameters are listed in Table 2.  
where \(X\) and \(\hat{X}\) denote the original frame (ground truth) and the reconstructed frame, respectively, \(L_{\rm adv}\) indicates the adversarial loss, i.e., the difference between the probability distributions and the other three terms \(L_{2}\), \(L_{\rm MS\_SSIM}\) and \(L_{\rm GDL}\) refer to the reconstruction data losses, as explained below. The coefficients \(\lambda_{1}\), \(\lambda_{2}\), \(\lambda_{3}\), and \(\lambda_{4}\) are non-negative weights for each of the loss terms. We have explained the general adversarial loss \(L_{\rm adv}\) in Section 2.1. Specifically in our work, the Binary Cross Entropy (BCE) loss function (Equation (4)), which measures the binary cross-entropy between the classification result \(D(\mathbf{x})\) and the target value \(\mathbf{y}\), is adopted. \[L_{\rm adv}(D(\mathbf{x}),\ \mathbf{y})=L_{\rm BCE}(d,\ \mathbf{y})=-\frac{1}{N}\sum_{n=1}^{N }(y_{n}\log(d_{n})+(1-y_{n})\log(1-d_{n})) \tag{4}\]  
The target value \(y_{n}\) represents the expected label data between 0 (false) and 1 (true) and \(d\) is the discrimination value for the frame \(\mathbf{x}\). This study falls into the category of nonrandom input approaches. The errors (reconstruction data losses) between the generated output and ground truth can be calculated for the training set. The first term is the \(L_{2}\) loss (mean square error, MSE)  
\[L_{2}(\mathbf{x},\ \mathbf{y})=\frac{1}{N}\sum_{n=1}^{N}(x_{n}-y_{n})^{2}, \tag{5}\]  
where \(\mathbf{x}\) and \(\mathbf{y}\) are the generated output and the ground truth, respectively, and \(N\) is the number of pixels within a frame. The \(L_{2}\) loss function performs the pixel-by-pixel comparison and helps to sharpen the output image.