# Efficient Video Frame Interpolation Using Generative Adversarial Networks
## 2 Related Works
### Image Generation and Video Frame Generation  
GANs have shown their efficiency in numerous image applications. In the beginning, many researchers used GANs to generate images from scratch or simple labeled data. Later, cross-domain GANs, which transform data from a domain to another, such as from sketch to an image [11], from description text to image [20; 21], and from photographs to painting [22], were developed. Other applications of GANs in image processing include super-resolution [4], image inpainting [23], image rendering [24], de-raining [25], and medical image processing [26]. [PERSON] et al. [6] proposed a conditional GAN structure for image processing based on the image-to-image translation.  
Video frame generation is a technique that produces new frames based on the existing image sequence with temporal semantics. There are two types of frame generation, frame extrapolation and frame interpolation as shown in Table 1.  
In this study, we focus on frame interpolation, where an intermediate frame is predicted from adjacent frames. Traditional interpolation techniques are realized by estimating the relative motion of pixels, blocks, or meshes. One classic pixel-based approach is the optical flow, however, which requires heavy computation and performs poorly for videos containing abrupt changes in brightness or motion. Consequently, gradient-based [27] and phase-based [28] approaches were proposed. Deep learning based approaches, instead, directly synthesize the intermediate frame using neural networks and generally provide more satisfactory results than conventional methods.  
The networks considered in video frame generation include the convolutional neural networks (CNNs) [32; 33], recurrent neural networks (RNNs), long short term memory networks (LSTM), and GANs. [PERSON] et al. [1] designed a fully-convolutional CNN and estimated pairs of 1D kernels for all pixels to produce the interpolated frame. [PERSON] et al. [34] proposed a video frame interpolation method based on the bi-directional optical flow and depth maps estimation. [PERSON] et al. [35] proposed a phase-based deep learning method that predicts the interpolated frame level by level with a phase-based loss function. For GAN-based approaches, [PERSON] and [PERSON] [14] proposed using the retrospective cycle GAN framework to predict a future frame by four previous frames. [PERSON] et al. [7] used a DCGAN based approach for frame interpolation to enhance video compression performance. In their work, the authors used a feed-forward network to predict a latent vector as the input for the generator and produced interpolated frames based on the linear interpolation in latent space. [PERSON] et al. [8] concatenated two GANs to generate the intermediate frames between keyframes, in which the first generator learns motions, and the second one adds more details into the first network's output. In that work, the authors also enhanced the input data with new frames that were obtained by pixel-wise linear interpolation. The method, application, dataset, and result of deep learning-based frame interpolation are summarized and listed in Table 1. Although the deep learning based approaches generally achieve good performance, most of them use complex networks and auxiliary data and therefore a long time for training and processing is required. Moreover, many GAN based approaches still have difficulties in characterize irregular motion and image details.