# GenConViT: Deepfake Video Detection Using Generative Convolutional Vision Transformer
## 3 Proposed Deepfake Detection Method
### Autoencoder and Variational Autoencoder  
An AE and a VAE consist of two networks: an Encoder and a Decoder. The Encoder of the AE maps an input image \(X\in\mathbb{R}^{H\times W\times C}\) to a latent space \(Z\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times K}\), where \(\kappa\) is the number of channels (features) in the output and \(h^{\prime}\) and \(w^{\prime}\) are the height and width of the output feature map, respectively. The Decoder of the AE maps the latent space \(Z\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times K}\) to an output image \(X^{\prime}\in\mathbb{R}^{H\times W\times C}\). The Encoder of the AE is composed of five convolutional layers with width starting from 3 and expanding up to 256, with kernels of size \(3\times 3\) and a stride of 2. Each convolutional layer is followed by ReLU non-linearity and Max pooling of kernel size \(2\times 2\) and stride 2. The output of the Encoder is a \(256\times 7\times 7\) down-sampled LF. The Decoder is composed of five transposed convolutional layers with a width starting at 256 and ending at 3, with kernels of size \(2\times 2\) and stride of 2. Each transposed convolutional layer is followed by ReLU non-linearity. The output  
\begin{table}
\begin{tabular}{l l} \hline
**Paper** & **Dataset(s) for Training and Evaluation** \\ \hline Image+Video Fusion [18] & DFDC \\ Selim EfficientNet B7 [69] & DFDC \\ CViT [60] & DFDC \\ Random cut-out [70] & DFDC (with face cut-out augmentation) \\ STDT [28] & FF++, Celeb-DF \\ ViT with distillation [61] & DFDC \\ [PERSON] et al. [16] & DFDC \\ [PERSON] et al. [62] & FF++, DFDC \\ [PERSON] et al. [54] & FF++, CelebA-HQ (FaceShifter GAN) \\ GenConViT (ours) & FF++, DFDC, TM, Celeb-DF (v2), TIMIT \\ \hline \end{tabular}
\end{table}
Table 1: Datasets used in selected deepfake detection methods.  
of the Decoder, \(I_{A}\), is a reconstructed feature space of the input image with dimensions \(H\times W\times C\). In this case, \(I_{A}\) has dimensions \(224\times 224\times 3\). The details of this configuration are shown in Table 2.  
The goal of the VAE is to learn a meaningful latent representation of the input image and reconstruct the input image by performing random sampling of the latent space while minimizing the reconstruction loss. The Encoder of the VAE maps an input image \(X\) to a probability distribution over a latent space \(Z\in\mathbb{R}^{K},Z\sim\mathcal{N}(\mu,\sigma^{2})\), in which \(\mu\) and \(\sigma^{2}\) are the mean and variance of the learned distribution, respectively.  
The Encoder of the VAE is composed of four convolutional layers with widths starting from 3 and expanding up to 128, with kernels of size \(3\times 3\) and a stride of 2. Each convolutional layer is followed by Batch Normalization (BN) and LeakyReLU non-linearity. The use of Batch Normalization also serves as a regularizer, mitigating the risk of overfitting during training. The Decoder is composed of four transposed convolutional layers with widths starting from 256 and reducing to 3, with kernels of size \(2\times 2\) and a stride of 2. Each transposed convolutional layer is followed by LeakyReLU non-linearity. The output of the Decoder, \(I_{B}\), is an image reconstructed from the input image with the dimensions \(H/2\times W/2\times C\). In this case, \(I_{B}\) has dimensions \(112\times 112\times 3\). The details of this configuration are shown in Table 3. The choice of convolutional layers for both the AE and VAE is due to the computing power and memory we had, model accuracy, the extensive experiment, and the training time. To find the optimal hyperparameters for both the AE and VAE components, we experimented with latent sizes ranging from 8 to 1024 (i.e., 8, 64, 128, 256, 512, 1024) and various downsampling widths. We settled on 128 latent channels and 5 downsampling layers, since that setup minimized both the validation MSE and Cross-Entropy loss.  
Figure 1: The proposed GenConViT deepfake detection framework.