# GenConViT: Deepfake Video Detection Using Generative Convolutional Vision Transformer
## 3 Proposed Deepfake Detection Method
### ConvNeXt-Swin Hybrid  
The ConvNeXt-Swin Transformer architecture is a hybrid CNN-Transformer model that combines the strengths of ConvNeXt [34] and Swin Transformer [35] architectures for deepfake detection tasks. The ConvNeXt model is a CNN architecture that has shown an impressive performance in image recognition tasks by extracting high-level features from images through a series of convolutional layers. The Swin Transformer is a transformer-based model that uses a self-attention mechanism to extract both local and global features. The GenConViT model leverages the strengths of both architectures by using ConvNeXt as the backbone for feature extraction and the Swin Transformer for feature processing. In our proposed method, the ConvNeXt architecture extracts high-level features from images, which are then passed through a HybridEmbed module to embed the features into a compact and informative vector. The resulting vector is then passed to the Swin Transformer model. The ConvNeXt backbone consists of multiple convolutional layers that extract high-level features from the input images and the LFs from the AE or VAE. We used pre-trained ConvNeXt and Swin Transformer models, which were trained on an ImageNet dataset. After extracting learnable features using the ConvNeXt backbone, we pass the feature maps through the HybridEmbed module. The HybridEmbed module is designed to extract feature maps from the ConvNeXt, flatten them, and project them to an embedding dimension of 768. It consists of a \(1\times 1\) convolutional layer which takes the feature maps from the backbone and reduces their channel dimension to the desired embedding dimension. The resulting feature maps are flattened and transposed to obtain a sequence of feature vectors, which are then further processed by the Swin Transformer. The GenConViT's network \(A\) consists of two Hybrid ConvNeXt-Swin models that take in an LF of size \(224\times 224\times 3\) generated by the AE (\(I_{A}\)) and an input image of the same size. The models output a feature space of size 1000, which is then concatenated. A linear mapping layer of 2 then transforms this combined feature vector into a class prediction, corresponding to the probabilities of the real and fake classes. Network \(B\) has the same configuration as \(A\), but it uses the VAE and outputs both a class prediction probability and a reconstructed image of \(224\times 224\times 3\) in size. Finally, the predictions of network A and network B are averaged to obtain the final real/fake prediction. In summary, our proposed GenConViT method introduces a hybrid architecture for deepfake detection, combining the feature extraction capabilities of the pre-trained ConvNeXt-Swin architecture with generative components that use an AE and VAE. This setup helps our model to capture subtle inconsistencies in deepfake videos. GenConViT has \begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|} \hline
**Network** & \multicolumn{8}{c|}{**AE Configuration**} \\ \hline
**Encoder** & \multicolumn{4}{c|}{**Conv**} & \multicolumn{1}{c|}{**Kernel**} & \multicolumn{1}{c|}{**Stride**} & \multicolumn{1}{c|}{**Activation**} & \multicolumn{1}{c|}{**Normalization**} \\ \cline{2-9}  & 3-16 & 16-32 & 32-64 & 64-128 & 128-256 & 3 & 1 & ReLU & - \\ \hline
**Decoder** & \multicolumn{4}{c|}{**ConvTranspose**} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c}{} \\ \cline{2-9}  & 256-128 & 128-64 & 64-32 & 32-16 & 16-3 & 2 & 1 & ReLU & - \\ \hline \end{tabular}
\end{table} 
Table 2: Configuration of GenConViT model’s Autoencoder.