Dynamic Mixed-Prototype Model for Incremental Deepfake Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 09:22:43

Abstract: The rapid advancement of deepfake technology poses significant threats to social trust. Although recent deepfake detectors have exhibited promising results on deepfakes of the same type as those present in training, their effectiveness degrades significantly on novel deepfakes crafted by unseen algorithms due to the gap in forgery patterns. Some studies have enhanced detectors by adapting to the continuously emerging deepfakes through incremental learning. Despite the progress, they overlooked the scarcity of novel samples that can easily lead to insufficient learning of forgery patterns. To mitigate this issue, we introduce the Dynamic Mixed-Prototype (DMP) model, which dynamically increases prototypes to adapt to novel deepfakes efficiently. Specifically, the DMP model adopts multiple prototypes to represent both real and fake classes, enabling learning novel patterns by expanding prototypes and jointly retaining knowledge learned in previous prototypes. Furthermore, we propose the Prototype-Guided Replay strategy and Prototype Representation Distillation loss, both of which effectively prevent forgetting learned knowledge based on the prototypical representation of samples. Our method surpasses existing incremental deepfake detectors across four datasets and exhibits superior generalizability to novel deepfakes through learning limited deepfake samples.

Main paper: 
 related works 2.1 deepfake detection: 
 The potential misuse of deepfakes presents significant risks to society, making deepfake detection an urgent yet challenging task. Early methods usually leveraged biological artifacts, such as lack of eye blinking [24] , abnormal head pose [25] , and atypical pupils [26] for detecting deepfake media. However, advancements in deep generative models have enabled deepfakes to avoid these obvious artifacts, rendering these methods ineffective. Subsequent works have shifted towards using end-to-end learning with handcrafted modules to detect deepfakes. For example, TALL [27] aggregated four frames and input them simultaneously into an image classifier for cross-frame comparison. AltFreezing [28] enhanced the learning of both temporal and spatial artifacts by alternately freezing the spatial and temporal modules during training. StyleGRU [29] represented the dynamic properties of style latent vectors to spot temporal distinctiveness artifacts in deepfake videos. These methods merely fit the forgery patterns within the training set, and thus their effectiveness on unseen deepfakes is limited. Another direction involves employing pretext tasks to learn audio-lip consistency representations capable of identifying deepfakes. Notable examples include lip-reading [12] and audio-visual contrastive learning [16] , which are effective only for discriminating talking face videos. In general, these deepfake detectors rely on detecting specific forgery patterns that may not exist in novel deepfakes, which leads to a decline in effectiveness. Therefore, incremental deepfake detection emerges as a more practical approach to discriminate deepfakes by directly adapting detectors to novel samples.The objective of incremental deepfake detection is to detect emerging deepfakes, which requires detectors to adapt to novel deepfakes while retaining learned forgery patterns to discriminate previously learned deepfakes. Specifically, the training procedure consists of a base training stage to obtain a base detector and multiple incremental training stages to simulate the detector's adaptation to novel deepfakes. Some related studies [30, 31] assumed the incremental stages have ample training samples. Nonetheless, as the algorithms used to generate online deepfakes are unknown, it is hard to synthesize a large volume of samples for adaptation. In practical scenarios, often only a few labeled samples are available, typically obtained through multimedia retrieval [19] or proactive deepfake watermarking [20, 21] . Therefore, in this paper, we assume only limited labeled deepfakes are available in incremental training stages. Existing few-shot incremental deepfake detection works [17, 18] emphasized retaining the learned knowledge through distillation and replay. In contrast, our method enhances the learning of novel deepfakes by modeling new deepfake patterns explicitly through prototypes and thus is more effective against emerging deepfakes. 
problem definition: 
 In incremental deepfake detection, an ideal detector is expected to continually identify new deepfakes by sequentially learning from novel samples while preserving the ability to spot previously learned deepfakes. Especially, considering the scarcity of available novel deepfakes, the detector is expected to learn novel patterns from limited samples. The continuous set of training data is formally given as 𝑆 𝑡 𝑁 1 , where 𝑆 𝑡 denotes the training data at 𝑡-th stage, and 𝑁 represents the total number of stages. Within the training data 𝑆 𝑡 for each stage, {𝑥, 𝑦} ∈ 𝑆 𝑡 represents the corresponding training image and label, where 𝑦 ∈ {0, 1} denotes real or fake category, respectively. The detector is trained across different stages sequentially to learn the forgery patterns associated with the training deepfakes at each stage. We designate the first training stage (𝑡=1) as the base stage and subsequent stages as incremental stages (𝑡 >1). In real-world scenarios, a detector is initially trained with ample data in the base stage, which represents the pre-deployment training phase. While in preceding incremental stages corresponding to the adaptation after deployment, available deepfakes with labels are limited. Thus, the quantity of training data in the base stage 𝑆 1 is larger than those in the incremental stages 𝑆 𝑡 𝑡 >1 . After training in each stage, the detector is evaluated using test sets that contain samples from the current and previous stages. This evaluation setting assesses both a detector's ability to adapt to unseen deepfakes and its ability to retain learned patterns. 
method: 
 We aim to enhance incremental deepfake detection, where we focus on effectively adapting detectors to limited novel deepfakes while retaining the knowledge of learned deepfakes. To adapt detectors to novel deepfakes, we introduce the Dynamic Mixed-Prototype (DMP) model, which represents a face as a Mixture of Prototypes (MoP) and adapts to novel deepfakes by expanding prototypes. To further retain learned knowledge in DMP, we propose Prototype-Guided Replay (PGR) and Prototype Representation Distillation (PRD), which are efficient in avoiding forgetting in DMP through replaying seen samples and adopting the MoP as intermediate representations for DMP to distill itself. This section is arranged as follows. We first formulate the proposed DMP model for incremental deepfake detection and detail the design motivations. Building upon the DMP model, we elaborate on the training pipeline for incrementally training the DMP model on novel deepfakes as illustrated in Figure 2 . Within this framework, we describe the PGR for data replay and PRD for knowledge distillation, which aid in retaining the model's memory of previously learned deepfake representations.Learning intrinsic deepfake patterns from limited labeled deepfakes is inherently challenging, as the model is prone to overfit to biased forgery patterns when sufficient samples are unavailable. We note that the prototypical network [22] can effectively facilitate few-shot learning by projecting samples into a metric space, where compact features are learned and represented as prototypes. Since the classification within a prototypical network is conducted by calculating the distances to the prototype representations of each class, prototypical networks reflect a simpler inductive bias to reduce overfitting, which is beneficial for incremental learning.Motivated by the advantages of prototypical networks, we introduce the Dynamic Mixed-Prototype (DMP) model to classify faces based on the distance from their embeddings to real and fake prototypes. Namely, if a face is close to a prototype in the embedding space, it contains the corresponding pattern. Previous prototypebased deepfake detection works typically adopted a single prototype for both classes to enforce a tight cluster for all deepfakes [32, 33] . We argue that each deepfake sample poses various patterns inherently, as various manipulation techniques can potentially be used to produce a single deepfake. In this circumstance, attributing all deepfakes to one fake prototype hinders the learning of richer forgery patterns. Therefore, unlike previous works, DMP adopts multiple prototypes for real or fake classes and apportions a face to multiple prototypes, i.e., Mixture of Prototypes (MoP). This mixture apportionment helps to mine both shared and unique patterns in deepfakes, thereby facilitating the learning of richer patterns. Furthermore, representing faces as MoP enables increasing prototypes to expand the representations for a category. Specifically, DMP incrementally learns novel deepfakes by incorporating new prototypes for novel forgery patterns in each stage while retaining previous prototypes to preserve learned knowledge. As for real faces, the prototype set remains fixed since the distribution of real samples is consistent across different learning stages.Formally, the proposed DMP model is built with an encoder 𝑓 and prototypes 𝑃. We denote the encoder that has been trained up to 𝑡-th stage as 𝑓 𝑡 and the prototypes as 𝑃 𝑡 . 𝑃 𝑡 is a set of vectors, i.e., 𝑃 𝑡 ∈ 𝑅 𝑁 𝑡 ×𝑑 , where 𝑁 𝑡 is the number of prototypes in stage 𝑡 and 𝑑 is the embedding dimension. We denote the first 𝑘 prototypes of 𝑃 𝑡 as 𝑃 𝑡 0 , i.e., prototypes for real patterns, and the rest of prototypes as 𝑃 𝑡 1 , i.e., prototypes for forgery patterns. Initially, 𝑃 0 contains 𝑘 real prototypes. Before DMP is trained in each stage, 𝑘 incremental prototypes are added to 𝑃 𝑡 to model the novel forgery patterns in the training data, i.e., 𝑁 𝑡 = (𝑡 + 1) × 𝑘. In DMP, a face 𝑥 is first fed to encoder 𝑓 into feature embedding 𝑓 (𝑥). As we aim to represent 𝑥 as MoP based on its similarity to 𝑃, its embedding is apportioned to different prototypes by calculating the distances from 𝑓 (𝑥) to 𝑃. Specifically, we consider the softmax of the negative distances from 𝑓 (𝑥) to each prototype in 𝑃 as the MoP, denoted as 𝐷 (𝑥, 𝑃). Its 𝑖-th entry 𝐷 (𝑥, 𝑃 𝑖 ) is formalized as:EQUATION)where 𝑑 is the cosine distance and 𝜏 = 1 is the softmax temperature. Herein, the negative distances can be considered as similarities, and the softmax operation in its entirety transforms similarities of 𝑓 (𝑥) to 𝑃 into normalized apportionment weights, i.e., MoP.The DMP further predicts the category of 𝑥 as class 𝑐 based on the weighted apportionment, i.e., entries in MoP 𝐷 (𝑥, 𝑃 𝑖 ), and the predicted classification probability 𝑝 (𝑦 = 𝑐 | 𝑥) is formalized as:EQUATIONwhere 𝑐 = 0 is for real and 𝑐 = 1 is for fake category. In training, we use cross-entropy loss L 𝐶𝐸 for optimizing the classification:EQUATIONDuring the optimization process, the feature representation 𝑓 (𝑥) of a sample is attracted towards the prototypes of its corresponding category 𝑃 𝑖 ∈ 𝑃 𝑡 𝑐 while being repelled from the prototypes of the opposite category. After training, DMP can represent 𝑥 as MoP 𝐷 (𝑥, 𝑃) based on its distance to prototypes, where the components of different prototypes within a sample are expressed as 𝐷 (𝑥, 𝑃 𝑖 ).Prototypical networks that rely on a single prototype per class typically establish prototypes as the centroid of sample embeddings within identical classes, which ensures that the intra-class distances of embeddings are compact. However, this approach is not feasible when representing classes using multiple prototypes, as it causes overlapping among different prototypes within the same category. To circumvent this limitation, we employ pre-assigned and fixed prototypes following Yang et al. [34] and further expand their method to the scenario where one category is represented using multiple prototypes. Therefore, training DMP involves aligning the encoder 𝑓 with these fixed prototypes for classification. This approach prevents the collapse issue that arises when assigning prototypes as the centroid of sample embeddings, where all prototypes within the same category collapse to a single point. Additionally, the pre-assigned prototypes are largely separated. This separation encourages the encoder to learn more diverse forgery patterns, enhancing the richness of the learned deepfake representations.We introduce the Prototype-Guided Replay (PGR) strategy, which is designed to select replay samples for the proposed DMP model to mitigate forgetting by stabilizing the learned deepfake representations. Specifically, PGR selects a subset of previously learned deepfake samples to be stored in a replay set, which is then incorporated into the training data for the subsequent stage. The procedure is shown in the left part of Figure 2 .Since the DMP model represents a sample as a Mixture of Prototypes, the key to maintaining the stability of representations lies in preserving the stability of prototypes. When applying PGR to select samples for replay, the closest 𝑛 samples to each prototype are selected at the end of each training phase. The reasons for replaying the closest samples to each prototype are twofold. First, by replaying the closest 𝑛 samples to each prototype, PGR helps to maintain the representational stability of the prototypes in subsequent incremental training stages. Second, since DMP uses pre-assigned and fixed prototypes, the encoder 𝑓 𝑡 is expected to map a replayed sample 𝑥 close to the prototype where it was initially embedded. The PGR strategy, along with the subsequent distillation loss, aids in guiding the encoder 𝑓 𝑡 to project replay samples near their original prototypes. Consequently, the classification accuracy on previously learned samples of the DMP model is maintained.The selection of the replay set is conducted at the end of each training stage. The selected replay set after stage 𝑡 is denoted as 𝑅 𝑡 +1 . The pool of candidate samples for the replay set is the union of the current replay set 𝑅 𝑡 and the current stage's training set 𝑆 𝑡 . For each prototype, 𝑛 nearest samples are selected from 𝑅 𝑡 ∪ 𝑆 𝑡 for replay, which are denoted as:EQUATIONwhere 𝑥 𝑡 ∈ 𝑅 𝑡 ∪𝑆 𝑡 . PGR contributes to retaining the representation associated with a prototype by replaying the samples nearest to it, thus mitigating catastrophic forgetting.To further mitigate catastrophic forgetting, we leverage 𝑓 EQUATION)where 𝐾𝐿 is the KL divergence. As the expansion of 𝑃 makes the dimensions of 𝐷 (𝑥, 𝑃 𝑡 -1 ) and 𝐷 (𝑥, 𝑃 𝑡 ) differs, we truncate 𝐷 (𝑥, 𝑃 𝑡 ) to match the size of 𝐷 (𝑥, 𝑃 𝑡 -1 ). The total loss L 𝑡𝑜𝑡𝑎𝑙 for optimizing DMP in stage 𝑡 > 1 is:EQUATIONWe set 𝛼 as 0.2 to make L 𝐶𝐸 and L 𝑃𝑅𝐷 converge to the same order of magnitude. 
experimental setup: 
 Dataset. We conduct evaluations on four widely used deepfake datasets, i.e., FaceForensics++ (FF++) [35] , DFDC preview (DFDCp) [36] , DFD [35] , and Celeb-DF v2 (CDF) [37] . FF++ is a deepfake dataset containing 4,000 forged videos and 1,000 real videos. All videos are provided in three compression levels: raw, high-quality (HQ), and low-quality (LQ). DFDCp is the preview dataset for the Deepfake Detection Challenge and contains over 4,000 face swap videos. Its fake videos are generated using two face swap algorithms. DFD is Google's supplement to the FF++ dataset and contains over 3,000 face swap videos. CDF contains 590 real and 5,639 fake videos corresponding to 59 celebrities using an improved face swap algorithm. We follow the splits in [18] for training and testing.Evaluation protocol. We follow the evaluation protocol in [18] . Specifically, the training sequence used in evaluation is 𝐷 𝐶 = {FF++, DFDCP, DFD, CDF}. Evaluated models are trained sequentially across four training sets of these stages. Upon completion of each training stage, the models are tested on the test sets of the datasets corresponding to all previously trained stages. In the base stage, the whole training set of FF++(HQ) is used for training. In incremental stages, 25 fake videos along with 25 real videos from the current incremental dataset, i.e., DFDCP, DFD, and CDF, are randomly sampled for training. Evaluation metric. We adopt four metrics in evaluating the proposed method following [18] , i.e., Accuracy, Average Accuracy, Average Forgetting, and Area Under Curve.(1) Accuracy (ACC) refers to the ratios of correctly predicted samples in all samples. It is calculated as ACC = 𝑇 𝑃 +𝑇 𝑁 𝑛 , where 𝑇 𝑃 (True Positives) refers to the number of samples that are correctly identified as deepfake, 𝑇 𝑁 (True Negatives) refers to the number of samples that are correctly identified as real, and 𝑛 represents the total number of images or samples that are evaluated.(2) Average Accuracy (AA) refers to the average accuracy of all previous stages. It is calculated as AA = 1 𝑁 𝑁 𝑖=1 ACC 𝑖 , where ACC 𝑖 is the accuracy of the 𝑖-th task and 𝑁 is the number of tasks.(3) Average Forgetting (AF) refers to the average forgetting rate of previous stages. It is calculated as AF = 1 (4) Area Under Curve (AUC) refers to the area under the Receiver Operating Characteristic (ROC) curve. We use AUC to evaluate the generalization of detectors across different deepfake datasets. Implementation Details. We use an Xception model pretrained on ImageNet [38] to initialize the DMP model. Our framework is trained for 40 epochs in the base stage and incrementally trained for 40 epochs in each incremental stage. The DMP model is trained using AdamW [39] optimizer with a learning rate of 2e-4 and a batch size of 64, and the learning rate is decayed to half every 5 epochs. Each batch consists of 32 real faces and 32 deepfake faces. The replay set size K is set to 500, and PGR selects 500//𝑁 𝑡 nearest samples for each prototype after stage 𝑡 for replay. In the PyTorch framework, training with automatic mixed precision requires approximately 20GB of GPU memory. Preprocess. We extract 20 frames at equal temporal intervals from each video in training and 10 frames from each video in testing. Then, the faces are aligned and cropped using Retinaface [40] , maintaining a 15% margin around each face. Then, all faces are resized to 299 × 299 as the inputs.We compare the proposed method with two general incremental learning methods, i.e., LWF [42] and DGR [41] , and two incremental deepfake detection methods, i.e., DFIL [18] and CoReD [17] . As all compared methods are model-agnostic, we use Xception as the backbone in all methods for fair comparison.The results are shown in Table 1 . Clearly, all compared methods obtained satisfying ACC on FF++ after initially training on ample data from FF++. However, when they were later incrementally trained with limited data from DFDCP, DFD, and CDF, their performance in detecting previously learned deepfakes deteriorated significantly. For instance, after training on the DFDCP dataset in the second training stage, the LWF method experienced a 7.69% decrease in detection accuracy for the FF++ dataset. After the training across all four stages, its ACC scores on the FF++ and DFDCP datasets are 67.34% and 67.43%, respectively. Namely, after incremental learning with new deepfake samples, LWF failed to accurately detect previously learned deepfakes. In stark contrast, the performance of DMP to learned deepfakes consistently maintained high ACC on all subsequent datasets. For instance, after being trained up to CDF, DMP achieved an ACC of 91.61% on FF++, outperforming the top result in compared methods. Moreover, after the final stage of incremental training on CDF, DMP achieved an Accuracy Forgetting (AF) rate of 4.08%. Compared to DFIL, the proposed DMP exhibited a lower AF with identical replay set sizes. This validates the efficiency of the proposed PRD and PGR in preserving the learned forgery representations.Besides, we note the proposed DMP is efficient in learning limited novel deepfakes. After training on the DFDCP and CDF datasets, our DMP achieved ACC scores of 89.72% and 91.67% on corresponding test sets, surpassing the highest ACC among all compared methods 88.87% (DFIL) and 87.90% (LWF). Furthermore, upon completing the training across all four stages, DMP attained an average accuracy (AA) of 89.99%, significantly outperforming all compared methods. These results validate the proposed DMP model is efficient in learning from limited novel deepfakes.Table 2 : Comparison of the generalization ability of different methods to DFDCP, DFD, and CDF. The metric is the AUC score. The upper shows the results of cross-dataset generalization, while the lower shows the results of incremental learning. Compared results are cited from [18] . Bold indicates top results.Test Set AUC(%)↑ DFDCP DFD CDF Xception [35] 72.20 70.50 65.50 LTW [43] 74.58 88.56 77.17 LRL [44] 76 We compare the proposed DMP with recent deepfake detectors in terms of generalization using AUC as the metric. All detectors are trained on FF++ and tested on DFDCP, DFD, and CDF. The compared methods includes generalizable deepfake detection methods [15, 35, [43] [44] [45] [46] and incremental deepfake detection methods [17, 18] . We evaluate the cross-dataset generalization performance of generalizable deepfake detection methods and the performance of incremental deepfake detection methods. The results are listed in Table 2 . It is evident that generalizable deepfake detectors performed inferiorly to incremental deepfake detectors on all three datasets, which is attributed to distribution shifts across datasets. Incremental deepfake detectors exhibited higher AUC scores by adapting to 25 videos from each dataset. This suggests that when a limited number of novel deepfake samples are available for adaptation, incremental deepfake detection is a more practical approach. Notably, our DMP demonstrates the best performance among the incremental methods on the two challenging datasets, DFDCP and CDF. Our model's AUC score surpasses the best among the comparative methods on the CDF dataset by 3.08%(92.84% v.s.89.76%). This indicates that our DMP model can effectively learn new forgery patterns that differ significantly from learned patterns through newly added prototypes. Modules.We conduct ablations to verify the effectiveness of the proposed DMP model, PGR, and PRD. For the DMP model, we use the following variants for comparison: (1) Vanilla Xception model with a fully connected layer as the classifier instead of prototypes. As it lacks prototypes, the proposed PGR and PRD are not applicable. We instead adopt a standard exemplar sample replay and distillation method following [47] . (2) A Mixed-Prototype model similar to the DMP model with a set of fake prototypes that is not incremental, which is denoted as MP. For the PGR strategy, we adopt the commonly used examplar sample replay proposed in [47] for comparison. For the PRD loss, we adopt the vanilla distillation loss in [47] for comparison. The results are presented in Table 3 , which indicates each element contributes to improving performance. Compared to vanilla Xception, DMP significantly enhances the AF, proving its advantages in retaining learned knowledge in prototypes. Compared to MP, DMP demonstrates a higher AA, which is attributed to the effectiveness of learning novel forgery patterns through new prototypes. Furthermore, our PRD and PGR better mitigate forgetting during incremental training compared to the replay and distillation method in [47] (fourth and fifth rows in Table 3 ). This is due to the fact that representing a deepfake sample as a MoP better aligns with the shared and unique features presented among different deepfake samples, thereby enhancing the effectiveness of replay and distillation. Distance Metric. The DMP model classifies samples based on the distance from the samples to the prototypes. The used distance metric 𝑑 specifies the modeling assumption about the class-conditional data distribution in the embedding space, making the selection of the distance function vital. Common distance metrics used in prototype learning include the Euclidean distance and cosine distance, and we employ the cosine distance in DMP. We compare it with the Euclidean distance to investigate their impacts on the performance of incremental deepfake detection. The results are presented in Table 4. Employing the cosine distance as the metric achieves higher AA across all four stages, proving to be more effective than using the Euclidean distance. The Euclidean distance corresponds to the assumption of a mixed density distribution, which indicates that the distribution of sample embeddings around deepfake patterns does not align closely with the mixed density distribution. We infer that the commonly used cosine distance performs effectively due to the use of fixed prototypes. In this context, the cosine distance for prototype-level classification is equivalent to a frozen linear layer, which reflects a simpler inductive bias to reduce overfitting in learning limited samples. This comparative experiment underscores the validity of our prototype layer design in the DMP model for learning limited novel samples. Numbers of Prototypes. The proposed DMP model adopts multiple prototypes to represent real and fake classes. The number of prototypes added before each stage, i.e., 𝑘, is hand-tuned as 4, where the number of prototypes 𝑁 𝑡 = (𝑡 + 1) × 𝑘. The parameter 𝑘 influences the DMP's ability to learn from new samples and retain knowledge about previously learned samples. This is because 𝑘 affects how the DMP model represents the samples as MoPs, which are then used for classification in Equation 2, replay in Equation 4, and distillation in Equation 5. We conducted ablations to assess the impact of 𝑘 on the performance of DMP and to verify the effectiveness of representing classes by multiple prototypes for incremental deepfake detection. The experimental results are presented in Table 5 . It is obvious that when 𝑘 = 1, the ACC in the base phase is comparable to that of using multiple prototypes 𝑘 > 1. However, the performance of the model with 𝑘 = 1 deteriorates notably on stage 𝑡 > 1, whereas increasing 𝑘 improves the AA. Beyond the results in Table 5 , we empirically found that further increasing 𝑘 brings marginal performance gain.In real-world scenarios, the size of the replay set can vary due to limitations in storage and computational resources. Moreover, the volume of accessible labeled novel deepfake samples may also change. Since the size of the replay set and the number of novel samples both impact the performance of incremental deepfake detection, we conducted sensitivity analyses on these factors separately by decreasing them. The results are shown in Figure 4 . It is apparent that decreasing the size of the replay set results in a growth in AF and a decrease in AA due to the reduced accuracy on previously encountered deepfakes. We also observed that diminishing the replay set reduces the efficiency of adaptation to novel deepfakes, which further decreases the AA. This is attributed to the role of distillation, which not only mitigates forgetting but CDF Fake DFDCP Fake DFD Fake FF++ Fake FF++ Real also facilitates the learning of novel deepfakes, underscoring the significance of replay and distillation for the adaptation to novel deepfakes. Besides, reducing the number of novel samples used for training also leads to a decrease in AA. The reduction in AF is less pronounced as the initial performance on novel samples is already lower due to the scarcity of available training samples and the ACC on previous test sets in subsequent training stages may improve, thereby compensating the forgetting ratio.Saliency Map As stated before, DMP represents each deepfake sample as a mixture of prototypes. Namely, different prototypes are expected to exhibit distinct saliency areas in a deepfake when discriminating it. We employ GradCAM [48] to visualize the component prototypes for real and fake samples to investigate their roles in discriminating deepfakes. For each sample, we visualize the corresponding saliency area of the three closest fake prototypes to its embedding in the feature space. The visualization results are shown in Figure 3 . It's obvious that different prototypes correspond to different forged areas in a deepfake sample, indicating each prototype represents a unique deepfake pattern. For instance, in the second column of samples from FF++ fake, the visualized prototypes' saliency areas respectively spot the distorted areas between the eyebrows (second row), the nose (third row), and the mouth (fourth row). Meanwhile, the combination of salient areas from different prototypes within a deepfake sample comprehensively marks major visible artifacts. This further confirms that modeling deepfakes as mixtures of prototypes aids in enriching learned deepfake representations. Besides, the saliency area visualizations for samples in FF++ real confirm that our fake prototypes avoid locating forged regions in real faces, demonstrating the efficiency of the fake prototypes in distinguishing between real and fake faces. 
conclusion: 
 The proposed framework in this paper enhances the incremental learning of limited novel deepfakes while avoiding forgetting previously learned deepfakes. The proposed DMP model represents faces as mixtures of prototypes to learn rich forgery representations and thus can explicitly model new deepfake patterns with new prototypes. Moreover, to prevent DMP from forgetting learned deepfake representations, we propose PGR and PRD tailored for DMP to replay representative samples and distill to maintain the stability of existing prototypes. Extensive evaluations verify our framework can learn incremental novel deepfake samples and avoid catastrophic forgetting efficiently. In scenarios where limited novel deepfake samples are available for adaptation, our framework demonstrates satisfactory performance, which is crucial for maintaining the effectiveness of deepfake detectors. Future work includes incorporating the latest deepfake technology, such as diffusion models, to build a more comprehensive evaluation protocol for incremental deepfake detection. 
: 
 to public safety and causing widespread concerns. Therefore, the development of effective deepfake detection methods has become more urgent. In addition, the continued evolution and proliferation of deepfake algorithms brings challenges to the generalizability and adaptability of deepfake detectors, requiring the ability to quickly detect novel deepfake contents.Previous works [10] [11] [12] have primarily focused on enhancing the generalizability of deepfake detectors in the absence of access to novel deepfakes. These methods improve performance on unseen deepfakes in the testing phase by guiding detectors to capture prior forgery patterns that may be shared among the training deepfakes and unseen deepfakes. Specifically, these methods incorporate strategies such as identifying face swap boundaries with synthesizing training data [10, 13, 14] , spotting texture anomalies and inconsistency [11, 15] , and perceiving audio-visual mismatch using pretext tasks [12, 16] . Nevertheless, different deepfake algorithms exhibit distinct forgery patterns, and learned patterns may not be universally applicable to expose various novel deepfakes. As a result, their reliance on prior patterns leads to compromised performance when applied to deepfakes that significantly deviate from these patterns.Some studies [17, 18] resort to utilizing a limited number of samples for adaptation to detect novel deepfakes, i.e., incremental deepfake detection, sidestepping the limitations of relying on prior patterns to detect unknown deepfakes. Namely, these methods assumed that a few labeled faces are available for adapting deepfake detectors, a scenario we consider plausible. For instance, multimedia retrieval [19] can be employed to retrieve the original image, thereby enabling verification of the consistency of an image with its original source to obtain its label. Additionally, proactive deepfake watermarking [20, 21] can aid in determining whether an image has been manipulated. With a small set of labeled samples, the detector can then incrementally learn to detect novel deepfakes during the inferencing phase, which is more suitable for practical application scenarios than enhancing detectors following generalizable deepfake detection methods.Incremental deepfake detection involves multiple learning stages, each corresponding to the process of adapting the detector to the distribution of different deepfake algorithms. This requires the network to efficiently learn a few novel deepfakes while avoiding the forgetting of previously acquired deepfake patterns. Current incremental deepfake detection methods [17, 18] focus on preventing catastrophic forgetting of learned deepfakes by utilizing knowledge distillation and data replay. For instance, CoReD [17] employs the previously trained model as a teacher network and uses a distillation loss to guide the student network in the subsequent stage, thereby preventing the forgetting of learned deepfakes. Compared to CoReD, DFIL [18] further employs hard example replay to preserve the deepfake detector's knowledge of previously learned forgery patterns. These methods typically utilized softmax loss to cluster limited novel deepfakes and previous deepfakes into aAmple base real sample Ample base fake sample Limited novel fake sample Base real prototype Base fake prototype Novel fake prototype New decision boundary Old decision boundary Figure 1 : Previous approaches incrementally learn novel deepfakes by adapting detectors through softmax loss, whereas the scarcity of novel deepfakes often leads to insufficient learning. In contrast, the proposed DMP model explicitly learns novel deepfakes through newly added prototypes to reduce overfitting, facilitating more effective classification in the prototype space.single group for shared features to adapt to novel deepfake samples. However, given the variety of potential deepfake patterns, such methods hinder the learning of novel patterns. Namely, these methods do not address the insufficient learning of limited novel deepfake samples.To adequately learn novel deepfake patterns using limited samples, we resort to prototypical networks [22, 23] . Prototypical networks classify samples in a metric space by computing distances to prototypes of each class. Compared to existing incremental deepfake detection works that cluster novel and previous deepfakes into a single group, prototypical networks reflect a simpler inductive bias that is advantageous when learning from limited samples [22] . To this end, we introduce the Dynamic Mixed-Prototype (DMP) model, which dynamically increases prototypes to learn novel patterns from limited incremental deepfakes efficiently, as illustrated in Figure 1 . Specifically, the DMP model leverages multiple prototypes for both real and fake classes, which enables the learning of novel forgery patterns by expanding the set of fake prototypes during incremental learning. In this manner, novel deepfake patterns are learned through newly added prototypes and knowledge learned in previous prototypes is retained jointly, greatly enhancing the effectiveness of incremental deepfake detection.During the training process on novel deepfakes, the model's ability to discriminate previously learned deepfakes can be compromised, i.e., catastrophic forgetting. This arises because the learned representations are susceptible to sequential training in incremental learning, which underscores the stability of the learned representations. As the proposed DMP model represents a sample as its distance to prototypes, the stability of representations necessitates that this distance remains stable after sequential training. To address this need, we propose the Prototype Representation Distillation (PRD) loss, which leverages DMP to distill itself in adapting to novel deepfakes and adopts the distance from sample to prototypes as the intermediate representations in distillation. In addition, we incorporate the Prototype-Guided Replay (PGR) strategy to further enhance the distillation by replaying learned deepfakes. Specifically, PGR selects several samples closest to each prototype to store in a replay set, which is replayed during the next phase of incremental learning. During distillation using PRD, PGR replays the samples closest to each prototype to maintain the stability of the learned prototypes. Adopting PRD and PGR demonstrates major advancement in maintaining learned representations and thus enhances the proposed DMP. Consisting of the DMP model, the PGR strategy, and the PRD loss, our method achieves the dual goals of mitigating catastrophic forgetting and efficiently learning from limited novel samples, comprehensively enhancing incremental deepfake detection. We conduct extensive experiments on four popular deepfake detection datasets, which shows that our method surpasses existing incremental deepfake detection methods across four datasets. The contributions of this paper can be summarized as follows:• We present the novel DMP model, which expands prototypes dynamically during incremental learning to adapt to new forgery patterns from limited samples for incremental deepfake detection.• We introduce a Prototype-Guided Replay strategy coupled with a Prototype Representation Distillation loss to maintain the learned forgery patterns based on the prototypical representation of samples, thereby effectively mitigating the catastrophic forgetting in the proposed DMP model.• We conduct extensive experiments to demonstrate the effectiveness of our method. The experimental results show that the proposed method surpasses existing incremental deepfake detection methods on multiple datasets. 
 
 Appendix:  


Decision: The paper presents the Dynamic Mixed-Prototype (DMP) Model for Incremental Deepfake Detection, designed to efficiently adapt to evolving deepfake technology by dynamically increasing prototypes to distinguish between real and fake facial media. The model utilizes multiple prototypes for each class and incorporates the Prototype-Guided Replay (PGR) strategy and Prototype Representation Distillation (PRD) loss to prevent knowledge forgetting. Testing across four datasets showed that the DMP model outperforms existing incremental deepfake detectors and demonstrates superior generalizability with limited deepfake samples.

Reviews: review: paper_summary: paper_summary: This paper titled "Dynamic Mixed-Prototype Model for Incremental Deepfake Detection" introduces the Dynamic Mixed-Prototype (DMP) model, which aims to address the challenges posed by the rapid advancement of deepfake technology. The paper highlights that while existing deepfake detectors show promising results on familiar deepfake types, their performance significantly degrades on novel deepfakes created by unseen algorithms. The DMP model dynamically increases prototypes to adapt to novel deepfakes efficiently. It employs multiple prototypes to represent both real and fake classes, enabling the learning of novel patterns by expanding prototypes and retaining knowledge from previous prototypes. The paper also proposes the Prototype-Guided Replay (PGR) strategy and Prototype Representation Distillation (PRD) loss to prevent forgetting learned knowledge. Extensive experiments demonstrate that the DMP model surpasses existing incremental deepfake detectors across multiple datasets, showing superior generalizability to novel deepfakes.
strength_weakness: strength_weakness: Novelty: The introduction of the Dynamic Mixed-Prototype (DMP) model is innovative. It addresses the limitations of existing deepfake detectors by dynamically adapting to novel deepfakes through the expansion of prototypes.
Theoretical Approach: The paper provides a robust theoretical foundation for the DMP model. The use of multiple prototypes to represent real and fake classes is a significant advancement over traditional methods.
Technical Correctness: The implementation of the Prototype-Guided Replay (PGR) strategy and Prototype Representation Distillation (PRD) loss is technically sound and well-justified.
Adequate Evaluation: The paper includes extensive experiments across four datasets, demonstrating the model's effectiveness and superior performance compared to existing methods.
Clarity: The paper is well-written and clearly explains the proposed methods, theoretical background, and experimental results.
Applications: The DMP model has practical applications in enhancing the robustness and generalizability of deepfake detection systems, which is crucial for maintaining public trust and safety.
limitations: limitations: Complexity: The introduction of multiple prototypes and the additional strategies (PGR and PRD) add complexity to the model, which may affect its scalability and ease of implementation.
Computational Overhead: The dynamic expansion of prototypes and replay strategies might introduce additional computational overhead, which could be a concern for real-time applications.
paper_summary: paper_summary: The paper is about a Dynamic Mixed-Prototype (DMP) Model for Incremental Deepfake
Detection. It addresses the challenge posed by the rapid advancement of deepfake technology,
which threatens social trust by creating highly convincing and realistic facial media that can be
used for malicious purposes. The paper introduces the DMP model, which dynamically increases
prototypes to adapt to novel deepfakes efficiently. This model uses multiple prototypes to
represent both real and fake classes, allowing it to learn new patterns by expanding prototypes
while retaining knowledge from previous prototypes.
The authors also propose two strategies to prevent forgetting learned knowledge: the Prototype-
Guided Replay (PGR) strategy and the Prototype Representation Distillation (PRD) loss.
These strategies are particularly effective for maintaining learned representations based on the
prototypical representation of samples.
The paper demonstrates that the DMP model, combined with the PGR strategy and PRD loss,
outperforms existing incremental deepfake detectors across four datasets and shows superior
generalizability to novel deepfakes through learning with limited deepfake samples.
strength_weakness: strength_weakness: The paper presents several strengths in its approach to incremental deepfake detection, which can
be highlighted as follows:
1. Novelty: The paper introduces a novel model that dynamically adapts to new deepfake
patterns by increasing prototypes. This approach is innovative as it moves beyond static
prototype representations to accommodate the evolving nature of deepfakes.
Prototype-Guided Replay (PGR) Strategy: This strategy is designed to replay samples that
are closest to the prototypes, which aids in stabilizing the learned representations and
mitigating catastrophic forgetting.
2. Theoretical Approach and Technical Correctness: The paper is theoretically sound, as it
builds upon the concept of prototypical networks, which are known for their effectiveness in
few-shot learning scenarios. The use of multiple prototypes for each class allows the model
to capture a richer set of forgery patterns.
The Prototype Representation Distillation (PRD) loss is a technically rigorous contribution
that aligns the model's representations across training stages, ensuring the stability of the
learned deepfake patterns.
3. Clarity: The paper is well-structured, with a clear introduction, detailed explanation of the
method, and a systematic presentation of the experimental results. This clarity aids in
understanding the contributions and the significance of the proposed model.
The use of figures, such as Figure 1, which contrasts the DMP model's approach with
previous methods, effectively illustrates the advantages of the proposed approach.
limitations: limitations: The paper introduces an incremental learning approach that aims to enhance the learning process
of existing detection models by incorporating a small number of new forged samples, thereby
effectively improving the model's detection generalization performance. However, there are
several issues that need further clarification in the paper:
1. It can be seen from Figure 4 that the curve does not appear to have reached its peak,
indicating that the current experimental setup may not fully reveal the potential of the
model. The ablation study presented in the figure does not adequately demonstrate the
impact of the additional samples on the model's performance. To more comprehensively
assess the model's performance, I suggest that the authors consider increasing the size of the
replay set and the number of training videos. Such improvements may help the curve to
reach a more pronounced local maximum, thereby providing a clearer performance
evaluation for the readers.
2. I have noticed that Table 2 compares the incremental learning model with existing
generalizable deepfake detection methods. To enhance the academic value and timeliness of
the paper, I suggest that the authors include at least one highly cited research work published
after 2023 in the comparison.
3. This paper presents an innovative Dynamic Mixed-Prototype model, aimed at enhancing the
application effects of incremental learning. To fully demonstrate the advantages of
incremental learning, it is suggested that the authors first present the generalization
performance of the model on cross-datasets before applying incremental learning in the
experimental section. Subsequently, the cross-dataset performance of the model on the same
datasets after incremental learning should be shown and compared. This will help to prove
that even with only a small amount of new forged method supervision data, the model can
significantly improve the performance of existing detection models.
4. In the incremental learning phase of the paper, the authors have employed a random
selection strategy to create an incremental training dataset consisting of 25 real videos and 25 fake videos. However, the impact of this random selection strategy on the stability of
detection performance is not adequately discussed in the paper. To provide a more
comprehensive assessment of the model's generalization capabilities and stability, it is
recommended that the authors conduct multiple validation experiments and report the mean
and variance of the results. This will help elucidate the effectiveness of improving the
model's generalization performance through the random selection of novel deepfake
supervised data in practical applications.
5. The DMP model has practical applications in scenarios where deepfake detectors need to be
continuously updated to handle new forgery techniques. Its incremental learning capability
makes it suitable for real-world deployment where access to novel deepfake samples might
be limited.
paper_summary: paper_summary: The Dynamic Mixed-Prototype (DMP) model addresses the challenge of detecting novel deepfakes by dynamically increasing prototypes to efficiently adapt to emerging forgery patterns. By incorporating Prototype-Guided Replay and Prototype Representation Distillation, it surpasses existing incremental detectors, demonstrating superior generalizability to novel deepfakes across multiple datasets.
strength_weakness: strength_weakness: The paper presents the novel DMP model, which expands prototypes dynamically during incremental learning to adapt to new forgery patterns from limited samples for incremental deepfake detection. 

The paper introduces a Prototype-Guided Replay strategy coupled with a Prototype Representation Distillation loss to maintain the learned forgery patterns based on the prototypical representation of samples.

The paper conducts extensive experiments to demonstrate the effectiveness of the method.
limitations: limitations: 1.  The paper emphasizes the use of cosine distance over Euclidean distance for classifying prototypes. However, this reliance might limit the flexibility of the model. If future deepfake patterns deviate significantly from the current distributions, it might be interesting to explore such experiments.

2. The DMP model, with its Prototype-Guided Replay (PGR) and Prototype Representation Distillation (PRD), involves maintaining and replaying a set of representative samples. As the diversity and volume of deepfake types grow, the storage and computational overhead for managing these prototypes and replay sets might become significant. Can be shown experimental results of some computation complexity?
paper_summary: paper_summary: This paper proposed a prototype-based model for incremental deepfake detection. The authors used multiple prototypes to represent one class. In addition to the traditional CE loss, the authors introduced a prototype-based knowledge distillation loss and they also used the prototypes to guide the replay set selection.
strength_weakness: strength_weakness: 1. The proposed method is technically sound and achieves good results on four benchmark datasets.
2. The authors performed various ablation studies to verify the effectiveness of their proposed components, such as the prototype-based replay set selection strategy and the impact of the prototype number.
3. The paper is generally well-written and easy to follow.
limitations: limitations: 1. Prototype-based models have been widely used in image classification for semi-supervised and incremental learning. The idea of prototype-based knowledge distillation is not new, so the novelty of this paper is limited to some extent. The authors applied existing techniques to a new application, i.e. incremental deepfake detection which is basically an image classification problem, with moderate modifications such as the replay set selection.
2. The results reported in Table 1 are generally promising. However, I noticed that compared to DFIL, the performance gain is small on FF++, DFDCP, and DFD, but more obvious on CDF. What is the reason that the proposed method works better on the last incremental learning stage? More insightful discussions are required to explain the advantages and limitations of the proposed method.
3. It is not very clear to me how to update the prototypes at each stage. After increasing the number of prototypes by K, do you only generate K new prototypes while keeping the past prototypes unchanged, or perform clustering from scratch to update all the prototypes?