AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-01 03:39:32

Abstract: The detection and localization of highly realistic deepfake audio-visual content are challenging even for the most advanced state-of-the-art methods. While most of the research efforts in this domain are focused on detecting high-quality deepfake images and videos, only a few works address the problem of the localization of small segments of audio-visual manipulations embedded in real videos. In this research, we emulate the process of such content generation and propose the AV-Deepfake1M dataset. The dataset contains content-driven (i) video manipulations, (ii) audio manipulations, and (iii) audio-visual manipulations for more than 2K subjects resulting in a total of more than 1M videos. The paper provides a thorough description of the proposed data generation pipeline accompanied by a rigorous analysis of the quality of the generated data. The comprehensive benchmark of the proposed dataset utilizing state-of-the-art deepfake detection and localization methods indicates a significant drop in performance compared to previous datasets. The proposed dataset will play a vital role in building the next-generation deepfake localization methods. The dataset and associated code will be made public.

Main paper: 
 introduction: 
 We are witnessing rapid progress in the domain of content generation technology, i.e., models trained on massive amounts of data that can produce highly realistic text [3, 51, 52] , video [18, 49, 59] and audio [27, 28, 45] . Consequently, discriminating between real and fake content is becoming increasingly more challenging even for humans [38, 67] . This opens the door for misuse of content generation technology for example to spread misinformation and commit fraud, rendering the development of reliable detection methods vital.The development of such methods is highly dependent on the available deepfake benchmark datasets, which led to the steady increase in the number of publicly available datasets that provide examples of visual-only [26, 33, 36] , audio-only [37, 62] , and audiovisual [29] content modification strategies (e.g., face-swapping, facereenactment, etc.). However, the majority of these datasets and methods assume that the entirety of the content (i.e., audio, visual, audiovisual) is either real or fake. This leaves the door open for criminals to exploit the embedding of small segments of manipulations in the otherwise real content. As argued in [6] , this type of targeted manipulation can lead to drastic changes in the underlying meaning as illustrated in Figure 1 . Given that most deepfake benchmark datasets do not include this new type of manipulation strategy, state-of-the-art [44] 2019 Cla V FS/RE -1,000 4,000 5,000 Google DFD [39] 2019 Cla V FS 5 363 3,068 3,431 DFDC [16] 2020 Cla AV FS 960 23,654 104,500 128,154 DeeperForensics [26] 2020 Cla V FS 100 50,000 10,000 60,000 Celeb-DF [36] 2020 Cla V FS 59 590 5,639 6,229 WildDeepfake [68] 2020 Cla ---3,805 3,509 7,314 FFIW 10𝐾 [67] 2021 Cla/SL V FS -10,000 10,000 20,000 KoDF [33] 2021 Cla V FS/RE 403 62,166 175,776 237,942 FakeAVCeleb [29] 2021 Cla AV RE 600+ 570 25,000+ 25,500+ ForgeryNet [21] 2021 SL/TL/Cla V Random FS/RE 5,400+ 99,630 121,617 221,247 ASVSpoof2021DF [37] detection methods might fail to perform reliably on this new type of deepfake content. This work addresses this gap by releasing a new large-scale audiovisual dataset called AV-Deepfake1M specifically designed for the task of temporal deepfake localization. To improve the realism and quality of generated content, the proposed data generation pipeline incorporates the ChatGPT 1 large language model. The pipeline further utilizes the latest open-source state-of-the-art methods for highquality audio [8, 31] and video [54] generation. The scale and novel modification strategies position the proposed dataset as the most comprehensive audio-visual benchmark as illustrated in Figure 1 , making it an important asset for building the next generation of deepfake localization methods. The main contributions of this work are,• We propose AV-Deepfake1M, a large-scale content-driven audio-visual dataset for the task of temporal deepfake localization.• We propose an elaborate data generation pipeline employing novel manipulation strategies and incorporating the state-ofthe-art in text, video and audio generation. • We perform comprehensive analysis and benchmark of the proposed dataset utilizing state-of-the-art deepfake detection and localization methods. 
related work: 
 The performance of any deepfake detection method is highly dependent on the quantitative and qualitative aspects of the datasets used for development. Over the past few years, many datasets (e.g., [21, 32, 38] ) have been proposed to support the research on deepfake detection. A comprehensive list of the relevant publicly available datasets is given in Table 1 . Most of the available datasets provide examples of face manipulations through either face swapping [16, 32, 67] or face reenactment [29, 33] techniques. In terms of the number of samples, earlier datasets are smaller due to the 1 https://chat.openai.com/ limited availability of face manipulation techniques. With the rapid advancements in content generation technology, several large-scale datasets such as DFDC [16] , DeeperForensics [26] , KoDF [33] , and DF-Platter [38] have been proposed. However, the task associated with these datasets is mainly restricted to coarse-level deepfake detection. Until this point manipulations are mainly applied only to the visual modality, and later, audio manipulations [37] and audio-visual manipulations [29] have been proposed to increase the complexity of the task.In 2022, LAV-DF [6] was introduced to become the first contentdriven deepfake dataset for temporal localization. However, the quality and scale of LAV-DF are limited, and the state-of-the-art methods designed for temporal localization [4, 65] are already achieving very strong performance. AV-Deepfake1M addresses these gaps by improving the quality, diversity, and scale of the previous datasets designed for temporal deepfake localization. Given that LAV-DF is the only available public dataset that has been designed for the same task as the dataset proposed in this paper, next we do a direct comparison of the two datasets. In addition to the fact that AV-Deepfake1M is significantly larger than LAV-DF, in terms of the number of subjects, and amount of real and fake videos, the following differences further highlight our contributions.• LAV-DF uses a rule-based system to find antonyms that maximize the change in sentiment in the transcript manipulation step. We argue that naively choosing the antonyms causes context inconsistencies and low diversity of the fake content. AV-Deepfake1M addresses this issue with the use of a large language model, which results in diverse and contextconsistent fake content. • The output quality of the visual generator Wav2Lip [42] and audio generator SV2TTS [25] used for generating LAV-DF is not sufficient for state-of-the-art detection methods. AV-Deepfake1M utilizes the latest open-source state-of-the-art methods for high-quality audio and video generation. [{"operation": "replace", "old_word": "great", "new_word": "terrible", "index": 4}, {"operation": "delete", "old_word": "not", "new_word": None, "index": 17}, {"operation": "insert", "old_word": None, "new_word": "not", "index": 24}] Insert Delete Fake Frames Fake Audio Figure 2 : Data manipulation and generation pipeline. Overview of the proposed three-stage pipeline. Given a real video, the pre-processing consists of audio extraction via FFmpeg followed by Whisper-based transcript generation. In the first stage, transcript manipulation, the original transcript is modified through word-level insertions, deletions, and replacements. In the second stage, audio generation, based on the relevant transcript manipulation, the audio is generated in both speaker-dependent and independent fashion. In the final stage, video generation, based on the generated audio, the subject-dependant video is generated with smooth transitions in terms of lip-synchronization, pose, and other relevant attributes.• LAV-DF includes only replacement as a manipulation strategy. AV-Deepfake1M includes two additional challenging manipulation strategies, deletion and insertion. 
av-deepfake1m dataset: 
 AV-Deepfake1M is a large-scale audio-visual deepfake dataset, including 1,886 hours of audio-visual data from 2,068 unique subjects captured in diverse background environments. This positions the proposed dataset as the most comprehensive audio-visual benchmark as illustrated in Figure 1 and Table 1 . The generated videos in AV-Deepfake1M preserve the background and identity present in the real videos, while the content is carefully manipulated with contentdriven audio-visual data. Following previous deepfake dataset generation research [6, 29] , the dataset includes three different combinations of modified modalities in the generated fake videos. Please note that here we also introduce the concept of content-driven modifications for unimodal as well as multimodal aspects. We further elaborate on this in the supplementary material.• Fake Audio and Fake Visual. Both the real audio and visual frames are manipulated. • Fake Audio and Real Visual. Only the real audio corresponding to replacements and deletions is manipulated. To further increase data quality, the fake audio, and the corresponding length-normalized real visual segments are synchronized. As for the insertions, new visual segments are generated based on the length of the fake audio and are lip-synced to the background noise (i.e., closed mouth).• Real Audio and Fake Visual. Only the real visual frames corresponding to replacements and deletions are manipulated.To further increase data quality, the length of the fake visual segments is normalized to match the length of the real audio.As for the insertions, background noise is inserted for the corresponding fake visual segments.The three-stage pipeline for generating content-driven deepfakes is illustrated in Figure 2 . A subset of real videos from the Voxceleb2 [14] dataset is pre-processed to extract the audio using FFmpeg [50] , followed by Whisper-based [43] real transcript generation.Manipulation Strategy. The first stage for generating content-driven deepfakes is transcript manipulation. We utilize ChatGPT for altering the real transcripts. Through LangChain [9] the output of ChatGPT is a structured JSON with four fields: 1) operation: This set contains replace, delete, and insert, which has been applied on the input; 2) old_word: The word in the input to replace or delete; 3) new_word: The word in the input to insert or replace; 4) index:The location of the operation in the input. The number of transcript modifications depends on the video length and is determined by the following equation 𝑀 = ceil(𝑡/10) where 𝑀 is the number of modifications and 𝑡 (sec) is the length of the video. We followed [3] and built a few-shot prompt template for ChatGPT. System: You are a helpful text modifier. Your target is to modify the provided text to invert its meaning to the opposite direction. Here is the transcript of the audio.Please use the provided operations to modify the transcript to change its meaning.The operation can be one of "delete", "insert" and "replace". Analysis. Figure 3 (a) illustrates a comparison of the frequencies of the top 20 words in AV-Deepfake1M and LAV-DF [6] . The results show that few words in LAV-DF have dominant frequencies (> 10%), whereas this issue is drastically reduced in AV-Deepfake1M.HumanOwing to the contribution of ChatGPT, we also observed a significant increase in unique new words (27.7 times more) in the modified transcripts compared to LAV-DF, illustrated in Figure 3 (b) . This statistical comparison shows that the proposed LLM-based transcript manipulation strategy generates more diverse content compared to the rule-based strategy employed in LAV-DF. We further elaborate on the advantages of using an LLM in this step in the supplementary material.Manipulation Strategy. The next stage is to generate high-quality audio with the same style as the speaker. The audio is first separated into background noise and speech using Denoiser [17] . Zero-shot voice cloning methods such as SV2TTS [25] utilized by previous datasets [6, 29] have low signal-to-noise ratio resulting in low-quality audio manipulations that are easily localized by BA-TFD [4] and UMMAFormer [65] . To increase the quality of the generated audio, we employ the identity-dependent text-to-speech method VITS [31] for a subset of the subjects. Further diversity in the audio generation was introduced by utilizing the identity-independent text-to-speech method YourTTS [8] for the rest of the subjects. Audio generation is slightly different for each of the manipulation strategies (i.e., replace, insert and delete). In the case of replace and insert, we need to generate new audio corresponding to new_word(s). Generally, there are two ways to generate the new_word(s): 1) Generate audio for the final fake transcript and crop it to get the audio for the required new_word(s) and 2) Generate audio only for the new_word(s). To bring further diversity and challenge, we use both strategies to generate audio for the new_word(s). In the case of delete, only the background noise is retained. After the audio manipulation, we normalized the loudness of the fake audio segments to the original audio to add more realism. Finally, to keep the consistency with the environmental noise, we add the background noise previously separated to the final audio output.Analysis. We evaluated the quality of the audio generation following previous works [7, 11] (note that for all datasets, we only evaluated the samples where the audio modality is modified). The results are shown in Table 2 . The first evaluation metric is speaker encoder cosine similarity (SECS) [53] . It measures the similarity of the speakers given a pair of audio in the range [-1, 1]. We also calculated the signal-to-noise ratio (SNR) for all fake audio and Fréchet audio distance (FAD) [30] . The results indicate that AV-Deepfake1M contains higher quality audio compared to other datasets.Manipulation Strategy. The final stage of the generation pipeline is visual content generation. After the audio is generated, the lipsynced visual frames are generated based on the subjects' original pose and the fake audio. We investigated several face reenactment strategies including EAMM [24] , AVFR-GAN [2] , DiffTalk [46] , AD-NeRF [19] and ATVGnet [10] and concluded that these methods are not well suited for zero-shot lip-synced generation of unseen speakers. Thus, we use TalkLip [54] for visual content generation which is primarily designed for zero-shot lip-sync scenarios. LipTalk is 1) Identity-independent, 2) Lip-syncing only without generating new poses, 3) Fast, 4) State-of-the-art, and 5) Open-source. This way we avoid the weaknesses of the aforementioned face reenactment strategies. The pre-trained TalkLip model is used to generate fake visual frames that are lip-synchronized with the input audio and can be used for insertion, replacement, and deletion. Analysis. To evaluate the visual quality of the proposed dataset, we used peak signal-to-noise ratio (PSNR), structural similarity The left three-row three-column histograms illustrate the fake segment absolute lengths (sec), the fake segment lengths proportion in videos (%) and the video lengths (sec) in the train, validation, and test sets. In the middle, the histograms illustrate the overall statistics for fake segment lengths, proportions and video lengths, compared with LAV-DF. For the fake segment lengths and proportions, the X-axis is in log scale and for video lengths, the X-axis is in linear scale. For all histograms, the Y-axis is in linear scale. The vertical dotted lines and numbers in histograms represent the mean value. On the right side, (a) The number of segments with different modifications and (b) The number of videos with different numbers of segments per video. index (SSIM) [58] and Fréchet inception distance (FID) [23] metrics as shown in Table 3 . Note that for a fair comparison, we preprocessed the videos to a common format. The videos of FF++ [44] and DFDC [16] are 'in-the-wild', whereas FakeAVCeleb [29] , LAV-DF [6] and AV-Deepfake1M are facial videos. Thus, we cropped the facial region for FF++ and DFDC for visual quality assessment. Since FakeAVCeleb, LAV-DF and AV-Deepfake1M are multimodal, for a fair comparison, we only used the samples with the visual modality modified to compute the metrics. The results indicate that AV-Deepfake1M is of higher visual quality compared to existing datasets.We split the dataset into train, validation, and test sets. We first randomly select 1,657 subjects for the train set and 411 subjects for the test set without any overlap. The validation set is selected randomly from the train subset. The test set contains only samples with VITS-based identity-dependent audio. The variation in the number of subjects and videos in different sets is presented in Table 4 and Figure 4 . Figure 5 illustrates the direct comparison of AV-Deepfake1M and LAV-DF [6] . The results indicate that AV-Deepfake1M is more diverse in terms of modifications, subjects, fake segment and video lengths, and a lower average proportion of fake segments, making the dataset a vital asset for building better deepfake localization methods. To investigate if humans can detect the deepfakes in AV-Deepfake1M, we also conducted a user study with 25 participants with prior experience in video manipulation in the computer vision domain (note that the authors did not participate in the study). 200 random samples that contain 0 or 1 modification were selected for the study, where 100 from LAV-DF and 100 from AV-Deepfake1M. Each participant was asked to classify 20 videos (5 real and 5 fake from LAV-DF dataset, 5 real and 5 fake from AV-Deepfake1M) as real or fake and propose the potential fake segment start and end point. The user study results presented in Table 5 indicate that the deepfake content in AV-Deepfake1M is very challenging to detect for humans, and AV-Deepfake1M is more difficult than LAV-DF.We spent around ∼600 GPU hours for speech recognition with Whisper [43] , ∼2100 GPU hours for training VITS [31] (each of the 721 VITS models requires ∼3hrs), and ∼300 GPU hours for data generation. Overall, we needed ∼3000 GPU hours to generate AV-Deepfake1M with NVIDIA RTX6000 GPUs. 
benchmarks and metrics: 
 This section outlines the benchmark protocol for AV-Deepfake1M along with the used evaluation metrics. The goal is to detect and localize content-driven audio, visual, and audio-visual manipulations.The dataset is organized in train, validation, and test sets, as described in Section 3.2. The original test set (all modifications) is referred to as fullset in the rest of the text. For a fair comparison with visual-only and audio-only methods, we also prepared subset V (by excluding the videos with audio-only modifications from fullset) and subset A (by excluding the videos with visual-only modifications from fullset).For benchmarking temporal deepfake localization, we consider the following state-of-the-art methods: Pyannote [41] is a pre-trained speaker diarization method. TriDet [47] and ActionFormer [63] are the state-of-the-art in the temporal action localization domain. Since these two methods require pre-trained features, we extracted the state-of-the-art features VideoMAEv2 [56] and InternVideo [57] for benchmarking. BA-TFD [6] , BA-TFD+ [4] , and UMMAFormer [65] are the state-of-the-art methods specifically designed for audiovisual temporal deepfake localization. We followed the original settings for BA-TFD and BA-TFD+. For UMMAFormer [65] , we implemented it using the InternVideo [57] visual features and BYOL-A [40] audio features. For image-based classification methods, we consider Meso4 [1] , MesoInception4 [1] , Xception [12] , Face X-Ray [34] , LipForensics [20] , EfficientViT [15] , and SBI [48]. We followed the procedure used in previous works [4, 66] to aggregate the frame-level predictions to segments for localization.For benchmarking deepfake detection, we trained the image-based models Meso4 [1] , MesoInception4 [1] , Xception [12] and Effi-cientViT [15] with video frames along with the corresponding labels. For the segment-based methods MDS [13] and MARLIN [5] , we used a sliding window to sample segments from the video for training and inference. During the inference stage, the frame-and segmentlevel predictions are aggregated to video-level by max voting. The aggregation strategy is discussed in Section 5. We also evaluated the zero-shot performance of several methods, including the LLM-based Video-LLaMA [64] , audio pre-trained CLAP [60] , M2TR [55] and LipForensics [20] pre-trained on FF++ [44] , Face X-Ray [34] and SBI [48] pretrained on blending images. For Video-LLaMA, we also evaluated 5 model ensembles (the majority vote of 5 model inferences). To investigate the impact of the level of label access, we designed 3 different label access levels for training: frame-level labels, segment-level labels only, and video-level labels only.Temporal Deepfake Localization. We use average precision (AP) and average recall (AR) as prior works [6, 21] . Deepfake Detection. We use the standard evaluation protocol [16, 44] and report video-level accuracy (Acc.) and area under the curve (AUC). 
results and analysis: 
 This section reports the performance of the state-of-the-art deepfake detection and localization methods described in Section 4.2 on AV-Deepfake1M. The reported performance is based on different subsets, described in Section 4.1, and different levels of label access during training, described in Section 4.2.The results of this benchmark are depicted in Table 6 . All state-of-theart methods achieve significantly lower performance compared to the performance reported on previous datasets [6, 21] . This significant drop indicates that existing temporal deepfake localization methods are falling behind with the rapid advancements in content generation. In other words, we can claim that the highly realistic fake content in AV-Deepfake1M will open an avenue for further research on temporal deepfake localization methods.Similarly to temporal deepfake localization, the results of the classical deepfake detection benchmark are summarized in Table 7 .Table 6 : Temporal deepfake localization benchmark. Performance comparison of state-of-the-art methods on the proposed AV-Deepfake1M dataset. The results are significantly low, indicating that AV-Deepfake1M is an important benchmark for this task.Mod. AP@0.5 AP@0.75 AP@0.9 AP@0.95 AR@50 AR@30 AR@20 AR@10 AR@5 Fullset PyAnnote (Zero-Shot) [ Models that have access only to the video-level labels during training and the zero-shot models all perform poorly on this task, except the Face X-Ray and SBI which are designed to be generalizable. Providing the fine-grained segment-level and frame-level labels during training brings an improvement in performance. However, even with the frame-level labels provided during training, the AUC of the best-performing methods is less than 70, due to the multimodal modifications present in AV-Deepfake1M.The frame-and segment-based deepfake detection methods can only produce frame-and segment-level predictions. Thus, a suitable aggregation strategy is required to generate the video-level predictions. We investigated several popular aggregation strategies, such as max (e.g., [6] ), average (e.g., [15, 22, 55] ), and the average of the highest 5 scores (e.g., [35] ) for video-level predictions. The results of the experiment are presented in Table 9 . The results show that max is the optimal aggregation strategy on AV-Deepfake1M for the considered deepfake detection methods.We also evaluated the performance on subset V and subset A, as described in Section 4.1. As expected, all visual-only methods consistently perform better on subset V compared to fullset for both temporal localization and detection. The same holds for subset A and audio-only methods.We conducted additional experiments (Tables 8 and 10 ) to compare the performance on temporal localization and classification on AV-Deepfake1M and LAV-DF [6] .There is a significant drop in BA-TFD [6] temporal localization performance as compared to LAV-DF (Table 8 ). A similar pattern is also observed for BA-TFD+ [4] (AP@0.5 96.30 → 44.42) and UMMAFormer [65] (AP@0.5 98.83 → 51.64). For classification (Table 10 ), the performance of Xception [12] , LipForensics [20] , Face X-Ray [34] , and SBI [48] also drops compared to LAV-DF. These additional results further validate that AV-Deepfake1M is more challenging than LAV-DF.We conduct the experiments using Xception and BA-TFD pretrained on AV-Deepfake1M then finetune and evaluate on LAV-DF, shown in Table 11 . We observe the performance improvements are significant for both temporal localization with BA-TFD and classification with Xception, when compared with models trained on LAV-DF from scratch. 
conclusion: 
 This paper presents AV-Deepfake1M, the largest audio-visual dataset for temporal deepfake localization. The comprehensive benchmark important asset for building the next-generation of deepfake localization methods.Limitations. Similarly to other deepfake datasets, AV-Deepfake1M exhibits a misbalance in terms of the number of fake and real videos. Broader Impact. Owing to the diversified and realistic, contentdriven fake videos, AV-Deepfake1M will support the development of robust, generalized, audio-visual deepfake detection and localization models.Ethics Statement. We acknowledge that AV-Deepfake1M may raise ethical concerns such as the potential misuse of facial videos of celebrities, and even the data generation pipeline could have a potential negative impact. Misuse could include the creation of deepfake videos or other forms of exploitation. To avoid such issues, we have taken several measures such as distributing the data with a proper end-user license agreement, where we will impose certain restrictions on the usage of the data, such as the data generation technology and resulting content being restricted to research purposes only.  
 
 Appendix:  


Decision: All the reviewers are recommending that the paper be accepted. I agree. It does not seem to me to be of sufficient quality to recommend oral presentation, but definitely merits a poster.

Reviews: review: paper_summary: paper_summary: This paper proposes the AV-Deepfake1M dataset that contains content-driven (i) video manipulations, (ii) audio manipulations, and iii) audio-visual manipulations for more than 2K subjects resulting in a total of more than 1M videos. The paper provides a thorough description of the proposed data generation pipeline accompanied by a rigorous analysis of the quality of the generated data. The comprehensive benchmark of the proposed dataset utilizing state-of-the-art deepfake detection and localization methods indicates a significant drop in performance compared to previous datasets. The proposed dataset will play a vital role in building the next-generation deepfake localization methods.
strength_weakness: strength_weakness: The proposed AV-Deepfake1M dataset has a significant advantage over the previous dataset LAV-DF in terms of the number of subjects and amount of real and fake videos. Besides, it utilizes LLM to increase content diversity and employs more advanced methods for audio and video generation. Statistical characterization and experimental results demonstrate that this dataset outperforms its predecessor, LAV-DF, in many respects.
limitations: limitations: - The authors can use more TTS models instead of just 2.
- The word "LLM-Driven" only appears once in the title, while in the text "content-driven" is used. It seems better to replace LLM-Driven with content-driven in the title.
paper_summary: paper_summary: To address  the problem of the localization of small segments of audio-visual manipulations, this paper emulates the process of such content generation and propose the AV-Deepfake1M dataset, including more than 2K subjects, and a total of more than 1M videos.
strength_weakness: strength_weakness: This paper provides a thorough description of the proposed data generation pipeline accompanied by a rigorous analysis of the quality of the generated data. Then, existing deepfake detection and localization methods are evaluated and show that their performances are all a significant drop.
limitations: limitations: (1)How to ensure consistency between audio and video in a synthesized audio-visual dataset? (2) The comprehensive description of multimodal Deepfake forensics methods is incomplete.
paper_summary: paper_summary: In this paper, the authors construct a large-scale deepfake dataset based on RE and TTS approaches. In particular, they first modify the transcript of videos with word replacement, and then manipulate the lip movements to align the content. The generated videos are high-quality, with partial manipulation, making detecting these videos a challenging task. The authors conduct extensive benchmark tests on this dataset.

In general, this paper has some merits on deepfake community. Therefore, the reviewer give a positive rating.
strength_weakness: strength_weakness: [+] The proposed dataset is large-scale and high-quality.
[+] The authors conduct extensive experiments on this dataset.
limitations: limitations: [-] Data coverage. Is it appropriate for this dataset to completely disregard FS-related technologies? The reviewer understands that the manipulations discussed in this paper need to ensure the consistency of identity in videos. However, FS, as a potentially dangerous method (especially in video calls), should not be completely ignored. Is it possible to consider applying modified transcripts to some high-quality face-swapped videos to construct other forged subsets?

[-] Assessment of data quality. The videos in the dataset that have only one modality modified, i.e., the audio-modified and video-modified subsets, should exhibit semantic conflicts between the transcripts obtained from speech recognition (e.g., Whisper) and lip reading (e.g., avsr) methods. A higher rate of such conflicts is preferable for these subsets. Conversely, for modifications involving multiple modalities, this rate of conflict should be as low as possible. The authors are advised to include an evaluation metric for this aspect.
paper_summary: paper_summary: The researchers present a large-scale temporal deepfake localization task dataset called AV-Deepfake1M.
strength_weakness: strength_weakness: 1、The writing in this article is smooth and easy to understand.  
2、From the writing of this paper, the work in this paper provides the largest and most challenging dataset to date on the Audio-Visual Deepfake task. It has the potential to contribute to the further development of deepfake localization methods.  
3、This paper performs extensive benchmarking on the AV-Deepfake1M dataset.
limitations: limitations: The motivation for this article is unknown. In the introductory section of the article, the researcher claims that most of the existing deepfake benchmark datasets do not include this novel manipulation strategy, so the researcher releases a new large-scale audiovisual dataset, AV-Deepfake1M, to address this gap. And as far as we know, [1] already includes this strategy, has [1] addressed this issue? Admittedly, we might consider this a writing problem.  

[1] Z. Cai, K. Stefanov, A. Dhall and M. Hayat, "Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization," 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, Australia, 2022, pp. 1-10, doi: 10.1109/DICTA56598.2022.10034605.