Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work introduces a novel DeepFake dataset containing both fake video and fake audio. Most prior DeepFake benchmarks do not contain fake audio; DFDC, one of the exceptions, contains fake audio with the video, but the dataset does not specify if the video is fake or the audio. The proposed dataset has collected a combination of $\left[ \text{real, fake video} \right] \times \left[ \text{real, fake audio} \right]$, and it is clearly specified whether the video or audio is fake. Fake audio is generated by an auto-encoding scheme with audio-text translation and TTS. Fake video is generated from a collection of common DeepFake methods. Prior datasets are compared against the proposed dataset using various detection methods, where the results suggest that FakeAVCeleb has a comparable complexity and difficulty against the recently published DeepFake datasets.
strengths: 1. The paper is well-written and organized. There is sufficient detail on data collection and processing procedures.
2. This is the first dataset with clearly labeled fake audio and videos. As malicious content can be created with just fake audio or video, the dataset can be useful to the forensics community.
3. The dataset is based on VoxCeleb, and one can expand the dataset more easily.
4. Face similarity matching using Face++, and manual inspection increases the quality of the dataset.
5. A comprehensive benchmarking is performed on the dataset. Comparisons against the prior DeepFake datasets are carried out, and the results suggest that FakeAVCeleb has a comparable complexity and difficulty against the recently published DeepFake datasets.
weaknesses: 1. The released dataset only contains cropped faces. It would be good to also release a version of the dataset with the uncropped video. This can be helpful because in-the-wild fakes are uncropped, and different DeepFake detection algorithms can use different face detection methods. Some potential methods might not even use cropped faces as inputs; for example, image splicing detection.
2. For the fake audio, will it be better to also include identity swaps? Currently, my understanding is that fake audio is generated by an auto-encoding process, so the fake audio contains the text and sound profile from the same person. Would it be possible to include audio with text from person B and sound profile from person A? For the case of (real video, fake audio) pair, it should be more reasonable to use the same sound profile as the person in the real video.
3. The labels in Figure 3 will need better descriptions. For example, result (A) in the middle column seems to be before lip-synced. Is result (A) included in the dataset?
correctness: Yes.
clarity: Yes, the paper is well written and easy to follow.
relation to prior work: Yes.
additional feedback: Some typos:

line 303: FakeAVCelebwith -> FakeAVCeleb with

line 305: FakeAVCelebis -> FakeAVCeleb is

line 328: FakeAVCelebis -> FakeAVCeleb is

line 330: indiidual -> individual

line 334: FakeAVCelebin -> FakeAVCeleb in

line 350: FakeAVCelebby -> FakeAVCeleb by

line 355: FakeAVCelebstill -> FakeAVCeleb still
review_title: A dataset with clearly labeled fake videos and fake audios

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presented a novel AV multimodal deepfake detection dataset, which will further the deepfake detection. This dataset has both audio and video with audio lip-synced with the video, unlike other multimodal dataset available. This paper also focused on removing bias unlike other publicly available dataset by choosing people from different ethnic backgrounds. The results from this paper indicate the need of similar multimodal datasets to further this area of research. Although they've released the dataset if they could've provided the data generator itself it would have been helpful for the people indulging in this area of research.
strengths: - Novelty in the dataset as it is a AV multimodal dataset
- Using publicly available and widely used datasets like VoxCeleb2 dataset for their data generation process
- Using latest  and sota methods for deepfake data generation
weaknesses: Using publicly available datasets and models comes with the downside of acquiring the limitations of those datasets into their own dataset such as bias, which they tried to address by making the dataset as diverse as they could.
correctness: Yes the claims made in the paper seems to correct except the calling of existing deepfake datasets racially biased for which they could've provided a study supporting it or named the specific datasets which a biased.
clarity: This paper has clearly describe the intend of the authors and what they were trying to achieve along with proper documentation and code for the general public understanding.
relation to prior work: This paper clearly furthers the previous efforts in this domain by creating a novel multimodal dataset.
additional feedback: If you could, in near future, provide the data generation pipeline for public use that will be a great contribution.
review_title: Better dataset for deepfake detection

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - The authors present a novel Audio-Video Multimodal Deepfake Detection dataset, which contains both, video and audio deepfakes with accurate lip-sync, and three different types of Audio-Video deepfakes
- The authors provide a comprehensive baseline benchmark evaluation and demonstrated the crucial need and usefulness for a multimodal deepfake dataset.
strengths: 1) The proposed Audio-Video Deepfake dataset (FakeAVCeleb) contains not only deepfake videos but also respective synthesized lip-synced fake audios, which is an interesting and vital task for security and privacy issues.
2) 
weaknesses: The reviewer was never involved in the related Deepfake field, thus only provides coarse insights.
1) Contribution/Comparison to prior datasets. The core of the work is the multimodal deepfake dataset with audio and video, but not much new insight in the experiments to support the contribution. Just using the existing DeepFake Detection Methods to obtain results on the proposed benchmark. 
2) The author provides coarse data statistics(e.g., 490+, 20,000) in Table 1. The detailed information for the dataset is important for a dataset paper.
correctness: Mostly correct.
clarity: Yes. The paper is well written.
relation to prior work: Yes. The paper clearly discusses the relation to prior works.
additional feedback: None.
review_title: A Novel Audio-Video Multimodal Deepfake Dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a new audio-video multimodal dataset, FakeAVCeleb, for Deepfake detection. The dataset contains three different types of audio-video forgeries on the combinations of video or audio manipulations (*i.e.*, real video & fake audio, fake video & real audio, fake video & fake audio) with accurate lip synchronization using existing popular Deepfake generation methods. The authors try to carefully select the source videos to make them diverse in ethnic backgrounds. They further benchmark seven baseline methods on FakeAVCeleb and several existing datasets for Deepfake detection.
strengths: - The potential misuse and harm of advanced deep learning-based manipulation approaches (*i.e.*, Deepfakes) is a pretty significant issue. The introduced dataset can serve as a good contribution to alleviate this problem. A multimodal dataset including fake audio manipulation is currently missing. This work tries to fill this gap, which is useful for academia and industry.

- The paper is generally clearly written, and the dataset construction information is provided in detail.

- Representative related studies have been appropriately cited and discussed.

- The authors provide benchmark results on various datasets, including the proposed FakeAVCeleb, using seven baseline methods. The results demonstrate the complexity and challenge of the proposed dataset.
weaknesses: - The benchmark verified the complexity and challenge of the proposed dataset. However, it would be better if the authors could actually examine the usefulness of FakeAVCeleb, e.g., providing statistics to show the improved performance of forgery detectors with the help of the additional multimodal information. The authors can make the background introduction more concise for additional space.

- Reporting the exact number of fake videos and total videos in FakeAVCeleb is important, and the number of each type in Figure 1 should also be stated clearly.

- Please add the citation for each dataset in Table 1. Besides, Google DFD also contains real videos of source actors [1], and the "Rights Cleared" of DeeperForensics should be Yes. In addition, it is better to add one column about whether containing fine-grained labeling of fake videos and audios to show the superiority of FakeAVCeleb over DFDC.

- In the benchmark, the baselines are a bit old. It would be better if the authors could provide results on more state-of-the-art baselines, such as Face X-ray [2], F3-Net [3], LipForensics [4], *etc*.

---
[1] https://github.com/ondyari/FaceForensics/tree/master/dataset#space-requirement

[2] Face X-ray for More General Face Forgery Detection.  In CVPR 2020.

[3] Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues. In ECCV 2020.

[4] Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection. In CVPR 2021.
correctness: Most of the claims in the submission are correct. Please refer to Weaknesses and Additional Feedback for some of my concerns.
clarity: The paper is generally clearly written and explained in detail. Some writing issues can be found in Weaknesses and Additional Feedback.
relation to prior work: The paper appropriately cited relevant studies and discussed how this work differs from previous contributions.
additional feedback: - The current version of FakeAVCeleb only contains videos on celebrities. Indeed, celebrities are the most common objects in Deepfakes. However, it would be better if the dataset could include more diverse objects (i.e., not only celebrities but ordinary people) in the future. Also, the languages of speaking and the fake audio generation methods can be further expanded.

- Why are existing Deepfake datasets racially biased? For example, DeeperForensics contains actors from 26 countries, covering four typical skin tones: white, black, yellow, brown.

- What about the lip-unsynced cases? Can they be recognized as fakes? The authors may discuss this.

- Line 95, 96 Typo: "UADF" -> "UADFV".

- Line 184, 303, 355 Typo: a missing space after "FakeAVCeleb".

- Line 322, 330 Typo: What are Figure 5 and Figure 6 in the main paper?

- Line 330 Typo: "indiidual" -> "individual".
review_title: A New Audio-Video Multimodal Dataset With Diverse Combinations of Manipulations for Deepfake Detection

Meta-review: The paper introduces a new dataset of DeepFake video, where each instance may have any combination of real or fake audio or video. Initial reviews were mixed, but the author responses and paper revisions were able to address all reviewer concerns and in the end all reviewers thought that the paper was above the bar for acceptance. Congratulations on having your paper accepted to the NeurIPS 2021 Datasets & Benchmarks Track!
Decision: Accept