Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an approach for generalizable Deepfake detection using a recently proposed one-shot test-time training strategy and a combination of meta learning. The approach is simple and a straightforward extension of the recently proposed test-time training framework, where a test-data sample itself is used to create a pseudo-training set, and the model parameters are updated. Experiments are presented on various datasets under different experimental settings.   

strengths_and_weaknesses: Strengths

This paper directly extends the test-time training (TTT) method from Efros’ group, to the Deepfake detection problem, and shows that this can be used for better generalization. Since the TTT method is recent, and has been well received, the authors have cleverly “struck the iron, when it’s hot” and have applied this recently proposed “hot” method to Deepfake detection.

The paper is clearly well and the methods are well explained, for the most part. 

Weakness

Visual examples are missing. Though the authors have done a good job in extending a recently proposed well received method to Deepfake detection, it doesn’t make sense why the authors have not included any good visual examples. Other than Figure 2 (which hardly illustrates the method), there are no visual images. Since the method takes a test image and blends with images from the training set using different blending methods, these can be easily visualized and thus provide better insights. 

A few experimental scenarios and details are missing. 


questions: The paper uses a face detector method (DLIB) to extract faces. There is not much information on how accurate this method is. The paper appears to assume that the face detection accuracy is 100%. Since face detection is critical to the proposed method, are there scenarios when the face detection algorithm can fail? Some discussion on this will help.

The paper mentions that the input face images are resized to a dimension of 256x256. It will be good to have a discussion on why this dimension is chosen, and if choosing a higher or lower dimension will have an impact on the performance of the proposed method. 

limitations: Lack of good visual examples is a big limitation (having which could have ended up in a higher rating). 

The paper uses ACC and AUC as evaluation metrics. It will be helpful to know how the proposed method works on both pristine and Deepfake images. Metrics like True positives, False positives, True Negatives and False Negatives could help here. Since only ACC and AUC are provided, this does not reflect how well this algorithm performs separately on pristine and Deepfake images. 

There is not much discussion on the computational complexity and time complexity of the proposed approach. Since the paper uses test-time training, a good practical system should also discuss the time and computational complexity at the test stage.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The major contributions of this paper is:
1) This work proposes a simple **One-Shot Test-Time training (OST)** framework to improve detection of Out-of-Distribution face forgery detection. OST obtains noticeable improvements in OOD detection.

strengths_and_weaknesses: **Strengths:**
1) This paper is written well. It is easy to follow.
2) The improvements in generalization are interesting and useful to the face forgery detection community.


**Weaknesses:**
1) Although the method gives improvement and I appreciate the authors' comprehensive experiments, these improvements are not surprising. I agree that it is different from other existing methods, but I feel that the contributions are limited. I.e.: Only Line 1 in Algorithm 1 is proposed in this work, Lines 2-4 are from existing meta-learning works.

2) For detecting CNN-generated images, there is a popular work that shows its possible to create a universal detector that generalizes to detecting CNN-generated images from unseen GANs with different architectures, datasets and loss functions (See [1]). This generalization is obtained by a simple threshold calibration with no gradient updates (See appendix Table 5 [1]) of this universal detector. Can the authors include some discussion / benchmark regarding this generalization [1]. 

3) Can the authors clarify as to why OST cannot be applied to GAN-synthesized images? I.e.:  The universal detector [1] generalizes to face forgery detection (See Table 2 for Cross-generator generalization results) although only trained using CNN-generated images.


Overall this is an interesting paper. In my opinion, the weaknesses of this paper outweigh the strengths. I’m willing to change my opinion based on the rebuttal. 


=====

[1] Wang, S. Y., Wang, O., Zhang, R., Owens, A., & Efros, A. A. (2020). CNN-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8695-8704).
questions: Please see Weaknesses section above for a list of questions. 
limitations: The authors have discussed the limitations / ethic statement in Section 6 (Main Paper).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies a Test-Time Training paradigm to improve the generalization ability of deepfake detection method to unseen forgery attacks. Specifically, for each coming test sample, the proposed method synthesizes a pseudo-training sample by blending the test samples with a randomly selected template image and finetunes the pretrained detector with it. To achieve a good initialization for test time training, this paper adopts the MAML-based meta-training to enable fast adaptation to different test samples. Experiments are carried out in several public deepfake datasets.
strengths_and_weaknesses: Strengths:

1. Studying Test-time Training to improve the generalization ability of deepfake detection is reasonable.

2. Generating pseudo-training samples in the proposed Online Test-Time Training is novel and is devised specifically for deepfake detection.

3. Evaluation is reasonably thorough and acceptable results are claimed.


Weaknesses：

1. Most of test time training/adaption works, such as Tent [1], emphasize its advantage of computationally efﬁciency and capability of fast adaptation. Comparatively, it seems very time-consuming in the proposed Online Test-Time Training as the proposed Generating pseudo-training samples involves facial alignment, mask refinement and facial region blending. It is hard to be carried out in an ‘online’ manner and more likely to be an offline pipeline. 

2. In line 136, the reviewer is not clear about how many x_r are sampled. It is likely to cause the problem of domain imbalance (domain here means template and synthesis domains) if many template images are sampled and only one synthesized image is generated.

3. Offline Meta-training is trivial and directly borrows the ideas of MAML.

4. Missing some SOTA test time training/adaption baselines, such as Tent [1].

[1] Tent: Fully Test-Time Adaptation by Entropy Minimization. ICLR 2021.

questions: 1. In line 136, how many x_r are sampled? Could the authors provide more details about the number or proportion of template images and the synthesized images in the One-shot online training?

2. What is the technical novelty of proposed Offline Meta-training?

3. How long does Online Test-Time Training take?

4. Why only perform a single-step gradient descent? How is the performance if the number of step gradient descent increases?

limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new method for generalized deepfake detection in which the testing has domain gap from the training. The proposed method is based on meta-learning on training set and one-shot test-time training on the synthesized pseudo samples. Extensive experiments show that the proposed method outperforms the state of the art. Ablation study proves the effectiveness of the proposed designs.
strengths_and_weaknesses: Strength:

1.The paper is well-written. It has good literature review which clearly shows the difference between this paper and related works.

2.The proposed method is interesting and well designed. The motivation is clear and reasonable. 

3.Extensive experiments with analysis are provided. The new method outperforms a number of state-of-the-art methods. 

4.The experiment details are complete.


Weaknesses:

I didn’t see obvious drawbacks. 

The authors can provide comparison to more SOTA methods in table 2. 

The paper can be further enhanced by illustrating the advantages brought by the proposed OST adaptation in visualized examples.

questions: Will the result differ a lot for using different x_r in OST? The authors can provide further analysis on it.
limitations: The authors have discussed the limitations. No potential negative societal impact.
review_title: null

Meta-review: metareview: The reviewers unanimously accept the paper, so is the final proposal. 
confidence: Certain
Decision: Accept