Title: Fair Deepfake Detectors Can Generalize
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deepfake Detection; Causal Inference; Generalization; Fairness; Back-door Adjustment
Abstract: Deepfake detection models face two critical challenges: generalization to unseen manipulations and demographic fairness among population groups. However, existing approaches often demonstrate that these two objectives are inherently conflicting, revealing a trade-off between them.
In this paper, we, for the first time, uncover and formally define a causal relationship between fairness and generalization.
Building on the back-door adjustment, we show that controlling for confounders (data distribution and model capacity) enables improved generalization via fairness interventions.
Motivated by this insight, we propose Demographic Attribute-insensitive Intervention Detection (DAID), a plug-and-play framework composed of: i) Demographic-aware data rebalancing, which employs inverse-propensity weighting and subgroup-wise feature normalization to neutralize distributional biases; and ii) Demographic-agnostic feature aggregation, which uses a novel alignment loss to suppress sensitive-attribute signals.
Across three cross-domain benchmarks, DAID consistently achieves superior performance in both fairness and generalization compared to several state-of-the-art detectors, validating both its theoretical foundation and practical effectiveness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission argues that when model capacity and data distribution are controlled for, improvements to the fairness of a deepfake detector lead to improvements in their generalizability. They provide theoretical arguments for this claim based on the back-door criterion, and develop a training regime which attempts to counteract these confounders. Their results show that their technique achieves SOTA fairness and generalizability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The writing is very clear.\n\n2. The proposed method achieves state of the art on both fairness and generalizability.\n\n3. The ablation study at the end is very comprehensive, and provides an exhaustive analysis of their proposal’s contributions and impact.\n\nWeaknesses:\n\nI am not fully convinced of the theoretical results. In particular:\n\n1. The backdoor criterion is defined with respect to a causal graph. Therefore, the theoretical results are built on the assumption that data distribution (DD) and model capacity (MC) are an exhaustive set of confounders. If a different researcher assumed a larger causal graph with more confounders, then the back-door adjustment formula, Eq (2), would not be correct (i.e. it would require more marginalization). \n\n2. While the DD variable is called “data distribution”, but my understanding of lines 151-155 is that it refers to the proportion of the data set belonging to each sensitive attribute. There are many other properties of the data distribution that can impact fairness, e.g. correlation between sensitive attribute and spurious correlations such as clothing, picture setting etc. This broader definition of data distribution is implied in the latent normalization of Sec. 3.3, and if it is believed to be a confounding variable should be added to the DAG and marginalized out in the causal analysis.\n\n3. Lines 158-161 seem to depart from the theory that precedes it. Rather than defining the do intervention as a successful flipping of F from 0 to 1, it is defined in terms of training with or without a resampling strategy. Without understanding the relationship this has to the eventual F (which is not rigorously defined in the main paper as the threshold between low and high fairness is not established) it is hard to accept the numeric conclusions of the analysis.\n\nIn conclusion, I remain unconvinced of the claimed causal relationship. I am, however, convinced of a weaker claim: when DD and MC are controlled for, improvements in fairness lead to improvements in test AUC in the setting studied. This weaker claim is enough to motivate experimental choices, but weakens some of the stated contributions.\n\nIn general, this paper is very well written and thorough in its examination of its theoretical results. I lack the confidence to assert whether the primary experiment is lacking baselines or data sets/settings, but based on the results shown the improvements to generalizability and fairness are clearly-important contributions.\n\nSmall notes:\n\n“it worth” line 212\n\nLines 200-209 are a bit confusing without the context of Eq (12), which comes later. e.g. on line 200, you state “We enforce” which is a confusing construction. What is being enforced? It might be more appropriate to say “we define”, or make clear what you are enforcing wrt L_attr.\n\n\n\n**EDIT BASED ON DISCUSSION WITH AC**\n\nThis submission has a great deal of merit but does not adequately establish the causal relationship claimed in the abstract and introduction. In particular\n\n- The back-door criterion only establishes a causal relationship with respect to an underlying DAG, i.e. it assumes the DAG is correct. It remains unproven that there are not other confounders.\n- In particular, dataset reweighting (lines 158-161) certainly has a positive effect on fairness, but it is not exactly a do intervention. It affects something upstream of fairness, and could have unexplored confounding effects.\n\nI think the emphasis should be shifted to instead focus on the fact that, when model capacity and data distribution are controlled for, fairness-promoting interventions also improve generalization. This is a sufficient observation to motivate the experimental sections of this work.\n\nThe broad framework of causal inference and back-door criterion does not need to be scrapped to perform this change-in-emphasis; it can be done by softening some of the claims. For example, I suggest text to the effect of \"we investigate two possible confounders (data distribution and model capacity) and construct the resulting causal graph. Using the back-door criterion, we demonstrate a causal relationship between fairness and generalization under this causal model\". That way, the framework used in Sec 3 can be retained, but the insufficiently-substantiated claim that the authors demonstrate a (general) causal relationship between fairness and generalization is replaced by a claim that limits the scope to the DAG in question.\n\nQUESTIONS:\na. Am I correct in understanding that the latent normalization is just performed once, as opposed to e.g. at every hidden layer of a neural network architecture?\n\nb. Can you explain the DD stratification going on in lines 151-155? (related to weakness 2).\n\nc. In Fig. 3 you show the relationship between hyperparameters (meant to improve fairness) and resulting generalizability. Would you be able to provide the relationship between the hyperparameters and the resulting fairness (skew) too?\n\nLIMITATIONS:\nYes, with the exception of potentially-unstated or understated key assumptions of their theoretical results that limit the scope of their causality claim.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method, namely DAID, for distinguishing between real and deepfake-generated images, while enforcing algorithmic fairness. The main contribution lies in framing the problem through a causal lens; the authors argue that the causal effect of fairness on generalization performance can be confounded by two observable confounders: (i) the distribution of sensitive attributes (e.g., gender, race) in the dataset, and (ii) the complexity of the predictive model. To address this, the paper investigates whether conditioning on these two confounders reveals a causal effect of fairness on generalization error. To this end, the proposed approach incorporates a loss function that (i) reweights data points based on group membership and (ii) includes a regularization term that enforces demographic invariance in the latent representations. The effectiveness of DAID is validated through empirical experiments on real-world datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** \n\n1. The paper is well-written and easy to read. \n2. The motivation for the problem is clear. \n\n\n**Weakness** \n\nWhile the paper brings together several components relevant to enforcing fairness, such as reweighting samples and encouraging latent representations of similar images (real or deepfake) to be close, I am a bit concerned about the novelty. From my perspective, the primary contribution appears to lie in combining two existing strands of fairness approaches in an attempt to improve performance, rather than introducing a fundamentally new methodological insight. Below, I outline my main concerns in more detail:\n\n1. The idea behind the proposed $L_{\\rm attr}$ loss appears closely related to the FACE method introduced in [1] for enforcing individual fairness. However, the motivation for introducing this loss is not clear; to my understanding, the goal of this loss function is to ensure all latent representation of same $Y$ should be close to each other irrespective of their sensitive attributes, but it not clear what type of fairness/model complexity normalization it would bring. I recommend that the authors include a discussion situating their approach within the broader landscape of fairness definitions, such as demographic parity, equalized odds, or individual fairness. This would help make its fairness objectives more transparent. \n\n2. The description of observations is not clear to me. What is $X$ and $Y$? Does $Y  = 1$ mean real or fake images? What are we observing? \n\n3. The ACE estimation result is not clearly presented. It is unclear what is meant by an \"ACE gain of 2.35 percentage points\", does this imply that the estimated ACE is 0.0235? Additionally, the dataset used for computing the bootstrap result is not specified. The authors should clarify both the interpretation of the reported ACE value and the experimental setup used to obtain the bootstrap estimate.\n\n4. I am also confused between $\\tilde h$ in Equation (10) and $\\hat h$. Which on is being used in the loss? It would be great if the authors write the loss function explicitly in terms of all the trainable parameters. \n\n5. In the experimental section, some key evaluation metrics appear to be missing. For instance, [2] reports standard metrics such as False Positive Rate, True Positive Rate, and Accuracy, which are generally more interpretable and informative. I suggest that the authors consider including these metrics for improved clarity and comparability. Additionally, there are some discrepancies in the reported AUC scores; for example, [2] reports an AUC of 94.88 for DAW-FDD on the DFDC dataset, whereas this paper reports a significantly lower value of 59.96. Similar inconsistencies are observed for some other values. Why is this discrepancy? \n\n[1] Two Simple Ways to Learn Individual Fairness Metrics from Data. Debarghya Mukherjee,  Mikhail Yurochkin, Moulinath Banerjee and Yuekai Sun.\n\n[2] Improving Fairness in Deepfake Detection. Yan Ju, Shu Hu, Shan Jia, George H. Chen, and Siwei Lyu.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel Demographic Attribute-insensitive Intervention Detection (DAID) approach to address the trade-off between improving fairness and generalization in Deepfake image detection. A causal relationship between fairness and generalization is assumed and established through back-door adjustment, implemented via two modules that counteract the confounders of data distribution and model capacity. Experiments are conducted on various datasets and DAID is compared against multiple baselines. The results suggest that the proposed DAID approach can improve both fairness and generalization simultaneously.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(S1) This paper demonstrates that a direct relationship between fairness and generalization can be established after reducing the influence of confounders, as shown by a preliminary experiment in Section 3.2. Based on this finding, the proposed method looks sound to improve generalization by promoting fairness through the two proposed confounder-reduction modules, and therefore address the trade-off between fairness and generalization, as found in previous works.\n\n(S2) The experiments are rich, and the authors provide valuable details and discussions on many aspects in the ablation studies.\n\n(S3) In general, the paper is easy to follow and understand. \n\nWeaknesses:\n\n(W1) While the authors claim that \"DAID elucidates the causal relationship between fairness and generalization during training,\" neither this causal relationship nor the confounder-reduced features are evaluated in the experiments, making it unclear whether the proposed two modules work as intended. Additional comments after the reviewers/ACs discussion stage: In my opinion, if the authors claim a specific approach leads to improved outcomes of something else, that approach itself should be directly evaluated, and evidence should be provided to show that it works as intended. In this work, while the results demonstrate great gains in model generalizability, the underlying causal relationship that is claimed to support these gains is not evaluated. The authors are encouraged to further investigate this matter, or to de-emphasize the claim about \"establishing a causal relationship\", to enhance the contributions of this work and to avoid any potential confusion.\n\n(W2) The pre-processing and annotation pipeline for the Deepfake datasets used in this paper's experiments is unclear, although the authors claim that they follow previous studies (line 225). For example, four races are considered in [24] (Asian, White, Black, and Others), but only three of them are considered in this paper (Asian, White, Black). The choice of subgroups is very important in fairness works and can have a large impact on evaluating the model performance.\n\n(W3) Table 1 and Table 2 do not report any standard deviations or statistical tests, making it unclear whether the minor number differences represent true marginal improvements or if they actually follow the same distribution. This affects the authors' claim that the proposed method outperforms the baselines.\n\nQUESTIONS:\n(Q1) I agree that it is somewhat intuitive to assume a causal relationship from fairness to generalization, but could the authors provide more explanation and literature to support their choice of this assumption, rather than the other way around?\n\n(Q2) The authors are encouraged to respond to the weaknesses mentioned above, especially (W1) and (W3), which will greatly affect my future decisions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper motivates that demographic fairness and generalization to unseen data are not always at odds with each other. Based on this motivation, the paper proposes DAID to improve fairness between demographic groups and generalization to unseen data on deepfake detection models. Experiments show the effectiveness of the method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- I like this work. The method proposed is well explained, and useful for the deepfake detection problem. The problem being addressed is an important one.\n- Experiments are thorough: Beyond basic experiments, effects of hyperparameters is shown. The comparison on demographic attributes is also interesting.\n\nWeaknesses:\n- This paper needs a discussion on why exactly fairness and generalization may be at odds with each other. Specifically, when is fairness and generalization at odds with each other? Is this a characteristic of the data, the model, or both? When can we observe that these results will hold and when is it simply not feasible to improve in both dimensions? I would like to see a detailed discussion on this.\n- There are several works in the fairness and robustness literature in general, eg., [1], targeted towards tabular dataset that shows models can be more robust and fair. While robustness in [1] is considered as adversarial robustness, more adversarially robust models could improve generalization. I think its important to discuss works in fairness and robustness and draw a (qualitative) comparison.\n- Minor issues in writing. Example: Line 273-\"it worth noting\". I also find it odd to explicitly list the venues of papers in Table 1; since the references are already there, it is not needed. \n\n[1] Sharma, Shubham, Alan H. Gee, David Paydarfar, and Joydeep Ghosh. \"Fair-n: Fair and robust neural networks for structured data.\" In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 946-955. 2021.\n\nQUESTIONS:\nPlease refer to weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}