Title: FairNet: Dynamic Fairness Correction without Performance Loss via Contrastive Conditional LoRA
Venue: NeurIPS 2025
Accepted: yes
Keywords: Fair machine learning

Fairness

Bias mitigation
Abstract: Ensuring fairness in machine learning models is a critical challenge. Existing debiasing methods often compromise performance, rely on static correction strategies, and struggle with data sparsity, particularly within minority groups. Furthermore, their utilization of sensitive attributes is often suboptimal, either depending excessively on complete attribute labeling or disregarding these attributes entirely. To overcome these limitations, we propose FairNet, a novel framework for dynamic, instance-level fairness correction. FairNet integrates a bias detector with conditional low-rank adaptation (LoRA), which enables selective activation of the fairness correction mechanism exclusively for instances identified as biased, and thereby preserve performance on unbiased instances. A key contribution is a new contrastive loss function for training the LoRA module, specifically designed to minimize intra-class representation disparities across different sensitive groups and effectively address underfitting in minority groups. The FairNet framework can flexibly handle scenarios with complete, partial, or entirely absent sensitive attribute labels. Theoretical analysis confirms that, under moderate TPR/FPR for the bias detector, FairNet can enhance the performance of the worst group without diminishing overall model performance, and potentially yield slight performance improvements. Comprehensive empirical evaluations across diverse vision and language benchmarks validate the effectiveness of FairNet. Code is available at \url{https://github.com/SongqiZhou/FairNet}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FairNet, a dynamic, instance-level fairness correction framework that selectively applies representation-level adjustments using contrastive Conditional LoRA modules, guided by a lightweight bias detector. Unlike static global debiasing methods, FairNet activates corrections only for instances predicted to be biased, preserving overall performance. A novel contrastive loss aligns intra-class representations across sensitive groups to mitigate minority underfitting. FairNet supports full, partial, or no access to sensitive attribute labels. Theoretical analysis shows that, under reasonable bias detector performance, FairNet can improve worst-group accuracy without reducing overall accuracy. Experiments on vision and language datasets (CelebA, MultiNLI, HateXplain) validate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tI find the idea of selectively applying fairness correction via conditional LoRA to be a clear advancement over traditional static interventions. The framework addresses over-/under-correction by activating the correction mechanism only on potentially biased instances.\n2.\tThe design accommodates settings with full, partial, or no access to sensitive attributes, which enhances its applicability in real-world scenarios where such labels are often incomplete or missing.\n3.\tThe use of parameter-efficient LoRA modules and compact bias detectors allows FairNet to be integrated into existing models with relatively low computational overhead, making it more practical than full model retraining or architecture redesign.\n\n\n\nWeaknesses \n\n1.\tI am not convinced that we can improve the fairness while keeping the main performance. There is a theoretical analysis of the performance preservation analysis. However, the assumption is strong and makes the application domain more limited. Specifically, the theoretical guarantee relies heavily on the bias detector achieving a high TPR/FPR ratio and assumes that LoRA corrections will significantly help minority groups while having little or no negative effect on the majority. These conditions may not hold in realistic scenarios, particularly when biases are complex or intersectional.\n2.\tFairNet’s core mechanism depends on the accuracy of its internal bias detector. In low-label or unlabeled settings, this module may perform poorly, triggering LoRA on inappropriate samples or missing biased ones. I would like to see a deeper analysis of how sensitive the system is to detector quality, especially in the FairNet-Partial and FairNet-Unlabeled configurations.\n3.\tWhile the experiments show promising results on CelebA, MultiNLI, and HateXplain, I find the scope of evaluation too limited to support the strong claims made. All datasets focus on relatively simple binary or categorical sensitive attributes. There is no evaluation on more complex tasks, real-world large-scale datasets, or scenarios involving multiple/intersectional sensitive attributes. The paper would be significantly strengthened by a broader empirical validation, including robustness to detector failures, ablations on thresholding strategies, and analysis under real-world deployment constraints.\n4.\tThe paper does not adequately address the computational complexity introduced by the bias detectors and per-attribute LoRA modules. It remains unclear how well this approach scales in practice, especially when dealing with many sensitive attributes or high-dimensional inputs.\n\nQUESTIONS:\n1.\tHow sensitive is FairNet to the accuracy of the bias detector? Have you evaluated how downstream performance is affected when the detector has suboptimal TPR/FPR, especially in the FairNet-Partial and FairNet-Unlabeled settings?\n2.\tHow do you choose the threshold for activating LoRA modules? Is it a fixed value, or tuned per dataset? How robust is the system to this choice?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a technique for adjusting classifier performance on disadvantaged and minority groups via low-rank adaptation. By training a discriminative model to predict when a data instance comes from a disadvantaged group, they can engage the LoRA augmentation on specifically those inputs and not degrade the performance of a fairness-agnostic model on members of the majority. They demonstrate this FairNet's performance on vision and text data, where they improve the model's general accuracy, its accuracy for the worst-performing group, and its equalized odds gap across groups.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The submission has strong experimental results, including strong supplementary results that cover intersections of sensitive attribute, effect of the activation threshold and more.\n\n2. The proposed technique is simple, lightweight, and adaptable. It is agnostic to model architecture, and can improve performance on specific groups without hurting performance on \n\nWeakness:\n1. There are moderate gaps between the motivation and application in some places. For example, I do not find the motivation provided in lines 84-90 regarding choice of fairness criteria persuasive. Other fairness criteria are described as \"potentially abstract [and] might not align with the specific application context.\" However, WGA and EOD are abstract and potentially-misaligned as well. The submission argues that it bypasses the fairness-accuracy tradeoff, but to me it seems that this is inherent to its choice of metrics: if a classifier is held constant for the majority group, improving its accuracy on the minority group (i.e. improving WGA) inherently improves the accuracy of the entire classifier. Thus I find this submission to be more focused on improving model performance on sub-populations than it is about fairness. In lines 222-227, authors argue that aligning representations has the effect of improving downstream fairness, but I do not see this in the experimental results. We do see EOD go down, but EOD trivially goes down when accuracy goes up. EOD is a fairness metric when it exposes asymmetry: e.g. group A is given a lot of false positive (bad outcomes) while group B is given more false negatives (good outcomes). By reporting an average EOD gap it's hard for the reader to tell if EOD shrinks because fairness is improving, or because accuracy is generally improving.\n\n2. Relatedly, I am unconvinced by section 4.3. When sensitive attributes are fully supervised, it seems trivial that holding a classifier constant for the majority group and training an augmented classifier for minority groups with worse performance would simultaneously improve WGA and total accuracy. From my reading, the inequality in Eq (8) boils down to: performance goes up for FairNet if the discriminator is accurate enough that performance gains on correctly-classified minority points outweigh the performance costs on majority points that are incorrectly-classified as members of the minority. \n\nI believe modest edits to the text and experiments can resolve these points. If the contrastive loss aligning the latent representations is having a desired effect on downstream fairness, then it should be demonstrated in the experiments and emphasized in the text over claims that WGA and accuracy can be improved in tandem, which seems evidently true. \n\n3. Two key components of the technique seem unexplained: (a) how contrastive pairs are chosen (e.g. lines 169-170 emphasizes that the selection is crucial, but I cannot find where the selection process is defined), and (b) how the FairNet-Unlabeled algorithm generates pseudo-sensitive attribute labels, beyond the high-level description given in lines 292-294.\n\n4. In the abstract, introduction, and conclusion, FairNet is often described as applying the LoRA augmentation to instances \"identified as (potentially) biased\" or \"predicted to be vulnerable\". I find this language misleading, as it implies that the model is identifying cases where unfair predictions are being made. Instead, the LoRA augmentation is applied to instances identified as being disadvantaged/members of a minority group. This is a large ethical limitation: in the fully-observed setting, this reduced to applying an augmented classifier to minority instances (i.e. violating group fairness) and in the partially-observed setting this reduces to predicting whether an instance is a member of the minority group to decide which classifier to apply, which comes with similar ethical concerns. I do not think these ethical concerns reduce this submission's contribution or render the technique inapplicable, but I think they should be made more clear.\n\nSmall notes:\n- The appendix notation table is wonderful; it would be helpful to reference it from the main text. In addition, some terminology is introduced in lines 200-205 but used much earlier, e.g. $x$ is used on line 138 without introduction.\n\n- The word \"jointly\" is confusing on line 176, when the various modules are being trained sequentially and not in a joint fashion. The use of snowflakes and fire in Fig 1 to demonstrate this is very nice, and would benefit from a single sentence in the figure caption defining what these two symbols mean.\n\nQUESTIONS:\n1. In your experiments, how many detectors are used? Based on your notation, each LoRA module is activated only if its corresponding Detector fires. So in Figure 1, should there be a detector at every blue layer? In your experiments, are the detectors generally consistent, with every layer making similar predictions about minority group membership? What would results look like if you ablated this choice of multiple detectors, and either aggregated the detectors' results to fire all LoRA modules or used a single detector?\n\n2. Line 337 \"We replace our contrastive loss with binary cross entropy loss\". Given that the binary cross $L_\\textnormal{task}$ already appears in the loss in Eq (3), would it be correct to rephrase this as \"we remove the contrastive loss from Eq. (3) and only train on $L_\\textnormal{task}$\", or is a loss being applied to each layer $(j)$? I would be interested to see what effect this ablation has in the fully-supervised CelebA setting. In that setting, if optimizing the contrastive loss leads to better minority accuracy (WGA) than directly optimizing the minority group's cross-entropy loss, there is an extremely strong argument for the claim that the minority group benefits from aligning with the majority group's representation. Even if the WGA is better for the $L_\\textnormal{task}$ optimizer in that setting, this ablation would help me understand why the contrastive loss is helpful in the detector setting, and the contrastive loss could be motivated instead by downstream fairness performance.\n\nLIMITATIONS:\nSee Weakness 4.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes FairNet, a framework to dynamically correct fairness biases at the instance level.  At inference time,  it employs a lightweight bias detector that assesses the likelihood of the instance belonging to a “sensitive group”, and selectively activates Low-Rank Adaptation (LoRA) based correction for instances identified as potentially belonging to the “sensitive group”.

At training,  both the lightweight bias detector and the  LoRA based correction model are trained together with the base model.  The training can be done on data with or without the sensitive group labels. A contrastive loss is used for training the LoRA module.

Experiments are run on vision and language benchmarks.

STRENGTHS AND WEAKNESSES:
The paper is generally well written.  It tackles the important problem of fairness in machine learning models. It runs at instance level and dynamically corrects fairness biases when a “sensitive” attribute is detected.  In training, a contrastive loss is used to train the LoRA module for correction. It can use data with or without the sensitive group labels, and the original accuracy performance of the base model is preserved. A theoretical analysis is also provided.

The proposed approach seems to be solid and its effectiveness is demonstrated on vision and language datasets. 

One concern of the approach is that, with the additional bias detector and the LoRA based correction, computational cost will increase at inference time. It would be desirable to provide some latency/computational cost analysis.

QUESTIONS:
With the additional bias detector and the LoRA based correction, computational cost will increase at inference time. Has any analysis been done on latency/computational cost?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
The paper proposes FairNet, an in-processing debiasing wrapper that inserts lightweight sensitive-attribute detectors and conditionally triggered LoRA adapters into a frozen backbone. A detector-gated contrastive loss pulls minority-group representations toward majority ones, and a simple bound shows worst-group accuracy can rise without harming overall accuracy provided the detector’s TPR/FPR exceeds a data-dependent threshold. Experiments on vision (CelebA) and language (MultiNLI, HateXplain) benchmarks deliver ≈3–4 pp gains in worst-group accuracy and equal-odds while matching ERM accuracy.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Conditionally applying LoRA adapters is a conceptually simple idea that is easy to retrofit to existing networks.
2. The performance preservation condition (eq. 8) makes it explicit when instance-level debiasing can avoid the usual accuracy-fairness trade-off.

Weaknesses:
1. Combining a learned detector with conditional parameter-efficient adapters is an incremental idea with limited novelty, similar to "detect-then-correct" idea that exist in selective prediction and test-time adaptation literature.

QUESTIONS:
1. For multiple sensitive attributes (e.g. intersecting race and gender), do you train separate detectors per attribute or one shared detector?
2. For the unlabeled detector, which unsupervised method is used? How is the threshold chosen? And what are the resulting TPR/FPR on held out data?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FairNet. FairNet is an in-processing method that attaches bias detection heads to intermediate representations and conditionally triggers low rank adaptation (LoRA) modules trained with a novel contrastive loss. The detector identifies potentially biased instances and only those examples receive a small weight update to remove bias, and the base model’s predictions for unbiased inputs remain untouched. Additionally, ablations confirm both the detector and the contrastive loss are essential.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. A single architecture handles fully labelled, partially labelled, and unlabeled settings.\n2. Experiments are done across vision and NLP, with baselines and ablations.\n3. A theoretical framework to generate a formal condition linking detector quality to non decreasing overall accuracy.\n4. Touching an important research question in fairness domain.\n\nWeaknesses:\n1. Guarantees are based on a moderate TPR/FPR for the bias detector; however, the paper gives no systematic error bars to confirm this in real world applications.\n2.  CelebA and other datasets are mid-scale; no large-scale dataset is tested.\n3.  No intersectional bias is discussed.\n4.  Adding multiple detectors may hurt the test time speed.\n5. Ablation study is done only for CelebA.\n\nQUESTIONS:\n1. Is the detector threshold dataset dependent (Fig 2)? If yes, how could one tune it? and how difficult and sensitive is it? How this affect its application in real world applications?\n2. What is the FLOPs / latency overhead of detectors and LoRA? I believe it is affecting the test time, not only the train time.\n3. Have you tried two-attribute detectors (e.g., race and gender) and measured fairness gaps? Is it even possible?\n4. In FairNet unlabel, LOF is used for pseudo-labels. How stable is performance, especially for outlier samples?\n5. Does FairNet still preserve accuracy when test-set distribution shifts? Any comments on this?\n\nLIMITATIONS:\nPlease see weaknesses and the questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}