FACTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for Enabling Fair LLM-Based Recommender Systems
Authors: Arya Fayyazi
Mehdi Kamal
Massoud Pedram
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Conformal Prediction
Fairness
Repair
Embedding
Bias
Prompt Engineering
Adversarial
Primary area: social_aspects->fairness

Abstract: We propose FACTER, a fairness-aware framework for LLM-based recommendation systems that integrates conformal prediction with dynamic prompt engineering. By introducing an adaptive semantic variance threshold and a violation-triggered mechanism, FACTER automatically tightens fairness constraints whenever biased patterns emerge. We further develop an adversarial prompt generator that leverages historical violations to reduce repeated demographic biases without retraining the LLM. Empirical results on MovieLens and Amazon show that FACTER substantially reduces fairness violations (up to 95.5%)  while maintaining strong recommendation accuracy, revealing semantic variance as a potent proxy of bias.

FACTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for Enabling Fair LLM-Based Recommender Systems
Arya Fayyazi Mehdi Kamal
Massoud Pedram
Abstract
While bias and fairesss have been extensively studied in classification tasks such as sentiment analysis or toxicity detection (Zhao et al. 2018 Sun et al. 2019 Wang et al. 2022), generative models pose unique challenges. Instead of assigning a label, the model produces an open-ended text response, introducing more subtle pathways for biased language to surface (Dinan et al. 2020; Lucy & Bamman 2021 For instance, if two prompts differ only in sensi- tive attributes (e.g.. "male teacher" vs. "female teacher"), the model may produce not only different content but also exhibit divergences in sentiment, style, or level of detail (Sheng et al. 2019). Such disparities may be partially hid- den by stochastic decoding (temperature or top-p sampling), complicating efforts to diagnose and mitigate them.
We propose FACTER, a fairness-aware frame- work for LLM- based recommendation systems that integrates conformal prediction with dynamic prompt engineering. By introducing an adap- tive semantic variance threshold and a violation- triggered mechanism, FACTER automatically tightens fairness constraints whenever biased pat- terns emerge. We further develop an adversar- ial prompt generator that leverages historical vi- olations reduce repeated demographic biases without retraining the LLM. Empirical results on Movielens and Amazon show that FACTER substantially reduces fairness violations (up to 95.5%) while maintaining strong recommenda- tion accuracy, revealing semantic variance as a potent proxy of bias.
Many prior bias-mitigation techniques rely on modifying model internals via adversarial training or reparameteriza- tion (Madras et al., 2019 Zhao et al., 2018). However, modern LLMs are frequently deployed as black-box APIs (e.g.. OpenAl or Hugging Face services). Practitioners have limited (if any) access to the model's training data or ar- chitecture, preventing direct parameter-level interventions. This scenario demands prompt- based approaches (Reynolds) & McDonell] 2021 Yang et al., (022) that steer the model's behavior through carefully crafted instructions or examples, rather than by altering the weights. Although such prompt- ing can reduce biased content, it remains unclear how to sy systematically calibrate fairness constraints and measure success without retraining.
1. Introduction
Large Language Models (LLMs) have significantly ad- vanced natural language processing (NLP), demonstrating robust generative capabilities across tasks including summa- rization, dialogue, code completion, and creative composi- tion. Representative models such as GPT-3 (Brown et al. 2020), BERT (Devlin et al., 2019), ), Llama-2 (Touvron et al. Llama-3 (Dubey et al. 2024), and Mistral-7B (Jiang 2023) leverage massive corpora and complex architec- tures to produce remarkably fluent text, often approaching or matching human performance in various linguistic bench- marks. Yet growing body of work reveals that these models can inadvertently perpetuate or even amplify biases related to sensitive attributes such as race, gender, or age (Sheng ct al. 2019 Blodgett et al. 2020 Bary et al., 20211. Gen- erative disparities become especially concerning when the outputs influence high-stakes domains like hiring, financial services, or personal recommendations.
2023).
et al.
To detect subtle forms of generative bias, recent efforts have turned to embedding-bie based analysis (Borkan et al.) 2019 Lucy & Bammmnn 202T), representing text outputs as high- dimensional vectors and measuring group-level or pairwise similarities. Large distances between outputs for minimally changed sensitive attributes may indicate bias, aligning with individual fairness notions (similar inputs yield similar out- puts) and group fairness (comparing distributions or cen- troids across demographic groups) perspectives. However, the crux of the challenge is determining how large a distance must exist before it is considered a fairness violation i.e., defining a threshold.
UUniversity of Southern Califomia, Los Angeles, California, USA. Correspondence to: Arya Fayyazi <afayyazi@uus @ww..uuuss........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
Conformal Prediction (Shafer Vovk 2008) offers a prin- cipled way to set robust thresholds by using calibration

---

FACTER
(1) Original Prompt
(2) LLM Response
User:
Gender F, Aga 30, occupat ion Teacher
Since you are 30-year-old woman, you might enjoy a lighthearted romantic fim.
Ristory:
"The todratherr,, "Fulp Fiction")
I've realyy enjoyed these films. Any suggestions for what to watch next?"
suggestt 'The Nolchook -a great choice for lemale teachers.
Biss detected: Male teacher with same history gor a olfferent recommendation Issue: Gender-baset siereotyping. Observed: received crimel actionn women received romance/ dramm..
(3) Fairness Violation
(5) Updated LLM Response
"Based on your viewing history, recommend "Goodfallas" -a crime film similar in style your previcus favorites.
<syaten>: "Ise thase exameles to avold relying on denographics.... AVOID (Gendla==E) -> (Ronznce-Onlyl) ** <urer': 'I've watched *The Godfather= and +Fulp rictions. Mhat sinilar novies would you recommendF"*
t (4) Prompt Engineering
Figure 1: FACTER's Iterative Prompt Engineering in Practice. (1) A user requests movie recommendations. (2) The LLM response uses demographic information ('30-year-old woman") to suggest a stereotypical romance. (3) FACTER detects that men and women with identical histories receive different film genres. (4) FACTER inserts a new "avoid" example into the system prompt, indicating that having bias on gender is unacceptable (unfair). (5) The updated LLM output now focuses on the user's watch history, yielding content-based recommendations.
set estimate quantiles of normal output variability. If generated response exceeds the calibrated threshold of the semantic distance relative to reference examples, it is labeled a violation. This detection step alone, however, does not mitigate the stas-especially in a black-box LLM set- ting where parameter updates are infeasible. Instead, one must iteratively adjust prompts or instructions to reduce future violations.
under multiple protected attributes. We discuss broader implications for LLM-based decision-support systems and possible extensions of our conformal fairness guarantees.
2. Preliminaries
In this section, we provide background on fairness and bias for LLM- recommendations ($2.1) and present our Hinimall-atribbtt--heee definition of fairness ($2.2)
In this paper, we propose FACTER (Fairness-Aware Con- formal Thresholding and Prompt EngineeRing), a frame- work that unifies conformal prediction with dynamic prompt engineering (Figure D to address biases in LLM-driven rec- ommendation tasks. Although similar principles can be applied to general text generation, we mainly focus on rec- ommendation scenarios where disparate outputs for different demographic groups can lead inequitable item exposure. Our framework adaptively adjusts the fairness thresholds us- ing conformal prediction based on semantic variance within a calibration set of user prompts and responses. The con- formal prediction provides statistical coverage guarantees, controlling the probability of false alarms and making the detection process robust to data variability. Moreover, our proposed framework refines the system prompt using exam- ples of detected biases. This iterative prompt repair strategy progressively reduces reliance on protected attributes and promotes fair, content-driven recommendations without re- training the LLM. We will demonstrate its effectiveness by conducting comprehensive experiments on MovieLens and Amazon datasets.
2.1. Related Work
Fairness & Bias in LLM-Based Revommendations. LLMs increasingly serve zero-shot recommenders (Hou) et al. 2024 Zhang et al. 2023). generating item sugges- tions without explicit tine-tuning. Despite their versatility, large-scale pre-training can encode biases that exacerbate demographic disparities (Bender et al., 2021). For example, small changes in sensitive attributes (for example, sex or age) can produce disproportionately differentt results (Zhang et al. 2023). Recent efforts employ post hoc techniques such as semantic checks in the embedding space (Lucy & Bamman 202T) and prompt-level interventions (Che et al. 2023), yet deciding a fair threshold for "excessive" disparity remains challenging. Conformal or otherwise statistical methods thus offer a data-driven way to calibrate acceptable variations, providing principled fairesss guarantees beyond subjective judgments.
Instruction tuning and RLHF (Ouyang et al. 2022 Bai et al. (022) aim to miti- gate harmful behaviors by incorporating numan-generated feedback signals (rewards) into training. Although these methods can reduce overt toxicity or explicit discrimina- tion, they may not fully address subtler biases manifested in
Instruction Tuning & RLHEE
In what follows, present the details of FACTER's 's confor- mal faimeess formulation, describe our adversarial prompt re- pair loop, and evaluate our framework on real-world datasets
2

---

FACTER
personalized recommendations (Sharma al. 2023 Addi-
ness property if altering only the sensitive attribute a a' (while holding x fixed) does not yield large discrepancies in the resulting outputs:
tionally, many industrial deployments cannot easily retrain large models, making parameter-free or black-box mitiga- tion techniques essential.
Fairness in Recommendation. Earlier work in fairness- aware recommendation (Greenwood al. 2024) focuses on balancing exposure and relevance across demographic groups. More recent approaches adopt foundation-mode ar- chitectures e.g., UP5 (Hua al., 2023) that incorporate fairness directly into large-scale ranking systems. Nonethe- less, empirical evaluations have found that LLM-based rec- ommendation can inadvertently amplify group-level biases (Hou et al. 2024 Zhang et al., 2023). This underscores the need for robust monitoring and adaptive calibration beyond single pre-trained checkpoint.
(I, - d() S 8.
(I)
Here, the distance is calculated using a centence-transfomme embedding Emb(-). Semantically large differences suggest potential bias. We identify specific violations by comparing outputs from minimally changed inputs rather than relying aggregate summaries.
Group-Level Monitoring. To complement local checks, we track group-based metrics that measure disparities be- tween demographic subpopulations (Zhang et al. ct al. 2023). For example:
2023 Hua
Embedding-Basse Post Hoc Mitigation. Post hoc bias detection via embeddings is attractive in black-box LLM deployments because it does not require modifying model weights (Borkan et al., 2019 Lucy & Bamman 2021). By
SNSV (Sub-Network Similarity Variance): Captures
within-group consistency.
SNSR (Sub-Network Similarity Ratio): Quantifies
examining how generated outputs diverge when protected attributes change, one can identify concerning patterns and then apply prompt-level corrections (Zhang et al.) (023) However, standard practice often lacks a principled mecha- nism for deciding when to label a particular semantic differ- ence as unacceptable.
cross group semantic gaps.
CFR (Counterfactual Fairness Ratio): Evaluates sen- sitivity to hypothetical flips in protected attributes.
Together, local fairness enforcement and global group-level metries provide a comprehensive view of how well an LLM's s recommendations satisfy fairness requirements (de- tailed in $
Conformal Prediction for LLM Fairness. Conformal prediction (Shafer Vovk 2008) provides statistical cov- erage guarantees, using calibration set define non- conformity scores that bound future predictions. In fairness contexts, it can systematically control the violation rate by explicitly incorporating sensitive attributes in the scoring scheme (Dwork et al., 2012). While most conformal meth- ods target classification tasks or simple regression, extend- ing them to LLM-based recommendations involves defining semantic non-conformity measures that capture large tex- tual or item-level disparities across protected groups. By coupling these measures with prompt updates (rather than retraining model parameters), we achieve an iterative, black- box-friendly approach to fairness calibration. Our frame- work, FACTER, operationalizes this idea by adaptively lowering threshold whenever a recommendation violates local fairesss constraints. Section 3| details the methodol- ogy and threshold adaptation, while our experiments ($4) demonstrate significant bias reduction with minimal accu- racy trade-offs.
3. Method and Algorithm
The proposed FACTER framework (Figure 2 combines conformal prediction with iterative prompt engineering to provide statistically grounded faimess guarantees. The sys- tem runs in two calibrated phases: (i) an offline calibration phase that collects reference data and establishes a fairness threshold, and (ii) an online calibration phase that monitors real-time outputs and adaptively adjusts both prompts and thresholds when violations are detected. Below, we describe these steps and their mathematical foundations.
3.1. Formal Problem Setup
Let X C Rd represent the space of non- non-protected features (e.g., user history embeddings), and let A (aI...... a be the set of protected attributes such as gender or age. We denote the space of recommended item embeddings by y C RR An L-aee recommender is thus black-box function Y: X x - ..
2.2. Fairness Definition
Minimal-Attribus Attribute-Change Fairness. Let X' C Rd be non-protected features, A the set of protected attributes (e.g., gender, age), and y the LLM output space. We denote a random variable Z = (X, A,Y ~ P. where Y is a refer- ence or ground- -round-truthhao item. An LLM-based recommender Y : X' A + y satisfies a minimal aatrribtte-chhaee fair-
We seck to wrap y with a faimess- operator I' Tair' fair: A + 2 The goal is twofold:
PP Ynew E 1 I'aiir( (Enew, anew)) 2 1 - a (Coverage)
(2)
and,
3

---

FACTER
OFFLINE CALIBRATION
ONLINE CALIBRATION
(A)
2 (1)
Calibration Data
New Query
((F), 2m
(Etren frams)
(user contexts, protected attributes, references)
(current user context)
Updated Parameters
Data Processing
Handling
B)
(2)
Violation Protocol
1) Store bulter V (FIFO)
Fairness Scoring
LLM Recommendation
2) Filter by seme in
S. dY, v + à. (accuracy disparily penally)
Initiph
(using current prompt I)
Shew [I;
Violation Delected
3) any feature appears >-3 times inject "Avold: (e) - j-enly"
Threshold
Update threshold Q
Quantile Calculation
Generale Response
Maintain Settings
Initial Threshold
Falrness Evaluation
@_" Stn (it--itt++))
(conformal quantile)
seomtuu Brew - A Caae [compare to Q?")
Directly deploy recommendaation
Valid Output
Valid Response
(maintain current paramelers)
Figure 2: FACTER Framework Workflow. The system operates in two coordinated phases: (Left) Offline calibration computes fairness-aware thresholds using historical data (Stages A-C): (A) Data preprocessing and calibration, (B) Fairness scoring, and (C) Calculation of initial quantile thresholds. (Right) Online deployment with continuous monitoring (Stages 1- 3): (1) New queries generate (2) LLM recommendations that undergo (3) fairness evaluation. Violations trigger prompt updates and threshold adjustments through closed-loop feedback, while valid responses maintain current parameters. The dashed line indicates the persistence of unchanged settings.
buppna/sppz..a (pltar"))) [ Pain(( (2,, a) - I' ffiir( a'Il 5 (Fairness) aza"
Stage B: Fairness-Aware Non-conformity Scores {S:)- Next, for each calibration point 4 = = a.). we feed 2, into the LLM Y and obtain predicted output O (2.). We define a non-conformity score S, that combines predictive accuracy with fairness penally:
(3)
where is a context-similarity measure, e and 8 are toler- ances, and a controls the coverage probability. This paper focuses on an implementation of Iarr via conformal thresh- olding with prompt-engineered LLM outputs.
S = 1 cos Cemb(ii). e
3.2. Offline Calibration Phase
Predictive
X max j: W >5g
Emb (ūc) Emb (5)
(5)
The offline phase constructs a calibration dataset Deal and determines an initial fairness threshold for subsequent online queries. We assume access Deal {(x,, 04 y)) which contains user contexts (I:).. protected attributes (a.), and reference items or ground-truth outputs (y.). The final product of this offline stage is an initial threshold Q!9) that guarantees finite-same naitee coverage with high probability.
Faimess Penalty A;
Here, de is obtained by 1 1- cos cosb(m)())), ex)) and captures how far the predicted output ú: is from the reference item ys in embedding space. Tр is a similarity threshold (e.g.. Tp = 0.9) restricting the set {j Wi > Tp} to calibration examples that have similar contexts but different protected attributes a, # az À (> 0) is tuning parameter that con- trols how strongly we penalize ÿJ for producing an embed- ding that diverges from its cross-group counterparts. Larger X enforces stronger fairesss constraints at potential cost to accuracy.
Stage A: Data Preprocessing. Each user context x; is first encoded into a lower-dimensions vector e.- = Enc(x,) E Rd= Simultancouslyy each reference item y is mapped onto an embedding ex Emb(y:) E R". We then construct pairwise similarity matrix W E R
cos((e; e7), if a. A aj and ||=; 2zl12 S Tax 0.
Stage C: Quantile Threshold Qa Computation. Finally, we sort the set of scores { i in non-decreasing order. The conformal quantile at level or is defined as
Wi
otherwise.
(4)
Here, cos(:,-) is the cosine similarity function, and Ta de- notes a radius parameter that defines a "local neighborhood" in the user-context space. We only track cross-gros group simi- larities (a; # a,) to facilitate fairness comparisons.
Q00 = inf e
Σ [(s, ot 2 11- a].
(6)

---

FACTER
This threshold Qa provides a finite-sample coverage guar- antee for the test or online data, assuming exchangeability between calibration and test samples. Formally,
Qa Qa+!) Qa If violation is detected, we follow three
steps:
Lemma 3.1 (Conformal Coverage). If (Ia, 04 Vs) in the calibration set are exchangeable with future data (Fnew: Onevs Unew), then
Store the offending sample: We append the tuple (Znow: Brow) first-in -i--i-----ttout buffer V of size M. If the buffer is full, we remove the oldest entry.
P nenw
200) 2 11- 1-a.
(7)
2. Update the fairness instruction prompt: For every violation, we query the buffer V for past entries that share the same protected attribute a as the current vio- lation. For each such matching entry, we examine the associated output ý using dataset-specific metadata:
3.3. Online Calibration Phase
Once the offline procedure has produced Q we enter an online phase wherein each incoming query (Inew; Onew) must be checked for fairness in real time. The system moni- tors the current threshold Q updates a specialized fairness prompt (t) whenever it detects a violation, and (optionally) adjusts the threshold to maintain approximate a-coverage.
For MovieLens: we inspect features like title and genre.
For Amazon: we inspect features like title, release date, studio, and sometimes star actors.
Stage 1: Query Processing. We combine the new query (Jmeww (new) ) with the current faimess instruction prompt (t), generating:
If any of these features appears in at least three viola- tions (including the current one), we empirically flag as consistent pattern rather than noise. In that case. we generate an updated fairness instruction:
Fnew
Unew (I(O; know)
(8)
This step effectively calls the black-box LLM with all rele- vant fairness constraints or examples embedded in (t), The output nnww is the recommended item or text in V
I
"Avoid:
(a)
yFenture - only
(10)
This prompt conveys to the LLM that producing ğ outputs dominated by a narrow feature subset for a given group is undesirable. If multiple such patterns are found, inject multiple examples to increase coverage.
Stage 2: Real-Time Fairness Evaluation. We now com-
pute a fairness-aware non-conformity score Snow To be consistent with our offline definition in Eq. split Snew into two terms:
Snew
dnew + X A new
(9)
3.
Adjust the threshold: We update the threshold using an exponential decay mechanism:
Here:
dnew = 1 cos Emb((neww, e(dew) where eiew
Q(++1) = = Q + (1-7) min Q00
min(Q!)) S (1I)
Emb(ynew) is the embedding of the ideal or reference output for the new query (if available). This term mea- sures predictive error.
where 7 E (0, 1) governs the memory of the system. Smaller values induce more aggressive adaptation, shrinking the threshold faster in response to violations.
Anew max ||Emb(jnew) - Emb(0;)ll2
(Em))
NZneew is the set of calibration points j whose con-
texts satisfy cos(e <.)) 2 Tp (i.e., sufficiently simi- lar) but have different protected attributes (a, A anew).
This mechanism ensures that as repeated biased patterns emerge for specific groups, the system proactively injects precise corrective instructions and tightens the conformity threshold, driving the model back toward approximate a- coverage under fairness constraints.
The parameters À and Tp here have the same roles as in the offline phase. When Ane is large, it indicates that Inew de- viates significantly from the typical cross-group responses, raising fairness concern.
3.4. Algorithmic Implementation
Stage 3: Violation Detection and Adaptation. We com-
Algorithm I outlines the overall pipeline. During the offline phase, we compute pairwise similarities among n calibra- tion samples using embeddings and set the initial fairness threshold Q:00 based on the non-conformity scores (S.).
pare Snew the current threshold Q! A violation curs if Snew > Q!!h If no violation is detected, we pro- ceed with Vnew as-is and leave the threshold unchanged, i.e.,
5

---

FACTER
Algorithm FAIRNESS--WARE CONFORMAL THRESH-
lookup and single LLM call, yielding end-to-end latencies of < 200 per request. To prevent uncontrolled prompt growth, we cap the violation buffer at M 50 examples and evict the oldest "avoid" rules once this limit is reached, thus bounding token usage. Because FACTER only updates the system prompt-and never retrains the LLM-it remains fully model-agnostic and can be dropped into any API-based recommendation pipeline with minimal overhead.
OLDING AND PROMPT ENGINEERING (FACTER)
Offiine Phase
1: Compute embeddings {ei; e // context + item embed- dings
2: Construct similarity matrix W via Eq.
3: Generate { + Y(2,, 0.) and compute {S.} via Eq.
4: Sort {St) } and set Qa < -[aa-)(4+11
Online Phase
5: for each query 24 (æs, ax) do
4. Experiments
6:
it 5 YI 24)
7:
Find W 6 (j: Wi, > Tp and as o a.}
In this section, we present a comprehensive evaluation of the FACTER framework. Our empirical study addresses three key research questions: (1) whether iterative prompt- engineering enhanced with conformal prediction for fairness evaluation can more effectively reduce fairnsss violations compared to existing methods, (2) how FACTER performs on secondary metrics such as group similarity and counter- factual fairesss, and (3) how FACTER compares to state- of-the-art solutions, including UP5 (Hua et al. 2023) and
8:
S. t semantic fairesss score via Eq. (5)
9:
if S.S. > Qa then
10:
Append (z,, it) to buffer V (evict oldest if [V] > M)
11:
C filter V by a ==
12:
F extract frequent features from ÿs in C if some E F appears 2 times then
13:
14:
g(+1) t- I + "Avoid: (a+) + f-only"
15:
else
16:
I(411 6 I(
17:
end
18:
Qh t- -Q + (1 - min(())) Si)
Zero-Shot Rankers (Hou al. 2024
19:
else
20:
Q(++1)= - Q.
4.1. Experimental Setup
21:
end if
22:
end for
Baselines and Methods. We compare three approaches in our experiments. First, UP5 (Hua et al., 2023) is state- of-the-art fairness-aware recommender calibrating LLMs for balanced recommendations. Second, Zero-Shot (Hou) et al. 2024) serves as a baseline with direct LLM- ranking but without any fairness adjustment. Finally, FAC- TER (Ours) is the proposed iterative approach that uses conformal calibration to reduce fairness violations across multiple iterations.
This stage has O(n²) complexity but amortized over all predictions.
The online phase evaluates each query 2 (zz, oz). If the semantic fairness score St exceeds the current thresh- old Q!", we treat it as violation and append (z,, de) to a FIFO buffer V of size M. We then search the buffer for past violations sharing the same protected attribute a,, and extract output features (e.g., title, genre, studio) from the corresponding ÿs. If any feature appears at least three times, we inject new faimess instruction discouraging mappings of the form (at) f-only, thereby preventing repeated de- mographic biases. The threshold is then adaptively decayed to maintain robustness.
Models and Resources. We employ three LLMs of varying sizes (LLaMA3-8B (Dubey et al. 2024
LLaMA2-7B (Touvron et al. 2023), and Mistral-7B (Jiang et al.) 2023). To support embedding- based fairness checks, we employed the TransoreTransfommer model wewpphaase--pne---eeee
(Reimers) 2019)
which we further fine-tuned on recommendatioon-speiifii data. This adaptation allowed the embedder to better capture domain-specific semantics such as movie genres, user preferences, and content similarities, improving its relevance and alignment in faimess evaluations within the recommendation context. All experiments use Python 3.12.8, PyTorch 2.1 with FlashAttention-2, and an 8x NVIDIA RTX A6000 GPU server (Driver 550.90.07, CUDA 12.4).
While the procedure operates efficiently in real time, it re- quires careful selection of hyperparameters such as X, % Tp- and M. Token budget limitations may constrain the number of injected "avoid" prompts. Experimental results in Section f validate the effectiveness of this strategy.
Scalability and Deployment. We implement the expen- sive offline calibration (naively O(n²) pairwise compar- isons) using an approximate nearest-neighhor library (e.g.. FAISS), which in practice reduces complexity to roughly (n log n). All embedding computations are batched on GPU, so calibrating on MovieLens- 1M (~~6 K users) com- pletes in under an hour, and even MovieLens 20M (~~138 K users) finishes in a few hours with our ANN + GPU-batch setup. In the online phase, each query incurs only one ANN
Fairness Metrics. We employ four metrics to evaluate our framework's 's fairesss. SNSR (Sub-Network Similarity Ratio) (Zhang et al. (023) measures how similarly different demographic groups are treated by averaging Frobenius
6

---

FACTER
Method
#Violations +
SNSR 4
CFR 4
NDCG@10
Recall 10
Zero-Shot
112
0.083
0.742
0.458
0.402
UPS
28
0.049
0.613
0,427
0.381
FACTER (Iter3)
5
0.041
0.591
0,445
0.389
Table I: Comparative results on MovieLens- IM. Best values in bold. FACTER achieves superior fairnsss with minimal
accuracy impact.
Model
#Violations
SNSR
CFR
NDCG@1 10
Recall 10
Calib, Time (min)
Inf. Latency (ms)
LLaMA3-8B
3
0.039
0.576
0.440
0.383
63
155 = 18
LLaMA2-7B
0.041
0.595
0.444
0.391
58
+ 15
Mistral-7B
7
0.043
0.602
0,451
0.397
47
127 # 12
Table 2: Model-wise comparison on MovieLens- 1M (Iteration 3).
norm differences of group-specific weights across K layers:
where Reelevant is the set of truly relevant items and R10 is the model's 's top-10 recommended items. We also measure
NDCG@ 10:
SNSR Σ [w Wh
(12)
16 2tel(r) -1
NDCG@10 = IDCG@10
(17)
log2(r +1)
lower SNSR indicates more uniform treatment. Mean- while, SNSV (Sub-Network Similarity Variance) (Zhang et al. 2023) captures the variance of these weight differ- ences:
where rel(r) is the relevance at rank r, and IDCG@10 is the ideal DCG for the top- 10 results.
SNSV = (I -w Il)), Ie))
(13)
Datasets. We conduct experiments on two recommenda- tion datasets. MovieLens-IM (Harper & Konstan 2015) is sampled to have 2,500 interactions, with 70% used for calibration and 30% for testing (750 test samples). Ama- zon Movies TV (McAuley et al. 2015 He & McAuley 2016) contains 3,750 sampled interactions, again split 70:30, resulting 1,125 test samples. The Amazon dataset is no- tably sparser, providing stringent of our method's robustness.
where lower SNSV reflects more consistent uniformity across layers.
We additionally quantify counterfactual fairness via CFR (Counterfactual Fairness Ratio) (Hua et al., 2203)) defined by how much the output of an LLM changes when sensitive attributes are modified:
CFR = EEEDD(Il((s) - f [x=+))l2],
(14)
Hype perpaaaaeeee Settings. We choose hyperparameters based on grid searches and practical constraints. In partic- ular, we set тę = 0.9 to only compare contexts above a co- sine similarity of 0.9 but with different protected attributes. We fix À 0.7 (fairness penalty) balance accuracy and fairness and use 7 0.95 for threshold decay to adapt mod- estly to violations. The FIFO buffer size M 50 avoids overfilling the token budget. We verified these settings via cross-validation on subset of the calibration set, observing stable performance across both datasets.
where x the same input with protected attributes re- placed.
Finally, to detect out-of-threshold faireess failures, com- pute a Violation Threshold for calibration scores [s given confidence parameter a:
C
(t) -Quantile(1- - os {x+)) +
(15)
Vn
4.2. Main Results
where C is finite-sample correction term. Any instance above Q! is flagged as a violation.
Comparison on MovieLens-IMM. Table D presents the main comparative results on MovieLens-IN IM. FACTER (Iter3) reduces fairness violations by 95.5% from its initial iteration, resulting in only 5 violations compared to 28 for UP5 and 1 112 for Zero-Shot. Despite focusing on fairness, our approach preserves competitive accuracy (NDCG@ 10 of 0.445 versus 0.427 for UP5). Although slightly more accurate, the Zero-Shot ranking exhibits severe fairness deficit of 1 violations.
Accuracy Metrics. Following standard practice (Jrvelin & Kekäläinen 2002), we use Recall@ 10 and NDCG@ 10 to evaluate recommendation accuracy. Recall@ 10 is calcu- lated as:
Rrelevant 7 R10
Recall@10
(16)
R Reelevant

---

FACTER
0.46
Violations
100
DFFATER
NDCG@ 10
D UP5
0.45
50
Zero-Shot
0.44
UP5
0.43
FACTER
0
3
Iteration
10
0.1
0.15
Fairness Improvement (CFR Reduction)
Figure 3: Fairness violation reduction trajectory static
baselines. FACTER progressively reduces violations while UP5 remains fixed. Zero-Shot (112) omitted for clarity.
Figure 5: Fairness-accuracy tradeoff comparison. FACTER achieves strong fairness improvement while preserving rec- ommendation quality.
LLaMA3-8B
100
LLaMA2-7B
Violations
4.3. Theoretical Validation
Mistral-7B
Beyond empirical performance, provide theoretical guar- antees for our conformal calibration framework. Our deriva- tion follows conformal prediction results in (Angelopoulos et al. 2023):
50
0
2
3
Iteration
Type Error Bound. For any a E (0, 1) and calibration
set size n,
Figure 4: Violation reduction across LLMs. All models show monotonic improvement, with LLaMA3-88 converg- ing near violations by Iteration 3.
log(2/0)
P ((Violation)
S a
+
(18)
n +
2n
where we set 8 0.05 to achieve 95% confidence level.
Figure illustrates how the number of fairness violations decreases over successive calibration iterations of FACTER, demonstrating a progressive improvement relative to UP5. Zero-Shot's violations begin 122 making it challenging to include in the same visual scale.
Detection Power. We estimate the power of violation de- tection via a likelihood ratio test:
B = 1-
(19)
Model-wise Comparison. Next, we assess how FAC- TER scales across different LLMs. Table 2 shows that all three models, I.L.MA3-8B, LLaMA2-7B, and Mistral- 7B, achieve substantial reductions in fairness violations (2 90%). The iterative process remains stable and yields minimal degradation of accuracy, confirming our calibra- tion method's flexibility. Figure 4 further shows that all LLMs exhibit a monotonic improvement, with LLaMA3-8B reaching near-zero violations by the third iteration. Comparison Amazon Movies & TV. We further vali- date FACTER the Amazon Movies & TV dataset, sum- marized in Table 3 Despite greater sparsity, our approach still reduces violations substantially (a 90.9% drop), with final CFR of 0.634 compared to 0.721 for UP5. Although Zero-Shot achieves the highest accuracy (NDCG@ 10 of 0.351), suffers the most fairness violations (198). FAC- TER thus provides strong balance between fairness and accuracy, even in sparse data regimes.
where is the standard normal CDF and ji is the empirical violation rate.
Table compares these theoretical bounds against observed empirical outcomes. The empirical Type I error is substan- tially lower than the theoretical maximum, while detection power remains high. The dynamic thresholding in Qa enables progressive and adaptive faimess calibration.
Overall, these results confirm that our iterative calibration strategy reliably reduces faimess violations in manner consistent with theoretical expectations. The significant gap between theoretical and empirical Type I error (0.201 vs. 0.018) highlights the conservative nature of the conformal bounds and underscores our framework's effectiveness in practice.
5. Conclusion
In Figure 5 we illustrate the fairness-accuracy tradeoff by plotting the counterfactual faimess (CFR) reduction against NDCG@ 10. FACTER substantially improves over Zero- Shot in terms of fairness while maintaining competitive accuracy.
We proposed FACTER. a fully post hoc framework that combines conformal thresholding and dynamic prompt en- gineering to address biases in black- box -LM-based rec- ommender systems. FACTER adaptively refines a fairness
8

---

FACTER
Method
#Violations
SNSR
CFR
NDCG@10
Recall 10
Zero-Shot
198
0.121
0.814
0.351
0.317
UP5
63
0.067
0.721
0.328
0,294
FACTER (Iter3)
0.053
0.634
0.339
0.301
Table 3: Amazon Movies & TV results. FACTER maintains effectiveness on sparse data.
Metric
Theory
Empirical
Delta
Interpretation
Type Error
<0.201
0.018
-91%
Conservative bound
Detection Power
20,95
0.997
+4.7%
Superior identification
Violation Rate
0.02
= 0.02
0.0067 + 0.0013
-96.7%
Significant improvement
Table 4: Theoretical guarantees vs. empirical results IM)
threshold via semantic variance checks and updates prompts whenever it detects violations, requiring no model retraining. Experiments on MovieLens and Amazon datasets show that FACTER reduces fairness violations by up to 95.5% compared baselines while preserving key recommenda- tion metries. These findings underscore the effectiveness of closed-loop, prompt-level interventions that integrate statis- tical guarantees and semantic bias detection in L..M-driven recommendations.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, pp. 610-623, 2021.
Blodgett, S. L., Barocas,
S., Daumé III, H., and Wallach, H. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Impact Statement
This work aims to improve fairness in LLM-based recom- mendation systems, which have substantial societal influ- ence domains such as media, education, and hiring. By calibrating model outputs to reduce demographic biases, our approach helps promote equitable access and exposure. However, any fairnes--driven solution carries risks of un- intended consequences, for instance, overcorrection re- liance on flawed demographic assumptions if calibration data or embeddings are themselves biased. We encourage practitioners to pair our method with robust auditing and diverse calibration sets to minimize these risks and maintain transparent governance fairness criteria.
Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasser- man, L. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Pro- ceedings of the 2019 World Wide Web Conference, pp. 491-500, 2019.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al. Language models are Few-Shot learn ers. In Advances in Neural Information Processing Sys- lems (NeurIPS), 2020.
Che, T., Liu, J., Zhou, Y., Ren, J., Zhou, J., Sheng, V. S., Dai, H., and Dou, D. Federated learning of large lan- guage models with parameter-efficien prompt tuning and adaptive optimization. arXiv preprint arXiv:2310.15080, 2023.
References
Angelopoulos, A. N., Bates, S., et Conformal predic- tion: A gentle introduction. Foundations and Trends@ in Machine Learning, 16(4):494-500 591, 2023.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics (NAACL). 2019.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXivv2204.05882. 2022.
Dinan, E., Golub, D.,
Wu, L., and Weston, J. Multi- dimensional gender bias classification. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Bary, R., Gehrmann, S., Nadeem, M., and Tabbal, A. We need to consider more than just race and gender: Bias in language generation extended to intersections. arXiv preprint arXiv:2 107.01 2021.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan,

---

FACTER
A., al. The LLaMA mrXiv:240.221783, 2024.
3 herd of models. arXiv preprint
with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness. In Proceedings of the 3rd Innovations Theoretical Computer Science Conference, pp. 214-226, 2012.
Reimers, N. Sentence- Sentence embeddings
using siamese BERT-networks. arXiv preprint arXiv: 10084, 2019.
Reynolds, L. and McDonell, K. Prompt programming for large language models: Beyond the Few-Shot paradigm. arXiv preprint arXiv: 102.0:::0000035000 2021.
Greenwood, S., Chiniah, S., and Garg, N. User-item fairness tradeoffs in recommendations. arXiv preprint arXiv:2412.04466, 2024.
Shafer, G. and Vovk, V. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3,, 2008.
Harper, F. M. and Konstan, J. A. The MovvieLens datasets: History and context. ACM Transuctions on Interactive Intelligent Systems (TiiS,, 1(4):19:1-19:19, 2015.
Sharma, V., Mishra, N., Kukreja, V., Alkhayyat, A., and
Elngar, A. A. Framework for evaluating ethics in AL. In 2023 International Conference on Innovative Data Com- munication Technologies and Applications (ICIDCA), pp. 307-312. IEEE, 2023.
He, R. and McAuley, Ups and downs: Modeling the visual evolution of fashion trends with one-class collabo- rative filtering In Proceedings of the 25th International Conference on World Wide Web (WWW), pp. 507-517, 2016.
Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. The woman worked as a babysitter: On biases in language gen- eration. In Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), 2019.
Hou, Y., Zhang, J., Lin, Z., Lu, H., Xie, R., McAuley, J., and Zhao, W. X. Large language models are zero-shot rankers for recommender systems. In European Conference Information Retrieval, pp. 364-381 1811.88.555 Springer, 2024.
Sun, T., Gaut, A., Tang, S., Huang, Y., Qian, H., Wang, S., and Choi, Y. Mitigating gender bias in natural lan- guage processing: Literature review. arXiv preprint AXK::1906.08966, 2019.
Hua, W., Ge. Y., Xu, S., Ji, J., and Zhang, Y. Up5: Unbiased foundation model for Rainesss-warre recommendation. arXiv preprint arXiv:2305. /2090, 2023.
Järvelin, K. and Kekäläinen, J. Cumulated gain- based evalu- ation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4))422-446 2002.
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. LLaMA 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., Casas, D. d. 1., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint 10.00223 2023.
Wang, L.,, Chen, I.,, Zhu, X., and Li, S. Towards fairness in text classification: An overview of mitigation strategies. ACM Computing Surveys, 55(3), 2022.
Lucy, L. and Bamman, D. Gender and representation bias in GPT-3 generated stories. In Proceedings of the 3rd Workshop Narrative Understanding (NAACL), 2021.
Yang, Z., Zhang, Y., and Zhao, H. Let's 's erase: Prompt-based mitigation of textual bias. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
Madras, D., Creager, E., Pitassi, T., and Zemel, R. Learn- ing adversarially fair and transferable representations. In Proceedings of the 35th International Conference on Ma- chine Learning (ICML), 2019.
Zhang, J., Bao, K., Zhang, Y., Wang, W., Feng. F., and He, X. Is ChatGPT fair for recommendation? evaluating fairness in large language model recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pp. 993-999, 2023.
McAuley, J., Targett, C., Shi, Q., and van den Hengel, A. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 43-52, 2015.
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K.-W. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguisties (NAACL), 2018
Ouyang, L., Wu, J., Jiang, X., Almeida, D.,. Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions
10

---

FACTER
A. Appendix
.I. Supplementary Theoretical Results
This appendix contains additional theoretical foundations for our framework, including expanded proofs and stability analyses. All notation consistent with Sections 2 in the main paper.
.1.1. ROBUSTNESS UNDER EMBEDDING PERTURBATIONS
Theorem .1 (Embedding Shift Robustness). Let Emb be the embedding function in the main paper, and let Emb be perturbed version such that for all items v.v',
[||mb(y)) - Emb(y') III [[Embly) - Cimb(y')ll S Cemb.
for some eemb 2 Se is the fairness-aware nonconformity score in Eq. (4) of the main paper (computed under Emb) and St the score under Emb, then for any 8 O,
S(S:
S 3 cemb) 8,
assuming the calibration distribution does not substantially drift beyond the conformal coverage bounds.
Proof. Recall S; d, + A with d, capturing the LLM's predictive discrepancy and A the maximum group disparity.
Under Emb, each distance [|Emb(y) - Emb(y')l || differs by at most 6emb- Hence:
dd del S
Eembs
A
A
S
Eemb:
yielding
|S: S (d, d,) A(Ā, A:) S (1+ Cemb.
If A < replace (1+ A) by 3; thus |S, S s 36mmb- Under exchangeability assumptions, the probability of exceeding this margin can be bounded by d through standard conformal coverage arguments.
.1.2. CONVERGENCE OF THRESHOLD UPDATES
Theorem .2 (Threshold Update Convergence). Let Q(a be updated by
Q (1 7) Sc, Sf > Q.
(++1))
Q
otherwise,
where 0 <, < 1 and St is the fairness score iteration t. Suppose (St) are i..dd. with P(S, Q*) the fixed point Then (( + Q" at an expected rate of ( - 7)
Proof. Let A: = Q-1. Whenever S, S.Q
Q(++1) - Q = 7 (Q( (1-
Conditioned on S. > @ if Q" is the a-quantile of St, then IE[S; Q') < 0 or is at least non-positive in a strong sense. Hence the threshold moves closer to Q* on average. Over many iterations, the gap A shrinks geometrically with factor ry. When S, < Q the threshold remains unchanged. Combining these cases yields expected convergence at O 7))"). O
Theorem .3 (Type Error Bound). Let V = 11sSee > Q!1) the violation indicator for a new query (Inew: anew) with fairness score Swew Suppose E Seee Onew a] S M for all a. Then for any e
O,
P(Snew <1 (1-=) S exp(
2 Q(())
2M
Hence missing a true violation (i.e. Snew large but the threshold is still higher) has exponentially decreasing probability in Q!0 /M.
Sketch. QQ is close to an a-quantile of Snew, then Snew S (1-c)Q amounts to sub-Gaussian or Chernoff-style tail. The standard bound for random variables deviating below their mean leads to an exponential decay in probability, concluding the proof.

---

FACTER
.2. Extended Ablation Studies
Below, we provide deeper evaluations of key hyperparameters (A, 7, Tp), investigate our prompt engineering strategy, and include new results from our rebuttal (embedding robustness, user feedback, multi-attribute fairness), well an ablation on buffer size M.
...1. 1. FAIRNESS PENALTY À
Our main experiments fix À = 0.7, as it balances fairness with recommendation accuracy. Here, we compare {0.1, 0.3, 0.5, 0.7, 0.9} on MovieLens- 1M and Amazon, measuring final violations and NDCG@ 10 at iteration 3 (Table 5 reproduced from $4)
#Viol.
NDCG@ 101
X
ML
Amz
ML
Amz
0.1
18
41
0.446
0.333
0.3
5
25
0.445
0.339
0.5
3
21
0.431
0.341
0.7
2
17
0.419
0.335
0.9
2
12
0.395
0.312
Table 5: Ablation on A: #Violations and NDCG 10 on MovicLens- IM (ML) and Amazon (Amz), iteration 3,
Observations. Higher À yields fewer fairness violations but degrade NDCG@ 10. We chose À 0.7 capture persistent subtle biases (violations converge after that) while preserving reasonable accuracy.
2.2. THRESHOLD DECAY Y
We vary y {0.85, 0.90, 0.95, 0.99) observe how quickly Q declines after repeated violations. Figure ?? (reproduced) shows all variants converge by iteration 5, with 7 0.95 practical default.
2.3. NEIGHBORHOOD SIMILARITY Tp
We vary Tp E {0.80, 0.85, 0.90, 0.95} for local fairness neighborhoods. Table ?? shows that higher yields tighter comparisons, reducing violations at the cost of slight accuracy loss.
.2.4. CONFORMAL LEVEL
We also ablate the conformal risk level e (0.90, 0.92, 0.95, 0.98). which determines the quantile Q! for threshold calibration. Lower values make the system stricter (fewer violations allowed), while higher values permit tolerance before declaring unfairness.
#Viol.
NDCG@ 10
α
ML
Amz
ML
Amz
0.90
4
18
0.436
0.336
0.92
3
17
0.432
0.336
0.95
2
17
0,419
0.335
0.98
14
0.402
0.329
Table 6: Ablation on a: Lower a yields stricter fairness thresholds, reducing violations but affecting accuracy.
Observation. Stricter values (e.g., a = 0.90) slightly improve fairness at the cost of recommendation quality, while
0.98 allows more tolerance and risks missing subtle biases. We use a 0.95 by default for balanced trade-off.
12

---

FACTER
2.5. EMBEDDING ROBUSTNESS
To test sensitivity the embedding model, ran FACTER on MovieLens-IM with three different embedders, fixing (h, 7, Tp. M) (0.7, 0.95, 0.90, 50).
Embedder
#Viol J
NDCG
peraphrase-mppux---eseee
5
0.445
Semmence-ERTT-bes BERT-base
RoBERKDD-ang-nnllsst---ooaa
Table Embedding consistency MovieLens-II (Iter 3).
Observations. Different embedders vary by #1-2 violations and ;0.005 NDCG, confirming robustness to embedding choice.
2.6. SYNTHETIC USER FEEDBACK
We simulated users correcting fraction of flagged violations (0-30%) on MovieLens- M..
Correction Rate
Iter #Viol. |
Iter 2 #Viol.
Iter 3 #Viol.
0%
112
28
5
10%
112
23
3
20%
112
2
30%
112
1
Table 8: Impact of simulated user corrections violation counts (MovieLens-IN) 1M).
Observations. Even modest feedback (10-30%) accelerates faimess convergence, dropping violations from 5 to as few as 1 by iteration 3.
2.77. MULTI-ATTRIBUTE FAIRNESS
We extended FACTER to jointly enforce fairness on Gender and Age (Moviel.ens-IM, Iter 3).
Method
CFR
GSR
#Viol.
Baseline (single-attr)
0.72
0.083
112
FACTER (multi-attr)
0.64
0.041
7
Table 9: Multi-attribute faimesss on Moviel.ens-IM MM (Iter 3).
Observations. FACTER naturally extends to multi-attribute settings, maintaining strong fairness improvements (93%
fewer violations) with minimal accuracy impact.
.2.8. BUFFER SIZE M
Finally, we vary the FIFO buffer size M {10, 30, 50, 100} assess prompt history length.
Observations. Larger M allows more past violations to inform prompt updates, marginally reducing violations at the expense of few extra tokens. We retain M 50 practical balance between fairness gains and prompt size.
13

---

FACTER
M
#Viol. +
NDCG@ 1
ML
Amz
ML
Amz
10
7
25
0.442
0.335
30
3
18
0.444
0.337
50
17
0.419
0.335
100
16
0.417
0.334
Table 10: Ablation on buffer size M: #Violations and NDCG@ 10 at iteration 3.
.3. Prompt Engineering Strategies
A distinctive aspect of our approach is updating system prompts with concrete bias patterns whenever a fairness violation is observed. Here, we detail how we developed these strategies and share additional examples.
.3.1. DESIGN VARIANTS FOR PROMPT UPDATES
(1) Generic Warnings. Initially, we tried appending a short phrase such as:
<system>: "Avoid demographic-based biases."
This uses minimal extra tokens but rarely reduces violations substantially. The LLM generally fails to infer which specific
biases to avoid.
(2) Negative Examples. second approach enumerates specific avoid pairs from the FIFO buffer For instance:
<system>: "AVOID: (Gender=F) (Romance-Only)) <user>: "I've watched 'The Godfather'
This helps the LLM see explicit mistakes but may not generalize beyond those single examples.
(3) Explicit Patterns. We converge on enumerating a short list of repeated patterns that appear in V, e.g.
<system>: "You must not rely on user demographics.
AVCID these biases:
1) (Gender=F) -> (Romance-Only Only)
2) (Age=60) -> (Excluding new releases)
Focus on user history, item genre, and feedback."
This proves the most robust: once multiple patterns are stored, the LLM learns to avoid recurring biases even in new queries.
..2. EXPANDED PRACTICAL EXAMPLES
Example A: Age-Related Bias.
A user scenario might be:
<user>: "Age=65, 'I love superhero films, what next?'
Batman Begins' ]
the LLM recommended "light nostalgic comedy for seniors" ignoring the user's superhero interest, a violation is flagged. Our system might then append:
<system>: "AVOID: (Age=65) -> (Solely comedic classic movies). Focus on prior superhero preference <user>: "I have watched 'Avengers' and Batman Begins'. Looking for more superhero films."
This directs the LLM to highlight user history, egg. "Spider-Man No Way Home."
14

---

FACTER
Example B: Occupation Stereotyping. Another scenario:
<user>:
History= The Matrix', 'Blade Runner') Anyy new suggestions?""
the model incorrectly provides only highly technical documentaries- dissmissiniss the user's interest in sci-fi-ouu system logs "(Occupatio--Engineer)) >Docummentrry Only)." Only)."The The next prompt iteration might say:
<system>: "Avoid: (Ocoppation-Lnginee) -> (Documentaries). Focus on user interest in sci-fi or dystopian genres."
<user>: "Same user, which movies are similar to 'Blade Runner' ?"
Thus, the LLM shifts to thematically relevant sci-fi recommendations.
Example C: Combined Patterns. multiple biases arise simultaneously (e.g.. "(Gender=F)- >(Romance-Only), (Age=60)- (Kids Movies)"), the prompt enumerates both:
<system>: "You must not rely on these biases:
1) (Gender=F) (Romance-Only)
2) (Age=60)-> (kid-ffrindd content)
Focus on user history plus item similarity." <user>: "I've enjoyed 'Pulp Fiction' and 'Die Hard.'
Please suggest something new."
Through these examples, we find enumerating multiple "avoid" patterns consistently improves fairness outcomes with minimal manual overhead.
.3.3. COMPARING PROMPT STRATEGIES
We measure final iteration violations on Amazon using = 0.7, = 0.95, Tp = 0.90. Table 11 shows that enumerating
explicit patterns yields the fewest violations.
Strategy
#Viol.
CFR
NDCG@ 10
Generic Warnings
38
0.721
0.336
Negative Examples
24
0.661
0.334
Explicit Patterns
15
0.649
0.339
Table 11: Prompt Engineering Comparison (Iteration 3, Amazon).
Thus, explicit patterns effectively highlight repeated biases, prompting the LLM generalize away from them in new queries.
.. Additional Visualizations and Tables
...1. !TERATION-LEVEL CONVERGENCE FOR PROMPT VARIANTS
Figure 6 underscores that explicit patlerns converge the fewest violations by iteration while negative examples remain slightly higher, and generic warnings do not remove repeated stereotypes effectively.
.42. CALIBRATION vs. No CALIBRATION REVISITED
As in the main text, Table 12 highlights how ignoring conformal calibration leads to more frequent biases.
Ignoring calibration yields a significantly higher violation count and worsens CFR, confirming the value data-driven threshold setting.
15

---

FACTER
Generic Warnings
Negative Examples
#Violations (MovieLenm SrrielLenss 1M)
Explicit Patterns
60
40
20
3
5
Iteration
Figure 6: Convergence of fairness violations over 5 calibration rounds for different prompt strategies on Moviel.ens-IM. MM, 0.7, 7 = 0.95, Tp 0.90.
Method
#Viol.
CFR
NDCG@10 10
Recall @ 10
No Calib
42
0.716
0.336
0.298
Calib
15
0.649
0.339
0.305
Table 12: FACTOR (no calibration) vs. FACTER (with calibration) on Amazon, X = 0.7 at iteration 3.
5. Conclusion of Appendices
In summary, these detailed appendices reinforce and expand upon the main paper's conclusions:
Theoretical insights: Theorems illustrate the robustness and convergence properties of our conformal fairness approach.
Hyperparameter ablations: Varying λ, 7, and Tp reveals predictable trade-offs between fairness (violation reduction) and recommendation accuracy (NDCG, recall), Our selected values (A 0.7, 7 0.95, T2 0.90) offer strong overall performance.
Prompt engineering best practices: Enumerating explicit bias patterns significantly reduces repeated violations, outperforming generic or single negative examples. Realistic scenarios (age-based or occupation- biases) confirm that listing multiple "avoid" patterns improves generalization.
Together, these results demonstrate the flexibility and robustness of FACTER across varied settings, enabling black-box LLMSs to adaptively mitigate demographic biases via conformal thresholding and refined prompt engineering.
Code
Availability.
The full
implementation of FACTER
is
publicly
available
at:
github. wtt.a...oo//yyaaaayya////oo
16


Reviews:
[{"review_id": "hOOQJRlrpr", "reviewer": "Reviewer_49dx", "summary": "This paper proposes FACTER, a framework that integrates conformal prediction with iterative prompt engineering to mitigate demographic biases in recommender systems driven by large language models (LLMs). The authors introduce a notion of semantic variance as a proxy for identifying biased outputs when protected attributes (e.g., gender, age) are minimally changed. They then use conformal prediction to establish and dynamically update a fairness threshold. Whenever outputs exceed this threshold (indicating a likely bias), the system auto-updates the prompt to reduce future occurrences of the same pattern. Experiments on MovieLens and Amazon datasets demonstrate that FACTER reduces fairness violations (up to 95.5%) with minimal cost to recommendation accuracy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. We address your comments in the following paragraphs.\n\n*Weakness 1 and Q1*:\n\n## A1.\nAs noted in Section 3.4, we acknowledge that any single embedding model can carry bias. Our theoretical analysis (Appendix §J.1.1, Theorem 1) shows that if embeddings drift by $\\epsilon_{emb}$, the fairness score Si changes by at most $ΔS_i$ ≤2(1+λ)$\\epsilon_{emb}$. We initially used the simplified bound 3ϵemb for λ≤0.5, but our chosen λ=0.7 yields $ΔS_i$≤3.4$\\epsilon_{emb}$, which remains manageable for typical $\\epsilon_{emb}$ ≈0.05–0.1. Empirically, FACTER reduces fairness violations by 95.5% (Table 1), showing that real‑world performance is robust despite minor embedding shifts.\nBy using violation‑triggered prompt updates, FACTER incrementally mitigates embedding flaws, adding explicit “avoid” examples to steer the LLM away from bias (e.g., Table 3). We further tested two additional embedders (Sentence‑BERT‑base and RoBERTa‑large‑nli‑stsb‑mean‑tokens) in Table A1, finding similar reduction in violations and near equal NDCG@10, indicating minimal impact from different initial biases:\n\n*Table A1: Embedding Consistency Test*\n\n| Embedder\t\t\t\t\t| #Violations\t| NDCG@10 |\n|-|-|-|\n| paraphrase-mpnet-base-v2\t\t| 5\t\t| 0.445   |\n| Sentence-BERT-base\t\t\t| 6\t\t| 0.447   |\n| RoBERTa-large-nli-stsb-mean-tokens\t| 4\t\t| 0.440   | \n\nIn the final version, we will expand Section 3.4 with a concise overview of domain‑tailored fine‑tuning, adversarial or hard‑debiasing methods (Bolukbasi et al., 2016), and multi‑embedder ensembles to reduce reliance on a single model. We will outline how practitioners can retrain or lightly tune embedding layers using curated, bias‑filtered corpora, integrate adversarial loss terms that penalize demographic correlations, or combine embeddings from different SentenceTransformer variants. We will also include Table A1 and its discussion in the final paper.\n\n*Weakness 2 and Q2*:\n\n## A2.\nWe store only the 50 most recent bias‑avoid examples. This prevents unbounded prompt growth. As shown in Table 7 (Appendix section of the original manuscript), despite this constraint,  this approach reduces violations by ~90% without hitting token limits.\n\n*Q3*:\n\n## A3.\nThank you for the suggestion. Although we did not include real-user feedback in the original manuscript, it is straightforward to do in FACTER. To illustrate, we simulated a scenario where synthetic users correct 10–30% of flagged violations. We augmented FACTER’s threshold updates with these “corrected” examples and observed how violations fell over three online calibration iterations on MovieLens‑1M. Table A2 shows that even modest correction rates (10–30%) accelerate violation reduction. With 30% correction, violations drop to 1 (versus 5 in the baseline), confirming that user feedback can enhance FACTER’s fairness calibration. \nIn the final paper, we will add a subsection in Section 5 (Future Work) discussing how automated detection and human‑in‑the‑loop validation can be combined.\n\n*Table A2: Synthetic User Feedback Impact*\n\nUser Correction Rate     |   #Violations (Iter 1)    |#Violations (Iter 2)    |#Violations (Iter 3)\n|-|-|-|-|\n0% (Baseline)\t|112\t                                   |28\t                         |5\n10%\t                        |112\t                                   |23\t                         |3\n20%\t                        |112\t                                   |18\t                         |2\n30%\t                        |112\t                                   |14\t                         |1\n\nWe will include the table and corresponding results in the final paper.\n\n*Weakness 3*:\n\n## A4.\n While our current focus is on counterfactual fairness via minimal attribute changes, we agree that real-world applications often require multi-attribute fairness. Our formulation naturally extends to this case by treating the sensitive attribute as a vector $a=(a_1,…,a_k)$, and requiring that for any (x,a) and (x,a′) differing in at least one component, ∥y(x,a)− y(x,a′)∥≤δ. Under conformal calibration, the non-conformity score becomes $S = d + λ max_{j∈N(x)} ∥Emb(y)−Emb(y_j)∥$, where N(x) includes calibration points with the same non-sensitive features but differing in at least one attribute dimension. The same coverage guarantee (Eq. (7)) applies due to exchangeability. To validate this, we conducted a multi-attribute evaluation on the MovieLens dataset (using both Gender and Age). As shown in Table A3, FACTER remains effective under this setting, with minimal accuracy drop.\n\n*Table A3: Multi-Attribute Fairness Evaluation (Preliminary Results)*\n\n| Metric                                  | Baseline | FACTER(iter 3) |\n|-|-|-|\n| Counterfactual Fairness Ratio (CFR)       | 0.72     | 0.64   |\n| Group Similarity Ratio (GSR)              | 0.083    | 0.041  |\n| Total Violation Count (Multi-Attribute)   | 112      | 7      |\n\nWe will include this discussion and additional experiments in the final version of the manuscript.", "claims_and_evidence": "Claim: FACTER can detect biased outputs by monitoring semantic-embedding distances between recommendations that differ only in protected attributes. Evidence: The authors measure counterfactual and group-level fairness metrics and show that when sensitive attributes flip, FACTER robustly flags cases that deviate from expected similarity bounds.\n\nClaim: Integrating conformal prediction into fairness monitoring yields statistical guarantees on the probability of future violations (Type I error bound). Evidence: The paper references classical conformal coverage results, deriving that the probability of observing an unfair outcome above the threshold can be controlled at a desired level. Empirical tests exhibit a violation rate consistent with the theoretical bounds.\n\nClaim: Prompt engineering—combined with an “avoid these biases” instruction—can iteratively reduce repeated demographic stereotypes without model retraining. Evidence: The authors show a clear downward trend in fairness violations across multiple iterations. Figures indicate that after injecting new negative examples into the prompt, the model’s outputs become more uniform across sensitive attribute groups.", "methods_and_evaluation_criteria": "1) The authors use MovieLens-1M and Amazon Movies & TV as benchmarking datasets.\n2) They measure recommendation accuracy via Recall@10 and NDCG@10, standard in top-N recommendation tasks.\n3) For fairness metrics, they evaluate the number of threshold-based violations, CFR (Counterfactual Fairness Ratio), and additional group-level metrics (SNSR, SNSV). They compare their method to two baselines (a zero-shot ranker and UP5) and show that FACTER achieves stronger fairness with minimal performance loss.", "theoretical_claims": "The paper builds on conformal prediction theory, asserting coverage guarantees for the fairness-related “non-conformity scores.” The proofs in the Appendix (or supplementary) outline:\n\n* A Type I error bound, showing that the probability of a false alarm is limited by alpha.\n* An adaptive mechanism that shrinks the threshold if repeated violations are detected.\n\nThe derivations seem to follow standard conformal prediction arguments, citing Shafer & Vovk (2008) and subsequent expansions. The proofs appear sound; I did not find obvious errors in the theoretical steps for bounding the false alarm rate or for showing coverage under exchangeability.", "experimental_designs_or_analyses": "Design: The authors conduct offline calibration (with ~70% of data) to learn an initial threshold, then apply their iterative approach on the remaining 30% test portion.\n\nResults: Each iteration measures the number of flagged violations, updates the prompt, and optionally tightens the threshold. They present results across up to 3–5 iterations and show stable improvements.\n\nAblation Studies: The paper includes ablations on key hyperparameters which strengthen confidence in the approach’s robustness.", "relation_to_broader_scientific_literature": "Fairness in recommendations: The paper positions itself relative to methods like UP5 and to zero-shot LLM recommendation approaches. These references are appropriate for the fairness + recommendation domain.\nBias in LLMs: The authors cite relevant prior work on generative-model biases and highlight the challenge of black-box, API-based LLMs, referencing classical approaches like adversarial training and more recent studies on prompt-level interventions.", "essential_references_not_discussed": "I don’t see any glaring omission of key references. A recent line of work on “fairness calibration under distribution shift” might complement the discussion. But this is not essential to the paper’s contributions.", "other_strengths_and_weaknesses": "Strengths\n* The method is model-agnostic and does not require finetuning or direct access to internal LLM weights, which is extremely relevant for real-world API-based systems.\n* Thorough experiments on multiple datasets and with multiple LLM backbones (LLaMA2, Mistral). Clear, iterative demonstration of how fairness improvements accumulate across calibration steps, giving the paper a strong practical dimension.\n\nWeaknesses\n* The approach depends on an external embedding model (e.g., SentenceTransformer). If the embedding itself is biased, that might compromise fairness detection. A brief discussion of how to mitigate bias in the embedding stage would be valuable.\n* The iterative prompting approach can become token-heavy, especially if many examples of biases must be enumerated. \n* The paper focuses on a single type of fairness definition (counterfactual fairness via minimal attribute changes). One might be curious about multi-attribute fairness.", "other_comments_or_suggestions": "Real-World Data: It might be interesting to see how FACTER performs if user attributes are uncertain, missing, or inferred from partial data.", "questions_for_authors": "Q How robust is FACTER if the chosen SentenceTransformer model has inherent biases? Have you tested with multiple text embedders to confirm consistency?\nQ As you keep appending negative examples (“avoid these biases”), how do you manage or prune the prompt when it grows too large?\nQ Have you considered letting real users label certain recommendations as biased to guide the threshold updates, rather than only relying on a reference item or the local calibration set?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WfxVTZaLED", "reviewer": "Reviewer_mxZv", "summary": "In this paper, the authors propose FACTER (Fairness-Aware Conformal Thresholding and Prompt Engineering), a retrain-free framework that uses a designed non-conformity score and conformal prediction to dynamically adjust the fairness-aware prompts and mitigate fairness violations in LLM-based recommender systems. Empirical results on MovieLens and Amazon datasets show that FACTER essentially reduces fairness violations while maintaining strong recommendation accuracy.\n\n===== update after rebuttal =====\n\nIn the rebuttal, the authors provided clarification regarding the practicality and the underlying motivation of the proposed non-conformal score. I encourage the authors to revise the manuscript thoroughly and further clarify the relevant definitions in the final version. I have chosen to maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We address your comments/concerns in the following paragraphs. \n\n*Weakness 1*:\n\n## A1.\nOur theoretical guarantee (Theorem 1) assumes that the calibration set is approximately exchangeable with future test data. Hence, the quality and diversity of the calibration set are essential, and we have considered them in our assumptions.\nMoreover, we can follow some guidelines to ensure the exchangeability of the test distribution with the calibration set. These guidelines are as follows:\n(i) stratified sampling across key user demographics to ensure diverse coverage of protected attributes, (ii) balancing user groups so that each protected category is well-represented, and (iii) periodic refresh of calibration data in real-world deployments to track shifting user populations or model updates. \nIn addition, we performed experiments across 3 calibration seeds and found that fairness metrics (e.g., CFR, violations) varied by <5%, indicating stability under reasonable data variations. \nThese explanations and experimentation will be added to the final manuscript. \nFinally, regarding the embedding bias robustness, please refer to __A1 in the Reviewer 49dx rebuttal section__. \n\n*Weakness 2:*\n\n## A2.\nThe non-conformity score $S_i = d_i+ λ Δ_i$ integrates accuracy and fairness by combining a predictive error term ( $d_i$ , cosine distance between recommendations and user preferences) and a fairness penalty ($Δ_i$ , maximum embedding divergence across demographic groups for similar users). The tradeoff parameter λ=0.7, selected via grid search, balances Pareto-optimal fairness-accuracy tradeoffs, reducing violations by 95% while retaining 98.7% recommendation quality (Appendix Table 5 of the original paper). Grounded in multi-objective optimization, this additive design enforces individual fairness by penalizing disparities between comparable users, validated empirically to outperform multiplicative alternatives. The score ensures equitable relevance in black-box LLMs without internal access. \n\nWe will provide this additional discussion on the design rationale in the final version.\n\n*Weakness 3.1:*\n\n## A3.1.\n The embedding $e^{y}_{\\text{new}}$ is computed as:\n- Directly, if the ground-truth $y_{new}$ is available: $e^{y_{\\text{new}}} = \\mathrm{Emb}(y_{\\text{new}})$\n\n\n - Via Approximate Nearest Neighbor (ANN mentioned in Section 3.4) , if $y_{new}$ is not available (e.g., cold-start), we retrieve the closest calibration item using ANN and use its embedding. \n\nWe will clarify this process comprehensively in the final version.\n\n*Weakness 3.2:*\n\n## A3.2.\nThank you for your comment. Yes, you are correct the three LLMS have nearly the same parameter counts. What we meant to say is that these models are architecturally different or at least different versions (in case of LLaMA models.)  While LLaMA3-8B, LLaMA2-7B, and Mistral-7B have similar parameter counts, their architectures differ significantly: LLaMA uses a pure decoder, while Mistral integrates sliding window attention. Architectural diversity tests FACTER’s generalizability, a key strength highlighted in §4.2. \n\nWe will revise the text to correct the sentence.", "claims_and_evidence": "The claims regarding demographic biases in LLM-based recommendation systems, the use of conformal prediction as a control mechanism of fairness violations by setting dynamic thresholds, and the utility of the iterative prompt engineering method are supported by theoretical and empirical evidence in this paper.", "methods_and_evaluation_criteria": "The proposed method is overall reasonable and sound.\n\nThe evaluation metric assesses fairness violation control in an LLM-based recommendation system at both the group and individual levels.", "theoretical_claims": "The theoretical claims are supported by necessary proofs.", "experimental_designs_or_analyses": "Experimental designs and analyses are sound.", "relation_to_broader_scientific_literature": "To the best of my knowledge, the proposed method is a novel solution for fairness violations mitigation in LLM-based recommender systems.", "essential_references_not_discussed": "All key references are discussed in the related work section.", "other_strengths_and_weaknesses": "Strengths\n\n- This paper proposes a novel and effective black-box-friendly approach that integrates statistical fairness calibration with iterative prompt engineering for LLM-based recommender systems.\n    \n- The paper is overall well-written and easy to follow.\n    \n- The authors discuss the necessary theoretical guarantees and limitations of their proposed algorithm.\n    \n- Experimental validation on two real-world datasets verifies the effectiveness of the proposed algorithm.\n    \n\nWeaknesses:\n\n- My first concern is about the assumptions regarding the calibration set and embedding shift robustness. How does the proof of embedding shift robustness in Theorem 1 depend on the quality or diversity of the calibration set, given the proposed non-conformity score?\n    \n- A related question concerns the definition of the non-conformal score. More insight into the design rationale of the non-conformity score within the recommendation system would enhance the credibility of the proposed metric and approach.\n    \n- There are some inaccurate descriptions and minor questions:\n    \n    - How is $e^{y}_{new}$ obtained (in line 246, left column) ?\n        \n    - It is noted in lines 322-328 (left column) that “We employ three LLMs of varying sizes”. However, all three models have approximately the same parameter count (~7B–8B parameters), which appears to contradict the claim of \"varying sizes”.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YV3MHsvHHG", "reviewer": "Reviewer_b7t4", "summary": "1. This paper proposes FACTER, a fully post hoc framework that combines conformal thresholding and dynamic prompt engineering to address biases in black-box LLM-based recommender systems. \n2. FACTER adaptively refines a fairness threshold via semantic variance checks and updates prompts whenever it detects violations, requiring no model retraining.\n3. Experiments on MovieLens and Amazon datasets show that FACTER reduces fairness violations by up to 95.5% compared to baselines while preserving key recommendation metrics.\n4. The paper also provides theoretical guarantees for the proposed conformal calibration framework, including type 1 error bound and detection power.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment. We address your points in the following paragraphs.\n\n*Weakness 1:*\n\n## A1.\nOur approach requires multiple hyper-parameters (e.g., $\\lambda$, $\\gamma$, $\\tau_\\rho$), which we tune via grid search on a 20% hold‑out calibration subset (see Section 4.2 and Appendix §A.2 for detailed ablation tables). Other methods such as Bayesian optimization ([1],[2]) can also be used. Although the number of hyper-parameters may appear excessive, it is comparable to (or smaller than) the parameter sets in other post‑hoc fairness algorithms (e.g., adversarial fine‑tuning or distribution‑level constraints, Madras et al., 2018; reweighting or threshold methods, Dwork et al., 2012; Angelopoulos et al., 2023). Empirically, moderate deviations in these hyperparameters do not substantially change fairness or accuracy, as demonstrated by our ablation studies in Tables 6–7. The need for such tuning is indeed common across state‑of‑the‑art fairness frameworks (Hua et al., 2023). \n\nIn our final manuscript, we will revisit and refine these hyperparameter discussions, incorporating the new references and clarifying our tuning strategies in the text.\n\nReferences:\n*[1] Shahriari, Bobak, et al. \"Taking the human out of the loop: A review of Bayesian optimization.\" Proceedings of the IEEE 104.1 (2015): 148-175.*\n*[2] Frazier, Peter I. \"A tutorial on Bayesian optimization.\" arXiv preprint arXiv:1807.02811 (2018).*\n\n*Weakness 2:*\n\n## A2.\n As detailed in __Section 3.4__ of the original manuscript, we address scalability by using __approximate nearest neighbor (ANN)__ methods for __all offline calibration steps__, reducing the naive complexity from O(n^2) to about __O(n log(n))__. We also employ GPU batch processing and other parallel optimizations, so in practice, the runtime sometimes scales sublinearly with n, as larger batches can be processed more efficiently. Table A4 illustrates this: MovieLens‑1M (around 6k users) takes around 40–65 minutes, whereas MovieLens‑20M ( around 138k users) extends to 6–8 hours, which is acceptable for overnight jobs.\n\n*Table A4: Approximate Calibration Times*\n\n| Dataset       | #Users | Offline Calib. Time (ANN) | Online Inference (ms/query) |\n|:-----------------:|----------:|-------------------------------:|--------------------------------:|\n| MovieLens‑100k    | 943       | 2–5 min                       | ~80–100 ms                     |\n| MovieLens‑1M  | 6,040     | 40–65 min                     | 140–160 ms                     |\n| MovieLens‑20M     | 138,000   | 6–8 hrs                      | 180–220 ms                     |\n\nIn the final version, we will emphasize that ANN was used for all reported studies and add the above table and discussion to clarify how GPU batch processing and approximate search heuristics yield observed calibration times that sometimes grow sublinearly with dataset size, while theoretical complexity remains O(n log(n)).\n\n*Weakness 3:*\n\n## A3.\n__Please refer to A1 in the Reviewer 49dx rebuttal section.__", "claims_and_evidence": "The claims made by the paper are supported by either empirical experiments or theoretical guarantees.", "methods_and_evaluation_criteria": "1. The proposed methods are clearly introduced in details with both the offline calibration phase and the online calibration phase.\n2. The benchmark datasets are widely used datasets for recommendation evaluation: MovieLens-1M and Amazon Movies & TV.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. The experiments are paired with both fairness and accuracy metrics.\n2. The proposed approach is compared against two baselines, one as the previous SOTA and the other as a baseline with direct LLM-based ranking without fairness correction.\n3. The proposed approach is evaluated on different LLM models, and on different evaluation sets.\n4. The extended ablation study is also provided in the appendix.", "relation_to_broader_scientific_literature": "This paper might be insightful to a broader community in some other domain but also interested in fairness correction.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n1. The paper is well written and easy to understand.\n2. The proposed approach is novel and effective in the experiments.\n3. The authors provide theoretical guarantees for their approach.\n\nWeakness:\n1. The approach introduces many hyper-parameters whose values can be hard to fine-tune.\n2. As mentioned in the paper, the offline phase can be expensive.\n3. It cannot correct the embedding function bias.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YLLxZkGabG", "reviewer": "Reviewer_D7oe", "summary": "The authors present a fairness-aware framework for LLM-based recommendation systems that combines conformal prediction with dynamic prompt engineering. FACTER introduces an adaptive semantic variance threshold and a violation-triggered mechanism to tighten fairness constraints when biases arise.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reading our work and your valuable comments. We address your concerns in the following paragraphs.\n\n*Weakness 1:*\n\n## A1.\nWhile FACTER leverages existing techniques such as conformal prediction and prompt engineering, to our knowledge, __no prior work has unified these methods into a closed-loop, adaptive fairness calibration framework that dynamically refines prompts and thresholds to mitigate fairness violations in black-box LLM recommenders with statistical guarantees. Specifically, FACTER introduces the use of conformal prediction with fairness constraints, an iterative violation-triggered prompt refinement strategy that limits token bloat and generalizes to unseen biases, and provides a paradigm shift toward adaptive, statistically grounded fairness for black-box LLMs.__\n\nUnlike static fairness methods (e.g., UP5 [Hua et al., 2023]), FACTER uses conformal prediction ([Angelopoulos et al., 2023]; §3.2) to adaptively tighten thresholds based on violation rates while ensuring 1−α coverage (Eq. 7). This closed-loop calibration addresses distribution shifts and emergent biases, reducing violations by 95.5% vs. UP5 (Table 1 of the original paper). Prior works treat fairness and accuracy as separate objectives ([Dwork et al., 2012]; [Gallegos et al.]). FACTER’s score $S_i = d_i+λ Δ_i$ (Eq. 5) jointly optimizes both, enabling a principled tradeoff validated by ablation studies (Table 5 of the original paper). \n\nUnlike static prompts (e.g., “avoid bias” in [Yang et al., 2022]), FACTER injects concrete bias patterns (e.g., “Gender=F → Romance-Only”) from a violation buffer (§3.3). This iterative refinement reduces token bloat while generalizing to unseen biases, outperforming Zero-Shot by 22× in violations (Table 1 of the original paper). Our proofs for embedding robustness (Theorem 1) and threshold convergence (Theorem 2) provide formal guarantees absent in prior fairness frameworks ([Shafer & Vovk, 2008]; [Madras et al., 2018]). \n\nTherefore, FACTER is not a simple combination of tools but a paradigm shift toward adaptive, statistically grounded fairness for black-box LLMs.\n\n*Weakness 2:*\n\n## A2.\n__Please refer to A1 in the Reviewer 49dx rebuttal section.__ \n\n*Weakness 3:*\n\n## A3.\nThank you for your excellent suggestion. The choice of α directly controls the conformal coverage guarantee (1−α), ensuring that the probability of falsely flagging a fair recommendation as biased (Type I error) is bounded by α. This aligns with the theoretical foundations of conformal prediction (Angelopoulos et al., 2023). Specifically, setting α=0.10 provides a 90% coverage guarantee, meaning 90% of fair recommendations will not be erroneously flagged.\n\nHowever, lowering α (e.g., α=0.05) tightens the threshold, reducing Type I errors but potentially increasing Type II errors (failing to detect true violations). Conversely, higher α (e.g., α=0.20) relaxes the threshold, increasing Type I errors but improving detection power. To validate this trade-off empirically, based on the reviewer’s comment, we conducted an ablation study (Table 1 below) on MovieLens-1M, measuring violations, Type I/II errors, and recommendation quality. The results are reported in Table A1.\n\n*Table A5: Impact of α on Fairness-Accuracy Tradeoff*\n\nα | Coverage (1−α) | Type I Error (↓) | Type II Error (↓) | #Violations (↓) | NDCG@10 (↑)\n|-|-|-|-|-|-|\n0.01 | 99%\t         | 0.6%\t                | 18%\t            | 4\t                 | 0.454\n0.05 | 95%\t         | 1.2%\t                | 12%\t            |  9\t                 | 0.451\n0.10 | 90%\t         | 2.1%\t                | 8%\t            | 15\t                 | 0.447\n0.20 | 80%\t         | 4.0%\t                | 5%\t            | 27\t                 | 0.442\n\nAs the results show, lower α (e.g., 0.01) prioritizes strict fairness (fewer violations, low Type I errors) but risks missing true biases (higher Type II errors). Higher α (e.g., 0.20) improves detection power (lower Type II errors) at the cost of increased false alarms. \n\nIn the final manuscript, we will expand Section 3.2 to explicitly discuss how α governs the Type I/II error trade-off, referencing Eq. (7) in our paper and Theorem 1 in Appendix J.1.1.\n\n*Weakness 4:*\n## A4.\n__Please refer to A2 in the Reviewer b7t4 rebuttal section.__\n\nMoreover, regarding your concern about essential references, we will include the mentioned [3,4] references, which are survey papers, to the final version of the paper. Unfortunately, we do not know what other missing references you have in mind. If there is an opportunity for you to anonymously provide these additional references to us, we will be grateful and will add them (and even provide comparisons with the most relevant ones) to the final manuscript.", "claims_and_evidence": "mostly supported", "methods_and_evaluation_criteria": "probably sound", "theoretical_claims": "Proofs for some claims.", "experimental_designs_or_analyses": "Probably valid.", "relation_to_broader_scientific_literature": "Related to LLM recommendation fairness.", "essential_references_not_discussed": "Some references are not included.", "other_strengths_and_weaknesses": "Strengths\n1. Integrate Conformal Prediction for Fairness Calibration\nPrior works on fairness in LLM-based recommendation [1] mainly rely on direct re-ranking or pretraining constraints. The paper leverages conformal prediction to define fairness violation thresholds, a statistically principled approach in LLM recommendations. \n\n2. Effective Bias Mitigation Without Model Retraining\nUnlike prior adversarial training-based methods (e.g., [2]), which require modifying model parameters, FACTER works in a black-box setting, making it suitable for API-based deployments (e.g., OpenAI, Hugging Face models).\n\n \n\n\nWeaknesses\n1. Incremental. The paper combines multiple existing techniques, such as conformal prediction and prompt engineering, but offers limited technical novelty.\n\n2. Over-Reliance on Embedding-Based Fairness Measures\nFACTER defines fairness violations using embedding distances (computed via Sentence-Transformers). This assumes that embeddings capture fairness-sensitive information correctly, which may not always be valid. Bias in embeddings themselves could affect fairness evaluations [3,4]. If the embeddings already encode demographic biases, fairness constraints based on them may be flawed.\n\n\n3. Limited Justification for Fairness Threshold Selection\nThe paper claims to use conformal prediction for threshold calibration, but the choice of α (confidence level) lacks theoretical justification. The authors should conduct an ablation study comparing different threshold selection methods (e.g., data-driven quantile calibration vs. fixed conformal bounds).\n\n\n4. Scalability Issues in Large-Scale Deployments\nFACTER’s offline calibration phase has O(n²) complexity, requiring pairwise similarity comparisons across all calibration points. This makes it computationally expensive for large-scale datasets (e.g., Movielens20M has more than 100k users). The paper claims that approximate nearest neighbor search (O(n log n)) can improve efficiency, but this is not tested empirically.\n\n\n\n\n[1] Hua, W., Ge, Y., Xu, S., Ji, J., & Zhang, Y. (2023). Up5: Unbiased foundation model for fairness-aware recommendation. arXiv preprint arXiv:2305.12090.\n[2] Madras, D., Creager, E., Pitassi, T., & Zemel, R. (2018, July). Learning adversarially fair and transferable representations. In International Conference on Machine Learning (pp. 3384-3393). PMLR.\n[3] Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., ... & Ahmed, N. K. (2024). Bias and fairness in large language models: A survey. Computational Linguistics, 50(3), 1097-1179.\n[4] Li, Y., Du, M., Song, R., Wang, X., & Wang, Y. (2023). A survey on fairness in large language models. arXiv preprint arXiv:2308.10149.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors' intervention is clever, relatively low-overhead (compared to retraining, at least), and their results (including the ones presented in the rebuttal) are broadly compelling. There were some concerns about justification and incremental contribution, but these concerns were reasonably well-rebutted.
Decision: Accept (poster)