Gydeai/complaint-classifier-us-cfpb

Gyde Complaint Classifier: us-cfpb

A LoRA adapter for Qwen3-8B that files a consumer financial complaint into the US Consumer Financial Protection Bureau's product (10 classes) and issue (82 classes).

Part of the Gyde Complaint Classifier family: each regulator's own complaint categories, mapped to one shared top-level taxonomy (universal-taxonomy.json, 11 categories found in every banking system), with one adapter per regulator, so complaints from different jurisdictions can be compared at that top level.

Jurisdiction	Taxonomy	Training data	Evaluated on	Status
US	CFPB product and issue	8,000 real public complaints	1,999 held-out real complaints	This release
India	RBI Integrated Ombudsman Scheme categories	Synthetic (written by Reflection Beam)	Synthetic complaints with AI-agreed (silver) labels; no real-complaint evaluation yet	Preview
Other regulators	e.g. FCA (UK), AFCA (Australia), MAS (Singapore)	—	—	On request

Results

1,999 held-out complaints from the CFPB Consumer Complaint Database, same test set for every model.

Model	Product accuracy	Issue accuracy	Both correct
Claude Sonnet 5.5, 5 retrieved training complaints as examples	84.7%	56.6%	55.8%
This adapter (Qwen3-8B + LoRA, rank 16)	83.5%	55.0%	54.4%
Reflection Beam (501B mixture of experts, 23B active), prompt only	82.9%	33.2%	31.5%
Claude Sonnet 5.5, prompt only	82.3%	44.1%	42.9%
Claude Haiku 5.5, prompt only	83.6%	37.8%	35.9%
Claude Haiku 5.5, 5 retrieved examples	84.4%	47.9%	—
TF-IDF + logistic regression (same 8,000 training complaints)	80.8%	52.6%	—

Product accuracy by class: Mortgage 94% (184 test complaints), Student loan 94% (69), Credit reporting 87% (487), Checking or savings account 85% (266), Debt collection 82% (422), Credit card or prepaid card 80% (354), Money transfer, virtual currency or money service 76% (94), Vehicle loan or lease 71% (83), Payday, title or personal loan 60% (40). "Debt or credit management" has no test complaints.

Product accuracy sits near 82-85% for every model we tried, including much larger ones: the label is the consumer's own choice and some products overlap (credit reporting vs debt collection). The 82-class issue label is where training on the data pays off: +11 points over Claude Sonnet, +17 over Claude Haiku and +22 over a 501B model, all prompted without examples. Sonnet given the five most similar training complaints as examples is 1.6 points ahead on issue, at roughly 900 times the cost per complaint (below). On the first 200 test complaints only, Claude Opus 5.5 prompted without examples reached 61.5% issue accuracy (this adapter: 51.5% on the same 200); not run on the full set.

Cost per 1,000 complaints (USD)

Model	Real-time	Batch
This adapter, self-hosted on one NVIDIA A30	~$0.012	~$0.014 (10,000 per batch) to ~$0.034 (1,000 per batch)
Claude Haiku 5.5, prompt only	~$2.6	~$0.7
Claude Sonnet 5.5, prompt only	~$5.6	~$0.9
Claude Sonnet 5.5, 5 examples	~$10.5	~$3.4

Estimates, not invoices. Self-hosted: one NVIDIA A30 in India at about $0.41/hour (list price ₹38.88, converted at USD 1 = ₹95), about 10 complaints/s, billed per minute; real-time assumes the GPU is busy (at low volume a dedicated GPU costs about $9.80/day whatever the traffic, so run in batches or share it across tasks); batch includes about 160 s of GPU start-up. API: list prices ($1/$5 and $3/$15 per million input/output tokens) on our measured tokens per complaint; batch applies the 50% batch discount and caches the shared instructions (the retrieved examples differ per complaint and are not cached). At bank volumes both are small in absolute terms; the adapter's case is fine-grained accuracy, data residency and control as much as cost.

Use

Base model Qwen/Qwen3-8B (revision b968826d9c46dd6066d109eabc6255188de91218). Greedy decoding (temperature 0), max_tokens 80, thinking disabled. The adapter was trained on exactly this prompt:

• System: You file US consumer finance complaints by product and issue. Reply with JSON only: {"product": ..., "issue": ...}.
• User: Complaint: {text}\nAnswer:

With vLLM:

vllm serve Qwen/Qwen3-8B --revision b968826d9c46dd6066d109eabc6255188de91218 \
  --enable-lora --max-lora-rank 16 --lora-modules us-cfpb=Gydeai/complaint-classifier-us-cfpb

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
r = client.chat.completions.create(
    model="us-cfpb", temperature=0, max_tokens=80,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    messages=[{"role": "system", "content": 'You file US consumer finance complaints by product and issue. '
                                            'Reply with JSON only: {"product": ..., "issue": ...}.'},
              {"role": "user", "content": "Complaint: My card was charged twice and the bank will not reverse it.\nAnswer:"}])
print(r.choices[0].message.content)   # {"product": "Credit card or prepaid card", ...}

labels.json lists the 10 products and 82 issues; universal-taxonomy.json maps each product to the shared categories.

Gyde also serves this model inside India with data-residency, policy and failover controls, and trains versions on your own categories: gyde.ai.

Training

LoRA rank 16, alpha 32, dropout 0.05 on all attention and MLP projections; 1 epoch over 8,000 complaints, learning rate 2e-4; about 40 minutes on one A100 40 GB (run.json).

Limits

• US complaints and US categories only. An Indian UPI or ATM complaint will be put in the nearest US category.
• English only. Narratives are as written by US consumers, with the CFPB's redactions.
• For triage and analytics, not for deciding a customer's case. Send low-confidence answers to a person.
• Base model origin: Qwen (Alibaba Cloud). A version on another base model is available on request.

Data and licence

Training and test data: CFPB Consumer Complaint Database, a US government work. Adapter: Apache 2.0. Base model: Qwen3-8B under its own licence (Apache 2.0).

Built by Gyde (DAI Labs Private Limited, Pune, India).