adapterops-drafting

Writes a customer-support reply to a request.

Part of AdapterOps: four LoRA adapters over one Qwen2.5-1.5B base, served together with vLLM multi-LoRA. Portfolio project — no real users or customer data.

The scores below describe revision 3778c895d3501f4e8d513252e8e217da4287e720 (adapter weights sha256 2e3eeb7e0aac6087…), the revision the project serves. Load that revision rather than main.

Prompt

Write a helpful customer-support reply to this request.
Request: {text}
Reply:

Raw text, no chat template. Greedy decoding, at most 448 new tokens. Replace {text} with the input.

Evaluation

Golden sets are frozen random held-out splits; every system below was run on the same items. The hard-cases split is mined from this adapter's own failures, so it is report-only and sits near zero by construction for classification.

system	split (n)	metric	score
this adapter	golden (300)	GPT-4o grade, 1–5	4.2367
base model, prompted	golden (300)	GPT-4o grade, 1–5	2.8567
GPT-4o-mini (frontier reference)	golden	GPT-4o grade, 1–5	4.5167
this adapter, run 1 / run 2	golden (300)	distilled judge	4.2709 / 4.2844
this adapter	hard cases (113), report-only	distilled judge	3.6918

Latency with all four adapters served at once on one A10 (vLLM, concurrency 16): P50 1,249 ms · P95 3,000 ms.

Caveats

• The distilled judge (the gate metric) tracks GPT-4o on this adapter's replies (Spearman 0.74) but not on another generator's (0.33). Compare models on GPT-4o grades.
• Replies can contain template slots such as {{Order Number}}, from the Bitext data.
• Share-alike: trained on CDLA-Sharing-1.0 data.

Training

QLoRA (4-bit NF4) on Qwen/Qwen2.5-1.5B-Instruct, LoRA rank 16, alpha 32, on all attention and MLP projections; prompt tokens masked from the loss. 6,000 training rows from bitext/Bitext-customer-support-llm-chatbot-training-dataset (cdla-sharing-1.0).

Full decision log, results and negative findings: https://github.com/tpawar03/AdapterOps.