QWEN_OPENIIT: Promise-to-Pay call copilot (Qwen2.5-0.5B + LoRA)

A small language model that reads a live, code-mixed (Hinglish / Kanglish / English) debt-collection call and, at any point in the conversation, classifies the promise-to-pay (PTP) under discussion into one of six behavioural types and proposes the next agent line.

It was built for the CreditNirvana problem statement "Real-Time Fake PTP Detection & In-Call Guidance" (Open IIT Data Analytics). Code, data pipeline, tabular models and the web app are in the project's GitHub repository (branch FINAL).

Base model	Qwen/Qwen2.5-0.5B-Instruct
Fine-tuning	LoRA r=16, α=32, dropout 0.05, all attention + MLP projections (8.8 M trainable params, 1.75%)
Training data	2,400 dialogue prefixes from 1,680 training-split accounts, 2 epochs, lr 2e-4, effective batch 16
Files	root = merged weights (bf16, ready to load); lora_adapter/ = the LoRA adapter only (34 MB)

Output format

The model answers <code>|<category>|<next agent line>. The first generated token is a digit 1–6, so the category, and a confidence from the softmax over the six digit logits, are available at time-to-first-token.

code	category	meaning
1	genuine_feasible	means it and can pay
2	genuine_infeasible	means it, but amount/date don't fit cash flow (hardship)
3	escape	says yes to end the call
4	third_party	a relative/colleague is promising on the borrower's behalf
5	repeat_promiser	long history of broken promises
6	agent_recorded_or_pushed	the agent forced or logged a date the borrower never gave

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Shubhbagaria12/QWEN_OPENIIT"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.float16).cuda().eval()

SYS = ("You are a debt-collection call copilot. Read the live call and answer as "
       "<code>|<category>|<next agent line>. Codes: 1=genuine_feasible 2=genuine_infeasible 3=escape "
       "4=third_party 5=repeat_promiser 6=agent_recorded_or_pushed. Never disclose debt details to a "
       "third party. Never threaten. Protect hardship cases.")
prompt = (
    "<|im_start|>system\n" + SYS + "<|im_end|>\n<|im_start|>user\n"
    "CTX lang=hinglish portfolio=credit_card prior_broken=2 salary_day=28 promise=Rs30900 on 06 Apr\n"
    "CALL:\n"
    "A: Good morning sir main se call kar raha hoon kya ji line pe hain\n"      # names are masked in the data
    "C: Main mummy hoon woh kaam pe gaya hai\n"
    "A: Kya aap unko message de sakte hain ki se call aaya tha?\n"
    "C: main bata dunga 6 tareekh tak ho jayega.\n"
    "<|im_end|>\n<|im_start|>assistant\n"
)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")

# category + confidence from the first token
code_ids = [tok(str(i), add_special_tokens=False)["input_ids"][0] for i in range(1, 7)]
with torch.no_grad():
    logits = model(**ids).logits[0, -1, code_ids].float()
probs = torch.softmax(logits / 1.156, -1)      # temperature fitted on validation (fp16)
print(probs.argmax().item() + 1, probs.max().item())

# full answer
out = model.generate(**ids, max_new_tokens=90, do_sample=False)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
# -> 4|third_party|Koi baat nahi. [NAME] ji se baat karne ka sahi samay kya rahega? ...

Prompt rules used in training:

• one line per utterance: A: agent, B: voice bot, C: customer;
• the CTX line gives language, portfolio, prior broken PTPs, salary day (? if unknown) and the promise under discussion;
• any prefix of the call can be scored, so it works turn by turn.

4-bit (fits < 1 GB VRAM): load with BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True) and use temperature 1.127.

Evaluation (held-out test split, 484 calls)

setting	accuracy	macro-F1	top-1 ECE (calibrated)
fp16, end of call	0.645	0.673	0.042
fp16, mid-call prefix	0.641	0.669	0.055
NF4 4-bit, end of call	0.605	0.642	0.039

Per class (fp16, end of call):

category	precision	recall	F1
genuine_feasible	0.561	0.696	0.621
genuine_infeasible	0.639	0.434	0.517
escape	0.606	0.548	0.576
third_party	0.949	0.982	0.966
repeat_promiser	0.464	0.553	0.505
agent_recorded_or_pushed	0.837	0.872	0.854

Generated text on 150 test calls: 100% valid format, 37% exact match with the approved policy script, 0 compliance violations (debt disclosure to a third party, threatening language).

Latency (RTX 3050 Laptop 4 GB, batch 1, HF eager, CUDA-synchronised, mean prompt 330 tokens):

precision	peak VRAM	time-to-first-token p50 / p95 (incremental KV cache)	decode
fp16	1.28 GB	27 / 38 ms	46 tok/s
NF4 4-bit	0.74 GB	47 / 57 ms	30 tok/s

Intended use and limitations

• Decision support only. On its own the model sends 17% of hardship (genuine_infeasible) cases to escape/repeat. In the reference system its category is a second opinion next to a calibrated LightGBM model, and deterministic guardrails can only soften treatment (hardship → restructuring/human; third party → no disclosure). Do not use it to make collection decisions alone.
• Never show free-generated text to agents or borrowers unchecked. The model can pair a code with the wrong script or invent dates. The reference system re-renders the approved script for the final, guard-railed category.
• Weaker on history-dependent classes (repeat_promiser, feasible vs infeasible), because it sees account history only as a one-line summary.
• Training data: CreditNirvana sandbox call transcripts (templated, romanised code-mixed text). Real ASR output, native scripts (Devanagari, Kannada) and other languages will need further fine-tuning.
• Labels come from a rule-based labeller plus self-training, whose agreement with human annotators is 0.67 (macro-F1 0.69); that bounds what any model trained on them can show.
• Regulatory context: RBI Fair Practices Code and India's DPDP Act. Voice-derived signals are sensitive processing; purpose and retention limits apply.