Fintech-Fine-Tune — Qwen3-0.6B + LoRA

A LoRA adapter for Qwen3-0.6B that turns natural-language fintech commands/questions into a fixed JSON schema, so a downstream system can act on them without a rule-based parser.

"can you pay the rent bill, it's $1,249?"
      -> {"intent": "pay_bill", "entity": "rent", "amount": 1249, "account": null}

"dump 20 shares of META"
      -> {"intent": "sell_stock", "entity": "META", "amount": 20, "account": "investment"}

Trained entirely on a local CPU (no GPU) using peft LoRA — full training code, dataset, and inference script: GitHub repo.

Output schema

{"intent": "<one of 20 intents>", "entity": "<string or null>", "amount": "<number or null>", "account": "checking | savings | credit | investment | null"}

Intents: check_balance, transfer_funds, pay_bill, dispute_transaction, freeze_card, unfreeze_card, report_lost_card, request_new_card, view_transaction_history, set_spending_alert, update_credit_limit, apply_for_loan, check_loan_status, buy_stock, sell_stock, check_portfolio, schedule_recurring_payment, cancel_recurring_payment, open_account, close_account.

Training

LoRA rank / alpha	8 / 16
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable params	5,046,272 / 601,096,192 (0.84%)
Epochs	3
Effective batch size	16 (batch 4 × grad-accum 4)
Learning rate	2e-4, cosine schedule, 3% warmup
Hardware	CPU only, float32
Data	450 train / 60 val, synthetic, 20 intents, roughly balanced

Epoch	eval_loss
1	0.0789
2	0.0265
3 (final)	0.0214

Known limitations

The eval loss above is measured on validation examples generated by the same synthetic process as training, so it mostly reflects whether the model learned the output format. Testing separately on 10 hand-written prompts with fresh wording/entities not in train or val told a more honest story:

• 10/10 produced syntactically valid JSON with exactly the 4 expected keys.
• 7/10 predicted a correct, valid intent.
• 3/10 hallucinated a plausible-looking intent value that isn't in the 20-label taxonomy at all (e.g. lock_renewable_document instead of freeze_card for "lock my visa, someone jacked it"; check_revenue instead of check_balance for "how much runway do I have left in checking").

Solid for cleanly-phrased commands close to the training distribution; not yet reliable enough to trust blindly on informal or unusual phrasing. Validate intent against the known list of 20 before acting on it downstream.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_MODEL = "Qwen/Qwen3-0.6B"
ADAPTER = "Gaurav8HF/Fintech-Fine-Tune"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, dtype=torch.float32)
model = PeftModel.from_pretrained(base_model, ADAPTER)

messages = [{"role": "user", "content": "freeze my card, I think I lost it"}]
prompt_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=False,
    enable_thinking=False,  # must match training -- Qwen3 normally "thinks" before answering
)
out = model.generate(prompt_ids, max_new_tokens=64, do_sample=False, pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(out[0, prompt_ids.shape[1]:], skip_special_tokens=True))

Framework versions

• PEFT 0.19.1