byosm-intent

Route a banking customer's message to one of 77 intents (Banking77).

A single-purpose System 1 model built with BYOSM: fine-tuned from sentence-transformers/all-MiniLM-L6-v2 for this task only, with a calibrated decision policy, shipped as a 22.74M-parameter ONNX bundle that runs on any CPU with onnxruntime.

task	text / classify
labels	Refund_not_showing_up, activate_my_card, age_limit, apple_pay_or_google_pay, atm_support, automatic_top_up, balance_not_updated_after_bank_transfer, balance_not_updated_after_cheque_or_cash_deposit, beneficiary_not_allowed, cancel_transfer, card_about_to_expire, card_acceptance, card_arrival, card_delivery_estimate, card_linking, card_not_working, card_payment_fee_charged, card_payment_not_recognised, card_payment_wrong_exchange_rate, card_swallowed, cash_withdrawal_charge, cash_withdrawal_not_recognised, change_pin, compromised_card, contactless_not_working, country_support, declined_card_payment, declined_cash_withdrawal, declined_transfer, direct_debit_payment_not_recognised, disposable_card_limits, edit_personal_details, exchange_charge, exchange_rate, exchange_via_app, extra_charge_on_statement, failed_transfer, fiat_currency_support, get_disposable_virtual_card, get_physical_card, getting_spare_card, getting_virtual_card, lost_or_stolen_card, lost_or_stolen_phone, order_physical_card, passcode_forgotten, pending_card_payment, pending_cash_withdrawal, pending_top_up, pending_transfer, pin_blocked, receiving_money, request_refund, reverted_card_payment?, supported_cards_and_currencies, terminate_account, top_up_by_bank_transfer_charge, top_up_by_card_charge, top_up_by_cash_or_cheque, top_up_failed, top_up_limits, top_up_reverted, topping_up_by_card, transaction_charged_twice, transfer_fee_charged, transfer_into_account, transfer_not_received_by_recipient, transfer_timing, unable_to_verify_identity, verify_my_identity, verify_source_of_funds, verify_top_up, virtual_card_not_working, visa_or_mastercard, why_verify_identity, wrong_amount_of_cash_received, wrong_exchange_rate_for_cash_withdrawal
parameters	22,743,629 (shipped as int8 ONNX, 23 MB)
backbone	hf:sentence-transformers/all-MiniLM-L6-v2 · embedding dim 384
objective	accuracy, primary suite b77
decision policy	threshold, threshold – (calibrated on b77's calib split)

Labels

• Refund_not_showing_up: Refund not showing up
• activate_my_card: activate my card
• age_limit: age limit
• apple_pay_or_google_pay: apple pay or google pay
• atm_support: atm support
• automatic_top_up: automatic top up
• balance_not_updated_after_bank_transfer: balance not updated after bank transfer
• balance_not_updated_after_cheque_or_cash_deposit: balance not updated after cheque or cash deposit
• beneficiary_not_allowed: beneficiary not allowed
• cancel_transfer: cancel transfer
• card_about_to_expire: card about to expire
• card_acceptance: card acceptance
• card_arrival: card arrival
• card_delivery_estimate: card delivery estimate
• card_linking: card linking
• card_not_working: card not working
• card_payment_fee_charged: card payment fee charged
• card_payment_not_recognised: card payment not recognised
• card_payment_wrong_exchange_rate: card payment wrong exchange rate
• card_swallowed: card swallowed
• cash_withdrawal_charge: cash withdrawal charge
• cash_withdrawal_not_recognised: cash withdrawal not recognised
• change_pin: change pin
• compromised_card: compromised card
• contactless_not_working: contactless not working
• country_support: country support
• declined_card_payment: declined card payment
• declined_cash_withdrawal: declined cash withdrawal
• declined_transfer: declined transfer
• direct_debit_payment_not_recognised: direct debit payment not recognised
• disposable_card_limits: disposable card limits
• edit_personal_details: edit personal details
• exchange_charge: exchange charge
• exchange_rate: exchange rate
• exchange_via_app: exchange via app
• extra_charge_on_statement: extra charge on statement
• failed_transfer: failed transfer
• fiat_currency_support: fiat currency support
• get_disposable_virtual_card: get disposable virtual card
• get_physical_card: get physical card
• getting_spare_card: getting spare card
• getting_virtual_card: getting virtual card
• lost_or_stolen_card: lost or stolen card
• lost_or_stolen_phone: lost or stolen phone
• order_physical_card: order physical card
• passcode_forgotten: passcode forgotten
• pending_card_payment: pending card payment
• pending_cash_withdrawal: pending cash withdrawal
• pending_top_up: pending top up
• pending_transfer: pending transfer
• pin_blocked: pin blocked
• receiving_money: receiving money
• request_refund: request refund
• reverted_card_payment?: reverted card payment?
• supported_cards_and_currencies: supported cards and currencies
• terminate_account: terminate account
• top_up_by_bank_transfer_charge: top up by bank transfer charge
• top_up_by_card_charge: top up by card charge
• top_up_by_cash_or_cheque: top up by cash or cheque
• top_up_failed: top up failed
• top_up_limits: top up limits
• top_up_reverted: top up reverted
• topping_up_by_card: topping up by card
• transaction_charged_twice: transaction charged twice
• transfer_fee_charged: transfer fee charged
• transfer_into_account: transfer into account
• transfer_not_received_by_recipient: transfer not received by recipient
• transfer_timing: transfer timing
• unable_to_verify_identity: unable to verify identity
• verify_my_identity: verify my identity
• verify_source_of_funds: verify source of funds
• verify_top_up: verify top up
• virtual_card_not_working: virtual card not working
• visa_or_mastercard: visa or mastercard
• why_verify_identity: why verify identity
• wrong_amount_of_cash_received: wrong amount of cash received
• wrong_exchange_rate_for_cash_withdrawal: wrong exchange rate for cash withdrawal

Comparison

Every model below, this one and the baselines, was scored by the same code on the same frozen test splits, with its decision policy calibrated on the same calib splits. Bold = this model.

model	params	builtin accuracy	b77 accuracy (primary)	builtin ROC-AUC	b77 ROC-AUC
final-s1 (this model)	22.74M	–	0.9374	–	–
ampemallrd-s1 (BYOSM experiment)	22.74M	0.9340	0.9337	–	–

Test evaluation (this model)

suite	dataset	test items	accuracy	95% CI	precision	recall	ROC-AUC	avg precision	decision threshold
b77	mteb/banking77	2460	0.9374	0.928-0.948	–	–	–	–	–

Usage

pip install huggingface_hub numpy onnxruntime tokenizers

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("bsye/byosm-intent")
sys.path.insert(0, path)
from byosm_rt import load

m = load(path)
print(m.predict("so what I wanted to ask was"))
emb = m.encode(...)                # the learned representation (JEPA-style), for probes / retrieval / other heads

Local HTTP server

python byosm_rt.py serve . --port 8800
# GET /v1/models · POST /v1/intent/predict · POST /v1/intent/encode

How it was built

1. Spec: labels, definitions, objective and data plan in spec.yaml (included in this repo).
2. Data (8,964 training / 984 dev examples):

• mteb/banking77: role labeled

3. Supervised training: layer-wise LR decay on pretrained encoders (floor 0.5), EMA of the weights, mixed precision, the benchmark's builtin test split folded into training; checkpoint chosen on dev (best epoch 3).
4. Evaluation: frozen eval suites; decision policy calibrated on each suite's calib split; baselines scored identically; promotion decided on calib, test only reported.
5. Improvement lineage (each step proposed by byosm improve from the previous board):

• spec.yaml: initial spec
• ampemallrd-s1 ← manual (from spec.yaml): train.layer_decay_floor=0.5 train.probe_epochs=0 train.ema=true train.seed=1
• final-s1 ← manual (from ampemallrd-s1): train.holdout_test=auto

Limitations

• Evaluated on b77 (2460 test items). Small suites have wide confidence intervals.

Reproduce

git clone https://github.com/bsye/BYOSM && cd BYOSM && pip install -e .
# put spec.yaml from this repo at specialists/intent/spec.yaml, then:
byosm build intent && byosm board intent && byosm export intent