bintanghutagalung/banking77-deberta-v3-base

BANKING77 Intent Classification with DeBERTa-v3-base

Fine-tuned microsoft/deberta-v3-base for 77-class banking intent classification using the BANKING77 dataset.

The model maps an English banking customer utterance to one of 77 intents such as card_arrival, cash_withdrawal, pending_transfer, cash_withdrawal_wrong_exchange_rate, and other banking-service categories defined by BANKING77.

Model Details

• Architecture: DeBERTa-v3-base sequence classification
• Base model: microsoft/deberta-v3-base
• Task: Multiclass single-label intent classification
• Number of labels: 77
• Language: English
• Dataset: BANKING77
• Framework: PyTorch + Transformers
• License: MIT

Dataset

BANKING77 contains 13,083 English banking queries across 77 fine-grained intents.

Official dataset split:

Split	Rows
Train	10,003
Test	3,080
Total	13,083

Dataset source:

• https://huggingface.co/datasets/PolyAI/banking77
• https://github.com/PolyAI-LDN/task-specific-datasets/tree/master/banking_data

The official training set was used for model development. The official test set was kept out of model selection and hyperparameter tuning.

For the main final evaluation, rows overlapping with development data and one duplicated test row were excluded, leaving 3,072 unique evaluation examples.

Training Procedure

The project used the following workflow:

1. Audit BANKING77 for missing values, empty text, duplicate text, normalized duplicates, and conflicting labels.
2. Build a group-aware train/validation split from the official training set.
3. Train a TF-IDF + Logistic Regression baseline.
4. Fine-tune microsoft/deberta-v3-base.
5. Select hyperparameters only from validation performance.
6. Freeze the experiment before reading the official test set.
7. Audit the official test set for development overlap and internal duplication.
8. Evaluate the final model on the leakage-clean, deduplicated test population.
9. Compare DeBERTa against the baseline using aggregate metrics, per-class metrics, paired bootstrap, and McNemar testing.

Selected hyperparameters:

Hyperparameter	Value
Learning rate	3e-5
Weight decay	0.01
Warmup ratio	0.1
Per-device batch size	8
Gradient accumulation steps	2
Epochs	3

Training was performed in Google Colab using an NVIDIA T4 GPU.

Evaluation Results

Main evaluation population: 3,072 examples across all 77 intents.

Metric	TF-IDF + Logistic Regression	DeBERTa-v3-base
Macro-F1	0.873659	0.922648
Accuracy	0.873372	0.922852
Macro Precision	0.880722	0.925809
Macro Recall	0.873347	0.922858
Weighted-F1	0.873635	0.922591
Error Rate	0.126628	0.077148
Log Loss	1.179109	0.326044
Brier Score	0.430436	0.121375
Expected Calibration Error	0.445879	0.033650

The Macro-F1 improvement over the baseline was +0.048989.

Paired bootstrap 95% confidence interval for the Macro-F1 difference:

• Lower bound: 0.038244
• Upper bound: 0.060958

McNemar comparison:

• DeBERTa-only correct cases: 233
• Baseline-only correct cases: 81
• The project evaluation marked the improvement over the baseline as statistically supported.

Per-intent comparison:

• 61 intents improved in F1
• 4 intents were unchanged
• 12 intents regressed in F1

Usage

Install the required packages:

pip install transformers torch sentencepiece

Load the model:

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_id = "bintanghutagalung/banking77-deberta-v3-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "I am still waiting for my card"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True
)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = int(logits.argmax(dim=-1).item())
predicted_intent = model.config.id2label[predicted_id]

print(predicted_intent)

For probabilities:

probabilities = torch.softmax(logits, dim=-1)[0]
confidence = float(probabilities[predicted_id])

print({
    "intent": predicted_intent,
    "confidence": confidence
})

Intended Use

This model is intended for:

• banking customer-service intent classification;
• routing banking queries to downstream workflows;
• research and benchmarking on BANKING77;
• experimentation with fine-grained intent detection.

Out-of-Scope Use

This model should not be treated as:

• a banking decision system;
• a fraud-detection model;
• a credit-scoring model;
• a financial-advice system;
• a replacement for human review in high-impact cases.

It predicts an intent label from text. It does not verify whether the customer's statement is true and does not determine the correct financial action by itself.

Limitations

• The model was evaluated on BANKING77, not production traffic from a specific bank.
• BANKING77 is English-language data; performance on Indonesian, multilingual, or code-switched queries has not been established.
• BANKING77 has documented potential label errors in part of its training data.
• DeBERTa did not outperform the baseline on every intent; 12 of 77 intents showed lower F1.
• Confidence scores are not guarantees of correctness.
• Production performance can degrade under distribution shift, new banking products, new terminology, adversarial inputs, or intent definitions that differ from BANKING77.
• A confidence threshold and human fallback should be validated on the target deployment data before operational use.

Reproducibility

The accompanying GitHub repository contains the training notebook and selected experiment artifacts, including:

• final evaluation metrics;
• per-class metrics;
• hyperparameter configuration;
• baseline comparison;
• statistical comparison;
• test-exclusion audit;
• label mapping;
• model metadata.

GitHub repository: https://github.com/BINTANNG99/klasifikasi-intent-banking77-deberta-v3-base

License

The fine-tuned model is released under the MIT License, consistent with the base model license.

BANKING77 is a separate dataset and remains subject to its own license and attribution requirements.

References

Casanueva, I., Temčinas, T., Gerz, D., Henderson, M., & Vulić, I. (2020).
Efficient Intent Detection with Dual Sentence Encoders.
https://aclanthology.org/2020.nlp4convai-1.5/

He, P., Gao, J., & Chen, W. (2021).
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing.
https://arxiv.org/abs/2111.09543

Ying, R., & Thomas, C. (2022).
Label Errors in BANKING77.
https://aclanthology.org/2022.insights-1.19/