Multi-lingual Query Scope Classifier (addyo07/query-scope-classifier)

A production-grade, fast, multi-lingual single-pass sequence classifier fine-tuned from answerdotai/ModernBERT-base to categorize incoming user queries into 4 distinct scope categories across English, Devanagari Hindi, and Hinglish.

🏷️ 4-Class Taxonomy

1. ChitChat (Label 0): Casual greetings, small talk, AI identity questions, emotional banter.
2. User (Label 1): Personal facts, user preferences, memory updates, user profile instructions.
3. Domain (Label 2, Primary Default): Code execution, math formulas, general domain task queries, technical instructions.
4. Temporal (Label 3): Time-sensitive queries, schedules, dates, past session history, reminders.

---

📊 Performance & SLA Benchmarks

• Base Architecture: answerdotai/ModernBERT-base (149M parameters, RoPE, Unpadded FlashAttention-2).
• Holdout Test Accuracy: 96.18% across 2,201 holdout samples.
• Macro F1 Score: 0.9619
• Calibrated Non-Default Precision: 98.01% at confidence threshold tau* = 0.81 (with automatic safe fallback to Domain when uncertain).
• Quantized INT8 ONNX File Size: 143.67 MB

Per-Class Recall Breakdown

Scope Class	Recall	Precision	F1-Score
ChitChat	98.00%	98.50%	0.9825
Temporal	97.28%	97.80%	0.9754
User	95.27%	97.73%	0.9648
Domain (Default)	94.18%	95.20%	0.9469

---

📁 Repository Structure

.gitattributes
README.md
model/
  onnx/
    config.json
    model_quantized.onnx         # 143.67 MB Dynamic INT8 ONNX model
  pytorch/
    config.json
    model.safetensors            # 571 MB PyTorch BFloat16 weights
    tokenizer.json
    tokenizer_config.json
scripts/                          # Full fine-tuning, dataset audit & quantization pipeline

---

💻 Python / PyTorch Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

MODEL_NAME = "addyo07/query-scope-classifier"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, subfolder="model/pytorch")
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME, subfolder="model/pytorch")

labels = ["ChitChat", "User", "Domain", "Temporal"]
query = "aaj sham ko mera schedule kya hai?"

inputs = tokenizer(query, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)
    pred_idx = torch.argmax(probs, dim=-1).item()

print(f"Predicted Scope: {labels[pred_idx]} (Confidence: {probs[0][pred_idx].item():.4f})")

---

⚡ ONNX Runtime Usage (Fast CPU Inference)

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("addyo07/query-scope-classifier", subfolder="model/pytorch")
session = ort.InferenceSession("model/onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])

query = "Remind me to submit the quarterly tax report tomorrow at 5pm"
inputs = tokenizer(query, return_tensors="np", max_length=64, truncation=True)

onnx_inputs = {
    "input_ids": inputs["input_ids"].astype(np.int64),
    "attention_mask": inputs["attention_mask"].astype(np.int64)
}
outputs = session.run(None, onnx_inputs)
logits = outputs[0][0]
probs = np.exp(logits) / np.sum(np.exp(logits))
pred_id = np.argmax(probs)

labels = ["ChitChat", "User", "Domain", "Temporal"]
print(f"Scope: {labels[pred_id]}, Confidence: {probs[pred_id]:.4f}")