NajahUniv/AraUni-MARBERTv2-Intent-Classifier

AraUni MARBERTv2 Multi-label Classifier

This is a 20-label Arabic intent classifier fine-tuned from UBC-NLP/MARBERTv2. It routes one question to zero, one, or several university-information labels. The model uses mask-aware mean pooling and independent sigmoid outputs; it is not a generative model.

Model details

• Architecture: MARBERTv2 encoder + dropout + linear multi-label head
• Pooling: attention-mask-aware mean pooling
• Maximum input length: 256 tokens
• Weights: SafeTensors
• Training data: NajahUniv/arabic-univeristy-chatbot-qa-cleaned
• Pinned dataset revision: 9136ae156c49e1cc014020cef75c20a8b8d9aae0
• Decision thresholds: selected on validation data and stored in config.json and thresholds.json

The repository includes custom Transformers modeling code so it loads through the standard AutoClass API. Review the code in this repository before enabling trust_remote_code; in a production deployment, pin revision to a reviewed model commit SHA.

Evaluation

Split	Macro F1	Micro F1	Subset accuracy	LRAP
Validation	0.9874	0.9881	0.9760	0.9955
Test	0.9880	0.9881	0.9760	0.9968

Threshold selection used only the validation split. The test split remained held out until the final comparison. Full aggregate and per-label results are available in validation_metrics.json and test_metrics.json.

Basic usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "NajahUniv/AraUni-MARBERTv2-Intent-Classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    trust_remote_code=True,
).eval()

text = "ما هي شروط القبول في الجامعة؟"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=model.config.max_length)
with torch.inference_mode():
    probabilities = torch.sigmoid(model(**inputs).logits[0])

result = {}
for index, probability in enumerate(probabilities.tolist()):
    label = model.config.id2label[index]
    threshold = model.config.thresholds[label]
    result[label] = {"probability": probability, "selected": probability >= threshold}

selected_labels = [label for label, value in result.items() if value["selected"]]
print(selected_labels)

A complete command-line example is included at examples/basic_inference.py:

python examples/basic_inference.py \
  --model-id NajahUniv/AraUni-MARBERTv2-Intent-Classifier \
  --text "ما هي شروط التسجيل؟"

Complete basic inference example

"""Run multi-label inference with the published Hugging Face model."""

from __future__ import annotations

import argparse
import json

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

DEFAULT_MODEL_ID = "NajahUniv/AraUni-MARBERTv2-Intent-Classifier"


def choose_device(requested: str) -> str:
    if requested != "auto":
        return requested
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model-id", default=DEFAULT_MODEL_ID)
    parser.add_argument("--revision", help="Pin a commit SHA in production")
    parser.add_argument("--text", required=True)
    parser.add_argument("--device", default="auto", choices=("auto", "cpu", "mps", "cuda"))
    parser.add_argument("--top-k", type=int, default=5)
    args = parser.parse_args()

    load_kwargs = {"revision": args.revision} if args.revision else {}
    tokenizer = AutoTokenizer.from_pretrained(
        args.model_id,
        trust_remote_code=True,
        **load_kwargs,
    )
    model = AutoModelForSequenceClassification.from_pretrained(
        args.model_id,
        trust_remote_code=True,
        **load_kwargs,
    )
    device = choose_device(args.device)
    model.to(device).eval()
    encoded = tokenizer(
        args.text,
        return_tensors="pt",
        truncation=True,
        max_length=model.config.max_length,
    ).to(device)
    with torch.inference_mode():
        probabilities = torch.sigmoid(model(**encoded).logits[0]).cpu()

    labels = [model.config.id2label[index] for index in range(model.config.num_labels)]
    thresholds = model.config.thresholds
    ranked = sorted(
        (
            {
                "label": label,
                "probability": float(probabilities[index]),
                "threshold": float(thresholds[label]),
                "selected": float(probabilities[index]) >= float(thresholds[label]),
            }
            for index, label in enumerate(labels)
        ),
        key=lambda item: item["probability"],
        reverse=True,
    )
    result = {
        "text": args.text,
        "selected_labels": [item["label"] for item in ranked if item["selected"]],
        "scores": ranked[: max(1, args.top_k)],
    }
    print(json.dumps(result, ensure_ascii=False, indent=2))


if __name__ == "__main__":
    main()

FastAPI hosting

The included service loads the model once, supports batching, uses MPS automatically on Apple Silicon, and optionally requires a bearer token:

pip install -r requirements.txt
MODEL_ID=NajahUniv/AraUni-MARBERTv2-Intent-Classifier MODEL_API_KEY=change-me \
  uvicorn examples.fastapi_app:app --host 0.0.0.0 --port 8000

curl http://localhost:8000/classify \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer change-me' \
  -d '{"texts":["كيف يمكنني دفع الرسوم؟"],"top_k":5}'

In Swagger UI at http://localhost:8000/docs, click Authorize and enter only the MODEL_API_KEY value (change-me in the example). Swagger adds the Bearer prefix.

Complete FastAPI server example

"""FastAPI service for the published multi-label classifier."""

from __future__ import annotations

import hmac
import os
import threading
from contextlib import asynccontextmanager
from typing import Annotated

import torch
from fastapi import Depends, FastAPI, HTTPException
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from pydantic import BaseModel, Field
from transformers import AutoModelForSequenceClassification, AutoTokenizer

DEFAULT_MODEL_ID = "NajahUniv/AraUni-MARBERTv2-Intent-Classifier"
MODEL_ID = os.getenv("MODEL_ID", DEFAULT_MODEL_ID)
MODEL_REVISION = os.getenv("MODEL_REVISION")
MODEL_DEVICE = os.getenv("MODEL_DEVICE", "auto")
MODEL_API_KEY = os.getenv("MODEL_API_KEY")
MAX_BATCH_SIZE = int(os.getenv("MAX_BATCH_SIZE", "64"))


def choose_device() -> str:
    if MODEL_DEVICE != "auto":
        return MODEL_DEVICE
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"


class ClassifyRequest(BaseModel):
    texts: list[str] = Field(min_length=1)
    top_k: int = Field(default=5, ge=1)
    threshold: float | None = Field(default=None, gt=0, lt=1)


class ModelRuntime:
    def __init__(self) -> None:
        load_kwargs = {"revision": MODEL_REVISION} if MODEL_REVISION else {}
        self.tokenizer = AutoTokenizer.from_pretrained(
            MODEL_ID,
            trust_remote_code=True,
            **load_kwargs,
        )
        self.model = AutoModelForSequenceClassification.from_pretrained(
            MODEL_ID,
            trust_remote_code=True,
            **load_kwargs,
        )
        self.device = choose_device()
        self.model.to(self.device).eval()
        self.lock = threading.Lock()

    def classify(self, request: ClassifyRequest) -> list[dict[str, object]]:
        if len(request.texts) > MAX_BATCH_SIZE:
            raise HTTPException(413, f"at most {MAX_BATCH_SIZE} texts are allowed per request")
        encoded = self.tokenizer(
            request.texts,
            return_tensors="pt",
            padding=True,
            truncation=True,
            max_length=self.model.config.max_length,
        ).to(self.device)
        with self.lock, torch.inference_mode():
            probabilities = torch.sigmoid(self.model(**encoded).logits).cpu()
        labels = [
            self.model.config.id2label[index]
            for index in range(self.model.config.num_labels)
        ]
        results = []
        for text, row in zip(request.texts, probabilities, strict=True):
            scores = []
            for index, label in enumerate(labels):
                threshold = (
                    request.threshold
                    if request.threshold is not None
                    else float(self.model.config.thresholds[label])
                )
                scores.append(
                    {
                        "label": label,
                        "probability": float(row[index]),
                        "threshold": threshold,
                        "selected": float(row[index]) >= threshold,
                    }
                )
            scores.sort(key=lambda item: item["probability"], reverse=True)
            results.append(
                {
                    "text": text,
                    "selected_labels": [item["label"] for item in scores if item["selected"]],
                    "scores": scores[: min(request.top_k, len(scores))],
                }
            )
        return results


runtime: ModelRuntime | None = None


@asynccontextmanager
async def lifespan(_: FastAPI):
    global runtime
    runtime = ModelRuntime()
    yield
    runtime = None


app = FastAPI(title="AraUni MARBERTv2 Multi-label Classifier", lifespan=lifespan)
bearer_scheme = HTTPBearer(
    auto_error=False,
    scheme_name="BearerAuth",
    description="Enter the MODEL_API_KEY value. Swagger adds the 'Bearer' prefix.",
)


def authorize(
    credentials: Annotated[
        HTTPAuthorizationCredentials | None,
        Depends(bearer_scheme),
    ],
) -> None:
    if MODEL_API_KEY is None:
        return
    if (
        credentials is None
        or credentials.scheme.lower() != "bearer"
        or not hmac.compare_digest(credentials.credentials, MODEL_API_KEY)
    ):
        raise HTTPException(
            401,
            "invalid bearer token",
            headers={"WWW-Authenticate": "Bearer"},
        )


@app.get("/health")
def health() -> dict[str, object]:
    return {"status": "ok", "model_id": MODEL_ID, "device": runtime.device if runtime else None}


@app.get("/labels", dependencies=[Depends(authorize)])
def labels() -> dict[int, str]:
    if runtime is None:
        raise HTTPException(503, "model is not ready")
    return dict(runtime.model.config.id2label)


@app.post("/classify", dependencies=[Depends(authorize)])
def classify(request: ClassifyRequest) -> list[dict[str, object]]:
    if runtime is None:
        raise HTTPException(503, "model is not ready")
    return runtime.classify(request)

For public production hosting, also add TLS, request-size/rate limits, monitoring, and a pinned MODEL_REVISION commit SHA. On macOS, use one Uvicorn worker so multiple processes do not each load a separate copy of the model into memory.

Labels

• 0: academic_calendar
• 1: academic_programs
• 2: admissions
• 3: campus_services
• 4: contact_and_location
• 5: courses_and_study_plans
• 6: exams_and_grades
• 7: general_university_information
• 8: graduation
• 9: library
• 10: news_and_events
• 11: out_of_scope
• 12: registration
• 13: research_and_postgraduate
• 14: scholarships_and_aid
• 15: staff_and_departments
• 16: student_services
• 17: technical_support
• 18: transfer_and_equivalency
• 19: tuition_and_payments

Intended use and limitations

The model is intended for routing Arabic university-chatbot questions within the label taxonomy above. It should not be treated as an authoritative source of admissions, academic, payment, or policy advice. The training data is synthetic and task-specific; the strong held-out scores may not transfer to other universities, taxonomies, dialect distributions, spelling patterns, or real production traffic. Inputs outside the training distribution can still receive confident scores. Evaluate on real, independently collected traffic and add human fallback/escalation before deployment. The sigmoid values are classification scores, not guaranteed calibrated probabilities.

Reproducibility and repository contents

• model.safetensors: complete encoder and classifier weights (the only weight copy)
• config.json: architecture, label mappings, pooling, maximum length, and thresholds
• tokenizer.json and tokenizer_config.json: tokenizer artifacts
• configuration_arauni.py and modeling_arauni.py: AutoClass code
• dataset_provenance.json, training_args.json, and metadata.json: provenance
• validation_metrics.json, test_metrics.json, and thresholds.json: evaluation artifacts

This model card reports the saved checkpoint artifacts; consult the base model card for MARBERTv2 pretraining details and limitations.