NajahUniv/AraUni-Granite-311M-Intent-Classifier

AraUni Granite 311M Intent Classifier

Release: v0.1.3 · Lifecycle: preview

This is a 20-label Arabic university intent classifier fine-tuned from ibm-granite/granite-embedding-311m-multilingual-r2. It routes one question to zero, one, or several university-information labels. The model uses first_token pooling and independent sigmoid outputs; it is not a generative model.

Model details

• Architecture: ibm-granite/granite-embedding-311m-multilingual-r2 encoder + dropout + linear multi-label head
• Pooling: first_token
• Maximum input length: 256 tokens
• Model release: v0.1.3
• Release lifecycle: preview
• Weights: SafeTensors
• Training data: NajahUniv/arabic-univeristy-chatbot-qa-cleaned
• Dataset release: v0.1.0
• Pinned dataset revision: 9136ae156c49e1cc014020cef75c20a8b8d9aae0
• Decision thresholds: selected on validation data and stored in config.json and thresholds.json

The repository includes custom Transformers modeling code so it loads through the standard AutoClass API. Review the code in this repository before enabling trust_remote_code; in a production deployment, pin revision to a reviewed model commit SHA.

Release notes

v0.1.3 is a packaging and documentation patch release. Its trained weights are byte-identical to the earlier v0.1.x releases, and it continues to use dataset release v0.1.0. It presents benchmark provenance through the public, immutable dataset release, exact split, and commit; the low-level prepared-workload hash remains available in benchmark_results.json.

Evaluation

Split	Macro F1	Micro F1	Subset accuracy	LRAP
Validation	0.9896	0.9897	0.9790	0.9970
Test	0.9878	0.9888	0.9780	0.9952

Threshold selection used only the validation split. The test split remained held out until the final comparison. Full aggregate and per-label results are available in validation_metrics.json and test_metrics.json.

Local inference speed benchmark

MPS

Model	Parameters	Batch 1 p50 (ms)	Batch 1 q/s	Batch 8 q/s	Batch 32 q/s	Peak RSS (MiB)
AraUni Granite 311M Intent Classifier	311.7M	10.9	92.1	387.4	629.9	3363
AraUni MARBERTv2 Intent Classifier	162.3M	10.1	113.3	460.1	807.6	1774
AraUni Granite 97M Intent Classifier	97.4M	3.7	170.3	889.9	1640.6	1552

CPU

Model	Parameters	Batch 1 p50 (ms)	Batch 1 q/s	Batch 8 q/s	Batch 32 q/s	Peak RSS (MiB)
AraUni Granite 311M Intent Classifier	311.7M	18.3	57.1	112.9	115.9	3261
AraUni MARBERTv2 Intent Classifier	162.3M	11.0	80.6	139.2	116.4	1771
AraUni Granite 97M Intent Classifier	97.4M	5.8	162.4	338.7	369.9	1551

Measured on Apple M5 Max (PyTorch 2.13.0, Transformers 5.14.1). Each batch size used 5 warm-up and 30 measured iterations over the same deterministically shuffled questions derived from the test split of NajahUniv/arabic-univeristy-chatbot-qa-cleaned@v0.1.0, pinned to commit 9136ae156c49e1cc014020cef75c20a8b8d9aae0, after applying the repository's standard dataset-preparation pipeline.

Numbers include tokenization, padding, device transfer, model forward pass, sigmoid, thresholding, and result construction. They exclude model loading and HTTP overhead. Latency and throughput depend on hardware, software versions, input lengths, batch size, and thermal state; compare only rows from this same run. Peak RSS is whole-process memory and MPS uses unified memory.

The full machine-readable benchmark report is included as benchmark_results.json. Use the direct benchmark for model-to-model speed comparisons; use the FastAPI load test below to size a specific deployment.

Basic usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "NajahUniv/AraUni-Granite-311M-Intent-Classifier"
model_revision = "v0.1.3"
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    revision=model_revision,
    trust_remote_code=True,
)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    revision=model_revision,
    trust_remote_code=True,
).eval()

text = "ما هي شروط القبول في الجامعة؟"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=model.config.max_length)
with torch.inference_mode():
    probabilities = torch.sigmoid(model(**inputs).logits[0])

result = {}
for index, probability in enumerate(probabilities.tolist()):
    label = model.config.id2label[index]
    threshold = model.config.thresholds[label]
    result[label] = {"probability": probability, "selected": probability >= threshold}

selected_labels = [label for label, value in result.items() if value["selected"]]
print(selected_labels)

A complete command-line example is included at examples/basic_inference.py:

python examples/basic_inference.py \
  --model-id NajahUniv/AraUni-Granite-311M-Intent-Classifier \
  --revision v0.1.3 \
  --text "ما هي شروط التسجيل؟"

Complete basic inference example

"""Run multi-label inference with the published Hugging Face model."""

from __future__ import annotations

import argparse
import json

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

DEFAULT_MODEL_ID = "NajahUniv/AraUni-Granite-311M-Intent-Classifier"


def choose_device(requested: str) -> str:
    if requested != "auto":
        return requested
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model-id", default=DEFAULT_MODEL_ID)
    parser.add_argument("--revision", help="Pin a commit SHA in production")
    parser.add_argument("--text", required=True)
    parser.add_argument("--device", default="auto", choices=("auto", "cpu", "mps", "cuda"))
    parser.add_argument("--top-k", type=int, default=5)
    args = parser.parse_args()

    load_kwargs = {"revision": args.revision} if args.revision else {}
    tokenizer = AutoTokenizer.from_pretrained(
        args.model_id,
        trust_remote_code=True,
        **load_kwargs,
    )
    model = AutoModelForSequenceClassification.from_pretrained(
        args.model_id,
        trust_remote_code=True,
        **load_kwargs,
    )
    device = choose_device(args.device)
    model.to(device).eval()
    encoded = tokenizer(
        args.text,
        return_tensors="pt",
        truncation=True,
        max_length=model.config.max_length,
    ).to(device)
    with torch.inference_mode():
        probabilities = torch.sigmoid(model(**encoded).logits[0]).cpu()

    labels = [model.config.id2label[index] for index in range(model.config.num_labels)]
    thresholds = model.config.thresholds
    ranked = sorted(
        (
            {
                "label": label,
                "probability": float(probabilities[index]),
                "threshold": float(thresholds[label]),
                "selected": float(probabilities[index]) >= float(thresholds[label]),
            }
            for index, label in enumerate(labels)
        ),
        key=lambda item: item["probability"],
        reverse=True,
    )
    result = {
        "text": args.text,
        "selected_labels": [item["label"] for item in ranked if item["selected"]],
        "scores": ranked[: max(1, args.top_k)],
    }
    print(json.dumps(result, ensure_ascii=False, indent=2))


if __name__ == "__main__":
    main()

FastAPI hosting

The included service loads the model once, supports batching, uses MPS automatically on Apple Silicon, and optionally requires a bearer token:

pip install -r requirements.txt
MODEL_ID=NajahUniv/AraUni-Granite-311M-Intent-Classifier MODEL_REVISION=v0.1.3 MODEL_API_KEY=change-me \
  uvicorn examples.fastapi_app:app --host 0.0.0.0 --port 8000

curl http://localhost:8000/classify \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer change-me' \
  -d '{"texts":["كيف يمكنني دفع الرسوم؟"],"top_k":5}'

In Swagger UI at http://localhost:8000/docs, click Authorize and enter only the MODEL_API_KEY value (change-me in the example). Swagger adds the Bearer prefix.

Complete FastAPI server example

"""FastAPI service for the published multi-label classifier."""

from __future__ import annotations

import hmac
import os
import threading
from contextlib import asynccontextmanager
from typing import Annotated

import torch
from fastapi import Depends, FastAPI, HTTPException
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from pydantic import BaseModel, Field
from transformers import AutoModelForSequenceClassification, AutoTokenizer

DEFAULT_MODEL_ID = "NajahUniv/AraUni-Granite-311M-Intent-Classifier"
MODEL_ID = os.getenv("MODEL_ID", DEFAULT_MODEL_ID)
MODEL_REVISION = os.getenv("MODEL_REVISION")
MODEL_DEVICE = os.getenv("MODEL_DEVICE", "auto")
MODEL_API_KEY = os.getenv("MODEL_API_KEY")
MAX_BATCH_SIZE = int(os.getenv("MAX_BATCH_SIZE", "64"))


def choose_device() -> str:
    if MODEL_DEVICE != "auto":
        return MODEL_DEVICE
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"


class ClassifyRequest(BaseModel):
    texts: list[str] = Field(min_length=1)
    top_k: int = Field(default=5, ge=1)
    threshold: float | None = Field(default=None, gt=0, lt=1)


class ModelRuntime:
    def __init__(self) -> None:
        load_kwargs = {"revision": MODEL_REVISION} if MODEL_REVISION else {}
        self.tokenizer = AutoTokenizer.from_pretrained(
            MODEL_ID,
            trust_remote_code=True,
            **load_kwargs,
        )
        self.model = AutoModelForSequenceClassification.from_pretrained(
            MODEL_ID,
            trust_remote_code=True,
            **load_kwargs,
        )
        self.device = choose_device()
        self.model.to(self.device).eval()
        self.lock = threading.Lock()

    def classify(self, request: ClassifyRequest) -> list[dict[str, object]]:
        if len(request.texts) > MAX_BATCH_SIZE:
            raise HTTPException(413, f"at most {MAX_BATCH_SIZE} texts are allowed per request")
        encoded = self.tokenizer(
            request.texts,
            return_tensors="pt",
            padding=True,
            truncation=True,
            max_length=self.model.config.max_length,
        ).to(self.device)
        with self.lock, torch.inference_mode():
            probabilities = torch.sigmoid(self.model(**encoded).logits).cpu()
        labels = [
            self.model.config.id2label[index]
            for index in range(self.model.config.num_labels)
        ]
        results = []
        for text, row in zip(request.texts, probabilities, strict=True):
            scores = []
            for index, label in enumerate(labels):
                threshold = (
                    request.threshold
                    if request.threshold is not None
                    else float(self.model.config.thresholds[label])
                )
                scores.append(
                    {
                        "label": label,
                        "probability": float(row[index]),
                        "threshold": threshold,
                        "selected": float(row[index]) >= threshold,
                    }
                )
            scores.sort(key=lambda item: item["probability"], reverse=True)
            results.append(
                {
                    "text": text,
                    "selected_labels": [item["label"] for item in scores if item["selected"]],
                    "scores": scores[: min(request.top_k, len(scores))],
                }
            )
        return results


runtime: ModelRuntime | None = None


@asynccontextmanager
async def lifespan(_: FastAPI):
    global runtime
    runtime = ModelRuntime()
    yield
    runtime = None


app = FastAPI(title="AraUni Multi-label Intent Classifier", lifespan=lifespan)
bearer_scheme = HTTPBearer(
    auto_error=False,
    scheme_name="BearerAuth",
    description="Enter the MODEL_API_KEY value. Swagger adds the 'Bearer' prefix.",
)


def authorize(
    credentials: Annotated[
        HTTPAuthorizationCredentials | None,
        Depends(bearer_scheme),
    ],
) -> None:
    if MODEL_API_KEY is None:
        return
    if (
        credentials is None
        or credentials.scheme.lower() != "bearer"
        or not hmac.compare_digest(credentials.credentials, MODEL_API_KEY)
    ):
        raise HTTPException(
            401,
            "invalid bearer token",
            headers={"WWW-Authenticate": "Bearer"},
        )


@app.get("/health")
def health() -> dict[str, object]:
    return {"status": "ok", "model_id": MODEL_ID, "device": runtime.device if runtime else None}


@app.get("/labels", dependencies=[Depends(authorize)])
def labels() -> dict[int, str]:
    if runtime is None:
        raise HTTPException(503, "model is not ready")
    return dict(runtime.model.config.id2label)


@app.post("/classify", dependencies=[Depends(authorize)])
def classify(request: ClassifyRequest) -> list[dict[str, object]]:
    if runtime is None:
        raise HTTPException(503, "model is not ready")
    return runtime.classify(request)

For public production hosting, also add TLS, request-size/rate limits, monitoring, and a pinned MODEL_REVISION commit SHA. On macOS, use one Uvicorn worker so multiple processes do not each load a separate copy of the model into memory.

Labels

• 0: academic_calendar
• 1: academic_programs
• 2: admissions
• 3: campus_services
• 4: contact_and_location
• 5: courses_and_study_plans
• 6: exams_and_grades
• 7: general_university_information
• 8: graduation
• 9: library
• 10: news_and_events
• 11: out_of_scope
• 12: registration
• 13: research_and_postgraduate
• 14: scholarships_and_aid
• 15: staff_and_departments
• 16: student_services
• 17: technical_support
• 18: transfer_and_equivalency
• 19: tuition_and_payments

Intended use and limitations

The model is intended for routing Arabic university-chatbot questions within the label taxonomy above. It should not be treated as an authoritative source of admissions, academic, payment, or policy advice. The training data is task-specific; consult the pinned dataset release for its synthetic and real-user composition. The strong held-out scores may not transfer to other universities, taxonomies, dialect distributions, spelling patterns, or production traffic. Inputs outside the training distribution can still receive confident scores. Evaluate on real, independently collected traffic and add human fallback/escalation before deployment. The sigmoid values are classification scores, not guaranteed calibrated probabilities.

Reproducibility and repository contents

• model.safetensors: complete encoder and classifier weights (the only weight copy)
• config.json: architecture, label mappings, pooling, maximum length, and thresholds
• tokenizer.json and tokenizer_config.json: tokenizer artifacts
• configuration_arauni.py and modeling_arauni.py: AutoClass code
• dataset_provenance.json, training_args.json, and metadata.json: provenance
• validation_metrics.json, test_metrics.json, and thresholds.json: evaluation artifacts
• benchmark_results.json: shared MPS/CPU speed comparison and environment metadata

This model card reports the saved checkpoint artifacts; consult the linked base-model card for its pretraining details, license, intended uses, and limitations.