badbat4560/gliclass-multilang-ultra-fp8w8a8

GLiClass Multilang Ultra · FP8 W8A8

Multilingual zero-shot classification with native FP8 inference, tested on a 6 GB laptop GPU.

This repository contains an FP8 E4M3 quantization of Knowledgator's GLiClass Multilang Ultra, an optimized inference adapter, and a reproducible evaluation. It classifies text against labels supplied at inference time. It is not a text-generation model.

Measured on RTX 4050 Laptop, batch 1	Result
Optimized FP8 request latency, median	16.10 ms
Speedup over BF16 with the same CUDA Graph wrapper	1.48×
Peak allocated GPU memory	2.17 GiB
Weight-file reduction	33.85%
Top-1 agreement with original BF16	659 / 664 · 99.25%

The speedup requires the supplied Triton + CUDA Graph adapter. Loading the checkpoint through a generic dequantizing loader does not reproduce this performance. Measurements are local, on one GPU and a small evaluation suite; they are not a universal performance or quality claim.

Quick start

Tested with Python 3.12.9 on Windows, PyTorch 2.11.0 + CUDA 12.8 and NVIDIA driver 610.62. RTX 4050 Laptop has compute capability 8.9. Other operating systems and GPUs have not been validated for this release.

Download the complete repository, including the weights and Python files, then run these commands inside the downloaded directory:

python -m venv .venv
# Windows PowerShell: .venv/Scripts/Activate.ps1
# Linux shell: source .venv/bin/activate
python -m pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt
python inference.py --text "NASA launched a spacecraft to study Mars." --labels space politics sports business

The Linux Triton dependency is provided for portability, but the measured environment is Windows. Installation and first-run compilation are not part of the latency numbers below.

Python usage:

import torch
from inference import load_pipeline

classifier = load_pipeline()
with torch.inference_mode():
    result = classifier(
        "NASA launched a spacecraft to study Mars.",
        ["space", "politics", "sports", "business"],
        batch_size=1,
        threshold=0.0,
    )
print(result)

The adapter supports sequential inference, batch 1, and at most 256 input tokens including labels and prompt formatting. The tokenizer truncates longer inputs. The first call for a new length bucket captures a graph; subsequent calls replay it with fresh input data. Training, concurrent access to one runner, hidden-state outputs and attention outputs are not supported by this adapter.

Latency and memory

Same hardware, tokenizer and request protocol. Each fixed-input measurement uses 10 initial warmup calls and 50 timed calls after evaluation, with CUDA synchronization around the complete request. Graph buckets are prepared outside timing. Tokenization and postprocessing are included.

Runtime	Median	p95	Peak allocated VRAM	Peak reserved VRAM
Original BF16, eager	24.18 ms	29.67 ms	3.219 GiB	3.428 GiB
Original BF16 + CUDA Graphs	23.79 ms	24.45 ms	3.238 GiB	3.537 GiB
FP8, original eager adapter	59.23 ms	66.36 ms	2.145 GiB	4.293 GiB
FP8 + fused Triton kernels	37.97 ms	46.62 ms	2.144 GiB	4.289 GiB
FP8 + Triton + CUDA Graphs	16.10 ms	16.44 ms	2.166 GiB	3.021 GiB

The optimized path is 3.68× faster than the initial native-FP8 adapter and 1.48× faster than the BF16 graph control. The initial adapter result is an implementation baseline, not the headline comparison against BF16.

Allocated memory measures live PyTorch tensors; reserved memory includes the allocator pool. Neither is total device memory reported by nvidia-smi. Initialization temporarily dequantizes weights to BF16 before replacing the projection modules, so 2.17 GiB is not a loading-memory requirement. Graph preparation took 2.18 seconds for FP8 in this run; model loading and cold Triton compilation are additional.

These are steady-state, batch-1, fixed-input latency results—not sustained throughput, large-batch performance or repeated-session confidence intervals. Laptop clocks and thermals were not locked.

Quality: paired evaluation on 664 examples

No fine-tuning was performed for this evaluation. Both models used the same texts, candidate labels and tokenizer. English labels were used for both SIB-200 languages. Macro-F1 is reported as a percentage.

Dataset	Examples	Original BF16	Optimized FP8	Difference
AG News, seeded test subset	256	79.08	79.49	+0.41 pp
SIB-200 English, full test split	204	84.57	84.04	−0.53 pp
SIB-200 Russian, full test split	204	84.09	83.42	−0.67 pp

5 of 664 top-1 predictions differ from BF16. This is agreement, not accuracy. BF16 with and without the graph wrapper produced identical top-1 predictions on all 664 examples. Small positive differences should not be interpreted as improved model quality.

This suite covers news/topic classification in English and Russian. It does not establish quality across all upstream languages, routing tasks, safety classification, long contexts or few-shot settings. These scores use our own protocol and should not be compared directly with the upstream model card's aggregate benchmarks.

Per-example results, paired accuracy differences and bootstrap intervals are in benchmarks/results.json and the *-predictions.jsonl files. The single-label pipeline stores only the winning label and probability; these files do not contain complete score distributions.

What is quantized?

• 168 projection matrices across 24 mT5 encoder blocks: attention q/k/v/o and feed-forward wi_0/wi_1/wo.
• Weights: FP8 E4M3, with BF16 per-output-channel scales.
• Activations: dynamic per-token FP8 E4M3.
• Embeddings, normalization and classification components remain BF16.
• Quantization recipe: FP8_DYNAMIC; no calibration data required by the saved recipe.

Weight artifact	Bytes
Original BF16	3,416,522,340
FP8 W8A8	2,259,902,516

The weight file is 33.85% smaller (1.51× compression). It contains 416 tensors: 168 FP8 tensors and 248 BF16 tensors, including scales. All 80 original unquantized tensors were verified byte-identical to the pinned BF16 comparison snapshot. The shared embedding alone occupies about 1.02 GB in decimal units.

How acceleration works

1. A Triton kernel combines per-token absmax, scale computation, FP8 conversion and row padding.
2. torch._scaled_mm performs native cuBLAS FP8 GEMM with FP32 output and use_fast_accum=False.
3. A second Triton kernel applies activation and weight scales in FP32 and casts the result to BF16.
4. CUDA Graphs replay the encoder for 64/128/192/256-token buckets. Padding is masked, and only the original token positions are passed to classification.

A separate untimed profiler pass recorded 168 hardware FP8 GEMM kernels:

sm89_xmma_gemm_e4m3f32_e4m3f32_f32_tn_n_..._cublas

GPU kernel count fell from 3269 to 1425 per profiled request. The compact evidence is in benchmarks/optimization-profiler.json. The adapter does not silently fall back to BF16 GEMM for these projections. Different arithmetic and padding shapes can change rounding; quality is measured separately rather than assuming numerical identity.

Reproducibility

See BENCHMARKS.md for dataset revisions, reconstruction commands, pinned baseline, timing details and dependencies. provenance.json records checkpoint lineage; SHA256SUMS covers the publication files.

The loader explicitly restores the mT5 shared-embedding alias required by the tested GLiClass/Transformers combination. Unexpected missing or extra keys abort loading.

Provenance and license

Base model: knowledgator/gliclass-multilang-ultra, by Knowledgator. The comparison snapshot is pinned to 33c3ffec64056b0d37173d2dd90b885ac29b72c3. The precise upstream revision used during the historical quantization was not recorded; unchanged-tensor identity was checked against that comparison snapshot.

Distributed under Apache-2.0, matching the pinned upstream model card. See NOTICE. This is an independent quantization and evaluation, not an official Knowledgator or NVIDIA release.