TigreGotico/openlid-v2-onnx

OpenLID-v2 - ONNX

ONNX export of laurievb/OpenLID-v2 model.bin, an improved fastText supervised language identifier covering 200 language varieties in iso639-3_Script label form (for example eng_Latn, por_Latn, glg_Latn), built on the OpenLID-v2 dataset, an updated version of Burchell et al., An Open Dataset and Model for Language Identification (ACL 2023).

The licence is GPL-3.0, inherited unchanged from the original model.

The upstream model card recommends normalising/cleaning text with openlid_normer.clean_line before classification for best results. This export does not vendor that normalizer - it hashes raw text exactly like TigreGotico/openlid-onnx and TigreGotico/glotlid-onnx do, and parity was measured against fastText's own predict() on raw (uncleaned) text, so the two stay comparable. Apply the same cleaning step yourself before calling the featurizer if you want to match the upstream-recommended pipeline.

What the graph does and does not do

fastText inference has five steps. The ONNX graph holds only the last two:

Step	Where
1. Tokenize the text	Python (openlid_v2_hash.py)
2. Character n-grams per word (minn..maxn)	Python
3. Hash the n-grams into buckets	Python
4. Average the embedding rows of all feature ids	ONNX
5. Multiply by the output matrix, then softmax	ONNX

Steps 1 to 3 are string processing. ONNX has no portable operator for fastText's FNV-1a byte hash over UTF-8, so that work stays in Python. The split keeps the graph a pure numeric pipeline:

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs

openlid_v2_hash.py is the reference implementation of steps 1 to 3, ported from the same recipe used for TigreGotico/glotlid-onnx. Two details are easy to get wrong:

• fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Without this, every non-ASCII n-gram lands in the wrong bucket.
• Each line ends with the </s> end-of-sentence token, and that token contributes its own vocabulary row.

This model uses plain softmax loss (not hierarchical softmax), so it does not need the Huffman-tree combination step that TigreGotico/lid176-onnx (fastText's classic lid.176.bin, trained with loss=hs) requires.

Files

File	Size	Purpose
openlid-v2.onnx	1.13 GB	fp32 graph
openlid-v2.int8.onnx	290 MB	dynamic int8 graph
labels.json	~4 KB	200 labels, in output order
vocab.txt	1.8 MB	185286 vocabulary words, in id order
config.json	122 B	dim, minn, maxn, bucket, nwords, nlabels, loss
openlid_v2_hash.py	5.2 KB	reference feature extractor

Model arguments

dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1  loss=softmax
nwords=185286  nlabels=200
input_matrix=(1185286, 256)   # nwords + bucket
output_matrix=(200, 256)

Tensors

Name	Direction	Type	Shape
input_ids	input	int64	[num_features]
probs	output	float32	[200]

probs is indexed by the order of labels.json.

Usage

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from openlid_v2_hash import FastTextFeaturizer

d = snapshot_download("TigreGotico/openlid-v2-onnx")
feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/openlid-v2.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    probs = sess.run(None, {"input_ids": feat(text)})[0]
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("O tempo está moi bo hoxe en Santiago"))
# [('__label__glg_Latn', 0.9...), ...]

Parity with fastText

59 short samples spanning 59 languages, chosen for script and resource diversity: Portuguese, Galician, Catalan, Basque, Spanish, English, French, German, Italian, Dutch, Arabic, Chinese, Japanese, Russian, Hindi, Greek, Ukrainian, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian, Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic, Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali, Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua, Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao, Burmese, Sinhala.

Each sample was compared against fastText's own f.predict(text + "\n", 1, 0.0, "strict") (the pybind entry point - fasttext-wheel's Python .predict() wrapper is broken under numpy>=2), on raw uncleaned text (see the note above about the upstream normalizer).

Model	Top-1 agreement
openlid-v2.onnx (fp32)	100.00 % (59/59)
openlid-v2.int8.onnx	100.00 % (59/59)

Citation

@inproceedings{burchell-etal-2023-open,
    title = "An Open Dataset and Model for Language Identification",
    author = "Burchell, Laurie and Birch, Alexandra and Bogoychev, Nikolay and Heafield, Kenneth",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    year = "2023",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.75",
}