TigreGotico/openlid-onnx

OpenLID - ONNX

ONNX export of laurievb/OpenLID model.bin, a fastText supervised language identifier covering 201 languages in iso639-3_Script label form (for example eng_Latn, por_Latn, glg_Latn), from Burchell et al., An Open Dataset and Model for Language Identification (ACL 2023).

The licence is GPL-3.0, inherited unchanged from the original model. There is a newer OpenLID-v2 covering slightly fewer but revised language varieties - use that one for new projects unless you need parity with the original OpenLID paper.

What the graph does and does not do

fastText inference has five steps. The ONNX graph holds only the last two:

Step	Where
1. Tokenize the text	Python (openlid_hash.py)
2. Character n-grams per word (minn..maxn)	Python
3. Hash the n-grams into buckets	Python
4. Average the embedding rows of all feature ids	ONNX
5. Multiply by the output matrix, then softmax	ONNX

Steps 1 to 3 are string processing. ONNX has no portable operator for fastText's FNV-1a byte hash over UTF-8, so that work stays in Python. The split keeps the graph a pure numeric pipeline:

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs

openlid_hash.py is the reference implementation of steps 1 to 3, ported from the same recipe used for TigreGotico/glotlid-onnx. Two details are easy to get wrong:

• fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Without this, every non-ASCII n-gram lands in the wrong bucket.
• Each line ends with the </s> end-of-sentence token, and that token contributes its own vocabulary row.

This model uses plain softmax loss (not hierarchical softmax), so it does not need the Huffman-tree combination step that TigreGotico/lid176-onnx (fastText's classic lid.176.bin, trained with loss=hs) requires.

Files

File	Size	Purpose
openlid.onnx	1.15 GB	fp32 graph
openlid.int8.onnx	293 MB	dynamic int8 graph
labels.json	4.1 KB	201 labels, in output order
vocab.txt	2.0 MB	201110 vocabulary words, in id order
config.json	122 B	dim, minn, maxn, bucket, nwords, nlabels, loss
openlid_hash.py	5.2 KB	reference feature extractor

Model arguments

dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1  loss=softmax
nwords=201110  nlabels=201
input_matrix=(1201110, 256)   # nwords + bucket
output_matrix=(201, 256)

Tensors

Name	Direction	Type	Shape
input_ids	input	int64	[num_features]
probs	output	float32	[201]

probs is indexed by the order of labels.json.

Usage

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from openlid_hash import FastTextFeaturizer

d = snapshot_download("TigreGotico/openlid-onnx")
feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/openlid.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    probs = sess.run(None, {"input_ids": feat(text)})[0]
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("O tempo está moi bo hoxe en Santiago"))
# [('__label__glg_Latn', 0.9...), ...]

Parity with fastText

59 short samples spanning 59 languages, chosen for script and resource diversity: Portuguese, Galician, Catalan, Basque, Spanish, English, French, German, Italian, Dutch, Arabic, Chinese, Japanese, Russian, Hindi, Greek, Ukrainian, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian, Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic, Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali, Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua, Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao, Burmese, Sinhala.

Each sample was compared against fastText's own f.predict(text + "\n", 1, 0.0, "strict") (the pybind entry point - fasttext-wheel's Python .predict() wrapper is broken under numpy>=2).

Model	Top-1 agreement
openlid.onnx (fp32)	100.00 % (59/59)
openlid.int8.onnx	100.00 % (59/59)

Citation

@inproceedings{burchell-etal-2023-open,
    title = "An Open Dataset and Model for Language Identification",
    author = "Burchell, Laurie and Birch, Alexandra and Bogoychev, Nikolay and Heafield, Kenneth",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    year = "2023",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.75",
}