TigreGotico/lid218e-onnx

fastText lid218e - ONNX

ONNX export of Meta's facebook/fasttext-language-identification (model.bin, known as lid218e). It covers 218 languages using the NLLB-200 taxonomy label codes (for example __label__eng_Latn, __label__por_Latn, __label__zho_Hans) - a distinct label set from lid176/GlotLID/OpenLID already published in this org, and the most-used LID model on the Hub (399k+ downloads at time of conversion).

The licence is CC-BY-NC-4.0 (non-commercial), inherited unchanged from the original Meta release. This is a re-export, not a re-training - keep attribution and licence terms intact if you redistribute it.

What the graph does and does not do

Unlike lid.176.bin, this model was trained with a plain softmax (loss=softmax, not hierarchical softmax), confirmed by inspecting model.f.getArgs().loss before export. So the simple GlotLID/OpenLID-style graph applies directly - no Huffman-tree combiner is needed here.

Step	Where
1. Tokenize the text	Python (lid218e_hash.py)
2. Character n-grams per word (minn..maxn)	Python
3. Hash the n-grams into buckets	Python
4. Average the embedding rows of all feature ids	ONNX
5. Multiply by the output matrix, then softmax	ONNX

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs

Two details that are easy to get wrong, both already handled:

• fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Missing this mis-hashes every non-ASCII n-gram.
• Each line ends with the </s> end-of-sentence token, which has its own vocabulary row and is a strong learned prior - always append it (the Python featurizer does this for you).

Files

File	Size	Purpose
lid218e.onnx + lid218e.onnx.data	1.09 GiB	fp32 graph (external data)
lid218e.int8.onnx	280 MiB	dynamic int8 graph
labels.json	4.6 KB	218 labels, in output order, __label__-prefixed
vocab.txt	1.3 MB	145940 vocabulary words, in id order
config.json	133 B	dim, minn, maxn, bucket, nwords, nlabels, loss
lid218e_hash.py	reference feature extractor (vendored from glotlid_hash.py)

Model arguments

dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1  loss=softmax
nwords=145940  nlabels=218
input_matrix=(1145940, 256)   # nwords + bucket
output_matrix=(218, 256)

Usage

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from lid218e_hash import GlotLIDFeaturizer as FastTextFeaturizer

d = snapshot_download("TigreGotico/lid218e-onnx")
feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/lid218e.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    probs = sess.run(None, {"input_ids": feat(text)})[0]
    top = np.argsort(-probs)[:k]
    return [(labels[i].replace("__label__", ""), float(probs[i])) for i in top]

print(detect("The weather is very nice today in London."))
# [('eng_Latn', 0.999...), ...]

For the int8 build load lid218e.int8.onnx instead.

Parity with fastText

59 short samples spanning 59 languages/scripts, compared against the original fastText model's own model.f.predict(text + "\n", 1, 0.0, "strict") (the pybind entry point - fasttext-wheel's plain Python .predict() wrapper is broken under numpy>=2, same issue documented in lid176-onnx).

Model	Top-1 agreement with fastText
lid218e.onnx (fp32)	100.00 % (59/59)
lid218e.int8.onnx	100.00 % (59/59)

Full greedy top-1 agreement in both precisions - no divergence observed on this sample set.

Citation

@article{joulin2016fasttext,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}
@misc{nllb2022,
  title={No Language Left Behind: Scaling Human-Centered Machine Translation},
  author={NLLB Team},
  year={2022}
}