TigreGotico/lid176-onnx

fastText lid.176 - ONNX

ONNX export of Facebook/Meta's classic fastText language identification model, lid.176.bin (also mirrored at julien-c/fasttext-language-id). It covers 176 languages with two-letter/three-letter fastText label codes (for example __label__en, __label__pt, __label__zh).

The licence is CC-BY-SA-3.0, inherited unchanged from the original Facebook release. This is a re-export, not a re-training - please keep attribution and licence terms intact if you redistribute it.

What the graph does and does not do - and a wrinkle this model has

fastText inference has five steps:

Step	Where
1. Tokenize the text	Python (lid176_hash.py)
2. Character n-grams per word (minn..maxn)	Python
3. Hash the n-grams into buckets	Python
4. Average the embedding rows of all feature ids	ONNX
5. Turn the averaged embedding into per-label probabilities	ONNX + Python

Steps 1-3 are string processing with no portable ONNX op for fastText's FNV-1a byte hash, so they stay in Python (lid176_hash.py), same as every other model in this hashing family.

Step 5 is where lid.176.bin differs from GlotLID and OpenLID: this model was trained with fastText's hierarchical softmax (loss=hs), not a flat softmax. Under hierarchical softmax the output matrix's rows are binary classifiers for internal nodes of a Huffman tree built over the label frequencies - a label's probability is the product of the sigmoid (or 1-sigmoid) values along the root-to-leaf path, not a single row of the output matrix. A plain MatMul -> Softmax graph (the GlotLID/OpenLID recipe) gives 0% agreement with fastText on this model - it was tried and verified wrong before switching to the approach below.

The ONNX graph therefore ends in Sigmoid over every Huffman node, and a small Python step (hs_tree.py + HSCombiner in lid176_hash.py) walks each label's path to combine node probabilities into per-label ones:

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0)
    -> MatMul(output_matrix) -> Sigmoid -> node_probs

node_probs = onnx_session.run(...)          # sigmoid per Huffman node
probs = HSCombiner.from_file("hs_tree.json")(node_probs)   # per label

The Huffman tree is not stored in fastText's .bin file - it is rebuilt deterministically from the label frequency counts every time the model loads. hs_tree.py reproduces that construction (HierarchicalSoftmaxLoss:: buildTree in fastText's Loss.cc) exactly, and hs_tree.json bakes the resulting per-label paths/codes in at export time so no fastText C++ code is needed at inference.

Two details that are easy to get wrong, both already solved here:

• fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Missing this mis-hashes every non-ASCII n-gram.
• Each line ends with the </s> end-of-sentence token, which has its own vocabulary row and is a strong learned prior - always append it (the Python featurizer does this for you).

Files

File	Size	Purpose
lid176.onnx	125 MB	fp32 graph, ends in Sigmoid
lid176.int8.onnx	31 MB	dynamic int8 graph
labels.json	2.6 KB	176 labels, in output/tree-leaf order
vocab.txt	323 KB	40010 vocabulary words, in id order
config.json	115 B	dim, minn, maxn, bucket, nwords, nlabels, loss
hs_tree.json	21 KB	precomputed Huffman paths/codes per label
hs_tree.py	1.7 KB	Huffman tree builder (reference/reproducibility only)
lid176_hash.py	5.6 KB	reference feature extractor + HSCombiner

Model arguments

dim=16  minn=2  maxn=4  bucket=2000000  wordNgrams=1  loss=hs
nwords=40010  nlabels=176
input_matrix=(2040010, 16)   # nwords + bucket
output_matrix=(176, 16)      # rows 0..174 are Huffman-node classifiers

Tensors

Name	Direction	Type	Shape
input_ids	input	int64	[num_features]
node_probs	output	float32	[176] (sigmoid per node; combine with HSCombiner)

Usage

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from lid176_hash import FastTextFeaturizer, HSCombiner

d = snapshot_download("TigreGotico/lid176-onnx")
feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
combine = HSCombiner.from_file(f"{d}/hs_tree.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/lid176.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    node_probs = sess.run(None, {"input_ids": feat(text)})[0]
    probs = combine(node_probs)
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("The weather is very nice today in London."))
# [('__label__en', 0.99...), ...]

Parity with fastText

59 short samples spanning 59 languages, chosen for script and resource diversity: Portuguese, Galician, Catalan, Basque, Spanish, English, French, German, Italian, Dutch, Arabic, Chinese, Japanese, Russian, Hindi, Greek, Ukrainian, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian, Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic, Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali, Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua, Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao, Burmese, Sinhala.

Each sample was compared against fastText's own f.predict(text + "\n", 1, 0.0, "strict") (the pybind entry point - fasttext-wheel's Python .predict() wrapper is broken under numpy>=2).

Model	Top-1 agreement
lid176.onnx (fp32)	100.00 % (59/59)
lid176.int8.onnx	96.61 % (57/59)

The two int8 mismatches are close calls between historically related languages/creoles (pms/ht, it/pt) that swap only after 8-bit weight rounding; fp32 gets both right.

Citation

If you use this model, please cite the original fastText/lid.176 authors:

@article{joulin2016fasttext,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}
@article{joulin2016fasttext2,
  title={FastText.zip: Compressing text classification models},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J\'egou, H\'erve and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1612.03651},
  year={2016}
}