py3langid-compact
The py3langid language identification model (langid.py by Marco Lui and Timothy Baldwin, as shipped in
rust-py3langid 0.4.1, BSD-3-Clause; 142 labels) in a
compact form for the browser: int8 weights with one scale per feature, and the byte automaton stored as each
row's differences from the start row (a 256-bit mask plus its targets). In memory it takes 25 MB instead of
97 MB, and this gzip file is 5.2 MB. It is the language detector of the metldr browser extension, read by
its vendored py3langid Rust crate.
Format (py3langid-compact.lidc.gz, gzip of a little-endian file)
Magic LIDC1\0\0\0; num_features, num_languages, num_states, num_rows, num_targets, history bytes (u32
each); the start row (256 u32); per row a 256-bit mask (4 u64); per row its first target (u32); the targets
(u32); each state's row (u16); each state's output feature (i32, -1 for none); priors (f32 per language);
scales (f32 per feature); weights (i8, features by languages); labels (u16 length and UTF-8 bytes each).
Converted from the upstream py3langid.lidg (SHA-256
da6860a9218a6122bcf26ba6e8752946336dc4f9c0816c6eb4fa790c7444dec2) by
pack-langid in.lidg out.lidc.
Evaluation (papluca/language-identification test, 10,000 texts, 20 languages)
| detector | sentence accuracy | four-word accuracy | time per text | memory |
|---|---|---|---|---|
| this model | 99.14% | 93.62% | 26 µs | 25 MB |
| py3langid upstream | 99.14% | 93.65% | 23 µs | 97 MB |
| lingua, high accuracy | 98.80% | 93.49% | 1,165 µs | 275 MB of models |
| fastText lid.176.ftz | 97.63% | 89.42% | 11 µs | 1 MB |
| langidentify lite | 94.34% | 91.93% | 27 µs | |
| whatlang | 90.59% | 77.81% | 39 µs |
It agrees with upstream on 19,981 of 20,000 texts (each sentence and its first four words).
License
BSD-3-Clause, as the upstream model.