crash-sv/scribe-fasttext-lid176

fastText lid.176 for Scribe SV

An unmodified, byte-for-byte copy of the official fastText language identification model lid.176.bin, published here so that Scribe SV — a Windows dictation and translation utility — can download it on demand from a repository we control. Nothing was retrained, quantized, converted or otherwise changed.

• Title: fastText lid.176 (language identification, 176 languages)
• Authors: Facebook AI Research — Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov et al.
• Original: https://fasttext.cc/docs/en/language-identification.html, downloaded from https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin
• License: Creative Commons Attribution-ShareAlike 3.0 (as stated by the original publisher). This copy keeps the same license; no additional terms or restrictions are applied.

File	Size	MD5	SHA-256
lid.176.bin	131 266 198 bytes	01810bc59c6a3d2b79c79e6336612f65	7e69ec5451bc261cc7844e49e4792a85d7f09c06789ec800fc4a44aec362764e

What Scribe SV uses it for

The translator works on a pair of languages chosen by the user (for example German ↔ French). For a piece of text the application asks one question: which of the two languages of the pair is this text in? It reads the full probability vector and picks the more probable of the two pair languages; the model's top-1 label is not used.

Usage

import fasttext  # pip install fasttext-predict
from huggingface_hub import hf_hub_download

model = fasttext.load_model(hf_hub_download("crash-sv/scribe-fasttext-lid176", "lid.176.bin"))
labels, probs = model.predict("Guten Morgen, wie geht es dir?", k=-1)

Citation

@article{joulin2016bag,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}

@article{joulin2016fasttext,
  title={FastText.zip: Compressing text classification models},
  author={Joulin, Armand and Grave, Edouard and Douze, Matthijs and J{\'e}gou, H{\'e}rve and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1612.03651},
  year={2016}
}