Package: fastspell
FastSpellTargetted language identifier, based on FastText and Hunspell.How it worksFastSpell will try to determine the language of a sentence by usingFastText.If the language detected is very similar to the target language (i.e. FastText detected Spanish, while the targetted language is Galician), extra checks are performed withHunspellto determine the language more precisely.Requirements & InstallationFastSpellcan be installed from PyPIpip install fastspellor directly from source:pip install .Note that it requires Python3.8 or higher, and thepython3-devpackage:sudo apt-get install python3-devIMPORTANT:
In some cases (for example, when using Python 3.10), thecyhunspellversion 2.0.2 installation will fail. If that's the case, you need to installcyhunspell==2.0.3before installingfastspell:First make sure build dependencies are installedsudo apt install build-essential autoconf autopoint libtoolthen install the pip packagepip install git+https://github.com/MSeal/cython_hunspell@2.0.3Model downloadTo trigger the FastText model download before running fastspell, run:fastspell-downloadSince version 0.7 all the dictionaries are installed automatically with pip and there is no need to do anything else.
For further explanation about how configuration works,see below.CondaAlso, you can install the conda package:conda install -c conda-forge -c bitextor fastspellAutomatic testingSome automatic tests are provided to check that the installation went fine. In order to check it, go to the/testsdirectory and run:python3 -m unittest discoverYou might need to istall theunittestpackage withpip, in  case you don't have it installed beforehand.ConfigurationA few configuration files are provided under thefastspell/configdirectory.
If you need to change default configuration, you can provide the path to your config directory with-c/--configor with the environment variableFASTSPELL_CONFIG.similar.yamlIn this dictionary-like file, similar languages are stored. These are the languages that are going to be "double-checked" with Hunspell after being identified with FastText. For example, see the linegl: [es, pt, gl]. This means that, when the targetted language is Galician, and FastText identifies a given sentence as Spanish, Portuguese or Galician, extra checks will be performed with Hunspell to confirm which of the three similar languages is more suitable for the sentence.Please note that you need Hunspell dictionaries for all the languages in this file (if you use thefastspell-downloadcommand, there is nothing else to do). This file can be modified to remove a language you are not interested in, or a language for which you don't have Hunspell dictionaries, or to add new similar or target languages.hunspell.yamlIn this file, the names of the dictionaries are stored. All similar languages must be in this list in order to properly work.For example, the first entry in thehunspell_codesisca: ca_ES, and the dictionary path is~/.local/share/fastspell/. That means that the Hunspell files for Catalan are~/.local/share/fastspell/ca_ES.dicand~/.local/share/fastspell/ca_ES.aff.By defaultdicpathis empty, which means FastSpell will look in these directories for the dictionaries:fastspell_dictionaries.__path__[0]~/.local/share/fastspell
~/.local/share/hunspell
$VIRTUAL_ENV/share/hunspell
/usr/share/hunspellTo use a custom path, put it indicpathand will be the first one to search.UsageModule:In order to useFastSpellas a Python module, just install and import it :from fastspell import FastSpellBuild a FastSpell object, like:fsobj = FastSpell.FastSpell("en", mode="cons")(learn more about modes in the section below)And then use thegetlangfunction with the sentences you want to identify, for example:fsobj.getlang("Hello, world")
#'en'
fsobj.getlang("Hola, mundo")
#'es'CLI:iusage: fastspell [-h] [--aggr] [--cons] [--hbs] [-q] [--debug]
                 [--logfile LOGFILE] [-v]
                 lang [input] [output]

positional arguments:
  lang
  input              Input sentences. (default: <_io.TextIOWrapper
                     name='<stdin>' encoding='UTF-8'>)
  output             Output of the language identification. (default:
                     <_io.TextIOWrapper name='<stdout>' mode='w'
                     encoding='UTF-8'>)

optional arguments:
  -h, --help         show this help message and exit
  --aggr             Aggressive strategy (more positives) (default: False)
  --cons             Conservative strategy (less positives) (default: False)
  --hbs              Return all Serbo-Croatian variants as 'hbs' (default:
                     False)

Logging:
  -q, --quiet        Silent logging mode (default: False)
  --debug            Debug logging mode (default: False)
  --logfile LOGFILE  Store log to a file (default: <_io.TextIOWrapper
                     name='<stderr>' mode='w' encoding='UTF-8'>)
  -v, --version      show version of this script and exitAggressive vs ConservativeFastSpell comes in two flavours: Aggressive and Conservative.TheAggressivemode is less hesitant to tag a sentence with the target language, and never has doubts. TheConservativeversion, on the other hand, is more reluctant to tag a sentence with the target language and will use theunk(unknown) tag in case of doubt (when there is a tie between the target language and other language, for example)BenchmarkUsage exampleInput text:19-01-2011 47 comentarios 7o Xornadas de Xardinería de Galicia (RE)PLANTEAR
• Proceso de valoración de idoneidade: entrevistas psicosociais e visita domiciliaria e aplicación de test psicolóxicos, se é o caso.
- Chrome e Firefox en MacOS non son compatibles (unicamente Safari é compatible con MacOS), pero invocarase PSAL ao intentar empregar Chrome ou Firefox.
Mago da luz / Maga da luz
Celebrada a homenaxe a Xosé Manuel Seivane Rivas
A instalación eléctrica en teletraballo
Saltar á navegación Navegación INICIO
Julio Freire, competidor da FGA, invitado polo Kennel club de Inglaterra, para participar nos Crufts 2014 (Birmingham, 6 - 9 de marzo).
25 de xullo - Truong Tan Sang toma posesión como presidente de Vietnam
Quen pode solicitar o dito financiamento?Command:fastspell  --aggr lang inputtext
fastspell  --cons lang inputtextAggressive output:19-01-2011 47 comentarios 7o Xornadas de Xardinería de Galicia (RE)PLANTEAR     gl
• Proceso de valoración de idoneidade: entrevistas psicosociais e visita domiciliaria e aplicación de test psicolóxicos, se é o caso.   gl
- Chrome e Firefox en MacOS non son compatibles (unicamente Safari é compatible con MacOS), pero invocarase PSAL ao intentar empregar Chrome ou Firefox.        gl
Mago da luz / Maga da luz       gl
Celebrada a homenaxe a Xosé Manuel Seivane Rivas        gl
A instalación eléctrica en teletraballo gl
Saltar á navegación Navegación INICIO   gl
Julio Freire, competidor da FGA, invitado polo Kennel club de Inglaterra, para participar nos Crufts 2014 (Birmingham, 6 - 9 de marzo). es
25 de xullo - Truong Tan Sang toma posesión como presidente de Vietnam  gl
Quen pode solicitar o dito financiamento?       glConservative output:19-01-2011 47 comentarios 7o Xornadas de Xardinería de Galicia (RE)PLANTEAR     unk
• Proceso de valoración de idoneidade: entrevistas psicosociais e visita domiciliaria e aplicación de test psicolóxicos, se é o caso.   gl
- Chrome e Firefox en MacOS non son compatibles (unicamente Safari é compatible con MacOS), pero invocarase PSAL ao intentar empregar Chrome ou Firefox.        gl
Mago da luz / Maga da luz       unk
Celebrada a homenaxe a Xosé Manuel Seivane Rivas        gl
A instalación eléctrica en teletraballo unk
Saltar á navegación Navegación INICIO   gl
Julio Freire, competidor da FGA, invitado polo Kennel club de Inglaterra, para participar nos Crufts 2014 (Birmingham, 6 - 9 de marzo). es
25 de xullo - Truong Tan Sang toma posesión como presidente de Vietnam  gl
Quen pode solicitar o dito financiamento?       glGetting stats:cat inputtext | fastspell --aggr $L | cut -f2 | sort | uniq -c | sort -nr
cat inputtext | fastspell --cons $L | cut -f2 | sort | uniq -c | sort -nrAggressive:9 gl
1 esConservative:6 gl
3 unk
1 esAll documents and software contained in this repository reflect only the authors' view. The Innovation and Networks Executive Agency of the European Union is not responsible for any use that may be made of the information it contains.