Package: py3langid
py3langidis a fork of the standalone language identification toollangid.pyby Marco Lui.Original license: BSD-2-Clause. Fork license: BSD-3-Clause.Changes in this forkExecution speed has been improved and the code base has been optimized for Python 3.6+:Import: Loading the package (import py3langid) is about 30% fasterStartup: Loading the default classification model is 25-30x fasterExecution: Language detection withlangid.classifyis 5-6x faster on paragraphs (less on longer texts)For implementation details see this blog post:How to make language detection with langid.py faster.UsageDrop-in replacementInstall the package:pip3 install py3langid(orpipwhere applicable)Use it:with Python:import py3langid as langidon the command-line:langidWith PythonBasics:>>>importpy3langidaslangid>>>text='This text is in English.'# identified language and probability>>>langid.classify(text)('en',-56.77429)# unpack the result tuple in variables>>>lang,prob=langid.classify(text)# all potential languages>>>langid.rank(text)More options:>>>frompy3langid.langidimportLanguageIdentifier,MODEL_FILE# subset of target languages>>>identifier=LanguageIdentifier.from_pickled_model(MODEL_FILE)>>>identifier.set_languages(['de','en','fr'])# this won't work well...>>>identifier.classify('这样不好')('en',-81.831665)# normalization of probabilities to an interval between 0 and 1>>>identifier=LanguageIdentifier.from_pickled_model(MODEL_FILE,norm_probs=True)>>>identifier.classify('This should be enough text.')('en',1.0)Note: the Numpy data type for the feature vector has been changed to optimize for speed. If results are inconsistent, try restoring the original setting:>>>langid.classify(text,datatype='uint32')On the command-line# basic usage with probability normalization$echo"This should be enough text."|langid-n('en',1.0)# define a subset of target languages$echo"This won't be recognized properly."|langid-n-lfr,it,tr('it',0.97038305)Legacy documentationThe docs below are provided for reference, only part of the functions are currently tested and maintained.Introductionlangid.pyis a standalone Language Identification (LangID) tool.The design principles are as follows:FastPre-trained over a large number of languages (currently 97)Not sensitive to domain-specific features (e.g. HTML/XML markup)Single .py file with minimal dependenciesDeployable as a web serviceAll that is required to runlangid.pyis Python >= 3.6 and numpy.The accompanying training tools are still Python2-only.langid.pyis WSGI-compliant.langid.pywill usefapws3as a web server if
available, and default towsgiref.simple_serverotherwise.langid.pycomes pre-trained on 97 languages (ISO 639-1 codes given):af, am, an, ar, as, az, be, bg, bn, br,
bs, ca, cs, cy, da, de, dz, el, en, eo,
es, et, eu, fa, fi, fo, fr, ga, gl, gu,
he, hi, hr, ht, hu, hy, id, is, it, ja,
jv, ka, kk, km, kn, ko, ku, ky, la, lb,
lo, lt, lv, mg, mk, ml, mn, mr, ms, mt,
nb, ne, nl, nn, no, oc, or, pa, pl, ps,
pt, qu, ro, ru, rw, se, si, sk, sl, sq,
sr, sv, sw, ta, te, th, tl, tr, ug, uk,
ur, vi, vo, wa, xh, zh, zuThe training data was drawn from 5 different sources:JRC-AcquisClueWeb 09WikipediaReuters RCV2Debian i18nUsagelangid [options]optional arguments:-h,--helpshow this help message and exit-s,--servelaunch web service--host=HOSThost/ip to bind to--port=PORTport to listen on-vincrease verbosity (repeat for greater effect)-mMODELload model from file-lLANGS,--langs=LANGScomma-separated set of target ISO639 language codes
(e.g en,de)-r,--remoteauto-detect IP address for remote access-b,--batchspecify a list of files on the command line-d,--distshow full distribution over languages-uURL,--url=URLlangid of URL--lineprocess pipes line-by-line rather than as a document-n,--normalizenormalize confidence scores to probability valuesThe simplest way to uselangid.pyis as a command-line tool, and you can
invoke usingpython langid.py. If you installedlangid.pyas a Python
module (e.g. viapip install langid), you can invokelangidinstead ofpython langid.py-n(the two are equivalent).  This will cause a prompt to
display. Enter text to identify, and hit enter:>>> This is a test
('en', -54.41310358047485)
>>> Questa e una prova
('it', -35.41771221160889)langid.pycan also detect when the input is redirected (only tested under Linux), and in this
case will process until EOF rather than until newline like in interactive mode:python langid.py < README.rst
('en', -22552.496054649353)The value returned is the unnormalized probability estimate for the language. Calculating
the exact probability estimate is disabled by default, but can be enabled through a flag:python langid.py -n < README.rst
('en', 1.0)More details are provided in this README in the section onProbability Normalization.You can also uselangid.pyas a Python library:# python
Python 2.7.2+ (default, Oct  4 2011, 20:06:09)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import langid
>>> langid.classify("This is a test")
('en', -54.41310358047485)Finally,langid.pycan use Python’s built-inwsgiref.simple_server(orfapws3if available) to
provide language identification as a web service. To do this, launchpython langid.py-s, and
accesshttp://localhost:9008/detect. The web service supports GET, POST and PUT. If GET is performed
with no data, a simple HTML forms interface is displayed.The response is generated in JSON, here is an example:{"responseData": {"confidence": -54.41310358047485, "language": "en"}, "responseDetails": null, "responseStatus": 200}A utility such as curl can be used to access the web service:# curl -d "q=This is a test" localhost:9008/detect
{"responseData": {"confidence": -54.41310358047485, "language": "en"}, "responseDetails": null, "responseStatus": 200}You can also use HTTP PUT:# curl -T readme.rst localhost:9008/detect
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
100  2871  100   119  100  2752    117   2723  0:00:01  0:00:01 --:--:--  2727
{"responseData": {"confidence": -22552.496054649353, "language": "en"}, "responseDetails": null, "responseStatus": 200}If no “q=XXX” key-value pair is present in the HTTP POST payload,langid.pywill interpret the entire
file as a single query. This allows for redirection via curl:# echo "This is a test" | curl -d @- localhost:9008/detect
{"responseData": {"confidence": -54.41310358047485, "language": "en"}, "responseDetails": null, "responseStatus": 200}langid.pywill attempt to discover the host IP address automatically. Often, this is set to localhost(127.0.1.1), even
though the machine has a different external IP address.langid.pycan attempt to automatically discover the external
IP address. To enable this functionality, startlangid.pywith the-rflag.langid.pysupports constraining of the output language set using the-lflag and a comma-separated list of ISO639-1
language codes (the-nflag enables probability normalization):# python langid.py -n -l it,fr
>>> Io non parlo italiano
('it', 0.99999999988965627)
>>> Je ne parle pas français
('fr', 1.0)
>>> I don't speak english
('it', 0.92210605672341062)When usinglangid.pyas a library, the set_languages method can be used to constrain the language set:python
Python 2.7.2+ (default, Oct  4 2011, 20:06:09)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import langid
>>> langid.classify("I do not speak english")
('en', 0.57133487679900674)
>>> langid.set_languages(['de','fr','it'])
>>> langid.classify("I do not speak english")
('it', 0.99999835791478453)
>>> langid.set_languages(['en','it'])
>>> langid.classify("I do not speak english")
('en', 0.99176190378750373)Batch Modelangid.pysupports batch mode processing, which can be invoked with the-bflag.
In this mode,langid.pyreads a list of paths to files to classify as arguments.
If no arguments are supplied,langid.pyreads the list of paths fromstdin,
this is useful for usinglangid.pywith UNIX utilities such asfind.In batch mode,langid.pyusesmultiprocessingto invoke multiple instances of
the classifier, utilizing all available CPUs to classify documents in parallel.Probability NormalizationThe probabilistic model implemented bylangid.pyinvolves the multiplication of a
large number of probabilities. For computational reasons, the actual calculations are
implemented in the log-probability space (a common numerical technique for dealing with
vanishingly small probabilities). One side-effect of this is that it is not necessary to
compute a full probability in order to determine the most probable language in a set
of candidate languages. However, users sometimes find it helpful to have a “confidence”
score for the probability prediction. Thus,langid.pyimplements a re-normalization
that produces an output in the 0-1 range.langid.pydisables probability normalization by default. For
command-line usages oflangid.py, it can be enabled by passing the-nflag. For
probability normalization in library use, the user must instantiate their ownLanguageIdentifier. An example of such usage is as follows:>> from py3langid.langid import LanguageIdentifier, MODEL_FILE
>> identifier = LanguageIdentifier.from_pickled_model(MODEL_FILE, norm_probs=True)
>> identifier.classify("This is a test")
('en', 0.9999999909903544)Training a modelSo far Python 2.7 only, see theoriginal instructions.Read morelangid.pyis based on published research. [1] describes the LD feature selection technique in detail,
and [2] provides more detail about the modulelangid.pyitself.[1] Lui, Marco and Timothy Baldwin (2011) Cross-domain Feature Selection for Language Identification,
In Proceedings of the Fifth International Joint Conference on Natural Language Processing (IJCNLP 2011),
Chiang Mai, Thailand, pp. 553—561. Available fromhttp://www.aclweb.org/anthology/I11-1062[2] Lui, Marco and Timothy Baldwin (2012) langid.py: An Off-the-shelf Language Identification Tool,
In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012),
Demo Session, Jeju, Republic of Korea. Available from www.aclweb.org/anthology/P12-3005