Package: eld
Efficient Language DetectorEfficient language detector (Nito-ELDorELD) is a fast and accurate language detector, is one of the fastest non compiled detectors, while its accuracy is within the range of the heaviest and slowest detectors.It's 100% Python, easy installation and no dependencies other thanregex.ELD is also available inJavascriptandPHP.This is the first version of a port made from the original version in PHP, the structure might not be definitive, the code can be optimized. My knowledge of Python is basic, feel free to suggest improvements.InstallationHow to useBenchmarksLanguagesInstallation$pipinstalleldAlternatively, download / clone the files can work too, by changing the import path.How to use?fromeldimportLanguageDetectordetector=LanguageDetector()detect()expects a UTF-8 string, and returns an object, with a 'language' variable, which is either anISO 639-1 codeorNoneprint(detector.detect('Hola, cómo te llamas?'))# Object { language: "es", scores(): {"es": 0.53, "et": 0.21, ...}, is_reliable(): True }# Object { language: None|str, scores(): None|dict, is_reliable(): bool }print(detector.detect('Hola, cómo te llamas?').language)# "es"# if clean_text(True), detect() removes Urls, domains, emails, alphanumerical & numbersdetector.clean_text(True)# Default is FalseTo reduce the languages to be detected, there are 3 different options, they only need to be executed once. (Check availablelanguagesbelow)lang_subset=['en','es','fr','it','nl','de']# Option 1# with dynamic_lang_subset(), detect() executes normally, and then filters excluded languagesdetector.dynamic_lang_subset(lang_subset)# Returns an object with a list named 'languages', with the validated languages or 'None'# Option 2. lang_subset() Will first remove the excluded languages, from the n-grams database# For a single detection is slower than dynamic_lang_subset(), but for several will be faster# If save option is true (default), the new Ngrams subset will be stored, and loaded next calldetector.lang_subset(lang_subset)# lang_subset(langs, save=True)# Returns object {success: True, languages: ['de', 'en', ...], error: None, file: 'ngramsM60...'}# To remove either dynamic_lang_subset() or lang_subset(), call the methods with None as argumentdetector.lang_subset(None)# Finally the optimal way to regularly use a language subset: we create the instance with a file# The file in the argument can be a subset by lang_subset() or another database like 'ngramsL60'langSubsetDetect=LanguageDetector('ngramsL60')BenchmarksI comparedELDwith a different variety of detectors, since the interesting part is the algorithm.URLVersionLanguagehttps://github.com/nitotm/efficient-language-detector-py/0.9.0Pythonhttps://github.com/nitotm/efficient-language-detector/1.0.0PHPhttps://github.com/pemistahl/lingua-py1.3.2Pythonhttps://github.com/CLD2Owners/cld2Aug 21, 2015C++https://github.com/google/cld3Aug 28, 2020C++https://github.com/wooorm/franc6.1.0JavascriptBenchmarks:Tweets:760KB, short sentences of 140 chars max.;Big test:10MB, sentences in all 60 languages supported;Sentences:8MB, this is theLinguasentences test, minus unsupported languages.Short sentences is whatELDand most detectors focus on, as very short text is unreliable, but I included theLinguaWord pairs1.5MB, andSingle words880KBtests to see how they all compare beyond their reliable limits.These are the results, first, accuracy and then execution time.1.Lingua could have a small advantage as it participates with 54 languages, 6 less.2.CLD2 and CLD3, return a list of languages, the ones not included in this test where discarded, but usually they return one language, I believe they have a disadvantage.
Also, I confirm the results of CLD2 for short text are correct, contrary to the test on theLinguapage, they did not use the parameter "bestEffort = True", their benchmark for CLD2 is unfair.Linguais the average accuracy winner, but at what cost, the same test that inELDorCLD2is below 10 seconds, in Lingua takes more than 5 hours! It acts like a brute-force software.
Also, its lead comes from single and pair words, which are unreliable regardless.The Python version ofNITO-ELDis not the fastest but is still considered fast, as it is faster than any other non compiled detector tested.I addedELD-Lfor comparison, which has a 2.3x bigger database, but only increases execution time marginally, a testament to the efficiency of the algorithm.ELD-Lis not the main database as it does not improve language detection in sentences.Here is the average, per benchmark, of Tweets, Big test & Sentences.LanguagesThese are theISO 639-1 codesof the 60 supported languages forNito-ELDv1'am', 'ar', 'az', 'be', 'bg', 'bn', 'ca', 'cs', 'da', 'de', 'el', 'en', 'es', 'et', 'eu', 'fa', 'fi', 'fr', 'gu', 'he', 'hi', 'hr', 'hu', 'hy', 'is', 'it', 'ja', 'ka', 'kn', 'ko', 'ku', 'lo', 'lt', 'lv', 'ml', 'mr', 'ms', 'nl', 'no', 'or', 'pa', 'pl', 'pt', 'ro', 'ru', 'sk', 'sl', 'sq', 'sr', 'sv', 'ta', 'te', 'th', 'tl', 'tr', 'uk', 'ur', 'vi', 'yo', 'zh'Full name languages:Amharic, Arabic, Azerbaijani (Latin), Belarusian, Bulgarian, Bengali, Catalan, Czech, Danish, German, Greek, English, Spanish, Estonian, Basque, Persian, Finnish, French, Gujarati, Hebrew, Hindi, Croatian, Hungarian, Armenian, Icelandic, Italian, Japanese, Georgian, Kannada, Korean, Kurdish (Arabic), Lao, Lithuanian, Latvian, Malayalam, Marathi, Malay (Latin), Dutch, Norwegian, Oriya, Punjabi, Polish, Portuguese, Romanian, Russian, Slovak, Slovene, Albanian, Serbian (Cyrillic), Swedish, Tamil, Telugu, Thai, Tagalog, Turkish, Ukrainian, Urdu, Vietnamese, Yoruba, ChineseFuture improvementsTrain from bigger datasets, and more languages.The tokenizer could separate characters from languages that have their own alphabet, potentially improving accuracy and reducing the N-grams database. Retraining and testing is needed.Donate / HireIf you wish to Donate for open source improvements, Hire me for private modifications / upgrades, or to Contact me, use the following link:https://linktr.ee/nitotm