Saravanan666/pypi-org huggingface dataset:
Package: lingpy
LingPy: A Python Library for Automatic Tasks in Historical LinguisticsThis repository contains the Python packagelingpywhich can be used for various tasks in computational historical linguistics.Authors (Version 2.6.12): Johann-Mattis List and Robert ForkelCollaborators:
Christoph Rzymski, Simon J. Greenhill, Steven Moran, Peter Bouda, Johannes Dellert, Taraka Rama, Tiago Tresoldi, Gereon Kaiping, Frank Nagel, and Patrick Elmer.LingPy is a Python library for historical linguistics. It is being developed for Python 2.7 and Python 3.x
usinga single codebase.All source code is available at:https://github.com/lingpy/lingpy.Documentation can be found at:http://lingpy.org.For a list of papers in which LingPy was applied, seehere.Quick InstallationFor our latest stable version, you can simply use pip or easy_install for installation:$pipinstalllingpyor$pipinstalllingpyDepending on which easy_install or pip version you use, either the Python2 or the Python3 version of LingPy will be installed.If you want to install the current GitHub version of LingPy on your system, open a terminal and type in the following:$gitclonehttps://github.com/lingpy/lingpy/
$cdlingpy
$pythonsetup.pyinstallIf the last command above returns you some error regarding user permissions (usually "Errno 13"), you can install
LingPy in your home Python setup:$ python setup.py install --userIn order to use the library, start an interactive Python session and import LingPy as follows:>>>fromlingpyimport*To install LingPy to hack on it, fork the repository on GitHub, open a terminal and type:$gitclonehttps://github.com/<your-github-user>/lingpy/
$cdlingpy
$pythonsetup.pydevelopThis will install LingPy in"development mode",
i.e. you will be able edit the sources in the cloned repository and import the altered code just as the regular Python package.
Package: lingtrain-aligner
Lingtrain AlignerML powered library for the accurate texts alignment in different languages.🔥How to create bilingual books. Part 2. Lingtrain Alignment Studio💡Lingtrain Aligner. How to make parallel books for language learning. Part 1. Python and Colab versionPurposeMain purpose of this alignment tool is to build parallel corpora using two or more raw texts in different languages. Texts should contain the same information (i.e., one text should be a translated analog oh the other text). E.g., it can be theDrei Kameradenby Remarque in German and theThree Comrades— it's translation into English.ProcessThere are plenty of obstacles during the alignment process:The translator could translate several sentences as one.The translator could translate one sentence as many.There are some service marks in the textPage numbersChapters and other section headingsAuthor and title informationNotesWhile service marks can be handled manually (the tool helps to detect them), the translation conflicts should be handled more carefully.Lingtrain Aligner tool will do almost all alignment work for you. It matches the sentence pairs automatically using the multilingual machine learning models. Then it searches for the alignment conflicts and resolves them. As output you will have the parallel corpora either as two distinct plain text files or as the merged corpora in widely used TMX format.Supported languages and modelsAutomated alignment process relies on the sentence embeddings models. Embeddings are multidimensional vectors of a special kind which are used to calculate a distance between the sentences. Supported languages list depend on the selected backend model.distiluse-base-multilingual-cased-v2more reliable and fastmoderate weights size — 500MBsupports 50+ languagesfull list of supported languages can be found inthis paperLaBSE (Language-agnostic BERT Sentence Embedding)can be used for rare languagespretty heavy weights — 1.8GBsupports 100+ languagesfull list of supported languages can be foundhereProfitParallel corpora by itself can used as the resource for machine translation models or for linguistic researches.My personal goal of this project is to help people building parallel translated books for the foreign language learning.
Package: lingtypology
lingtypologyLingtypology is a Python3 tool for linguistic interactive mapping and online linguistic databases API.
It is inspired byR-packagecreated byAgrizolamz.
It uses the same phylosophy and provides similar functionality.InstallationThe package is available in PyPI.
Therefore, you can install it by running:Locally:pip3 install lingtypology --userGlobally (not recommended):sudo pip3 install lingtypologyUsageLingtypology package containsLingMapclass that allows to draw interactive maps,glottologlibrary that allows to interact with Glottolog data anddb_apisthat allows to interact with different other linguistic databases.For more informations consultLingtypology: Documentation.
Package: lingua.autotranslate
Introductionlingua.autotranslate is an autotranslation toolkit for .po files using Google
Translate API.UsageTo use this package you should add the following part to your buildout:[buildout]
...
parts =
    ...
    autotranslate
...

[autotranslate]
recipe = zc.recipe.egg
eggs =
    lingua.autotranslateWhen runningbin/buildoutit will create a new console script for you,bin/autotranslatewhich requires at least two parameters:~/workspace/my.product $ bin/autotranslate -h
usage: autotranslate [-h] -i <locale> <po file> [-s LANGUAGE] [-u]

Translate Po files.

optional arguments:
  -h, --help            show this help message and exit
  -i <locale> <po file>
                        Locale and filename of po-file to process
  -s LANGUAGE           Source language to translate from.
  -u                    Force updating translations by retranslating all
                        msgids.Example:See translation example below.:~/workspace/my.product $ bin/autotranslate -i de src/my/product/locales/de/LC_MESSAGES/my.product.po
[ Success ] New product -> Neues Produkt
[ Success ] Specification -> Beschreibung
[ Success ] Language -> Sprache
[ Found variable(s) ] ${back}
[ Success ] Back to product ${back} -> Zurück zum Produkt $ { back }
...AuthorDaniel Widerin  <daniel@widerin.net>ContributorsDaniel Widerin  [saily]  <daniel@widerin.net>Changelog0.2.1 (2014-03-17)Update docs.
[saily]0.2 (2014-03-17)PrintFound variablesmessage to inform users to validate their
i18n variables, because they get translated too. You will need to manually
fix them after runninglingua.autotranslate.
[saily]Allow configurable source language to start autotranslation from languages
different to English.
[saily]Python 2.7 shipsargparsebut Python 2.6 does not. Add it as a
installation requirement when running on Pyton 2.6.
[saily]0.1 (2013-09-06)Package created using templer
[saily]
Package: linguadoc
Installationpip3 install --verbose linguadocUsagePlease refer toapi docs.Execute usageConvert json into docxgen_lingua_docx --sourcejson [source.json] --lang [en/de/es] --destdocx [output.docx] --title [test_title]Package usageDevelopmentClone projectgit clone https://github.com/qishe-nlp/linguadoc.gitInstallpoetryInstall dependenciespoetry updateTestpoetry run pytest -rP --capture=syswhich run tests undertests/*Executepoetry run gen_lingua_docx --helpCreate sphinx docspoetry shell
cd apidocs
sphinx-apidoc -f -o source ../linguadoc
make html
python -m http.server -d build/htmlHost docs on github pagescp -rf apidocs/build/html/* docs/BuildChangeversioninpyproject.tomlandlinguadoc/__init__.pyBuild python package bypoetry buildGit commit and pushPublish from local dev envSet pypi test environment variables in poetry, refer topoetry docPublish to pypi test bypoetry publish -r testPublish through CIGithub action build and publish package totest pypi repogit tag [x.x.x]
git push origin masterManually publish topypi repothroughgithub action
Package: linguaf
LinguaFLinguaF provides an easy access for researchers and developers to methods of quantitative language analysis, such as: readability, complexity, diversity, and other descriptive statistics.Usagedocuments=["Pain and suffering are always inevitable for a large intelligence and a deep heart. The really great men must, I think, have great sadness on earth.","To go wrong in one's own way is better than to go right in someone else's.","The darker the night, the brighter the stars, The deeper the grief, the closer is God!"]Descriptive StatisticsThe following descriptive statistics are supported (descriptive_statistics.pymodule):Number of characterschar_countNumber of lettersletter_countNumber of punctuation characterspunctuation_countNumber of digitsdigit_countNumber of syllablessyllable_countNumber of sentencessentence_countNumber of n-syllable wordsnumber_of_n_syllable_wordsAverage syllables per wordavg_syllable_per_wordAverage word lengthavg_word_lengthAverage sentence lengthavg_sentence_lengthAverage words per sentenceavg_words_per_sentenceAdditional methods:Get lexical items (nouns, adjectives, verbs, adverbs)get_lexical_itemsGet n-gramsget_ngramsGet sentencesget_sentencesGet wordsget_wordsTokenizetokenizeRemove punctuationremove_punctuationRemove digitsremove_digitsExample:fromlinguafimportdescriptive_statisticsasdsds.words_per_sentence(documents)# Output: 15Syntactical ComplexityThe following syntactical complexity metrics are supported (syntactical_complexity.pymodule):Mean Dependency Distance (MDD)mean_dependency_distanceExample:fromlinguafimportsyntactical_complexityasscsc.mean_dependency_distance(documents)# Output: 2.307306255835668Lexical DiversityThe following lexical diversity metrics are supported (lexical_diversity.pymodule):Lexical Density (LD)lexical_densityType Token Ratio (TTR)type_token_ratioHerdan's Constant or Log Type Token Ratio (LogTTR)log_type_token_ratioSummer's Indexsummer_indexRoot Type Token Ratio (RootTTR)root_type_token_ratioExample:fromlinguafimportlexical_diversityasldld.log_type_token_ratio(documents)# Output: 94.03574963462502ReadabilityThe following readability metrics are supported (readability.pymodule):Flesch Reading Ease (FRE)flesch_reading_easeFlesch-Kincaid Grade (FKG)flesch_kincaid_gradeAutomated Readability Index (ARI)automated_readability_indexSimple Automated Readability Index (sARI)automated_readability_index_simpleColeman's Readability Scorecoleman_readabilityEasy Listening Scoreeasy_listeningExample:fromlinguafimportreadabilityasrr.flesch_kincaid_grade(documents)# Output: 4.813333333333336InstallVia PIPpipinstalllinguafLatest version from GitHubgitclonehttps://github.com/Perevalov/LinguaF.gitcdLinguaF
pipinstall.Language SupportAt the moment, library supports English and Russian languages for all the methods.Open APIThe Swagger UI of the API is located here:http://webengineering.ins.hs-anhalt.de:41008/docsThere is currently a request limit set up for 50 requests per minute. If you are beyond the limit, requests are moved to the queue.CitationTBD
Package: lingualytics
Lingualytics : Easy codemixed analyticsLingualytics is a Python library for dealing with code mixed text.Lingualytics is powered by powerful libraries likePytorch,Transformers,Texthero,NLTKandScikit-learn.FeaturesPreprocessingRemove stopwordsRemove punctuations, with an option to add punctuations of your own languageRemove words less than a character limitRepresentationFind n-grams from given textNLPClassification using PyTorchTrain a classifier on your data to perform tasks like Sentiment AnalysisEvaluate the classifier with metrics like accuracy, f1 score, precision and recallUse the trained tokenizer to tokenize textSome pretrained Huggingface models trained on codemixed datasets you can usebert-base-multilingual-codemixed-cased-sentimentInstallationUse the package managerpipto install lingualytics.pipinstalllingualyticsUsagePreprocessingfromlingualytics.preprocessingimportremove_lessthan,remove_punctuation,remove_stopwordsfromlingualytics.stopwordsimporthi_stopwords,en_stopwordsfromtexthero.preprocessingimportremove_digitsimportpandasaspddf=pd.read_csv("https://github.com/lingualytics/py-lingualytics/raw/master/datasets/SAIL_2017/Processed_Data/Devanagari/validation.txt",header=None,sep='\t',names=['text','label'])# pd.set_option('display.max_colwidth', None)df['clean_text']=df['text'].pipe(remove_digits)\.pipe(remove_punctuation)\.pipe(remove_lessthan,length=3)\.pipe(remove_stopwords,stopwords=en_stopwords.union(hi_stopwords))print(df)ClassificationThe train data path should have 4 files
- train.txt
- validation.txt
- test.txtYou can just downloaddatasets/SAIL_2017/Processed Data/Devanagarifrom the Github repository to try this out.fromlingualytics.learnerimportLearnerlearner=Learner(model_type='bert',model_name='bert-base-multilingual-cased',dataset='SAIL-2017')learner.fit()Find topmost n-gramsfromlingualytics.representationimportget_ngramsimportpandasaspddf=pd.read_csv("https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv")ngrams=get_ngrams(df['text'],n=2)print(ngrams[:10])ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.LicenseMIT
Package: linguaml
LinguAMLLinguAML is an innovative project that combines the power of Large Language Models (LLMs) with traditional automated machine learning (AML) techniques, with a primary focus on tuning machine learning models for various tasks. By integrating LLMs' capabilities into the AML workflow, LinguAML aims to enhance the performance and efficiency of machine learning models, specifically by automating the process of hyperparameter tuning.Development NotesCheck the development notes 👉here.
Package: lingua-nostra
Lingua NostraLingua_Nostra is chatterbox's natural language parser, it converts natural
language into data structures, and data structures into natural language!The main use case of lingua_nostra is handling dates and numbersLatest version: 0.4.3Supported LanguagesTODO here will be a nice table comparing the state of language supportUsagesee the extensive unittests!fromlingua_nostra.formatimportnice_duration,nice_date,nice_date_time,\nice_number,nice_time,pronounce_numberfromlingua_nostra.parseimportextract_datetime,extract_number,\extract_numbers,extract_duration,normalizepronounce_number(100034000000299792458,short_scale=False)#"one hundred trillion, thirty four thousand billion, "#"two hundred and ninety nine million, seven hundred and ninety "#"two thousand, four hundred and fifty eight"extract_numbers("1 dog, seven pigs, macdonald had a farm, ""3 times 5 macarena")# [1, 7, 3, 5]
Package: linguappt
Installationpip3 install --verbose linguapptUsagePlease refer toapi docs.Execute usageValidate ppt templatelingua_pptx_validate --pptx [pptx file]Convert vocabulary csv file into ppt filelingua_vocabppt --sourcecsv [vocab csv file] --lang [language] --title [title shown in ppt] --destpptx [pptx file]Convert phrase csv file into ppt filelingua_phraseppt --sourcecsv [phrase csv file] --lang [language] --title [title shown in ppt] --destpptx [pptx file]Convert ppt into pdflingua_pptx2pdf --sourcepptx [pptx file] --destdir [dest directory storing pdf and images]Package usagefrom linguappt import SpanishVocabPPT, EnglishVocabPPT
from linguappt import EnglishPhrasePPT, SpanishPhrasePPT

def vocabppt(sourcecsv, title, lang, destpptx):
  _PPTS = {
    "en": EnglishVocabPPT,
    "es": SpanishVocabPPT
  }

  _PPT = _PPTS[lang]

  vp = _PPT(sourcecsv, title)
  vp.convert_to_ppt(destpptx)

def phraseppt(sourcecsv, title, lang, destpptx):
  _PPTS = {
    "en": EnglishPhrasePPT,
    "es": SpanishPhrasePPT
  }

  _PPT = _PPTS[lang]

  vp = _PPT(sourcecsv, title)
  vp.convert_to_ppt(destpptx)DevelopmentClone projectgit clone https://github.com/qishe-nlp/linguappt.gitInstallpoetryInstall dependenciespoetry updateTestpoetry run pytest -rP --capture=syswhich run tests undertests/*Executepoetry run lingua_pptx_validate --help
poetry run lingua_vocabppt --help
poetry run lingua_phraseppt --help

poetry run lingua_pptx2pdf2images --help
poetry run lingua_csv2media --helpCreate sphinx docspoetry shell
cd apidocs
sphinx-apidoc -f -o source ../linguappt
make html
python -m http.server -d build/htmlHost docs on github pagescp -rf apidocs/build/html/* docs/BuildChangeversioninpyproject.tomlandlinguappt/__init__.pyBuild python package bypoetry buildGit commit and pushPublish from local dev envSet pypi test environment variables in poetry, refer topoetry docPublish to pypi test bypoetry publish -r testPublish through CIGithub action build and publish package totest pypi repogit tag [x.x.x]
git push origin masterManually publish topypi repothroughgithub action
Package: lingua-py
Lingua-pyLingua-py is an Python wrapper for theRustlinguacrate withPyO3/Maturin.SummaryLingua is a NIF-based bridge for thelinguaRustlanguage detection library.UsageIn imitation ofUpstream, introduce basic usage.
Alternatively please seetestssection.10.1Basic usagelanguages:list[Language]=[Language.English,Language.Japanese]detector:LanguageDetector=LanguageDetectorBuilder.from_languages(languages).build()expected:Language=Language.Japaneseactual:Language=detector.detect_language_of("これは何語ですか？")assertactual==expectedpytest commandryesync--no-lock
ryerunpython-mpytest-s-vvpython/tests/small_usage/test_basic.py::test_basicLicenseApache License v2, and lingua-rs is alsoApache License v2.
Package: lingua-py-unofficial
Lingua-pyLingua-py is an Python wrapper for theRustlinguacrate withPyO3/Maturin.
This project is informal. It is expected that Python-binding will be developedofficiallyin the future.SummaryLingua is a NIF-based bridge for thelinguaRustlanguage detection library.UsageIn imitation ofUpstream, introduce basic usage.
Alternatively please seetestssection.10.1Basic usagelanguages:list[Language]=[Language.English,Language.Japanese]detector:LanguageDetector=LanguageDetectorBuilder.from_languages(languages).build()expected:Language=Language.Japaneseactual:Language=detector.detect_language_of("これは何語ですか？")assertactual==expectedpytest commandryesync--no-lock
ryerunpython-mpytest-s-vvpython/tests/small_usage/test_basic.py::test_basicLicenseApache License v2, and lingua-rs is alsoApache License v2.
Package: linguarita-lucien-rowan
LinguaritaWhen gettext doesn't work and you are sick of writing your own language engine for every program, you came to the right place! Linguarita is a generic language engine retrieving translations from JSON files.The changelog can be found athttps://codeberg.org/lucien-rowan/linguarita/src/branch/main/CHANGELOG.mdInstalling the moduleYou can either copy the source code and directly include linguarita.py into your program but I find that difficult. Instead, you can just open your virtual environment and typepip install linguarita-lucien-rowaninto the terminal.Using the moduleBasically, your project needs the following files:main.py (can be another name as well, just for execution)translations/(lang).json (the translations are stored in JSON files, just replace (lang) with the actual language like en for English, de for German or ua for Ukrainian)The translations occur withlinguarita.translateText(og_str, langcode)with og_str being the string you want to translate (must be the same as in the JSON file) and the langcode being the filename of the JSON file before the dot.ContributingOf course, this module also needs some attention.Which attention?Enhancing it with useful featuresFixing the bugs that might occurDocumenting its APITranslating that documentation to other languagesImplementing it into Python applicationsWhile Linguarita is a simple module from its code, it might seem a bit hard to implement it in real life scenarios. Therefore, documentation and example applications will follow.
Package: linguars
lingua-pylingua-rsPython binding. An accurate
natural language detection library, suitable for long and short text alike.InstallationpipinstalllinguarsUsageimportlinguarsdetector=linguars.LanguageDetector()print(detector.detect('我们中出了一个叛徒'))print(detector.confidence('我们中出了一个叛徒'))LicenseThis work is released under the MIT license. A copy of the license is provided in theLICENSEfile.
Package: lingueeParser
Linguee ParserThis is a parser for the Linguee dictionary. It is prohibited by the terms and conditions to use this script to forward their service to third parties. For more information see below.This script parses the translation based on the html IDs and classes. Therefore, if Linguee decides to change the HTML this script is going to brake.How to use the script?Download the script with pip or gitImport it:from lingueeParser import lingueeCreate an instance of the object:parser = linguee.Linguee(("english", "german"))Search for terms:term = parser.search('spam')Notes:You can access all your searches withparser.get_dictionary()Search gives only the term the same would beparser.get_dictionary()['spam']Terms and ConditionsIf you use the API, make sure you comply with LingueeTerms and Conditions, and in particular with that clause:Both private and business usage of linguee.com services is free of charge. It is however strictly prohibited to forward on our services to third parties against payment
Package: linguin
Linguin AI Python wrapperThis is a Python wrapper for theLinguin AIAPI (seeAPI docs) providing Language Detection as a Service.Linguin AI is free for up to 100 detections per day. You can get your API keyhere.Installationpip3 install linguinUsageGet started with just a few lines of code:from linguin import Linguin

# go to https://linguin.ai to get your key
linguin = Linguin("YOUR_API_TOKEN")

response = linguin.detect_language("test")

response.is_success
# >> True

response.result
# >> {'results': [{ 'lang': 'en', confidence: 1.0 }, ...]}

response = linguin.detect_profanity("you moron")

response.is_success
# >> True

response.result
# >> {'score':  0.9981}If anything goes wrong for example: empty query string:response = linguin.detect_language(" ")
    
response.is_success
# >> False
   
print(response.error)
# >> Error code: 400. The language of an empty text is more of a philosophical question.

response.error.status
# >> 400

response.error.message
# >> The language of an empty text is more of a philosophical question.If you prefer to handle exceptions instead:response = linguin.detect_language(" ", raise_on_error=True)
# >> raises LinguinInputErrorSee the list of all exceptionshere.Bulk detectionTo detect the language of multiple texts with one API call, you can pass them as an array to thebulk_detect_languagemethod.
The results will be returned in the same order as the texts. All texts have to not be empty.response = linguin.bulk_detect_language(["test", "Bahnhof", "고마워요"])

response.is_success
# >> True

response.result
# >> {'results': [[{'lang': 'en', 'confidence': 1.0}, ...], [{...}], ...]}To check multiple texts for profanity with one API call, you can pass them as an array to thebulk_detect_profanitymethod.
The results will be returned in the same order as the texts. All texts have to not be empty.response = linguin.bulk_detect_profanity(["a test", "you are a moron"])

response.is_success
# >> True

response.result
# >> {'scores': [0.0124, 0.9981]}Account statusYou can fetch the status of your account:response = linguin.status()

response.result
# >> {'daily_limit': 50000, 'detections_today': 4500, 'remaining_today': 45500}
# for unlimited usage we return -1Language listYou can fetch the list of supported languages:from linguin import Linguin

languages = Linguin.languages()
# >> { 'de': ["German", "Deutsch"], ... }DevelopmentAfter checking out the repo, runpip3 install -r requirements.txtto install dependencies.
Run tests withpython3 -m unittestContributingBug reports and pull requests are welcome on GitHub athttps://github.com/LinguinAI/linguin-python. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to thecode of conduct.LicenseThe gem is available as open source under the terms of theMIT License.Code of ConductEveryone interacting in the Linguin project's codebases, issue trackers, chat rooms and mailing lists is expected to follow thecode of conduct.