Saravanan666/pypi-org huggingface dataset:
Package: corpusit
Corpusitcorpusitprovides easy-to-use dataset iterators for natural language modeling
tasks, such as SkipGram.It is written in rust to enable fast multi-threading random sampling with
deterministic results. So you dont have to worry about the speed /
reproducibility.Corpusit does not provide tokenization functionalities. So please usecorpusiton tokenized corpus files (plain texts).EnvironmentPython >= 3.6Installation$pipinstallcorpusitOn Windows and MacOSPlease installrustcompiler before
executingpip install corpusit.UsageSkipGramEach line in the corpus file is a document, and the tokens should be separated by whitespace.importcorpusitcorpus_path='corpusit/data/corpus.txt'vocab=corpusit.Vocab.build(corpus_path,min_count=1,unk='<unk>')dataset=corpusit.SkipGramDataset(path_to_corpus=corpus_path,vocab=vocab,win_size=10,sep=" ",mode="onepass",# onepass | repeat | shufflesubsample=1e-3,power=0.75,n_neg=1,)it=dataset.positive_sampler(batch_size=100,seed=0,num_threads=4)fori,pairinenumerate(it):print(f'Iter{i:>4d}, shape={pair.shape}. First pair: 'f'{pair[0,0]:>5}({vocab.i2s[pair[0,0]]:>10}), 'f'{pair[0,1]:>5}({vocab.i2s[pair[0,1]]:>10})')# Return:# Iter    0, shape=(100, 2). First pair:    14 (        is),    10 ( anarchism)# Iter    1, shape=(100, 2). First pair:     8 (        to),   540 (      and/)# Iter    2, shape=(100, 2). First pair:   775 (constitutes),    34 (anarchists)# Iter    3, shape=(100, 2). First pair:    72 (     other),   214 (  criteria)# Iter    4, shape=(100, 2). First pair:   650 (  defining),   487 ( companion)# ...SkipGram with negative samplingit=dataset.sampler(100,seed=0,num_threads=4)fori,resinenumerate(it):pair,label=resprint(f'Iter{i:>4d}, shape={pair.shape}. First pair: 'f'{pair[0,0]:>5}({vocab.i2s[pair[0,0]]:>10}), 'f'{pair[0,1]:>5}({vocab.i2s[pair[0,1]]:>10}), 'f'label ={label[0]}')# Returns:# Iter    0, shape=(200, 2). First pair:    15 (        is),    10 ( anarchism), label = True# Iter    1, shape=(200, 2). First pair:     9 (        to),   722 (      and/), label = True# Iter    2, shape=(200, 2). First pair:   389 (constitutes),    34 (anarchists), label = True# Iter    3, shape=(200, 2). First pair:    73 (     other),   212 (  criteria), label = True# Iter    4, shape=(200, 2). First pair:   445 (  defining),   793 ( companion), label = True# ...RoadmapGloVeLicenseMIT
Package: corpus-patterns
corpus-patternsA preparatory utils library.Create a custom tokenizerfromcorpus_patternsimportset_tokenizernlp=spacy.blank("en")nlp.tokenizer=set_tokenizer(nlp)The tokenizer:Removes dashes from infixesAdds prefix/suffix rules for parenthesis/bracketsAdds special exceptions to treat dotted text as a single tokenUse with modified config file:@spacy.registry.tokenizers("test")# type: ignoredefcreate_corpus_tokenizer():defcreate_tokenizer(nlp):returnset_tokenizer(nlp)returncreate_tokenizernlp=spacy.load("en_core_web_sm",config={"nlp":{"tokenizer":{"@tokenizers":"test"}}},)Add .jsonl files to directoryEach file will contain lines of spacy matcher patterns.fromcorpus_patternsimportcreate_rulesfrompathlibimportPathcreate_rules(folder=Path("location-here"))# check directoryUtilsannotate_fragments()- given an nlp object and some*.txtfiles, create a single annotation*.jsonlfileextract_lines_from_txt_files()- accepts an iterator of*.txtfiles and yields each line (after sorting the same and ensuring uniqueness of content).split_data()- given a list of text strings, split the same into two groups and return a dictionary containing these groups based on the ratio provided (defaults to 0.80)
Package: corpus-pax
corpus-paxUsing Github API (to pull individuals, orgs, and article content), setup a local sqlite database, syncing images to Cloudflarel utilized in theLawSQL dataset.DocumentationSeedocumentation.DevelopmentCheckout code, create a new virtual environment:poetryaddcorpus-pax# python -m pip install corpus-paxpoetryupdate# install dependenciespoetryshellRun tests:pytest
Package: corpus-preprocess
corpus-preprocessUtility functions to preprocess Phil. legalese inweasel-based flows:lexcat-proj; andlexcat-multi[!IMPORTANT]
Requires privatecorpus-assetsfolder and sqlite3 db incitelaws-datato be cloned locally.- corpus-assets: # folder structure
  - concept: # must be two-level nested patterns.json + q.txt
  - artifact: # single folder patterns.json + q.txt
  - text: # each file is a .txtLanguage customizationAssuming familiarity with spacy:nlp.tokenizer=customize_tokenizer(nlp,special_token_rules)# custom tokenizerruler=nlp.add_pipe("span_ruler",config={"spans_key":"ruler","phrase_matcher_attr":"LOWER","spans_filter":{"@misc":"spacy.first_longest_spans_filter.v1"},# longest spans only},)ruler.add_patterns(patterns)# created patterns from this library and corpus-assets[!NOTE]
Loading model with 130k pattern lines takes ~2 min.Training dataConcept spansforfolderinget_concepts(asset_dir.joinpath("concept")):bn=DocBin()# use q.txt as queries to the db# number of segments per q.txt to fetchdocs=apply_concept_q_filter(nlp,db_file,filter_path=folder,max_segments=500)fordocindocs:bn.add(doc)bn.to_disk(asset_dir.joinpath(f"train/{folder.stem}.spacy"))Each concept_dir contains subtopics:- corpus-assets: # folder structure
  - concept: # must be two-level nested
    - political: # main subject category
        - bill_of_rights: # sub-topic
            - patterns.json # contains matcher files
            - q.txt # contains lines which can be used to query the databaseBecause of this structure, it's possible to train atextcat_multilabelcomponent:textcat_options=[concept["id"].split("/")[0]forconceptinconcept_patterns]@Language.factory(name="add_cats_from_spans")classAddTextCatComponent:def__init__(self,nlp:Language,name:str,options:list[str]):self.nlp=nlpself.options=optionsdef__call__(self,doc)->Doc:doc.cats={op:0.0foropinself.options}forspanindoc.spans["sc"]:ifspan.id:# some spans won't have an idvalue=self.nlp.vocab.strings[span.id]if"/"invalue:# e.g. political/bill_of_rightsmain_topic=value.split("/")[0]# just politicalifmain_topicinself.options:ifdoc.cats[main_topic]==0.0:doc.cats[main_topic]=1.0returndocNon-concept spansAlthough patterns fromset_patterns()are included in the constructednlpobject,
can ensure that a certain of rows (filter_count) are fetched from the database that have spans which are labeledtitleand/orserial, etc.forlabelin{"unit","ref","serial","title","axiom","date","juridical"}:bn=DocBin()docs=apply_label_filter(nlp,db_file,filter_labels={label},filter_count=1500)fordocindocs:bn.add(doc)bn.to_disk(asset_dir.joinpath(f"train/{label}.spacy"))
Package: corpus-replicator
Corpus ReplicatorCorpus Replicator is a corpus generation tool that enables the creation of multiple
unique output files based on templates. The primary intended use case is the
creation of a seed corpus that can be used by fuzzers. Support for additional output
formats can be added via the creation ofRecipes. If a desired format is unsupported,
support can be added via the creation of aCorpusGenerator.The goal is to create an efficient corpus that maximizes code coverage and minimizes
file size. Small unique files that execute quickly are preferred.Currently four media types can be generatedanimation,audio,imageandvideo.RequirementsCorpus Replicator relies onFFmpeg.Installationpip install corpus-replicatorExampleThis is an examplerecipefile.# "base" contains required entries and default flagsbase:codec:"h264"# name of the codeccontainer:"mp4"# container/file extensionlibrary:"libx264"# name of librarymedium:"video"# supported mediumtool:"ffmpeg"# name of supported tooldefault_flags:encoder:# "encoder" flag group["-c:v","libx264"]resolution:# "resolution" flag group["-s","320x240"]# variations allow flags to be added and overwritten# one file will be generated for each entry in a flag groupvariation:resolution:# flag group - overwrites default flag group in "base"-["-s","640x480"]-["-s","32x18"]-["-s","64x64"]monochrome:# flag group - adds new flag group-["-vf","hue=s=0"]Running the recipe will generate a corpus:$ corpus-replicator example.yml video -t test
Generating templates...
1 recipe(s) will be used with 1 template(s) to create 4 file(s).
Generating 4 'video/libx264/h264/mp4' file(s) using template 'test'...
Optimizing corpus, checking for duplicates...
Done.Resulting corpus:$ ls generated-corpus/
video-h264-libx264-test-monochrome-00.mp4
video-h264-libx264-test-resolution-01.mp4
video-h264-libx264-test-resolution-00.mp4
video-h264-libx264-test-resolution-02.mp4A more complex corpus can be generated by using multipleRecipesandTemplatesat
once.Recipes are stored insrc/corpus_replicator/recipes.
Package: corpus-similarity
corpus_similarityMeasure the similarity between two corpora (text datasets). The measures work best when each corpus is at least 10k words. This package support 74 languages.from corpus_similarity import Similarity
cs = Similarity(language = "eng")

result = cs.calculate(corpus1, corpus2)The package contains all preprocessing and training. Only the language needs to be specified. A list of supported languages is provided below.InputTheSimilarity.calculatemethod requires two input corpora. These can be a list of strings or a filename (supports .txt and .gz files).OutputThe output is a scalar measure of how similar the two corpora are. The values fall between 0 (very different) and 1 (very similar). The values are consistent within languages, but not across languages. For example, Swedish has higher relative similarity than Estonian.Installationpip install corpus_similarity

pip install git+https://github.com/jonathandunn/corpus_similarity.gitHow It WorksThe corpus similarity measure is a simple character n-gram comparison, with the best performance coming with using Spearman's Rho as a measure of correlation. The original idea for this kind of corpus comparison comes from Adam Kilgarriff (https://kilgarriff.co.uk/Publications/2001-K-CompCorpIJCL.pdf).Recent work inLinguahas evaluated the measures used in this package extensively in a multi-lingual setting (https://arxiv.org/abs/2206.04332). These measures have since been used to model the relationship between registers in a multi-lingual setting (https://arxiv.org/abs/2209.09813) and to validate geo-referenced corpus collections (https://arxiv.org/abs/2104.01294). Other work has modelled the relationship between corpus similarity (upstream) and embedding similarity (downstream) (https://arxiv.org/abs/2206.04330). These papers provide further details for the theory and evaluation behind this package.Languagesamh, Amharicara, Arabicaze, Azerbaijaniben, Bengalibul, Bulgariancat, Catalanceb, Cebuanoces, Czechcha, Chamorrodan, Danishdeu, Germanell, Greekeng, Englishest, Estonianeus, Basquefas, Farsifij, Fijianfin, Finnishfra, Frenchgle, Gaelicglg, Galicianguj, Gujaratihat, Hatianhaw, Hawaiianheb, Hebrewhin, Hindihmo, Hiri Motuhun, Hungarianilo, Ilocanoind, Indonesianisl, Icelandicita, Italianjav, Javanesejpn, Japanesekan, Kannadakat, Georgiankor, Koreanlav, Latvianlit, Lithuanianmal, Malayalammar, Marathimkd, Macedonianmlg, Malagasymon, Mongolianmri, te reo Maorimsa, Malaynld, Dutchnor, Norwegianpan, Punjabipol, Polishpor, Portugueseron, Romanianrus, Russiansin, Sinhalaslk, Slovakslv, Sloveniansmo, Samoansom, Somalispa, Spanishsqi, Albanianswe, Swedishtah, Tahitiantam, Tamiltel, Telugutgl, Tagalogtha, Thaiton, Tongantur, Turkishtvl, Tuvaluanukr, Ukrainianurd, Urduuzb, Uzbekvie, Vietnamesezho, Chinese
Package: corpus-statistics
Corpus Statistics: A Very Basic (for now) spaCy Pipeline ComponentIf you want to know what tokens your spaCy pipeline has seen, this is the component for you.pipinstallcorpus-statistics# OR for latestpipinstallgit+https://github.com/pmbaumgartner/corpus_statistics⚡️Examplefromspacy.lang.enimportEnglish# Use some example datafromdatasetsimportload_datasetdataset=load_dataset("imdb")texts=dataset["train"]["text"]nlp=English()# or spacy.load('a_model')nlp.add_pipe("simple_corpus_stats")# ✨ start the magicfordocinnlp.pipe(texts):# ➡️ do your pipeline stuff! ➡️passcorpus_stats=nlp.get_pipe("simple_corpus_stats")# check if a token has been processed through this pipelinetoken="apple"iftokenincorpus_stats:token_count=corpus_stats[token]print(f"'{token}' mentioned{token_count}times")# 'apple' mentioned 24 timesIt's got all your favoritelegomenalikehapaxanddis.only_seen_once=len(corpus_stats.hapax_legomena)percent_of_vocab=only_seen_once/corpus_stats.vocab_sizeprint(f"{percent_of_vocab*100:.1f}% tokens only occurred once.")# 47.6% tokens only occurred once.only_seen_twice=len(corpus_stats.dis_legomena)percent_of_vocab_2x=only_seen_twice/corpus_stats.vocab_sizeprint(f"{percent_of_vocab_2x*100:.1f}% tokens occurred twice.")# 12.3% tokens occurred twice.We counted some things too:# corpus_stats.vocabulary is a collections.Counter 🔢print(*corpus_stats.vocabulary.most_common(5),sep="\n")# ('the', 289838)# (',', 275296)# ('.', 236702)# ('and', 156484)# ('a', 156282)mean_doc_length=sum(corpus_stats.doc_lengths)/corpus_stats.corpus_lengthprint(f"Mean doc length:{mean_doc_length:.1f}")# Mean doc length: 272.5Use in Model Training and Config FilesThis can be quite helpful if you wanted to know what tokens were seen in your training data. You can include this component in your training config as follows....
[nlp]
lang = "en"
pipeline = ["simple_corpus_stats", ...]
...

[components]

[components.simple_corpus_stats]
factory = "simple_corpus_stats"
n_train = 1000  # This is important! See below⚠️ 🔁 If you use this component in a training config, your pipeline will see the same docs multiple times, due to the number of training epochs and evaluation steps, so the vocab counter will be incorrect. To correct for this, you need to specify the number of examples in your training dataset as then_trainconfig parameter.importspacynlp=spacy.load("your_trained_model")corpus_stats=nlp.get_pipe("simple_corpus_stats")assertmin(corpus_stats.vocabulary.values())==1# value from configassertlen(corpus_stats.doc_lengths)==1000
Package: corpus-writer
corpus-writerUpdate pre-existing opinion file (in markdown front-matter) with statutes, citations, short title.Thedecisions/statutesdirectories needs to be configured via an.envfile, seejust dumpenvfor anStepsDelete the statutes database since this may have been previously updated via their original yaml filesThe lack of a statutes database will cause a rebuild of the sameRun the following python command:fromcorpus_writerimportget_opinion_files_by_year,update_markdown_opinion_fileforiinrange(1900,1946):forfileinget_opinion_files_by_year(year=i):update_markdown_opinion_file(file=file)Note that for cases likeam/97-9-282-rtc/1998-04-22/main-123.mdwhere it takes too long to determine citations (likely a recursive regex error), a timer of 5 seconds is used byupdate_markdown_opinion_file.
Package: corpy
Installation$python3-mpipinstallcorpyOnly recent versions of Python 3 (3.10+) are supported by design.Help and feedbackIf you get stuck, it’s always a good idea to start by searching the
documentation, the short URL to which ishttps://corpy.rtfd.io/.The project is developed onGitHub. You can ask for help viaGitHub
discussionsand report bugs and give other kinds of feedback viaGitHub
issues. Support is provided gladly, time and other engagements permitting, but
cannot be guaranteed.What is CorPy?A fancy plural forcorpus;) Also, a collection of handy but not especially
mutually integrated tools for dealing with linguistic data. It abstracts away
functionality which is often needed in practice for teaching and/or day to day
work at theCzech National Corpus, without aspiring to
be a fully featured or consistent NLP framework.Here’s an idea of what you can do with CorPy:add linguistic annotation to raw textual data using eitherUDPipeorMorphoDiTaeasily generate word cloudsrun code ina sanitized global environment(useful for
debugging in interactive sessions, e.g. with Jupyter notebooks inJupyterLab)generate phonetic transcripts of Czech textswrangle corpora in the vertical formatdevised originally
forCWB, used also by(No)SketchEngineplus somecommand line utilitiesNoteShould I pick UDPipe or MorphoDiTa?Both are developed atÚFAL MFF UK.UDPipehas more features at the cost
of being somewhat more complex: it does bothmorphological tagging
(including lemmatization) and syntactic parsing, and it handles a
number of different input and output formats. You can also downloadpre-trained modelsfor many
different languages.By contrast,MorphoDiTaonly haspre-trained models for Czech and English, and only performsmorphological tagging (including lemmatization). However, its
output is more straightforward – it just splits your text into tokens and
annotates them, whereas UDPipe can (depending on the model) introduce
additional tokens necessary for a more explicit analysis, add multi-word
tokens etc. This is because UDPipe is tailored to the type of linguistic
analysis conducted within theUniversalDependenciesproject, using theCoNLL-Udata format.MorphoDiTa can also help you if you just want to tokenize text and don’t have
a language model available.DevelopmentDependencies and building the docscorpyneeds to be installed in the ReadTheDocs virtualenv forautodocto
work. The optional dependencies in thedocgroup are also needed. This is
all configured in.readthedocs.yml.LicenseCopyright © 2016–presentÚČNK/David LukešDistributed under theGNU General Public License v3.
Package: corrai
Measured Data Exploration for Physical and Mathematical ModelsThis Python library is designed to handle measured data from test benches or Building Energy Management Systems (BEMS).
It offers physical model calibration frameworks and original AI methods.FeaturesThe library includes the following features:Data cleaning:Based onPandas, it usesScikit-learnframework
to simplify data cleaning process through the creation of pipelines for time series.Data plotting:Generates plots of measured data, visualizes gaps and cleaning methods effects.Physical model calibration:Provides base class to define calibration problem, usesPymoooptimization methods for parameters identification.Building usage modeling:Generates time series of occupancy-related usage
(Domestic Hot water consumption, grey water use...).AI tools for HVAC FDD:Includes artificial intelligence tools for
Heating Ventilation and Air Conditioning (HVAC) systems fault detection and diagnostics (FDD).Getting startedThe source code is currently hosted on GitHub at:https://github.com/BuildingEnergySimulationTools/corraiTutorials are available in the dedicated folder.Released version are available at the Python Package Index (PyPI):# PyPI
pip install corrai
Package: corral-pipeline
The Powerful Pipeline FrameworkCorral will solve your pipeline needs by merging a database full connection
interface with a MVC model, by making you able of editing your custom schemas
and adding the possibility of writting specific processing steps following a
intuitive data handling model.FeaturesCommand line interface with standard utilitiesEditable commands on your applicationCompletely built on SQLAlchemy ORMAllows the introduction of StepsFollows the MVC (Model View Controller) patternSupportFull Documentation and Tutorial:http://corral.readthedocs.io/Community Forum:https://groups.google.com/forum/#!forum/corral-users-forumIssues:If you have issues please report them as a issue
here:https://github.com/toros-astro/corral/issuesUser InstallationThe easiest way to install corral is using pip$pipinstall-Ucorral-pipelineDevelopmmentInstall project by cloning fromCorral github:$gitclonehttps://github.com/toros-astro/corral.gitand by makingpip install-e., or the classicpython setup.py install.You can also run the install by giving the github link directly to pip:$pipinstall-egit+https://github.com/toros-astro/corral.gitLicenseBSD-3:https://opensource.org/licenses/BSD-3-ClauseBSD-3 Licence Explained:https://tldrlegal.com/license/bsd-3-clause-license-(revised)
Package: correct
correctIn what followspythonis an alias forpython3.6orpypy3.6or any later version (python3.7,pypy3.7and so on).InstallationInstall the latestpip&setuptoolspackages versionspython-mpipinstall--upgradepipsetuptoolsUserDownload and install the latest stable version fromPyPIrepositorypython-mpipinstall--upgradecorrectDeveloperDownload the latest version fromGitHubrepositorygitclonehttps://github.com/lycantropos/correct.gitcdcorrectInstall dependenciespython-mpipinstall-rrequirements.txtInstallpythonsetup.pyinstallDevelopmentBumping versionPreparationInstallbump2version.Pre-releaseChoose which version number category to bump followingsemver
specification.Test bumping versionbump2version--dry-run--verbose$CATEGORYwhere$CATEGORYis the target version number category name, possible
values arepatch/minor/major.Bump versionbump2version--verbose$CATEGORYThis will set version tomajor.minor.patch-alpha.ReleaseTest bumping versionbump2version--dry-run--verbosereleaseBump versionbump2version--verbosereleaseThis will set version tomajor.minor.patch.Running testsInstall dependenciespython-mpipinstall-rrequirements-tests.txtPlainpytestInsideDockercontainer:withCPythondocker-compose--filedocker-compose.cpython.ymlupwithPyPydocker-compose--filedocker-compose.pypy.ymlupBashscript:withCPython./run-tests.shor./run-tests.shcpythonwithPyPy./run-tests.shpypyPowerShellscript:withCPython.\run-tests.ps1or.\run-tests.ps1cpythonwithPyPy.\run-tests.ps1pypy
Package: correcthorse
Correct Horse: A memorable passphrase generatorCorrect Horse is a tool for generating resonably secure but reasonably
memorable passphrases. It does this by picking a set of words from a
dictionary and then sorting them into an order that plausibly makes
sense as a phrase. It can be used either as a command line tool to
generate and print a passphrase that one might enter into another
application or through an API to pick new passphrases for something
like a rotating code on a WiFi network.UsageCommand linecorrecthorse [-h] [-n MAX_WORDS] [-N MAX_LETTERS] [-l] [-c] [-u] [-s] [-j] [-H]
                   [-S SEPARATOR] [-f FILENAME] [-L LOCALE]Use the-n  WORD_COUNTor--max_words WORD_COUNToption to set the maximum number of words in the passphrase. Use the-N  LETTER_COUNTor--max_letters LETTER_COUNToption to set the maximum number of letters in the passphrase. You may set bothThe-lor--lower-casewill cause all of the words in the passphrase to be printed in lower case while the-cor--capitaliseflag will cause the first letter of each word to be capitalised. The default is to capitalise the first letter.When the words of the passphrase are printed a seperator string will be placed between then. This can be set using the-S SEPARATORor--separator SEPARATORoption. For convenience you can use-uor--underscoreto join the words with an underscore,-sor--spaceto join the words with a space,-Hor--hyphento join the words with a hyphen or-jor--jointo simply run the words together without a separator.By defaultcorrecthorsewill detect your locale and pick a word set that matches your local language. You can override the detected locale using the-L LOCALEor--locale LOCALEoptions. While great in theory, unless your language is English the chances are that currently there is no built in word file for your language. You can however specify a custom word file using the-f FILENAMEor--word-file FILENAMEoptions.Use as a python moduleTo use thecorrecthorsemodule through the API create an instance of theWordSetusingWordSet(filename=None, locale=None, encoding='UTF-8')and then callrandom_phrase(max_words=4).
Package: correct-horse
This program allows you to easily create strong, memorable passphrases.It reads a specified number of bits of data from your system’s CSPRNG, turrns it
into a very large number, and prints it out in base 7776.The result will be something likegreet say yaw dunham italy pass navel. If
your system has a good random number generator, this should be a reasonably secure
password, and also reasonably easy to memorize.see also:http://world.std.com/~reinhold/diceware.htmlhttps://www.rempe.us/diceware/https://xkcd.com/936/https://github.com/bbiw/mempassThe Portuguese translation is mostly from translate.google.com.
Estou certo que se pode melhorar.Additional translations are welcome.