Saravanan666/pypi-org huggingface dataset:
Package: preprocessingtext
# preprocessingtextA tool short, but very usefull to help in pre-processing data from texts.## How to Install>> pip install --user preprocessingtext## Usage#### Using stem_sentence()>> from preprocessingtext import CleanSentence>> cleaner =  CleanSentence(idiom='portuguese')>> cleaner.stem_sentence(sentence="String", remove_stop_words=True, remove_punctuation=True, normalize_text=True, replace_garbage=True)To init a a class, you need to pass the idiom that you want to work. The custom value, is "portuguese".Before, you can instance a new object from CleanSentence, and call the method stem_sentence. You can choose in use"remove_stop_words" from string (pass True or False) and "remove_punctuation" from string (pass True or False),"replace_garbage" (True or False) removing values from data, and "normalize_text" (True or False) to normalize text.#### Usage of list_to_replaceYou can improve what you need to replace (clean) in your data.  You can use "cleaner.list_to_replace.append('what_you_need_to_add')",or you can pass a new list of values:  cleaner.list_to_replace = ['item1', 'item2', 'item3']# Custom value of list_to_replace>> list_to_replace = ['https://', 'http://', '$']# Adding new valueslist_to_replace.append('item1')['https://', 'http://', 'R$', '$', 'item1']# Replacing values>> list_to_replace = ['item1', 'item2', 'item3']#### Using tokenizer()>> cleaner.tokenizer('Um exemplo de tokens.')>> ['Um', 'exemplo', 'de', 'tokens']## Example## Using all parameters of stem_sentence()>> string = "Eu sou uma sentença comum. Serei pré-processada com este modulo, veremos a serguir usando os métodos disponiveis">> cleaner.stem_sentence(sentence=string,remove_stop_words=True,remove_punctuation=True,normalize_text=True,replace_garbage=True)>> sentenc comum pre-process modul ver segu us metod disponi## Don't using remove_stop_words>> print(cleaner.stem_sentence(sentence=string,remove_stop_words=False,remove_punctuation=True,normalize_text=True,replace_garbage=True))>> eu sou uma sentenc comum ser pre-process com est modul ver a segu us os metod disponi## Tokenizer>> print(cleaner.tokenizer('Um exemplo de tokens.'))>> ['Um', 'exemplo', 'de', 'tokens']## Cleaning garbage words>> string_web = 'Acesse esses links para ganhar dinheiro: https://easymoney.com.net and http://falselink.com'>> cleaner.stem_sentence(sentence=string_web,remove_stop_words=False,remove_punctuation=True,replace_garbage=True)>> acess ess link par ganh dinh easymoney.com.net and falselink.com## English example>> en_cleaner = CleanSentences(idiom='english')>> string_web = 'Access these links to gain money: https://easymoney.com.net and http://falselink.com'>> print(en_cleaner.stem_sentence(sentence=string_web,remove_stop_words=True,remove_punctuation=True,replace_garbage=True))>> acc link gain money easymoney.com.net falselink.com# Author{'name': Everton Tomalok,'email': evertontomalok123@gmail.com}
Package: preprocessor
### PreprocessorSuited for simple preprocessing of Python files. The original use case of the package was annoying Cython work to do with data types. Notice that Preprocessor can run on any type of file (e.g. TSV, C++ sources etc.). To avoid clashes with other programming language preprocessor prefix and suffix is fully tunable.#### ExampleHere's a small file where we replicate the same Cython code 3 times - once for every of int, float and double datatypes. You can find the code for `typed_expression` function in `sample_utils.py` in this repo.```Pythonpypfrom sample_utils import typed_expressionypycdef class Mat:cdef CMat[dtype] matinternalint dtypedef sum(Mat self):# Inline preprcoessor expression. Executes a function.# Generally inline expressions should be one line,# But can be extended to multiple where all the data# on second line and following will be captured into# a string argument passed as last positional argument# to a functionpypinline typed_expression(pyp, "self.matinternal", "CMat",print('siema')return WrapMat(TYPED_EXPRESSION.sum())ypy```The output of the preprocessed file looks like this:```Pythoncdef class Mat:cdef CMat[dtype] matinternalint dtypedef sum(Mat self):# Inline preprcoessor expression. Executes a function.# Generally inline expressions should be one line,# But can be extended to multiple where all the data# on second line and following will be captured into# a string argument passed as last positional argument# to a functionif self.dtype == np.int32:print('siema')return WrapMat((<CMat[int]>(self.matinternal)).sum())elif self.dtype == np.float32:print('siema')return WrapMat((<CMat[float]>(self.matinternal)).sum())elif self.dtype == np.float64:print('siema')return WrapMat((<CMat[double]>(self.matinternal)).sum())else:raise ValueError("Invalid dtype:" + self.dtype + " (should be one of int32, float32, float64)")```#### Command Line InterfacePreprocessor comes with a command line script, to easily run the preprocessor form environments outside of Python```bashpreprocessor --input sample.py.pre --output sample.py```
Package: preprocessutils
preprocessutilsThis packages has data processing tools which can be used to make developer work easierInstallationpip install preprocessutilsUsagefrom preprocessutils import fix_imbalanceOVER-SAMPLEbalanced_dataframe = fix_imbalance(data=df, target=target, threshold=10.0, oversample=True)OVER-SAMPLE SMOTEbalanced_dataframe = fix_imbalance(data=df, target=target, threshold=10.0, smote=True)UNDER-SAMPLEbalanced_dataframe = fix_imbalance(data=df, target=target, threshold=10.0, oversample=False)
Package: preprocmachine
preprocmachineA tool to automate preprocessing phase.Free software: MIT licenseDocumentation:https://preprocmachine.readthedocs.io.FeaturesTakes a pandas dataframe as input along with target attributeExpandable library of self-contained modules containing preprocessing techniques available to useNormalizationImputationFeature SelectionOutlier DetectionDuplicate DetectionProvides flexibility to include a user defined preprocessing method if requiredAlgorithm is flexible enough to work with any user defined ML algorithmUtilizes Epsilon greedy algorithm to determine the preprocessing stepsOnce algorithm execution is complete, it will return following:Processed Datasetmetric to evaluate the performance  with comparison to performance before preprocessingSeries of Preprocessing Steps involved to reach this performanceCreditsInspiration was taken from a published paper and open source project Learn2Clean:https://github.com/LaureBerti/Learn2CleanHistory0.0.1 (2021-12-01)First release on PyPI.
Package: prep_source_repos
prep_source_repos-----------------Introduction============This repository contains scripts for managing multiple outstanding patchesto a gerrit based project. It was initially developed for managing TripleOdeployments, and still makes certain TripleOish assumptions (patches welcomeif you find the tool more generally useful)The source repo includes:- tooling to combine arbitrary unmerged gerrit patches (prep_source_repos)which will also export an rc file with git refs based on the combinedbranches- a sample config file that we're using for our TripleO deployments(repo_refs.yaml)Usage=====* create a repo_refs.yaml (see the one in the root of this repositoryfor inspiration).* run prep_source_repos $YOUR\_REFS\_FILE $DESTINATIION\_DIR to checkout andupdate the repositories specified by the refs file (in a TripleO context,$DESTINATION\_DIR will usually be "$TRIPLEO\_ROOT").Note that local edits are saved via git stash whenever you refresh yoursource repos, and restored after the update (which may, of course,fail). This provides a convenient way to use local edits / work inprogress for repositories that are used directly (vs e.g. those that arecloned into images).* (optional) source YOUR_REFS_FILE.variables to configure TripleO scripts touse your freshly integrated branches* proceed with any tripleo activies you might have (building images, deploying,etc etc).Advanced use============Refs that don't match the xx/yy/zz form of gerrit refs are presumed to belocal work-in-progress branches. These are not fetched, but are merged intothe rollup branch along with all the other patches. With a little care thispermits working effectively with multiple patchsets in one project withoutthem being made into a stack in gerrit.Refs of the form xx/yy/0 are late-bound references to gerrit - they will usethe gerrit REST API to find out the latest version and will use that.When running prep-source-repos any additional arguments after the refs andoutput dir are used to filter the repositories to fetch - so when working on(say) two local orthogonal patches to nova, and you need to update yourrollup branch just do::prep-source-repos foo bar novaand only nova will be updated.
Package: preptext
preptextUtilities of preparing datasets for deep learning models.
Can easily convert your text-formatted data into aDataLoaderofpytorch.Installationpipinstallpreptextor build from source code using:gitclonehttps://github.com/styxjedi/preptext.gitcdpreptext
pythonsetup.pyinstallUsageEach dataset consists of many entries, and each entry contains many fields.This tool is built following the logic above.For a specific dataset, you just need to follow the steps below.1. Define some fieldsSuppose we are working on a translation task that contains two fields,srcandtrg, in the dataset.importpreptextfields=preptext.Fields()fields.add_field("src",# nameinit_token="<bos>",# bos tokeneos_token="<eos>",# eos token# ... ...)fields.add_field("trg",# nameinit_token="<bos>",# bos tokeneos_token="<eos>",# eos token# ... ...)You would see more parameters when using.Basicly, the same field used in different datasets (train, valid, test) would share the same vocab. And vocabs of different fields are isolated from each other.2. Add each entry to a DataStoragedatastorage=preptext.DataStorage()forsrc,trgindataset:entry=preptext.Entry([src,trg],fields)datastorage.add_entry(entry)datasethere is your own translation dataset.Or you can collect all entries into alist, then convert the list into a DataStorage.data=[]forsrc,trgindataset:entry=preptext.Entry([src,trg],fields)data.append(entry)datastorage=preptext.DataStorage(data)3. Build VocabularyCreate a vocab for each field:fields.src.build_vocab(datastorage.src)fields.trg.build_vocab(datastorage.trg)Create a shared vocab:fields.src.build_vocab(datastorage.src,datastorage.trg)fields.trg.vocab=fields.src.vocabSpecify pretrained word vectors:vec=preptext.Vectors('file/path/to/the/word2vec.txt')# need to be txt formatfields.src.build_vocab(datastorage.src,vetors=vec)fields.trg.build_vocab(datastorage.trg,vectors=vec)NOTE:If vectors are specified, this package will cache the vector file into a folder named.vector_cacheautomatically.Now you're almost ready with your data. The following methods is ready to be used.# save the prepared data into a binary filedatastorage.dump('data.pkl')# load from a binary filedatastorage=preptext.DataStorage.load('data.pkl')# get the i-th data of text formati=0datastorage.get_entry(i)# get the i-th data of numpy array formatdatastorage.get_array(i)# get vocab matrixsrc_matrix=datastorage.fields.src.vocab.vectorstrg_matrix=datastorage.fields.trg.vocab.vectors# convert into pytorch dataloaderdl=preptext.converter.to_dataloader(datastorage,bach_size=50,shuffle=True,num_workers=4)# convert into bucketdataloader (minimized padding in each minibatch)bucketdl=preptext.converter.to_bucketdataloader(datastorage,key=lambdax:len(x.src),batch_size=50,shuffle=True,num_workers=4)# convert into distributed dataloaderdistributeddl=preptext.converter.to_distributeddataloader(datastorage,1,# world_size1,# rankbatch_size=50,num_workers=4)Enjoy!
Package: prepull-singularity
Application to populate the singularity cache. Useful for HPC clusters where
multiple nodes may pull to the same cache at the same time if images are not
yet present.Installationprepull-singularity is tested on python 3.5, 3.6, 3.7 and 3.8. It can be
installed with:pip installprepull-singularityHow to runMake a YAML file listing the images to be pulled.alpine:"alpine:latest"debian:"debian:stretch-slim"Run the programprepull-singularityimages.ymlUsageusage: prepull-singularity [-h] [-a MAX_ATTEMPTS] [-p PREFIX]
                           [--stop-on-failure] [--show-output-on-failure]
                           [--show-output-on-success]
                           [--singularity-exe SINGULARITY_EXE] [--use-digest]
                           input

Pull images from listed in a YAML file, so they get cached and can be run
without pulling later.

positional arguments:
  input                 A YAML file listing the images to be pulled, either as
                        a map or list.

optional arguments:
  -h, --help            show this help message and exit
  -a MAX_ATTEMPTS, --max-attempts MAX_ATTEMPTS
                        Maximum number of times to attempt pulling each image;
                        defaults to 3.
  -p PREFIX, --prefix PREFIX
                        Prefix for the image url; defaults to 'docker://'.
  --stop-on-failure     Stop when pulling an image fails; by default all
                        images will be attempted to be pull even if one fails.
  --show-output-on-failure
                        Print the stderr and stdout when pulling an image
                        fails.
  --show-output-on-success
                        Print the stderr and stdout when pulling an image
                        succeeds.
  --singularity-exe SINGULARITY_EXE
                        The command for running singularity; defaults to
                        'singularity'
  --use-digest          Retrieve the image digestes from dockerhub or quay.io
                        and use those (instead of the tags) to pull the
                        images. Only usable with docker images.
Package: prepup
prepupPrepup is a free open-source package that lets you inspect, explore, visualize, and perform pre-processing tasks on datasets in your windows/macOS terminal.InstallationPrepup can be installed using the Pip package manager.!pip install prepupMotivationDeveloping an efficient and user-friendly command line tool for data pre-processing to handle various tasks such as missing data, data formatting, and cleaning, with a simple interface and scalability for large datasets.File Format SupportedCSVEXCELPARQUETWhy you should use Prepup?It's Super fastPrepup is built on Polars which is alternative to pandas and helps you load and manipulate the DataFrames faster.AnalyticalPrepup handles tasks ranging from shape of data to the Standardizing the feature before training the model. It does it right on the terminal.CompatiblePrepup supports CSV, EXCEL and PARQUET formats making it compatible to go with different file formats.Non DestructivePrepup doesn't alters your raw data, It saves pre-processed data only when user specifies the path.Lives in your TerminalPrepup is terminal based and has specific entry points designed for using it instantly.Command Line Arguments available in PREPUPprepup "File name or File path" -inspectinspect flag takes the dataframe and returns the Features available, Features datatype and missing values present in the Dataset.prepup "File name or File path" -exploreexplore flag takes the dataframe and returns the Features available, Features datatype, Correlation between features, Detects Outliers, Checks Normal Distribution, Checks Skewness, Checks Kurtosis and also allows the option to check if the dataset is Imbalanced.prepup "File name or File path" -visualizevisualize flag plots the feature distribution directly on the terminal.prepup "File name or File path" -imputeThere are 8 different strategies available to impute missing data using PrepupOption 1 - Drops the Missing DataOption 2 - Impute Missing values with a Specific valueOption 3 - Impute Missing values with Mean.Option 4 - Impute Missing values with Median.Option 5 - Impute Missing value based on the distribution of existing columns.Option 6 - Impute Missing values based on Forward Fill Strategy where missing values are imputed based on the previous data points.Option 7 - Impute Missing values based on Backward Strategy where missing values are imputed based on the next data points.Option 8 - Impute missing values based on K-Nearest Neighbors.prepup "File name or File path" -standardizeStandardize allows you to standardize the dataset using two different methods:Robust ScalerStandard ScalerRobust Scaler is recommended if there are outliers present and you feel they can have influence on the Machine Learning model.Standard Scaler is go to function if you want to standardize the dataset before training the model on it.LicenseFree software: MIT licensePackage LinkGithub:https://github.com/sudhanshumukherjeexx/prepupDocumentation:https://sudhanshumukherjeexx.github.io/prepup
Package: prepush
pre-pushcheck what CI workflows are gonna be triggered after your pushHow it worksbasically we are taking every modified file path, and check if it matches to any of the path-strings in the current git repository workflowsworkflows locationyaml files in.github/workflows/directorychanged fileschanged files are the one, which are added to the latest commit from the previous commitusagerun theprepushcommand, from anywhere in the git repository
Package: prequel
Preql (pronounced: Prequel) is an interpreted relational query language.It is designed for use by data engineers, analyists and data scientists.Compiles to SQL at runtime. It has the performance and abilities of SQL, and much more.Support for Postgres, MySQL and Sqlite. (more planned!)Escape hatch to SQL, for all those databse-specific features we didn't think to includeProgrammer-friendly syntax and semantics, with gradual type-checking, inspired by Typescript and PythonInterface through Python, HTTP or a terminal environment with autocompletionNote: Preql is still work in progress, and isn't ready for production use, or any serious use yetDocumentationRead hereGet startedSimply install via pip:pipinstall-UprqlThen just run the interpeter:preqlRequires Python 3.8+Read moreQuick Example// Sum up all the squares of an aggregated list of numbers// Grouped by whether they are odd or evenfuncsqrsum(x)=sum(x*x)funcis_even(x)=x%2==0print[1..100]{is_even(item)=>sqrsum(item)}// Result is:┏━━━━━━━━━┳━━━━━━━━┓┃is_even┃sqrsum┃┡━━━━━━━━━╇━━━━━━━━┩│0│166650││1│161700│└─────────┴────────┘In the background, this was run by executing the following SQL code (reformatted):WITHrange1AS(SELECT1ASitemUNIONALLSELECTitem+1FROMrange1WHEREitem+1<100),subq_3(is_even,sqrsum)AS(SELECT((item%2)=0)ASis_even,SUM(item*item)ASsqrsumFROMrange1GROUPBY1)SELECT*FROMsubq_3LicensePreql uses an “Interface-Protection Clause” on top of the MIT license.See:LICENSEIn simple words, it can be used for any commercial or non-commercial purpose, as long as your product doesn't base its value on exposing the Preql language itself to your users.If you want to add the Preql language interface as a user-facing part of your commercial product, contact us for a commercial license.
Package: prereceivecli
A cli that implements a gitlab git server side pre-receive hook that gets driven from dynamodb and reports to slack offending pushes.Documentation:https://prereceivecli.readthedocs.org/en/latestDevelopment WorkflowThe workflow supports the following stepslinttestbuilddocumentuploadgraphThese actions are supported out of the box by the corresponding scripts under _CI/scripts directory with sane defaults based on best practices.
Sourcing setup_aliases.ps1 for windows powershell or setup_aliases.sh in bash on Mac or Linux will provide with handy aliases for the shell of all those commands prepended with an underscore.The bootstrap script creates a .venv directory inside the project directory hosting the virtual environment. It uses pipenv for that.
It is called by all other scripts before they do anything. So one could simple start by calling _lint and that would set up everything before it tried to actually lint the projectOnce the code is ready to be delivered the _tag script should be called accepting one of three arguments, patch, minor, major following the semantic versioning scheme.
So for the initial delivery one would call$ _tag –minorwhich would bump the version of the project to 0.1.0 tag it in git and do a push and also ask for the change and automagically update HISTORY.rst with the version and the change provided.So the full workflow after git is initialized is:repeat as necessary (of course it could be test - code - lint :) )codelinttestcommit and pushdevelop more through the code-lint-test cycletag (with the appropriate argument)buildupload (if you want to host your package in pypi)document (of course this could be run at any point)Important InformationThis template is based on pipenv. In order to be compatible with requirements.txt so the actual created package can be used by any part of the existing python ecosystem some hacks were needed.
So when building a package out of thisdo notsimple call$ python setup.py sdist bdist_eggas this will produce an unusable artifact with files missing.Instead use the provided build and upload scripts that create all the necessary files in the artifact.Project FeaturesCan protect directories and files from tampering by checking hash entries for them in a dynamodb
Please refer to USAGE.rst for setup details.History0.0.1 (26-02-2019)First code creation0.1.0 (26-02-2019)First public release0.1.1 (29-09-2020)Updated quaratine path logic0.1.2 (02-10-2020)Fix for get_project method0.1.3 (02-10-2020)Fixed project_group name logic0.2.0 (16-10-2020)Implemented web identity authentication.0.2.1 (16-10-2020)Bumped dependencies1.0.0 (30-12-2020)Replace hyphens with underscores in all project names for dynamo lookup.1.0.1 (07-01-2021)Made error message friendlier.1.0.2 (09-03-2021)Explicitly targeting the copy git repo for the latest gitlab.1.1.0 (16-03-2021)Implemented archive instead of checkout as checkout corrupted repositories in Gitlab after 13.8 and the exposure of GIT_DIR environment variable.1.1.1 (22-03-2021)Handling special 00000 ref case by not handling it.1.1.2 (26-04-2021)Bumped dependencies.1.1.3 (08-06-2021)Bumped dependencies.