Saravanan666/pypi-org huggingface dataset:
Package: nlp.lmd
lmdCode for paper titled "Language Model Decomposition: Quantifying the Dependency and Correlation of Language Models" (accepted EMNLP 2022). The arxiv version is here:InstallCreate virtual env if neededpython3 -m venv .venv
source .venv/bin/activateInstall from pippip install nlp.lmdInstall from sourcegit clone git@github.com:haozhg/lmd.git
cd lmd
pip install -e .To use lmd cli, runlmd --helporpython -m lmd.cli --help$ lmd --help
usage: Language Model Decomposition [-h] [--target TARGET] [--basis BASIS]
                                    [--tokenizer-name TOKENIZER_NAME]
                                    [--max-seq-length MAX_SEQ_LENGTH]
                                    [--batch-size BATCH_SIZE]
                                    [--dataset-name DATASET_NAME]
                                    [--dataset-config-name DATASET_CONFIG_NAME]
                                    [--val-split-percentage VAL_SPLIT_PERCENTAGE]
                                    [--test-split-percentage TEST_SPLIT_PERCENTAGE]
                                    [--max-train-samples MAX_TRAIN_SAMPLES]
                                    [--max-val-samples MAX_VAL_SAMPLES]
                                    [--max-test-samples MAX_TEST_SAMPLES]
                                    [--preprocessing-num-workers PREPROCESSING_NUM_WORKERS]
                                    [--overwrite_cache OVERWRITE_CACHE]
                                    [--preprocess-dir PREPROCESS_DIR]
                                    [--embedding-dir EMBEDDING_DIR]
                                    [--results-dir RESULTS_DIR]
                                    [--models-dir MODELS_DIR] [--alpha ALPHA]
                                    [--log-level LOG_LEVEL]
                                    [--try-models TRY_MODELS]
                                    [--pre-select-multiplier PRE_SELECT_MULTIPLIER]
                                    [--seed SEED]ResultsTo reproduce the results in Appendix B of the paper, runbash scripts/run.sh. The results are also stored inresults/128k
Package: nlp-model-gen-plugins
# NLP Model admin library pluginsThis project is being developed under supervision of Universidad Nacional de Mar del Plata and Infolab Mar del Plata.### Objetive
This package offers different plugins for using NLP admin library.### Limits### Current status & features### Requirements### Installation### License
MIT LicenseCopyright (c) 2018 The Python Packaging AuthorityPermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.This is a university project. It’s usage is thought for profesionals, no further help or usage guides will be provided.
Package: nlp-models
NLP ModelsA repository for building transformer based nlp modelsInstallationInstall from PyPipip install nlp-modelsInstall from sourcegit clone git@github.com:minggnim/nlp-models.git
pip install -r requirementsLlama2 Quantization model on consumer CPURun Chat applications on CPUStreamlit UIcd apps
streamlit run chat.pyCommand linellm_app chat -s 'hi there'Run Q&A application on CPUSteamlit UIcd apps
streamlit run qa.pyModelsbert_classifierA wrapper package around BERT-based classification modelsTraining exampleInference examplemulti_task_modelAn implementation of multi-tasking model built on encoder modelsZero-shot multi-task modelTraining exampleInference exampleQqrant Vector DBGPT-2Training GPT-2 modelFalcon 7BRunning Falcon 7b modelQuantized Llama2 modelsRun Llama2 chat on CPURun Llama2 QA on a custom pdf document on CPU
Package: nlpo3
nlpO3 Python bindingPython binding for nlpO3, a Thai natural language processing library in Rust.FeaturesThai word tokenizersegment()- use maximal-matching dictionary-based tokenization algorithm and honor Thai Character Cluster boundaries2.5x fasterthan similar pure Python implementation (PyThaiNLP's newmm)load_dict()- load a dictionary from plain text file (one word per line)Dictionary fileFor the interest of library size, nlpO3 does not assume what dictionary the developer would like to use.
It does not come with a dictionary. A dictionary is needed for the dictionary-based word tokenizer.For tokenization dictionary, trywords_th.txfromPyThaiNLP- around 62,000 words (CC0)word break dictionaryfromlibthai- consists of dictionaries in different categories, with make script (LGPL-2.1)Installpipinstallnlpo3UsageLoad filepath/to/dict.fileto memory and assign a namedict_nameto it.
Then tokenize a text with thedict_namedictionary:fromnlpo3importload_dict,segmentload_dict("path/to/dict.file","custom_dict")segment("สวัสดีครับ","dict_name")it will return a list of strings:['สวัสดี','ครับ'](result depends on words included in the dictionary)Use multithread mode, also use thedict_namedictionary:segment("สวัสดีครับ",dict_name="dict_name",parallel=True)Use safe mode to avoid long waiting time in some edge cases
for text with lots of ambiguous word boundaries:segment("สวัสดีครับ",dict_name="dict_name",safe=True)BuildRequirementsRust 2018 EditionPython 3.6 or newerPython Development HeadersUbuntu:sudo apt-get install python3-devmacOS: No action neededPyO3- already included in Cargo.tomlsetuptools-rustStepspython-mpipinstall--upgradebuild
python-mbuildThis should generate a wheel file, indist/directory, which can be installed by pip.IssuesPlease report issues athttps://github.com/PyThaiNLP/nlpo3/issues
Package: nlp-ood
APIIntroductionWe have researched and tested the studied solution. After testing, we built an api that quickly deploys the application on other environments.DependenciesWe use anaconda to create python environment:conda create --name ood_api python=3.8Activate enviromentconda activate ood_apiInstall all required libraries:pip install -r requirements.txtHow to runpython application.pyLicenseFor open source projects, say how it is licensed.
Package: nlppln
nlppln is a python package for creating NLP pipelines usingCommon Workflow Language(CWL).
It provides steps for (generic) NLP functionality, such as tokenization,
lemmatization, and part of speech tagging, and helps users to construct workflows
from these steps.A text processing step consist of a (Python) command line tool and a CWL
specification to use this tool.
Most tools provided by nppln wrap existing NLP functionality.
The command line tools are made withClick, a Python
package for creating command line interfaces.To create a workflow, you have to write a Python script:from nlppln import WorkflowGenerator

with WorkflowGenerator() as wf:
  txt_dir = wf.add_input(txt_dir='Directory')

  frogout = wf.frog_dir(in_dir=txt_dir)
  saf = wf.frog_to_saf(in_files=frogout)
  ner_stats = wf.save_ner_data(in_files=saf)
  new_saf = wf.replace_ner(metadata=ner_stats, in_files=saf)
  txt = wf.saf_to_txt(in_files=new_saf)

  wf.add_outputs(ner_stats=ner_stats, txt=txt)

  wf.save('anonymize.cwl')The resulting workflow can be run using a CWL runner, such ascwltool:cwltoolanonymize.cwl--txt_dir/path/to/directory/with/txt/files/For creating new (e.g., project specific) NLP functionality, you can usenlppln-gento generate boilerplate (i.e., empty) command line tools and CWL specifications.The full documentation can be found onRead the Docs.InstallationInstall nlppln using pip:pipinstallnlpplnPlease check theinstallation guidelinesfor additional required software.LicenseCopyright (c) 2016-2018, Netherlands eScience Center, University of TwenteLicensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Package: nlppreprocess
NLPPREPROCESSNLPPREPROCESS is a preprocessing package for NLP task. The main objective of the package is to reduce time consumed for preprocessing by using ready made functions.RequirementsPython 3.4 or higherInstallationUsing PIP via PyPI$ pip install nlppreprocessManually via GIT$ git clone git://github.com/gaganmanku96/nlppreprocess
$ cd nlppreprocess
$ python setup.py installFunctionalitiesReplaces wordsRemove stopwordsRemove numbersRemove HTML tagsRemove punctationsLemmatize words either by Wordnet or SnowballUsage>>> from nlpuitls import NLP
>>> obj = NLP()Parameters>>> obj = NLP(
       replace_words=True,
       remove_stopwords=True,
       remove_numbers=True,
       remove_HTML_tags=True,
       remove_punctation=True,
       lemmatize=False,
       lemmatize_method='wordnet'
      )Using with Pandas Library>>> dataFrame['text'] = dataFrame['text].apply(obj.process)Using with plain textx>>> print(obj.process("Pass a text here"))Add more stopwords>>> obj = NLP()
>>> obj.add_stopword(['this', 'and this'])Add more replace words>>> obj = NLP()
>>> obj.add_replacement([this="by this", this="by this"])
Package: nlprep
Featurehandle over 100 datasetgenerate statistic report about processed datasetsupport many pre-processing waysProvide a panel for entering your parameters at runtimeeasy to adapt your own dataset and pre-processing utilityOnline Explorerhttps://voidful.github.io/NLPrep-Datasets/DocumentationLearn more from thedocs.Quick StartInstalling via pippipinstallnlprepget one of the datasetnlprep--datasetclas_udicstm--outdirsentimentYou can also try nlprep in Google Colab:Overview$ nlprep
arguments:
  --dataset     which dataset to use     
  --outdir      processed result output directory       
  
optional arguments:
  -h, --help    show this help message and exit
  --util        data preprocessing utility, multiple utility are supported 
  --cachedir    dir for caching raw dataset
  --infile      local dataset path
  --report      generate a html statistics reportContributingThanks for your interest.There are many ways to contribute to this project. Get startedhere.LicenseLicenseIcons referenceIcons modify fromDarius Danfromwww.flaticon.comIcons modify fromFreepikfromwww.flaticon.com
Package: nlprov
NLProv: Natural Language Processing ToolNLProv is a Python library developed by Johnson & Johnson's Advanced Analytics
team that combines existing libraries for common Natural Language Processing tasks.
It combines several existing open-source libraries such as pandas, spaCy, and
scikit-learn to make a pipeline that is ready to process text data. There are
many user defined parameters depending on your type of project such as the
ability to choose stemming or lemmatization. Or, you might want to define
explicitly what to substitute with NaN text fields. Overall, it is a way to get
you started in your NLP task, no matter what you need.A tutorial on how to use this package can be foundhere.Installation InstructionsUsing pip with Python version 3.7 or higher:pipinstallnlprovFor more information on installing packages using pip, clickhere.ContributingTo help develop this package, you'll need to install a conda virtual
environment defined by our dev_environment.yml file using the below command.condaenvcreate-fdev_environment.ymlThen, just activate the environment when attempting to develop or run tests
using the below command.condaactivatenlp_envWhen you're all done developing or testing, just deactivate the environment
with the below command.condadeactivateDocker ConfigurationThis codebase is dockerized to build, run all of the unit tests usingpytest, and perform pip packaging.In order to run the docker container, ensure you haveDockerinstalled and running on your local machine.To start the docker container locally, simply navigate to the root of the
project directory and type:docker-composeup--buildNote:docker-composeis included in the Docker desktop installation link
above for MacOS and Windows based systems. If you have issues executingdocker-compose,Navigate Hereto ensure docker-compose is supported on your system.A Notey-er note: You can usedocker-compose up --buildduring development
to quickly run the tests after code changes without setting up/running a local
conda environment.GitHub Action CI ConfigurationEvery commit to this repository will trigger a build in GitHub Actions following the
.github/workflows/pythonapp.yml located in the root of this project.GitHub Actions is used to build and lint the NLProv package, run the tests, and perform pip packaging.If the environment name or version changes, the pythonapp.yml file will need to be updated to
follow the new pattern.Our WorkflowOur Methods and ToolsStyle Guide -PEP8 / pycodestyleGit Strategy -Git FlowUpcoming FeaturesHere is a roadmap of features to be implemented in this package. If you have any
ideas for additional features, please let us know!PreprocessingAbility to use custom stop wordsIncorporation of bi-gramsAbility for user to chose which langauge detection package to useVectorizationspaCy pre-trained modelsspaCy custom modelsSimilarity MetricsAdditional pairwise distancesLevenshtein DistanceWord Mover's DistanceVisualizationsTF-IDFJaccard
Package: nlp-rules
A sample Python projectA sample project that exists as an aid to thePython Packaging User
Guide'sTutorial on Packaging and Distributing
Projects.This project does not aim to cover best practices for Python project
development as a whole. For example, it does not provide guidance or tool
recommendations for version control, documentation, or testing.The source for this project is available here.Most of the configuration for a Python project is done in thesetup.pyfile,
an example of which is included in this project. You should edit this file
accordingly to adapt this sample project to your needs.This is the README file for the project.The file should use UTF-8 encoding and can be written usingreStructuredTextormarkdownwith the appropriatekey set. It will be used to generate the project webpage on PyPI and will be
displayed as the project homepage on common code-hosting services, and should be
written for that purpose.Typical contents for this file would include an overview of the project, basic
usage examples, etc. Generally, including the project changelog in here is not a
good idea, although a simple “What's New” section for the most recent version
may be appropriate.
Package: nlpsandbox-client
NLP Sandbox ClientOverviewNLP Sandbox Client Library for Python.  This repository consists of 2 packagesnlpsandbox: Python SDK client that wraps all the NLP Sandbox APIsnlpsandboxclient- Convenience functions that infer user behavior of the SDK client.SpecificationAPI version: 1.2.0Docker image:nlpsandbox/cliDocumentationnlpsandboxclientfunctionality is documentedhereInstallationpip install nlpsandbox-client
nlp-cli --versionContributing and developingThinking about contributing tonlpsandbox-client? Get started by reading ourContributor Guide.
Package: nlp_segmentation
分词工具python版(有node版)安装pip install nlp_segmentationcommand-line搜狗分词$sougou_fenci武汉市长江大桥武汉市n长江n大桥n百度分词$baidu_fenci武汉市长江大桥武汉地名长江大桥地名sougou 分词 codeimportnlp_segmentationresp=nlp_segmentation.sougou("武汉市长江大桥")forresult_iteminresp.result:print(row_format.format(result_item[0],result_item[1]))# 结 果# 武汉市         n# 长江         n# 大桥         nbaidu 分词 codeimportnlp_segmentationresp=nlp_segmentation.baidu("武汉市长江大桥")forresult_iteminresp.result:print(row_format.format(result_item[0],result_item[1]))# 结 果# 武汉市         地名# 长江大桥         地名
Package: nlpsig
nlpsigNLPSig (nlpsig) is a Python package for constructing streams/paths of
embeddings obtained from transformers. The key contributions are:A simple API for taking streams of textual data and constructing streams of
embeddings from transformersThenlpsig.SentenceEncoderandnlpsig.TextEncoderclasses allow you to pass in a corpus of text data (in a variety of formats)
and obtain corresponding embeddings using thesentence-transformerand HuggingFacetransformerslibraries,
respectively.Thenlpsig.PrepareDataallows you to easily construct paths/streams of embeddings which can be used
for several downstream tasks.Simple API for performing dimensionality reduction withnlpsig.DimReduceon the embeddings obtained from transformers by some simple wrappers over
popular dimensionality reduction algorithms such as PCA, UMAP, t-SNE, etc.This is particularly useful if we wish to use path signatures in any
downstream model since the dimensionality of the embeddings obtained from
transformers is usually very high.We present someSignature Networkmodels for longitudinal NLP tasks in thesig-networkslibrary which
uses these paths constructed in this library as inputs to neural networks
which utilise path signature methodology.We also havesimple classesfor constructing train/test splits of the data and for K-fold cross-validation
in which are general and are applied to examples in the Signature Networks in
thesig-networkslibrary.InstallationNLPSig is available on PyPI and can be installed withpip:pip install nlpsigContributingTo take advantage ofpre-commit, which will automatically format your code and
run some basic checks before you commit:pip install pre-commit  # or brew install pre-commit on macOS
pre-commit install  # will install a pre-commit hook into the git repoAfter doing this, each time you commit, some linters will be applied to format
the codebase. You can also/alternatively runpre-commit run --all-filesto run
the checks.SeeCONTRIBUTING.mdfor more information on running the test
suite usingnox.
Package: nlpstack
📚 NLPSTACKA Python Library for Natural Language ProcessingNLPSTACK is a Python library designed to make natural language processing (NLP) tasks more accessible and efficient.
With its key features listed below, NLPSTACK aims to become a go-to resource for developers working in NLP:Built on top of PyTorch, NLPSTACK offers ready-to-use modules for a seamless NLP experience.Provides a scikit-learn-compatible interface, ensuring familiar and easy usage.Allows for seamless integration of models provided by the Transformers library, enabling the use of powerful pre-trained models in your NLP workflows.Adopts a dependency injection design pattern, which ensures high extensibility and makes it easy to modify specific processes.🚧Currently in its early development stage, NLPSTACK plans to offer user-friendly interfaces for a wide range of NLP tasks in the future, including text classification, sequence labeling, and language model training.
Package: nlpstreamline
NLP Streamline ToolStreamline basic text & NLP processingHow to UseAfter installing the package, import the functions you need:from nlpstreamline import (
    basic_text_processing,
    nlp_text_processing,
    n_gram_generation
)Use following code to do basic text processing(lowercasing, removing urls, tokenizing, remove stopwords and punctuations), nlp processing (stemming and lemmatizing), and n grams generation.print(basic_text_processing('Good Morning! How are you today?'))

print(nlp_text_processing('Good Morning! How are you today?'))

print(n_gram_generation('Good Morning! How are you today?', 2))
Package: nlp-tarte
TarteA secondary layer for pie for disambiguationWhat it aims to doThis tagger is supposed to come as a secondary layer for lemma that should be disambiguated.Its core object (Tarte) should filter things that need to be disambiguatedIts training capacities should reorganize a training set so that it dispatch training samples across all sets and
it should not care about sample not containing unambiguous tokens.It takes POS, lemma context and form characters into the network to predict the disambiguated form.NotesGiven that not all sentences will have things to disambiguate, pretraining vector might be an important task. It
ispossible with PyTorchto load Gensim data easily. This wouldrequireto generate temps file where lemma AND pos are fed to
fake sentences.
Package: nlptest
NLP Test: Deliver Safe & Effective Language ModelsProject's Website•Key Features•How To Use•Community Support•Contributing•Mission•LicenseProject's WebsiteTake a look at our official page for user documentation and examples:nlptest.orgKey FeaturesGenerate and execute more than 50 distinct types of tests only with 1 line of codeTest all aspects of model quality: robustness, bias, representation, fairness and accuracy.​Automatically augment training data based on test results (for select models)​Support for popular NLP frameworks for NER and Text-Classifcation: Spark NLP, Hugging Face & Transformers.Support for testing LLMS ( OpenAI, Cohere, AI21, Hugging Face Inference API and Azure-OpenAI LLMs) for question answering and summarization task.How To Use# Install nlptest!pipinstallnlptest# Import and create a Harness objectfromnlptestimportHarnessh=Harness(task='ner',model='dslim/bert-base-NER',hub='huggingface')# Generate test cases, run them and view a reporth.generate().run().report()NoteFor more extended examples of usage and documentation, head over tonlptest.orgCommunity SupportSlackFor live discussion with the NLP Test community, join the#nlptestchannelGitHubFor bug reports, feature requests, and contributionsDiscussionsTo engage with other community members, share ideas, and show off how you use NLP Test!MissionWhile there is a lot of talk about the need to train AI models that are safe, robust, and fair - few tools have been made available to data scientists to meet these goals. As a result, the front line of NLP models in production systems reflects a sorry state of affairs.We propose here an early stage open-source community project that aims to fill this gap, and would love for you to join us on this mission. We aim to build on the foundation laid by previous research such asRibeiro et al. (2020),Song et al. (2020),Parrish et al. (2021),van Aken et al. (2021)and many others.John Snow Labshas a full development team allocated to the project and is committed to improving the library for years, as we do with other open-source libraries. Expect frequent releases with new test types, tasks, languages, and platforms to be added regularly. We look forward to working together to make safe, reliable, and responsible NLP an everyday reality.ContributingWe welcome all sorts of contributions:IdeasFeedbackDocumentationBug reportsDevelopment and testingFeel free to clone the repo and submit pull-requests! You can also contribute by simply opening an issue or discussion in this repo.ContributorsWe would like to acknowledge all contributors of this open-source community project.LicenseNLP Test is released under theApache License 2.0, which guarantees commercial use, modification, distribution, patent use, private use and sets limitations on trademark use, liability and warranty.