Saravanan666/pypi-org huggingface dataset:
Package: nlpcommon
nlpcommonnlpcommon, Python Text Tool. Python3开发。GuideFeatureInstallUsageDatasetContactCiteReferenceFeaturenlpcommonis a python Open Source Toolkit for text classification. The goal is to implement
text analysis algorithm, so as to achieve the use in the production environment.nlpcommonhas the characteristics
of clear algorithm, high performance and customizable corpus.Functions：ClassifierLogisticRegressionRandom ForestDecision TreeK-Nearest NeighboursNaive bayesXgboostSupport Vector Machine(SVM)TextCNNTextRNN_AttFasttextBertClusterMiniBatchKmeansWhile providing rich functions,nlpcommoninternal modules adhere to low coupling, model adherence to inert loading, dictionary publication, and easy to use.InstallRequirements and Installationpip3 install nlpcommonorgit clone https://github.com/shibing624/nlpcommon.git
cd nlpcommon
python3 setup.py installUsagedataStopwrodsexamples/base_demo.py:importsyssys.path.append('..')fromnlpcommonimportstopwordsif__name__=='__main__':print(len(stopwords),stopwords)output:2438{'．','大家','孰知','至于','./','知道','二话没说','一何','从宽','especially'...}ContactIssue(建议)：邮件我：xuming:xuming624@qq.com微信我：加我微信号：xuming624, 进Python-NLP交流群，备注：姓名-公司名-NLPCite如果你在研究中使用了nlpcommon，请按如下格式引用：@software{nlpcommon,
  author ={Xu Ming},
  title ={nlpcommon: A Tool for Text NLP},
  year ={2021},
  url ={https://github.com/shibing624/nlpcommon},}License授权协议为The Apache License 2.0，可免费用做商业用途。请在产品说明中附加nlpcommon的链接和授权协议。Contribute项目代码还很粗糙，如果大家对代码有所改进，欢迎提交回本项目，在提交之前，注意以下两点：在tests添加相应的单元测试使用python setup.py test来运行所有单元测试，确保所有单测都是通过的之后即可提交PR。Referencepytextclassifier
Package: nlp-data
普强内部NLP数据存储分享工具安装pypi安装# 安装基本功能pipinstallnlp-data# 安装全部功能pipinstallnlp-data[all]普强源安装pipinstallnlp-data--upgrade-ihttp://192.168.130.5:5002/simple/--trusted-host192.168.130.5--extra-index-urlhttps://mirrors.aliyun.com/pypi/simple使用Store的使用# Store相当于是S3对象存储的一个Bucket的封装,每个数据类型对应一个Bucketfromnlp_dataimportNLUDocStore# 查看文档NLUDocStore.list()# 获取文档docs=NLUDocStore.pull('xxx')# 推送文档NLUDocStore.push(docs=docs,name='xxx')Doc的使用# Doc是nlp-data的一个存储结构,可以用来存储该格式的数据,以及对数据进行一些操作# DocList是Doc的集合,可以用来存储多个Doc,相当于一个python List,有几本的append,extend等类方法, 但不同的DocList有特定的方法用来处理# 该数据类型# 以NLUDoc为例,该文档里面有domain,slots,intention等字段,可以用来存储NLU的结果fromnlp_dataimportNLUDoc,NLUDocList# 创建一个NLUDocdoc=NLUDoc(text='添加明天上午跟张三开会的提醒')doc.set_domain('schedule_cmn')doc.set_intention('add_schedule')doc.set_slot(text='明天上午',label='date')doc.set_slot(text='跟张三开会',label='title')# 创建一个NLUDocList,并添加docdocs=NLUDocList()docs.append(doc)# 从abnf句式输出文件中批量初始化docs=NLUDocList.from_abnf_output(output_dir='your/dir',domain='schedule_cmn')# 上传到bucketfromnlp_dataimportNLUDocStoreNLUDocStore.push(docs=docs,name='xxx')Augmentor的使用# Augmentor是nlp-data的一个数据增强工具,可以用来对数据进行增强fromnlp_dataimportGPTAugmentor,NLUDocStore,DialogueDocList,DialogueDoc# 创建一个Augmentoraugmentor=GPTAugmentor(api_key='xxx')# 广东话或者四川话增强NLUDocdocs=NLUDocStore.pull('xxx')aug_docs=augmentor.augment_nlu_by_localism(docs,'广东话')# 根据主题和情景生成多轮对话dialogue_docs=augmentor.generate_dialogue_docs(theme='添加日程',situation='用户正在驾驶车辆与车机系统丰田进行语音交互')# 对多轮对话数据增强dialogue_docs=DialogueDocList()dialogue_docs.quick_add(theme='添加日程',situation='用户正在驾驶车辆与车机系统丰田进行交互',conversation=['你好,丰田','在呢,有什么可以帮助你的','我要添加一个明天上午跟张三开会的日程','好的已为您添加成功'])aug_dialogue_docs=augmentor.augment_dialogue(dialogue_docs)S3的使用s3是基础的S3对象存储的封装,可以用来创建bucket,上传下载文件等# 初始化s3=S3Storage()# 列出所有buckets3.list_buckets()# 创建buckets3.create_bucket('test')# 列出bucket下所有文件s3.list_files('test')# 上传文件s3.upload_file(file_path='./test.txt',bucket_name='test')# 下载文件s3.download_file(object_name='./test.txt',bucket_name='test')# 删除文件s3.delete_file(bucket_name='test',file_name='test.txt')# 上传文件夹s3.upload_dir(bucket_name='test',dir='./tests')# 下载文件夹s3.download_dir(bucket_name='test',object_name='./tests',save_dir='./')# 删除文件夹s3.delete_dir(bucket_name='test',dir_name='tests')# 删除buckets3.delete_bucket('test')命令行# 查看帮助nlp-data--help# 下载文件,当xxx为一个s3中的文件夹时,会下载该文件夹下所有文件nlp-datadownloadxxx.xxx--bucketxxx--save_pathxxx# 上传文件, 当xxx为一个文件夹时,会上传该文件夹下所有文件nlp-datauploadxxx--bucketxxx# 删除文件, 当xxx为一个文件夹时,会删除该文件夹下所有文件nlp-datadeletexxx--bucketxxx
Package: nlp-dataset-readers
Dataset Readers for NLPUtility for easily parsing common datasets for NLP.InstallationFromPyPI:pipinstallnlp-dataset-readersFrom source:gitclonehttps://github.com/Riccorl/nlp-dataset-readerscdnlp-dataset-readers
pipinstall-e.Class and DatasetsSemantic Role LabelingDataset readers forCoNLL-2009CoNLL-2012UniteD-SRLWord Sense DisambiguationDataset readers forWSD Evaluation FrameworkXL-WSDWiCDataset readers forXL-WiCMCL-WiCUsageToDo
Package: nlp-datasets
datasetsA dataset utils repository based ontf.data.For tensorflow>=2.0 only!Requirementspython 3.6tensorflow>=2.0Installationpipinstallnlp-datasetsUsageseq2seq modelsThese models has an source sequencexand an target sequencey.fromnlp_datasetsimportSeq2SeqDatasetfromnlp_datasetsimportSpaceTokenizerfromnlp_datasets.utilsimportdata_dir_utilsasutilsfiles=[utils.get_data_file('iwslt15.tst2013.100.envi'),utils.get_data_file('iwslt15.tst2013.100.envi'),]x_tokenizer=SpaceTokenizer()x_tokenizer.build_from_corpus([utils.get_data_file('iwslt15.tst2013.100.en')])y_tokenizer=SpaceTokenizer()y_tokenizer.build_from_corpus([utils.get_data_file('iwslt15.tst2013.100.vi')])config={'train_batch_size':2,'predict_batch_size':2,'eval_batch_size':2,'buffer_size':100}dataset=Seq2SeqDataset(x_tokenizer,y_tokenizer,config)train_dataset=dataset.build_train_dataset(files)print(next(iter(train_dataset)))print('='*120)eval_dataset=dataset.build_eval_dataset(files)print(next(iter(eval_dataset)))print('='*120)predict_files=[utils.get_data_file('iwslt15.tst2013.100.envi')]predict_dataset=dataset.build_predict_dataset(predict_files)print(next(iter(predict_dataset)))print('='*120)sequence match modelsThese models has two sequences as input,xandy, and has an labelz.fromnlp_datasetsimportSeqMatchDatasetfromnlp_datasetsimportSpaceTokenizerfromnlp_datasets.utilsimportdata_dir_utilsasutilsfiles=[utils.get_data_file('dssm.query.doc.label.txt'),utils.get_data_file('dssm.query.doc.label.txt'),]x_tokenizer=SpaceTokenizer()x_tokenizer.build_from_vocab(utils.get_data_file('dssm.vocab.txt'))y_tokenizer=SpaceTokenizer()y_tokenizer.build_from_vocab(utils.get_data_file('dssm.vocab.txt'))config={'train_batch_size':2,'eval_batch_size':2,'predict_batch_size':2,'buffer_size':100,}dataset=SeqMatchDataset(x_tokenizer,y_tokenizer,config)train_dataset=dataset.build_train_dataset(files)print(next(iter(train_dataset)))print('='*120)eval_dataset=dataset.build_eval_dataset(files)print(next(iter(eval_dataset)))print('='*120)predict_files=[utils.get_data_file('dssm.query.doc.label.txt')]predict_dataset=dataset.build_predict_dataset(predict_files)print(next(iter(predict_dataset)))print('='*120)sequence classify modelThese models has a input sequencex, and a output labely.fromnlp_datasetsimportSeqClassifyDatasetfromnlp_datasetsimportSpaceTokenizerfromnlp_datasets.utilsimportdata_dir_utilsasutilsfiles=[utils.get_data_file('classify.seq.label.txt')]x_tokenizer=SpaceTokenizer()x_tokenizer.build_from_corpus([utils.get_data_file('classify.seq.txt')])config={'train_batch_size':2,'eval_batch_size':2,'predict_batch_size':2,'buffer_size':100}dataset=SeqClassifyDataset(x_tokenizer,config)train_dataset=dataset.build_train_dataset(files)print(next(iter(train_dataset)))print('='*120)eval_dataset=dataset.build_eval_dataset(files)print(next(iter(eval_dataset)))print('='*120)predict_files=[utils.get_data_file('classify.seq.txt')]predict_dataset=dataset.build_predict_dataset(predict_files)print(next(iter(predict_dataset)))print('='*120)
Package: nlp-dedup
NLPDedupRemove duplicates and near-duplicates from text corpora, no matter the scale.Developers:Dan Saattrup Nielsen (dan.nielsen@alexandra.dk)Kenneth Enevoldsen (kennethcenevoldsen@gmail.com)InstallationThe package is available on PyPI, so you can install the package using your favourite
package manager. For instance,pip install nlp_deduporpoetry add nlp_dedup.Quick StartIf the corpus is stored ascorpus.txt(bothtxtandjsonlfiles are supported),
the following deduplicates the corpus and stores the deduplicates corpus into the
folderdeduplicated:$ dedup corpus.txt deduplicatedThis defaults to deduplicating based on blocks of 13 consecutive words, where two
documents are considered near-duplicate if they have more than 80% of these blocks in
common. This can all be changed to your specific needs, however. See$ dedup --helpfor more information on all the settings.Deduplication can also be done directly from Python:>>> from nlp_dedup import Deduper
>>> deduper = Deduper()
>>> corpus = ["Test", "Another test", "Test"]
>>> deduper.deduplicate(corpus=corpus)Herecorpusdoes not have to be a list, but can also be an iterable or generator of
strings, if the corpus is too big to be stored in memory. Dictionaries are also
supported instead of strings, in which case thetextentry in the dictionaries will
be used (change this with thetext_columnargument when callingdeduplicate).See more inthe documentation.
Package: nlp-emotion-analysis-core
nlp-emotion-analysis-coreCore NLP library for extracting and analysing emotion and group dynamics in a group setting.Key functionalities1. Emotion extractionEmotion extraction is based the eight basic emotions model proposed by Plutchik[1]. The extraction is based on a vocabulary
curated using wordembedding models. Online conversations related to health were used to train the word embedding model which capture the
contextual relationships among terms. The trained model is then used to query for similar terms. Scripts to train the word embedding model,
query for similar terms and visualizations are included in core/emotions/ folder.
Emotion extraction is enriched with handling negation, intensifiers and is able to calculate the intensity of each emotion.Image retrieved from [2].2. User enagagement measuresUser engagement in a group is measured based on two main components:
i. Emotion enagagement
ii. Participation based on number of posts etc.3. Clinical information extractionClinical information can be extracted using a curated vocabulary of clinical terms. Physcial symptom extractor is added as a demonstration.4. Keyphrase extractionKeyphrase extraction is based on RAKE [3], which automatically extracts significant phrases from text.Examples for emotion extraction, physical symptoms extraction and keyphrase extraction are included in the main.pyReferences:[1] Plutchik, R. (1984). Emotions: A general psychoevolutionary theory. Approaches to emotion, 1984, 197-219.[2] Retrieved from:https://www.researchgate.net/profile/Gerardo_Maupome/publication/258313558/figure/fig1/AS:297350959517696@1447905401049/Plutchiks-wheel-of-emotions-with-basic-emotions-and-derivative-emotions-and-cone-like.png[3] Rose, S., Engel, D., Cramer, N., & Cowley, W. (2010). Automatic keyword extraction from individual documents. Text mining: applications and theory, 1, 1-20.
Package: nlpengine
NLP Enginenlpengineis a collection of Natural Language Processing functions. Its designed to make a NLP engineer's life easier by bundling some of the everyday tasks like data cleaning, feature extraction, text classification, etc. This module is built on top of other popular open source python libraries.InstallationpipinstallnlpengineFeaturesThe key featues in the initial release ofnlpengineincludes:Easy to use text classifiersVectorization of textsActive Learning (in development)Several utility functionsText ClassifierA wrapper on top of Facebook'sFastTexttext classifier to build a text classifier with just few lines of codes.texts=["sample sentence one","just another sentence!","is this a sentence?"]labels=["not question","not question","question"]fromnlpengine.classifiersimportFastTextClassifierclf=FastTextClassifier()model=clf.fit(text,labels)Convert texts to vectorsThis module helps convert a corpus of texts to a vector matrix easily. This extracted matrix could be used for further downstream tasks such as text similiarity, vector decomposition & visualization, etc.fromnlpengine.feature_extractionimportdefget_glove_embeddings_from_sentencestexts=["a great sentence","and a meaningful one"]vectors=get_glove_embeddings_from_sentences(texts,download_model=True)=======History=======0.0.1(2021-05-22)------------------*FirstreleaseonPyPI.
Package: nlpertools
Current version1.0.5备注最新版本在nlpertools里，不在src/nlpertools里IntroductionThis is a package of functions commonly used by NLP/data workers, which can simplify some reading and writing operations
and make the code more readable. It mainly includes two parts: basic reading and writing tools and data processing
functions commonly used in machine learning/deep learning.io# 基本的读写工具。包括了文件读写、文件夹读写、数据读写、词频统计等功能。dir# 文件夹读写file# 文件读写other# 其他ml# 机器学习/深度学习工作中常用的数据处理函数。包括划分十折交叉数据、常见json格式数据读取等功能。openApi# 网络开源接口plugin# 常用插件reminder# 提示other# 其他一些未分组功能Documenthttps://nlpertools.readthedocs.io/en/latest/Development Guideimport 全部放在一个文件里面，所有都从里面找，通过utils可以输出所用到的所有import然后运行脚本，生成打包到pypi里的nlpertools因为里面有很多的引用，需要依赖，不可能全部都安装类似paddle、ltp的import需要判断是否使用才import，因为import的时间太长
例子：git commit guidereadthedoc 检查文档构建状况classSTEM(object):fromltpimportLTPdef__init__(self,IPT_MODEL_PATH):self.ltp=LTP(IPT_MODEL_PATH)Emampleimportnlpertoolsa=nlpertools.readtxt_list_all_strip('res.txt')# 或b=nlpertools.io.file.readtxt_list_all_strip('res.txt')pypi用setup.cfgpyproject.toml文档用mkdocs.yml.readthedocs.yml
Package: nlp-feature-extractors
NLP Feature Extractors SDK for PythonThis package has been tested with Python 3.6, 3.7, 3.8 and 3.9The NLP Feature Extractors project enables the tabular components in the RAI tools to be applied to Natural Language Processing (NLP)
tasks on unstructured text.Please see the main documentation website:https://responsibleaitoolbox.ai/The open source code can be found here:https://github.com/microsoft/responsible-ai-widgets
Package: nlp-fenci
分词工具python版(有node版)安装pip install nlp_fencicommand-line搜狗分词$sougou_fenci武汉市长江大桥
武汉市n
长江n
大桥n百度分词$baidu_fenci武汉市长江大桥
武汉地名
长江大桥地名sougou 分词 codeimportsougou_fenciresp=sougoou_fenci.sougou("武汉市长江大桥")forresult_iteminresp.result:print(row_format.format(result_item[0],result_item[1]))# 结 果# 武汉市         n# 长江         n# 大桥         nbaidu 分词 codeimportsougou_fenciresp=sougoou_fenci.baidu("武汉市长江大桥")forresult_iteminresp.result:print(row_format.format(result_item[0],result_item[1]))# 结 果# 武汉市         地名# 长江大桥         地名
Package: nl-philter
PHILTERCLUT/HALD image generation and processing toolkit.(Based on imlementation of@hommwith some usability and accuracy improvements.)usage: philter [cmd]

A CLI toolkit for creating LUTs

subcommands:

    gen            Generate a fotified identity THALD image (somewhat immune
                   to compression artifacts) which can be used to capture
                   color transformations.

    clut           Generate CLUT image and CUBE map file from a fortified
                   THALD image.

    cube           Generate CUBE file from a NON fortified HALD image.
Package: nlpia2-wikipedia
WikipediaWikipediais a Python library that makes it easy to access and parse
data from Wikipedia.Search Wikipedia, get article summaries, get data like links and images
from a page, and more. Wikipedia wraps theMediaWiki
APIso you can focus on using
Wikipedia data, not getting it.>>> from wikipedia import wikipedia as wiki
>>> print wiki.summary("Wikipedia")
Wikipedia (/ˌwɪkɨˈpiːdiə/ or /ˌwɪkiˈpiːdiə/ WIK-i-PEE-dee-ə) is a collaboratively edited, multilingual, free Internet encyclopedia supported by the non-profit Wikimedia Foundation...

>>> wiki.search("Barack")
['Barak (given name)', 'Barack Obama', ... 'Presidential transition of Barack Obama']
>>> titles = wiki.search("New York")
>>> page = wiki.page(titles[1])
>>> page.title
'New York (State)'
>>> page.url
'http://en.wikipedia.org/wiki/New_York_(state)'
>>> page.content
'New York, sometimes called New York State...'
>>> page.summary
'New York, sometimes called New York State...'
>>> page.links[0]
'10 Hudson Yards'

>>> wiki.set_lang("fr")
>>> wiki.summary("Facebook", sentences=1)
Facebook est un service de réseautage social en ligne sur Internet permettant d'y publier des informations (photographies, liens, textes, etc.) en contrôlant leur visibilité par différentes catégories de personnes.Note: this library was designed for ease of use and simplicity, not for
advanced use. If you plan on doing serious scraping or automated
requests, please usePywikipediabot(or one of the other more advancedPython MediaWiki API
wrappers),
which has a larger API, rate limiting, and other features so we can be
considerate of the MediaWiki infrastructure.InstallationTo install Wikipedia, simply run:$ pip install nlpia2-wikipediaWikipedia is compatible with Python 2.6+ (2.7+ to run unittest discover)
and Python 3.3+.DocumentationRead the docs athttps://wikipedia.readthedocs.org/en/latest/.QuickstartFull APITo run tests, clone therepository on
GitHub, then run:$ pip install -r requirements.txt
$ bash runtests  # will run tests for python and python3
$ python -m unittest discover tests/ '*test.py'  # manual stylein the root project directory.To build the documentation yourself, after installing requirements.txt,
run:$ pip install sphinx
$ cd docs/
$ make htmlLicenseMIT licensed. See theLICENSE
filefor
full details.Creditswiki-apiby
@richardasaurus for inspiration@nmoroze and @themichaelyang for feedback and suggestionsTheWikimedia Foundationfor giving the world free access to data
Package: nlpie
NLPIE: A Lightweight Entity LinkerA lightweight entity linking tool using distilled clinical language models from Huggingface and spaCy/ScispaCy intended to connect clinical notes to UMLS codes. Additional mapping is available for ICD-10 and SNOMED CT entries.Repository Structure.
├── README.md
├── entity_linker.py
├── entity_linker_hf.py
├── mappings-from-UMLS
│ ├── cui_to_icd10_EXACT.json
│ ├── cui_to_icd10_RO.json
│ ├── cui_to_snomed_EXACT.json
│ └── cui_to_snomed_RO.json
├── preprocessor.py
├── query_filter.py
└── requirements.txtInstallationInstall the required packages by running:pipinstallnlpieUsagePreprocessorUse theInputPreprocessorclass inpreprocessor.pyto preprocess an input CSV file:frompreprocessorimportInputPreprocessorinput_path='input.csv'output_path='preprocessed.csv'preprocessor=InputPreprocessor(input_path)preprocessed_data=preprocessor.preprocess_input_file(output_path)Query FilterUse theQueryFilterclass inquery_filter.pyto filter a preprocessed CSV file based on a text query:fromquery_filterimportQueryFilterquery="pneumonia"input_csv_path="preprocessed.csv"query_filter=QueryFilter()query_umls_codes=query_filter.process_query(query)filtered_data,negated_rows=query_filter.filter_rows(query_umls_codes,input_csv_path)query_filter.save_filtered_data_to_csv(filtered_data,negated_rows,query,input_csv_path)
Package: nlpio
NLP.io : the simple Python library to clean text dataHow to use it ?pip install nlpiofromnlpioimportstrip_html_tagsprint(strip_html_tags('<h1>Hello World</h1>'))# "Hello World"Deploy upgrade on PyPiBuildpython setup.py sdist bdist_wheel
twine upload dist/*Testpython -m pip install -e .
python

>>> from nlpio import strip_html_tags
>>> print(strip_html_tags('<h1>Hello World</h1>'))LICENCEThis works is under MIT License.
Package: nlpip
nlpip - A natural language based unix pipeline utilitynlpiptakes a text written in natural language as input, and performs a
specified command also written in natural language, and then outputs the result
in natural language.Here's a demo:InstallationFirst, you need to installnlpipusingpip:pip install nlpipThen, set your OpenAI's API key as an environment variable:export OPENAI_API_KEY=WRITE_YOUR_KEY_HEREHow to usenlpipor justnlpfor short, provides following default commands:keywords: Extract keywords from input textpoem: Write poem using input textsummarize: Summarize input textsentiment: Analyze sentiment of input textFor example, if you want summarizeinput.txtand turn it into a poem, you can
run the following command:cat input.txt | nlp summarize | nlp poemHow to create a new commandTo create a new command, you just need to create a text file containing the
command in natural language. For example, if you want to create a command that
translates input text into French, you can create a file named~/.nlpip/prompts/fr.txtin with:Translate into FrenchThen, you can run the following command:echo Hello | nlp frAnd you will get the following output:BonjourYou may override default commands by creating a file with the same name in~/.nlpip/prompts/.Creditnlpipis inspired bySchillace Laws of Semantic AI.
Package: nlpir-python
NLPIR-python  A python wrapper and toolkit for NLPIRnlpir-python 是一个NLPIR的python调用包nlpir-python is a python wrapper for NLPIR modules.Documentation:https://nlpir-python.rtfd.ioGithub:https://github.com/NLPIR-team/nlpir-pythonSupport:https://github.com/NLPIR-team/nlpir-python/issuesFree software:MIT licenseAbout本模块提供两种调用方式:This package includes two level of method:1. Native call from Dynamic Link Library(DLL) 原生的直接调用DLL的调用方式These methods are native method directory from DLL, you can easily use them if you are familiar with the NLPIR modules.原生方法是直接调用的NLPIR中的api,并进行了部分简化处理,和python化.fromnlpir.nativeimportICTCLAStest_str="法国启蒙思想家孟德斯鸠曾说过：“一切有权力的人都容易滥用"\"权力，这是一条千古不变的经验。有权力的人直到把权力用到"\"极限方可休止。”另一法国启蒙思想家卢梭从社会契约论的观点"\"出发，认为国家权力是公民让渡其全部“自然权利”而获得的，"\"他在其名著《社会契约论》中写道：“任何国家权力无不是以民"\"众的权力（权利）让渡与公众认可作为前提的”。"ictclas=ICTCLAS()ictclas.paragraph_process(test_str,0)2. High-level pythonic method 整合后的更加Python的调用方式However, the native methods are not very friendly to the beginners. These methods provide a wrapper and tools for the
native call, make it easier to use.然而,对于一般用户来说,原生api功能强大但是却不是很友好.这里nlpir-python对原生api就行包装, 并提供了一些工具方法,使其更利于使用.fromnlpirimportictclas,toolstools.update_license()test_str="法国启蒙思想家孟德斯鸠曾说过：“一切有权力的人都容易滥用"\"权力，这是一条千古不变的经验。有权力的人直到把权力用到"\"极限方可休止。”另一法国启蒙思想家卢梭从社会契约论的观点"\"出发，认为国家权力是公民让渡其全部“自然权利”而获得的，"\"他在其名著《社会契约论》中写道：“任何国家权力无不是以民"\"众的权力（权利）让渡与公众认可作为前提的”。"forword,posinictclas.segment(test_str,pos_tagged=True):print(word,pos)NOTE: This module only support python3.6+NOTE: This repo use the git-lfs, please install lfs when pull this repoSupported TableNativeNative DocNative TestHigh-LevelHigh-Level DocHigh-Level TestTutorialICTCLAS✔✔✔✔✔✔✔NewWordFinder✔✔✔✔✔✔KeyExtract✔✔✔✔✔✔Summary✔✔✔✔✔✔SentimentNew✔✔✍SentimentAnalysis✔✔✍✔✔Classify✔✔✍DeepClassify✔✔✍Cluster✔✔✔✔EyeChecker✔✔✔✔✔✔DocCompareDocExtractor✔✔✔✔DocParseriEncoderHTMLParserKeyScanner✔✔✔✔RedupRemoverSpellCheckerSplitSentenceTextSimilarity✔✍✔✔Word2vec
Package: nlp-jp-gears
NLP JP GearsOverview日本語の自然言語処理で頻出の前処理をまとめたものです。pipelineをしいて、複数の処理をまとめることができます。APIpipelineの作成: composer.Composer全角英数字記号を半角に変換: zenhan.ZenToHanConverter半角英数字記号を全角に変換: zenhan.HanToZenConverter括弧とその間のテキストを削除: remover.TextBtwBracketsRemoverRequirementsPython 3.6+Installationpipinstallnlp-jp-gearsExamplefromnlp_jp_gearsimportComposerfromnlp_jp_gearsimport(ZenToHanConverter,TextBtwBracketsRemover)txt_btw_brackets_remover=TextBtwBracketsRemover()zenhan_converter=ZenToHanConverter()composer=Composer(txt_btw_brackets_remover,zenhan_converter)text="Ｐｙｔｈｏｎ（パイソン）で自然言語処理？"out=composer(text)print(out)Then, input text is preprocessed.Pythonで自然言語処理?And you can check what is removed and converted, as follows,print(txt_btw_brackets_remover.removes)print(zenhan_converter.converts)<{[(「『（［〈《〔｛«‹

！＂＃＄％＆＇（）＊＋，－．／：；＜＝＞？＠［＼］＾＿｀｛｜｝～０１２３４５６７８９ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ
Package: nlpkit
nlpkit is a collection of command line utillities and Python libraries
intended to helpful in performing natural language processing tasks.What is an NLP related task, then? In my experience it mostly has to do
with feeding picky command line tools with awkwardly formatted input
and then afterwards try to decipher their output. That’s at least what
nlpkit will help you do.Although some of the modules in nlpkit fit nicely together, it’s not
intended as a framework. Note that dependencies on other Python
modules are not handled in the package and must be installed separately.
Package: nlpknife
项目说明我的瑞士军刀安装与部署在线安装pythonsetup.pyinstall离线安装在项目根目录进行打包sh./shell/package.sh注意，这里的离线构建环境应当与部署总体环境一致，包括python版本，系统等随后将得到的 nlpknife.tar.gz 迁移到部署环境，解压并安装对应依赖项就可以。
解压：tar-zxvfnlpknife.tar.gz安装sh./package/install.sh