Package: nlpytools
[nlptools] Python NLP ToolsA straightforward Natural Language Processing ToolboxNLP Tools is a set of tools written in python that covers the most common NLP tasks with an easy and clear to understand style of code.It is being developed together with a Series of Articles about NLP by the main author in Medium. You can find the articles attfduque.medium.comInstallationInstalling with pippipinstallnlpytoolsUsage exampleTokenizationUsing the tokenizer:fromnlptools.core.structuresimporttokenizetokenize("This is a sentence")[<SOS>,this,is,a,sentence,<EOS>]Using sentence/document format:fromnlptools.core.structuresimportDocumentdoc=Document("This is a sentence. This is another sentence.")forsentenceindoc:print(sentence,sentence.tokens)Thisisasentence.[<SOS>,This,is,a,sentence,.,<EOS>]Thisisanothersentence.[<SOS>,This,is,another,sentence,.,<EOS>]NormalizationThese are the currently available normalization steps:pre_tokenization_functions={'simplify_punctuation':simplify_punctuation,'normalize_whitespace':normalize_whitespace}post_tokenization_functions={'normalize_contractions':normalize_contractions,'spell_correction':spell_correction,'remove_stopwords':remove_stopwords}Usage:fromnlptools.preprocessing.normalizationimportNormalizernormalizer=Normalizer(pre_tokenization_steps=['simplify_punctuation','normalize_whitespace'],post_tokenization_steps=['normalize_contractions','spell_correction'])norm.normalize_string("This is a nnormalized sentence!!!!         Yeah,,!!")# one can also use normalize_document'This is a normalized sentence! Yeah,!'Stemming:fromnlptools.preprocessing.stemmingimportPorterStemmerfromnlptools.core.structuresimporttokenizestemmer=PorterStemmer()tokens=tokenize("The words in this sentence will be stemmed.")stemmed_tokens=[stemmer.stem(token)fortokenintokens]['<sos>','the','word','in','thi','sent','will','be','stem','.','<eos>']Lemmatizing and TaggingFirst: taggingfromnlptools.preprocessing.taggingimportMLTaggertagger=MLTagger()tag_pairs=tagger.tag("Tag this sentence")fortagintag_pairs:print(tag,tag.PoS)<SOS>NoneTagNNPthisDTsentenceNN<EOS>NoneEvery token carries its own Part of Speech in the PoS attribute after the tagging.Then, after tagging, we can do Lemmatizationfromnlptools.preprocessing.taggingimportMLTaggertagger=MLTagger(force_ud=True)# Force UD format to use compatible tagstag_pairs=tagger.tag("The cars are running")lemmatized_words=[lemmatizer.lemmatize(word,word.PoS)forwordintag_pairs.tokens]print(" ".join(lemmatized_words[1:-1]))thecararerunFeaturizationfromnlptools.preprocessing.featurizationimportTfidftfidf=Tfidf()tfidf.fit(["The first sentence","The second sentence","The third sentence","First, second, third."])tfidf.transform(["The first sentence","The second sentence","The third sentence","First, second, third."])#or just go with fit_transformmatrix([[0.30543024,0.,0.,0.,0.,0.07438118,0.,0.07438118],[0.,0.30543024,0.,0.,0.,For more examples and usage, please refer to themedium series.Release History0.1.0Pypi releaseMetaTiago Duque –medium websiteDistributed under the MIT license. SeeLICENSEfor more information.Check me at githubCheck me at LinkedinContributingFork it (https://github.com/yourname/yourproject/fork)Create your feature branch (git checkout -b feature/fooBar)Write understandable code!!!Commit your changes (git commit -am 'Add some fooBar')Push to the branch (git push origin feature/fooBar)Create a new Pull Request