Package: textpipe
textpipe: clean and extract metadata from texttextpipeis a Python package for converting raw text in to clean, readable text and
extracting metadata from that text. Its functionalities include transforming
raw text into readable text by removing HTML tags and extracting
metadata such as the number of words and named entities from the text.Vision: the zen of textpipeDesigned for use in production pipelines without adult supervision.Rechargeable batteries included: provide sane defaults and clear examples to adapt.A uniform interface with thin wrappers around state-of-the-art NLP packages.As language-agnostic as possible.Bring your own models.FeaturesClean raw text by removingHTMLand other unreadable constructsIdentify the language of textExtract the number of words, number of sentences, named entities from a textCalculate the complexity of a textObtain text metadata by specifying a pipeline containing all desired elementsObtain sentiment (polarity and a subjectivity score)Generates word countsComputes minhash for cheap similarity estimation of documentsInstallationIt is recommended that you install textpipe using a virtual environment.First, create your virtual environment usingvirtualenvorvirtualenvwrapper.Using Venv if your default interpreter is python3.6python3-mvenv.venvUsing virtualenv.virtualenvvenv-ppython3.6Using virtualenvwrappermkvirtualenvtextpipe-ppython3.6Install textpipe using pip.pipinstalltextpipeInstall the required packages usingrequirements.txt.pipinstall-rrequirements.txtA note on spaCy download model requirementWhile the requirements.txt file that comes with the package calls for spaCy's en_core_web_sm model, this can be changed depending on the model and language you require for your intended use. SeespaCy.io's page on their different modelsfor more information.Usage example>>>fromtextpipeimportdoc,pipeline>>>sample_text='Sample text! <!DOCTYPE>'>>>document=doc.Doc(sample_text)>>>print(document.clean)'Sample text!'>>>print(document.language)'en'>>>print(document.nwords)2>>>pipe=pipeline.Pipeline(['CleanText','NWords'])>>>print(pipe(sample_text)){'CleanText':'Sample text!','NWords':3}In order to extend the existing Textpipe operations with your own proprietary operations;test_pipe=pipeline.Pipeline(['CleanText','NWords'])defcustom_op(doc,context=None,settings=None,**kwargs):return1custom_argument={'argument':1}test_pipe.register_operation('CUSTOM_STEP',custom_op)test_pipe.steps.append(('CUSTOM_STEP',custom_argument))ContributingSeeCONTRIBUTINGfor guidelines for contributors.Changes0.12.1Bumps redis, tqdm, pyling0.12.0Bumps versions of many dependencies including textacy. Results for keyterm extraction changed.0.11.9Exposes arbitrary SpaCyentsproperties0.11.8Exposes SpaCy'scatsattribute0.11.7Bumps spaCy and redis versions0.11.6Fixes bug where gensim model is not cached in pipeline0.11.5Raise TextpipeMissingModelException instead of KeyError0.11.4Bumps spaCy and datasketch dependencies0.11.1Replaces codacy with pylint on CIFixes pylint issues0.11.0Adds wrapper around Gensim keyed vectors to construct document embeddings from Redis cache0.9.0Adds functionality to compute document embeddings using a Gensim word2vec model0.8.6Removes non standard utf chars before detecting language0.8.5Bump spaCy to 2.1.30.8.4Fix broken install command0.8.3Fix broken install command0.8.2Fix copy-paste error in word vector aggregation (#118)0.8.1Fixes bugs in several operations that didn't accept kwargs0.8.0Bumps Spacy to 2.10.7.2Pins Spacy and Pattern versions (with pinned lxml)0.7.0change operation's registry from list to dictglobal pipeline data is available across operations via thecontextkwargload custom operations usingregister_operationin pipelinecustom steps (operations) with arguments