Package: cbc-nlp
cbc-nlp : The "consileon NLP framework"InstallationInstall via:py -m pip install --index-url https://test.pypi.org/simple/ --no-deps cbc-nlpor using therequirements.txtInstall the relevantspaCymodel through$ python -m spacy download [model]. For further details, see thespaCy WebsiteWhy Consileon NLP Framework?NLP models are developed based on text sources which contain (long) sequences of texts. A major part of the development is the pre-processing of input data. Most effort and time is spent on transforming text into other objects (lists of tokens) in order to be handled by NLP algorithms. This is where Consileon’s NLP Framework comes into play.Consileon NLP Framework contains packages that simplify the development of NLP models through modularization and encapsulation of frequent pre-processing tasks. In that way, you avoid repeating yourself or ending up with a bulk of unstructured sample code that you might not understand or be able to explain later on. Focus on your concept and leave the implementation on us.Features:Consileon NLP Framework offers all preprocessing tasks you need to develop your own NLP Model:Splittexts into smaller chunks (sentences, paragraphs)Split chunks of text intotokens(e.g. single words)Bring tokens into a canonical form (lower-casing)Filterout unwanted tokens andremove stop words."Lemmatization":  map words to their base/dictionary form (imported also for many non-english languages)Perform(other kinds of)mappingsto tokensRemove "garbage", i.e. artifacts which are contained in the source but don’t add meaning to the use case at hand (e.g. remove tables of numbers from texts when spoken language is required)Appendtags to tokens (e.g. specify the source or some semantic information)Choose subsetsof the input sequence for development (or other) reasonsMergeseveral data sources.andmany more.All these transformation steps can be pipelined in few coding lines and fed into NLP-algorithms to generate your NLP model.Getting started:The following tutorial will walk you through developing your own NLP-Model using Consileon’s NLP Framework:Seegetting_startet.ipynbLicensecbc-nlpis licensed under Apache 2.0 as described in theLICENSEfile.Developer NotesSet-upCreate a virtual environmentpy -3 -m venv .venv
.venv\scripts\activateNow install the package i) as an editible install (so code changes come into effect without a re-install) and ii) with the dev option (to have access to dev requirements such aspytest)python -m pip install -e .[dev]Distribution/ VersioningIf necessary, update the version number in thepyproject.toml.Next, update the software and build package indist\folderpipinstall--upgradebuild
python-mbuildFinally, upload to the distribution archive usingtwine. Note, for experimental changes you can upload totestPyPIfirst, before uploading toPyPI.pipinstall--upgradetwine
python-mtwineupload--repositorytestpypidist/*When asked, set username to "__token__" and your password to the respective token.If this doesn't work, add token directly into CLI commandpython -m twine upload --repository testpypi dist/* -u __token__ -p YOUR_RESPECTIVE_TOKENrequirements.txt fileFor development purposes, there also exists a set ofrequirements.txtfiles, where thedev-requirements.txtfile again includes additional packages such aspytest.Generally, therequirements.txtare maintained and updated viapip-compileusing the following commandpip-compile--no-annotate--output-file=requirements.txtpyproject.tomlTo update thedev-requirements.txt, usepip-compile--no-annotate--extradev--output-file=dev-requirements.txtpyproject.toml