Package: nlpeasy
NLPeasyBuild NLP pipelines the easy wayDisclaimer:This is in Alpha stage, lot of things can go wrong.
It could possibly change your Elasticsearch Data the API is not fixed yet
and even the name NLPeasy might change.Free software: Apache Software License 2.0UsageFor this example to completely work you need to have Python at least in Version 3.6 installed.
Also you need to have install and start eitherDockerhttps://www.docker.com/get-started, direct download links forMac (DMG)andWindows (exe).ElasticsearchandKibana:https://www.elastic.co/downloads/orhttps://www.elastic.co/downloads/elasticsearch-oss(pure Apache licensed version)Then on the terminal issue:python-mvenvvenvsourcevenv/bin/activate
pipinstallnlpeasyscikit-learn
python-mspacydownloaden_core_web_mdThe packagescikit-learnis just used in this example to get the newsgroups data and preprocess it.
The last command downloads a spacy model for the english language -
for the following you need to have at least it'smd(=medium) version which has wordvectors.importpandasaspdimportnlpeasyasnefromsklearn.datasetsimportfetch_20newsgroups# connect to running elastic or else start an Open Source stack on your dockerelk=ne.connect_elastic(docker_prefix='nlp',elk_version='7.4.0',mount_volume_prefix=None)# If it is started on docker it will on the first time pull the images (1.3GB)!# Setting mountVolumePrefix="./elastic-data/" would keep the data of elastic in your# filesystems and then the data survives container restarts# read data as Pandas data framenews_raw=fetch_20newsgroups(remove=('headers','footers','quotes'))news_groups=[news_raw['target_names'][i]foriinnews_raw['target']]news=pd.DataFrame({'newsgroup':news_groups,'message':news_raw['data']})# setup NLPeasy pipeline with name for the elastic index and set the text columnpipeline=ne.Pipeline(index='news',text_cols=['message'],tag_cols=['newsgroup'],elk=elk)pipeline+=ne.VaderSentiment('message','sentiment')pipeline+=ne.SpacyEnrichment(nlp='en_core_web_md',cols=['message'],vec=True)# do the pipeline - just for first 100, the whole thing would take 10 minutesnews_enriched=pipeline.process(news.head(10000000),write_elastic=True)# Create Kibana Dashboard of all the columnspipeline.create_kibana_dashboard()# open Kibana in webbrowserelk.show_kibana()Let's have some fun outside of Elastic/Kibana - but this needspip install matplotlibimportnumpyasnpfromscipy.cluster.hierarchyimportdendrogram,linkageimportmatplotlib.pyplotaspltgrouped=news_enriched.loc[~news_enriched.message_vec.isna()].groupby('newsgroup')group_vec=grouped.apply(lambdaz:np.stack(z.message_vec.values).mean(axis=0))clust=linkage(np.stack(group_vec),'ward')# calculate full dendrogramplt.figure(figsize=(10,10))plt.title('Hierarchical Clustering Dendrogram Newsgroups')plt.xlabel('sample index')plt.ylabel('distance')dendrogram(clust,leaf_rotation=0.,# rotates the x axis labelsleaf_font_size=8.,# font size for the x axis labelslabels=group_vec.index,orientation='left')plt.show()InstallationPrerequisites:Python 3 (we use Python 3.7)Elastic: Several possibilitiesHave Docker installed - needs to have the docker package installed (see below).Install and start Elasticsearch and Kibana:https://www.elastic.co/downloads/orhttps://www.elastic.co/downloads/elasticsearch-oss(pure Apache licensed version)Use any running Elasticsearch and Kibana (on premise or cloud)...Pretrained Models: See below for Spacy Language Models and WordVectorsIt is recommended to use a virtual environment:cd$PROJECT_DIRpython-mvenvvenvsourcevenv/bin/activateThe source statement has to be repeated whenever you open a new terminal.Then installpipinstallnlpeasyOr the development version from GitHub:pipinstall--upgradegit+https://github.com/d-one/nlpeasyIf you want to use spaCy language models download them (90-200 MB), e.g.python-mspacydownloaden_core_web_md# and/orpython-mspacydownloadde_core_news_mdIf you want to use pretrainedFastText-Wordvectors(each ~7GB):curl-Ohttps://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.en.zip
curl-Ohttps://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.de.zipIf you want to use Jupyter, install it to the virtual environment:pipinstalljupyterlabDevelopmentTo install this module in Dev-mode, i.e. change files and reload module:gitclonehttps://github.com/d-one/nlpeasycdnlpeasyIt is recommended to use a virtual environment:python-mvenvvenvsourcevenv/bin/activateInstall the version in edit mode:pipinstall-e.In Jupyter you can have reloaded code when you change the files as in:%load_extautoreload%autoreload2FeaturesPandas based pipelineSupport for any extensions - now includes some for Regex, spaCy, VaderSentimentWrite results to ElasticSearchAutomatic Kibana dashboard generationHave Elastic started in Docker if it is not installed locally or remotelyApache License 2.0CreditsThis package was created withCookiecutterand the [audreyr/cookiecutter-pypackage]https://github.com/audreyr/cookiecutter-pypackageproject template.