Package: excelcy
ExcelCyExcelCy is a NER trainer from XLSX, PDF, DOCX, PPT, PNG or JPG. ExcelCy uses spaCy framework to match Entity with PhraseMatcher or Matcher in regular expression.ExcelCy is convenienceThis is example taken from spaCy documentation,Simple Style Training. It demonstrates how to train NER using spaCy:importspacyimportrandomTRAIN_DATA=[("Uber blew through $1 million a week",{'entities':[(0,4,'ORG')]}),# note: it is required to supply the character position("Google rebrands its business apps",{'entities':[(0,6,"ORG")]})]# note: it is required to supply the character positionnlp=spacy.blank('en')optimizer=nlp.begin_training()foriinrange(20):random.shuffle(TRAIN_DATA)fortext,annotationsinTRAIN_DATA:nlp.update([text],[annotations],sgd=optimizer)nlp.to_disk('test_model')TheTRAIN_DATA, describes sentences and annotated entities to be trained. It is cumbersome to always count the characters. With ExcelCy, (start,end) characters can be omitted.# install excelcy# pip install excelcy# download the en model from spacy# python -m spacy download en"# run this inside python or filefromexcelcyimportExcelCy# Test: John is the CEO of this_is_a_unique_company_nameexcelcy=ExcelCy()# by default it is assume the nlp_base using model en_core_web_sm# excelcy.storage.config = Config(nlp_base='en_core_web_sm')doc=excelcy.nlp('John is the CEO of this_is_a_unique_company_name')# it will show no company entitiesprint([(ent.label_,ent.text)forentindoc.ents])# run this in root of repo or https://github.com/kororo/excelcy/raw/master/tests/data/test_data_01.xlsxexcelcy=ExcelCy.execute(file_path='tests/data/test_data_01.xlsx')# use the nlp object as per spaCy APIdoc=excelcy.nlp('John is the CEO of this_is_a_unique_company_name')# now it recognise the company nameprint([(ent.label_,ent.text)forentindoc.ents])# NOTE: if not showing, remember, it may be required to increase the "train_iteration" or lower the "train_drop", the "config" sheet in ExcelExcelCy is friendlyBy default, ExcelCy training is divided into phases, the example Excel file can be found intests/data/test_data_01.xlsx:1. DiscoveryThe first phase is to collect sentences from data source in sheet "source". The data source can be either:Text: Direct sentence values.Files: PDF, DOCX, PPT, PNG or JPG will be parsed usingtextract.Note: See textract source examples intests/data/test_data_03.xlsxNote: Dependencies "textract" is not included in the ExcelCy, it is required to add manually2. PreparationNext phase, the Gold annotation needs to be defined in sheet "prepare", based on:Current Data Model: Using spaCy API ofnlp(sentence).entsPhrase pattern: Robbie, Uber, Google, AmazonRegex pattern: ^([0-1]?[0-9]|2[0-3]):[0-5][0-9]$All annotations in here are considered as Gold annotations, which described inhere.3. TrainingMain phase of NER training, which described inSimple Style Training.
The data is iterated from sheet "train", check sheet "config" to control the parameters.4. ConsolidationThe last phase, is to test/save the results and repeat the phases if required.ExcelCy is flexibleNeed more specific export and phases? It is possible to control it using phase API.
This is the illustration of the real-world scenario:Train fromtests/data/test_data_05.xlsx# download the dataset$wgethttps://github.com/kororo/excelcy/raw/master/tests/data/test_data_05.xlsx# this will create a directory and file "export/train_05.xlsx"$excelcyexecutetest_data_05.xlsxOpen the result in "export/train_05.xlsx", it shows all identified sentences from source given. However, there is error in the "Himalayas" as identified as "PRODUCT".To fix this, add phrase matcher for "Himalayas = FAC". It is illustrated intests/data/test_data_05a.xlsxTrain again and check the result in "export/train_05a.xlsx"# download the dataset$wgethttps://github.com/kororo/excelcy/raw/master/tests/data/test_data_05a.xlsx# this will create a directory "nlp/data" and file "export/train_05a.xlsx"$excelcyexecutetest_data_05a.xlsxCheck the result that there is backed up nlp data model in "nlp" and the result is corrected in "export/train_05a.xlsx"Keep training the data model, if there is unexpected behaviour, there is backup data model in case needed.ExcelCy is comprehensiveUnder the hood, ExcelCy has strong and well-defined data storage. At any given phase above, the data can be inspected.fromexcelcyimportExcelCyfromexcelcy.storageimportConfig# Test: John is the CEO of this_is_a_unique_company_nameexcelcy=ExcelCy()excelcy.storage.config=Config(nlp_base='en_core_web_sm',train_iteration=10,train_drop=0.2)doc=excelcy.nlp('John is the CEO of this_is_a_unique_company_name')# showing no ORGprint([(ent.label_,ent.text)forentindoc.ents])excelcy.storage.source.add(kind='text',value='John is the CEO of this_is_a_unique_company_name')excelcy.discover()excelcy.storage.prepare.add(kind='phrase',value='this_is_a_unique_company_name',entity='ORG')excelcy.prepare()excelcy.train()doc=excelcy.nlp('John is the CEO of this_is_a_unique_company_name')# ORG now is recognisedprint([(ent.label_,ent.text)forentindoc.ents])# NOTE: if not showing, remember, it may be required to increase the "train_iteration" or lower the "train_drop", the "config" sheet in ExcelFeaturesLoad multiple data sources such as Word documents, PowerPoint presentations, PDF or images.Import/Export configuration with JSON, YML or Excel.Add custom Entity labels.Rule based phrase matching usingPhraseMatcherRule based matching usingregex + MatcherTrain Named Entity Recogniser with easeInstallEither use the famous pip or clone this repository and execute the
setup.py file.$pipinstallexcelcy# ensure you have the language model installed before$spacydownloadenTrainTo train the spaCy model:fromexcelcyimportExcelCyexcelcy=ExcelCy.execute(file_path='test_data_01.xlsx')Note:tests/data/test_data_01.xlsxCLIExelCy has basic CLI command for execute:$excelcyexecutehttps://github.com/kororo/excelcy/raw/master/tests/data/test_data_01.xlsxTestRun test by installing packages and run tox$pipinstallpoetrytox
$tox
$tox-epy36--tests/test_readme.pyFor hot-reload development coding$npmi-gnodemon
$nodemonData DefinitionExcelCy has data definition which expressed inapi.yml.
As long as, data given in this specific format and structure, ExcelCy will able to support any type of data format.
Check out, the Excel file format inapi.xlsx.
Data classes are defined withattrs,
check instorage.pyfor more detail.Publishing# this is note for contributors# ensure locally tests all runningnpmruntest# prepare for new versionpoetryversion0.4.1
npmrunexport# make changes in the git, especially release branch and check in the travis# https://travis-ci.com/github/kororo/excelcy# if all goes well, push to masterFAQWhat is that idx columns in the Excel sheet?The idea is to give reference between two things. Imagine in sheet "train", like to know where the sentence generated
from in sheet "source". And also, the nature of Excel, you can sort things, this is the safe guard to keep things in
the correct order.Can ExcelCy import/export to X, Y, Z data format?ExcelCy has strong and well-defined data storage, thanks toattrs.
It is possible to import/export data in any format.Error: ModuleNotFoundError: No module named 'pip'There are lots of possibility on this. Try to lower pip version (it was buggy for version 19.0.3).ExcelCy accepts suggestions/ideas?Yes! Please submit them into new issue with label "enhancement".AcknowledgementThis project uses other awesome projects:attrs: Python Classes Without Boilerplate.pyexcel: Single API for reading, manipulating and writing data in csv, ods, xls, xlsx and xlsm files.pyyaml: The next generation YAML parser and emitter for Python.spacy: Industrial-strength Natural Language Processing (NLP) with Python and Cython.textract: extract text from any document. no muss. no fuss.