Package: extractnet
ExtractNetBased on the popular content extraction packageDragnet, ExtractNet extend the machine learning approach to extract other attributes such as date, author and keywords from news article.Example code:Simply use the following command to install the latest released version:pipinstallextractnetStart extract content and other meta data passing the result html to functionimportrequestsfromextractnetimportExtractorraw_html=requests.get('https://currentsapi.services/en/blog/2019/03/27/python-microframework-benchmark/.html').textresults=Extractor().extract(raw_html)Why don't just use existing rule-base extraction method:We discover some webpage doesn't provide the real author name but simply populate the author tag with a default value.For exampleltn.com.tw,udn.comalways populate the same author value for each news article while the real author can only be found within the content.ExtractNet uses machine learning approach to extract these relevant data through visible section of the webpage just like a human.What ExtractNet is and isn'tExtractNet is a platform to extract any interesting attributes from any webpage, not just limited to content based article.The core of ExtractNet aims to convert unstructured webpage to structured data without relying hand crafted rulesExtractNet do not support boilerplate content extractionExtractNet allows user to add custom pipelines that returns additional data through a list ofcallbacks functionPerformanceResults of the body extraction evaluation:We use the same body extraction benchmark fromarticle-extraction-benchmarkModelPrecisionRecallF1AccuracyOpen SourceAutoExtract0.984 ± 0.0030.956 ± 0.0100.970 ± 0.0050.470 ± 0.037✗Diffbot0.958 ± 0.0090.944 ± 0.0130.951 ± 0.0100.348 ± 0.035✗ExtractNet0.922 ± 0.0110.933 ± 0.0130.927 ± 0.0100.160 ± 0.027✔boilerpipe0.850 ± 0.0160.870 ± 0.0200.860 ± 0.0160.006 ± 0.006✔dragnet0.925 ± 0.0120.889 ± 0.0180.907 ± 0.0140.221 ± 0.030✔html-text0.500 ± 0.0170.994 ± 0.0010.665 ± 0.0150.000 ± 0.000✔newspaper0.917 ± 0.0130.906 ± 0.0170.912 ± 0.0140.260 ± 0.032✔readability0.913 ± 0.0140.931 ± 0.0150.922 ± 0.0130.315 ± 0.034✔trafilatura0.930 ± 0.0100.967 ± 0.0090.948 ± 0.0080.243 ± 0.031✔Results of author name extraction:ModelF1ExtractNet: fasttext embeddings + CRF0.904 ± 0.10List of changes from DragnetUnderlying classifier is replaced byCatboostinstead ofDecision Treefor all attributes extraction for consistency and performance boost.Updated CSS features, added text+css latent featureIncludes aCRFmodel that extract names from author block text.Trained on 22000+ updated webpages collected in the late 2020,20 timesof dragnet data.GETTING STARTEDInstalling and extractionpip install extractnetimportrequestsfromextractnetimportExtractorraw_html=requests.get('https://apnews.com/article/6e58b5742b36e3de53298cf73fbfdf48').textresults=Extractor().extract(raw_html)forkey,valueinresults.items():print(key)print(value)print('------------')CallbacksExtractNet also support the ability to add callbacks functions to inject additional features during extraction processA quick glance of usage : each callbacks will be able to access the raw html string provided during the extraction process. This allows user to extract addtional information such as language detection to the final resultsdefmeta_pre1(raw_html):return{'first_value':0}defmeta_pre2(raw_html):return{'first_value':1,'second_value':2}deffind_stock_ticker(raw_html,results):matched_ticker=[]forticketinre.findall(r'[$][A-Za-z][\S]*',str(results['content'])):matched_ticker.append(ticket)return{'matched_ticker':matched_ticker}extract=Extractor(author_prob_threshold=0.1,meta_postprocess=[meta_pre1,meta_pre2],postprocess=[find_stock_ticker])The extracted results will containlike,first_valueandsecond_value. Do note callbacks are executed by the given order ( which means meta_pre1 will be executed first followed by meta_pre2 ), any results passed from theprevious stage will not be overwritten by later stageraw_html=requests.get('https://apnews.com/article/6e58b5742b36e3de53298cf73fbfdf48').textresults=extract(raw_html)In this example the value for first_value will remain 0 even though meta_pre2 also returns first_value=1 because meta_pre2 callbacks already assign first_value as 0.ContributingWe love contributions! Open an issue, or fork/create a pull
request.Develop LocallySince extractnet relies on several C++ modules, before starting to run locally you need to compile them firstUsually what you need would be this commandmakeHowever, you can try to build itSupress logging errorSetting the level to critical will suppress any logging outputfromextractnetimportExtractorfromextractnet.blocksimportBlockifyErrorlogging.getLogger('extractnet').setLevel(logging.CRITICAL)extractor=Extractor()More details about the code structureComing soonReferenceContent extraction using diverse feature sets[1] Peters, Matthew E. and D. Lecocq,Content extraction using diverse feature sets@inproceedings{Peters2013ContentEU,
  title={Content extraction using diverse feature sets},
  author={Matthew E. Peters and D. Lecocq},
  booktitle={WWW '13 Companion},
  year={2013}
}Bag of Tricks for Efficient Text Classification[2] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov,Bag of Tricks for Efficient Text Classification@article{joulin2016bag,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}