Package: metadoc
MetadocMetadoc is a post-truth era news article metadata retrieval service. It does social media activity lookup, source authenticity rating, checksum creation, json-ld and metatag parsing as well as information extraction for named entities, pullquotes, fulltext and other useful things based off of arbitrary article URLs. Also, Metadoc is built to be relatively fast.ExampleYou just throw it any news article URL, and Metadoc will yield.frommetadocimportMetadocurl="https://theintercept.com/2016/11/17/iphones-secretly-send-call-history-to-apple-security-firm-says"metadoc=Metadoc(url=url)res=metadoc.query()=>{'__version__':'0.9.0','authors':['Kim Zetter'],'canonical_url':'https://theintercept.com/2016/11/17/iphones-secretly-send-call-history-to-apple-security-firm-says/','domain':{'credibility':{'fake_confidence':'0.00','is_blacklisted':False},'date_registered':None,'favicon':'https://logo.clearbit.com/theintercept.com?size=200','name':'theintercept.com'},'entities':{'keywords':['cellebrite','fbi','skype','intercept'...]}},'image':'https://theintercept.imgix.net/wp-uploads/sites/1/2016/11/GettyImages-578052668-s.jpg?auto=compress%2Cformat&q=90&fit=crop&w=1200&h=800','language':'en','modified_date':None,'published_date':'2016-11-17T11:00:36+00:00','scraped_date':'2018-07-10T12:13:46+00:00','social':[{'metrics':[{'count':7340,'label':'sharecount'}],'provider':'facebook'}],'text':{'contenthash':'940a62c70db255b4aec378529ae7a2c8','fulltext':'a guardian of user privacy this year after fighting FBIdemandstohelpcrackintoSanBernardinoshooterSyed...','reading_time':439,'summary':'Your call logs get sent to Apple’s servers whenever iCloud is on — something Apple does not disclose.'},'title':'iPhones Secretly Send Call\xa0History to Apple, Security Firm Says','url':'https://theintercept.com/2016/11/17/iphones-secretly-send-call-history-to-apple-security-firm-says'}Trustworthiness CheckMetadoc does a basic background check on article sources. This means a simple blacklist-lookup viawhoisdata on the domain. Blacklists taken into account include the controversialPropOrNot. Thus, only if a domain is found on every blacklist do we spit out afake_confidenceof 1. The resulting metadata should be taken with a grain of salt.Part-of-speech taggingFor speed and simplicity, we decided againstnltkand instead rely on the Averaged Perceptron as imagined by Matthew Honnibal@explosion. The pip install comes pre-trained with aCoNLL 2000training set which works reasonably well to detect proper nouns. Since training is non-deterministic, unwanted stopwords might slip through. If you want to try out other datasets, simply replacemetadoc/extract/data/training_set.txtwith your own and runmetadoc.extract.pos.do_train.PurposeThis library is used in the context of a news-related software undertaking calledPraise. We're building the first social network dedicated to quality journalism recommendations. Synthesizing what we dub "audience-evaluated content" with automated metadata. If you're intrigued and might want to work with us, feel free to drop a line toa@praise.press.InstallRequires python 3.5.Using pippipinstallmetadocDevelopMac OSbrewinstallpython3libxml2libxsltlibtifflibjpegwebplittle-cms2Ubuntuapt-getinstall-ypython3libxml2-devlibxslt-devlibtiff-devlibjpeg-devwebpwhoisFedora/Redhatdnfinstalllibxml2-devellibxslt-devellibtiff-devellibjpeg-devellibjpeg-turbo-devellibwebpwhoisThenpip3install-rrequirements-dev.txt
pythonserve.py=>serving@6060Testpy.test-vtestsIf you happen to run into an error with OSX 10.11 concerning a lazy bound library in PIL,just remove/PIL/.dylibs/liblzma.5.dylib.TodoPage concatenation is needed in order to properly calculate wordcount and reading time.Authenticity heuristic with sharecount deviance detection (requires state).Perf: Worst offender is nltk's pos tagger. Roll own w/ Average Perceptron.Newspaper's summarize produces pullquotes, fulltext takes a while. Move to libextract?ContributorsMartin BorhoPaul SolbachMeteadoc is a software product of Praise Internet UG, Hamburg.Metadoc stems from a pedigree of nice libraries likegoose3,langdetectandnltk.Metadoc leans onthisperceptron implementation inspired by Matthew Honnibal.Metadoc is a work-in-progress.