Package: newsfeed
NewsFeedIntroductionNewsfeed based on GDELT ProjectInstallationcondacreate-nnewsfeedpython=3.7
pythonsetupinstallGDELT APIBased on thegdelt-doc-api, we consider a continuous querying mechanism by spliting the time range into multiple sub range (default setting is every 60 minutes).FIPS 2 letter Contries list: please check:LOOK-UP COUNTRIESGKG Themes list: please check:LOOK-UP THEMESThe URL encoding reference:url encodeGDELT DOC 2.0 APIGDELT GEO 2.0 API# BETA VERSIONGDELT TV 2.0 API# NOT YETGDELT Database QueryGDELT 1.0GDELT Events Database 1.0GDELT Global Knowledge Graph 1.0GDELT 2.0GDELT Events Database 2.0GDELT Mentions Database (new)GDELT Global Knowledge Graph 2.0GDELT OthersGDELT Global Entity GraphGDELT Visual Global Entity GraphGDELT Different GraphGDELT Global Frontpage GraphHOWTOAPIsFor Article query:fromnewsfeed.news.apis.filtersimport*fromnewsfeed.news.apis.queryimport*f=Art_Filter(keyword=["Exchange Rate","World"],start_date="20211231000000",end_date="20211231010000",country=["China","US"])articles_30=article_search(query_filter=f,max_recursion_depth=100,time_range=30)articles_60=article_search(query_filter=f,max_recursion_depth=100,time_range=60)For Timeline query:fromnewsfeed.news.apis.filtersimport*fromnewsfeed.news.apis.queryimport*f=Art_Filter(keyword=["Exchange Rate","World"],start_date="2021-12-31-00-00-00",end_date="2021-12-31-01-00-00",country=["China","US"])timelineraw=timeline_search(query_filter=f,max_recursion_depth=100,query_mode="timelinevolraw")For GEO query:fromnewsfeed.news.apis.filtersimport*fromnewsfeed.news.apis.queryimport*f=Art_Filter(keyword=["Exchange Rate","World"],country=["China","US"])geo_7d=geo_search(query_filter=f,sourcelang="english",timespan=7)query_mode:artlist:article_searchtimeline:timelinevol,timelinevolraw,timelinetone,timelinelang,timelinesourcecountrymost of the parameters are the same withgdelt-doc-api, however, to specify the precise date range, we remove thetimespanand usestart_dateandtime_rangefor iteratively collecting articles.Database QueryFor event database (both V1 and V2):fromnewsfeed.news.db.eventsimport*# GDELT Event Database Version 1.0gdelt_events_v1_events=EventV1(start_date="2021-01-01",end_date="2021-01-02")results_v1_events=gdelt_events_v1_events.query()results_v1_events_nowtime=gdelt_events_v1_events.query_nowtime()# GDELT Event Database Version 2.0 - Eventgdelt_events_v2_events=EventV2(start_date="2021-01-01-00-00-00",end_date="2021-01-02-00-00-00")results_v2_events=gdelt_events_v2_events.query()results_v2_events_nowtime=gdelt_events_v2_events.query_nowtime()# GDELT Event Database Version 2.0 - Mentionsgdelt_events_v2_mentions=EventV2(start_date="2021-01-01-00-00-00",end_date="2021-01-02-00-00-00",table="mentions")results_v2_mentions=gdelt_events_v2_mentions.query()results_v2_mentions_nowtime=gdelt_events_v2_mentions.query_nowtime()For GKG databse (both V1 and V2):fromnewsfeed.news.db.gkgimport*# GDELT GKG Database Version 1.0gdelt_events_v1_gkg=GKGV1(start_date="2021-01-01",end_date="2021-01-02")results_v1_gkg=gdelt_events_v1_gkg.query()results_v1_gkg_nowtime=gdelt_events_v1_gkg.query_nowtime()fromnewsfeed.news.db.gkgimport*# GDELT GKG Database Version 2.0gdelt_events_v2_gkg=GKGV2(start_date="2021-01-01-00-00-00",end_date="2021-01-02-00-00-00")results_v2_gkg=gdelt_events_v2_gkg.query()results_v2_gkg_nowtime=gdelt_events_v2_gkg.query_nowtime()For GEG, VGEG and GDG:fromnewsfeed.news.db.othersimport*# GDELT Global Entity Graphgdelt_v3_geg=GEG(start_date="2020-01-01",end_date="2020-01-02")gdelt_v3_geg_result=gdelt_v3_geg.query()# GDELT Visual Global Entity Graphgdelt_v3_vgeg=VGEG(query_date="2020-01-01",domain="CNN")gdelt_v3_vgeg_result=gdelt_v3_vgeg.query()# GDELT Global Difference Graphgdelt_v3_gdg=GDG(query_date="2018-08-27-14-00-00")gdelt_v3_gdg_result=gdelt_v3_gdg.query()# GDELT Global Frontpage Graphgdelt_v3_gfg=GFG(query_date="2018-03-02-02-00-00")gdelt_v3_gfg_result=gdelt_v3_gfg.query()UtilitiesFull-text downloader (based onnewspaper3kandWayback Machine)fromnewsfeed.utilsimportfulltextasftart=ft.download(url="https://english.news.cn/20220205/a4e93df9162e4053af64c392b5f5bfec/c.html")print("full text:\n{}".format(art.text))