Package: wikirepo
Python based Wikidata framework for easy dataframe extractionwikirepois a Python package that provides a framework to easily source and leverage standardizedWikidatainformation. The goal is to create an intuitive interface so that Wikidata can function as a common read-write repository for public statistics.See thedocumentationfor a full outline of the package including usage and available data.ContentsInstallationDataQuery DataUpload Data (WIP)Maps (WIP)ExamplesTo-DoInstallation⇧wikirepo can be downloaded from PyPI via pip or sourced directly from this repository:pipinstallwikirepogitclonehttps://github.com/andrewtavis/wikirepo.gitcdwikirepo
pythonsetup.pyinstallimportwikirepoData⇧wikirepo's data structure is built aroundWikidata.org. Human-readable access to Wikidata statistics is achieved through converting requests into Wikidata's Quantity IDs (QIDs) and Property IDs (PIDs), with the Python packagewikidataserving as a basis for data loading and indexing. See thedocumentationfor a structured overview of the currently available properties.Query Data⇧wikirepo's main access function,wikirepo.data.query, returns apandas.DataFrameof locations and property data across time.Each query needs the following inputs:locations: the locations that data should be queried forStrings are accepted forEarth, continents, and countriesGet all country names withwikirepo.data.incl_lctn_lbls(lctn_lvls='country')The user can also pass Wikidata QIDs directlydepth: the geographic level of the given locations to queryA depth of 0 is the locations themselvesGreater depths correspond to lower geographic levels (states of countries, etc.)A dictionary of locations is generated for lower depths (see second example below)timespan: start and enddatetime.dateobjects defining when data should come fromIf not provided, then the most recent data will be retrieved with annotation for when it's frominterval:yearly,monthly,weekly, ordailyas stringsFurther arguments: the names of modules inwikirepo/datadirectoriesThese are passed to arguments corresponding to their directoriesData will be queried for these properties for the givenlocations,depth,timespanandinterval, with results being merged as dataframe columnsQueries are also able to access information in Wikidata sub-pages for locations. For example: if inflation rate is not found on the location's main page, then wikirepo checks the location's economic topic page asinflation_rate.pyis found inwikirepo/data/economic(seeGermanyandeconomy of Germany).wikirepo further provides a unique dictionary class,EntitiesDict, that stores all loaded Wikidata entities during a query. This speeds up data retrieval, as entities are loaded once and then accessed in theEntitiesDictobject for any other needed properties.Examples ofwikirepo.data.queryfollow:Querying Information for Given Countriesimportwikirepofromwikirepo.dataimportwd_utilsfromdatetimeimportdateents_dict=wd_utils.EntitiesDict()# Strings must match their Wikidata English page namescountries=["Germany","United States of America","People's Republic of China"]# countries = ["Q183", "Q30", "Q148"] # we could also pass QIDs# data.incl_lctn_lbls(lctn_lvls='country') # or all countries`depth=0timespan=(date(2009,1,1),date(2010,1,1))interval="yearly"df=wikirepo.data.query(ents_dict=ents_dict,locations=countries,depth=depth,timespan=timespan,interval=interval,climate_props=None,demographic_props=["population","life_expectancy"],economic_props="median_income",electoral_poll_props=None,electoral_result_props=None,geographic_props=None,institutional_props="human_dev_idx",political_props="executive",misc_props=None,verbose=True,)col_order=["location","qid","year","executive","population","life_exp","human_dev_idx","median_income",]df=df[col_order]df.head(6)locationqidyearexecutivepopulationlife_exphuman_dev_idxmedian_incomeGermanyQ1832010Angela Merkel8.1752e+0779.98780.92133333GermanyQ1832009Angela Merkelnan79.83660.917nanUnited States of AmericaQ302010Barack Obama3.08746e+0878.54150.91443585United States of AmericaQ302009George W. Bushnan78.39020.91nanPeople's Republic of ChinaQ1482010Wen Jiabao1.35976e+0975.2360.706nanPeople's Republic of ChinaQ1482009Wen Jiabaonan75.0320.694nanQuerying Information for all US Counties# Note: >3000 regions, expect a 45 minute runtimeimportwikirepofromwikirepo.dataimportlctn_utils,wd_utilsfromdatetimeimportdateents_dict=wd_utils.EntitiesDict()country="United States of America"# country = "Q30" # we could also pass its QIDdepth=2# 2 for counties, 1 for states and territoriessub_lctns=True# for all# Only valid sub-locations given the timespan will be queriedtimespan=(date(2016,1,1),date(2018,1,1))interval="yearly"us_counties_dict=lctn_utils.gen_lctns_dict(ents_dict=ents_dict,locations=country,depth=depth,sub_lctns=sub_lctns,timespan=timespan,interval=interval,verbose=True,)df=wikirepo.data.query(ents_dict=ents_dict,locations=us_counties_dict,depth=depth,timespan=timespan,interval=interval,climate_props=None,demographic_props="population",economic_props=None,electoral_poll_props=None,electoral_result_props=None,geographic_props="area",institutional_props="capital",political_props=None,misc_props=None,verbose=True,)df[df["population"].notnull()].head(6)locationsub_lctnsub_sub_lctnqidyearpopulationarea_km2capitalUnited States of AmericaCaliforniaAlameda CountyQ10714620181.6602e+062127OaklandUnited States of AmericaCaliforniaContra Costa CountyQ10805820181.14936e+062078MartinezUnited States of AmericaCaliforniaMarin CountyQ10811720182638862145San RafaelUnited States of AmericaCaliforniaNapa CountyQ10813720181412942042NapaUnited States of AmericaCaliforniaSan Mateo CountyQ10810120187741551919Redwood CityUnited States of AmericaCaliforniaSanta Clara CountyQ11073920181.9566e+063377San JoseUpload Data (WIP)⇧wikirepo.data.uploadwill be the core of the eventual wikirepo upload feature. The goal is to record edits that a user makes to a previously queried or baseline dataframe such that these changes can then be pushed back to Wikidata. With the addition of Wikidata login credentials as a wikirepo feature (WIP), the unique information in the edited dataframe could then be uploaded to Wikidata for all to use.The same process used to query information from Wikidata could be reversed for the upload process. Dataframe columns could be linked to their corresponding Wikidata properties, whether the time qualifiers are apoint in timeor spans usingstart timeandend timecould be derived through the defined variables in the module header, and other necessary qualifiers for proper data indexing could also be included. Source information could also be added in corresponding columns to the given property edits.Pseudocodefor how this process could function follows:In the first example, changes are made to adf.copy()of a queried dataframe.pandasis then used to compare the new and original dataframes after the user has added information that they have access to.importwikirepofromwikirepo.dataimportlctn_utils,wd_utilsfromdatetimeimportdatecredentials=wd_utils.login()ents_dict=wd_utils.EntitiesDict()country="Country Name"depth=2sub_lctns=Truetimespan=(date(2000,1,1),date(2018,1,1))interval='yearly'lctns_dict=lctn_utils.gen_lctns_dict()df=wikirepo.data.query()df_copy=df.copy()# The user checks for NaNs and adds datadf_edits=pd.concat([df,df_copy]).drop_duplicates(keep=False)wikirepo.data.upload(df_edits,credentials)In the next exampledata.data_utils.gen_base_dfis used to create a dataframe with dimensions that match a time series that the user has access to. The data is then added to the column that corresponds to the property to which it should be added. Source information could further be added via a structured dictionary generated for the user.importwikirepofromwikirepo.dataimportdata_utils,wd_utilsfromdatetimeimportdatecredentials=wd_utils.login()locations="Country Name"depth=0# The user defines the time parameters based on their datatimespan=(date(1995,1,2),date(2010,1,2))# (first Monday, last Sunday)interval='weekly'base_df=data_utils.gen_base_df()base_df['data']=data_for_matching_time_seriessource_data=wd_utils.gen_source_dict('Source Information')base_df['data_source']=[source_data]*len(base_df)wikirepo.data.upload(base_df,credentials)Put simply: a full featuredwikirepo.data.uploadfunction would realize the potential of a single read-write repository for all public information.Maps (WIP)⇧wikirepo/mapsis a further goal of the project, as it combines wikirepo's focus on easy to access open source data and quick high level analytics.• Query MapsAs inwikirepo.data.query, passing thelocations,depth,timespanandintervalarguments could access GeoJSON files stored on Wikidata, thus providing mapping files in parallel to the user's data. These files could then be leveraged using existing Python plotting libraries to provide detailed presentations of geographic analysis.• Upload MapsSimilar to the potential of adding statistics throughwikirepo.data.upload, GeoJSON map files could also be uploaded to Wikidata using appropriate arguments. The potential exists for a myriad of variable maps givenlocations,depth,timespanandintervalinformation that would allow all wikirepo users to get the exact mapping file that they need for their given task.Examples⇧wikirepo can be used as a foundation for countless projects, with its usefulness and practicality only improving as more properties are added and more data is uploaded toWikidata.Current usage examples include:Sample notebooks for the Python packagepoli-sci-kitshow how to use wikirepo as a basis for political election and parliamentary appointment analysis, with those notebooks being found in theexamples for poli-sci-kitor onGoogle ColabPull requests with other examples will gladly be acceptedTo-Do⇧Please see thecontribution guidelinesif you are interested in contributing to this project. Work that is in progress or could be implemented includes:Expanding wikirepoCreating an outline of the package's structure for the readme(see issue)Integrating current Python tools with wikirepo structures for uploads to WikidataAdding a query of property descriptions todata.data_utils.incl_dir_idxs(see issue)Adding multiprocessing support to thewikirepo.data.queryprocess anddata.lctn_utils.gen_lctns_dictPotentially convertingwikirepo.data.queryanddata.lctn_utils.gen_lctns_dictover to generated Wikidata SPARQL queriesOptimizingwikirepo.data.query:Potentially convertingEntitiesDictandLocationsDictto slotted object classes for memory savingsDeriving and optimizing other slow parts of the query processAdding access to GeoJSON files for mapping viawikirepo.maps.queryDesigning and adding GeoJSON files indexed by time properties to WikidataCreating, improving and sharingexamplesImprovingtestsfor greatercode coverageImprovingcode qualityby refactoring large functions and checking conventionsExpanding WikidataThe growth of wikirepo's database relies on that ofWikidata. Throughdata.wd_utils.dir_to_topic_pagewikirepo can access properties on location sub-pages, thus allowing for statistics on any topic to be linked to. Beyond including entries for already existing properties (seethis issue), the following are examples of property types that could be added:Climate statistics could be added todata/climateThis would allow for easy modeling of global warming and its effectsPlanning would be needed for whether lower intervals would be necessary, or just include daily averagesThose for electoralpollingandresultsfor locationsThis would allow direct access to all needed election information in a single function callA property that links political parties and their regions indata/politicalFor easy professional presentation of electoral results (ex: loading in party hex colors, abbreviations, and alignments)data/demographicproperties such as:age, education, religious, and linguistic diversities across timedata/economicproperties such as:female workforce participation, workforce industry diversity, wealth diversity, and total working age population across timeDistinct properties for Freedom House and Press Freedom indexes, as well as other descriptive metricsThese could be added todata/institutionalSimilar ProjectsPythonhttps://github.com/dahlia/wikidatahttps://github.com/SuLab/WikidataIntegratorhttps://github.com/siznax/wptoolsJavaScripthttps://github.com/maxlath/wikibase-clihttps://github.com/maxlath/wikibase-edithttps://github.com/maxlath/wikibase-dump-filterhttps://github.com/maxlath/wikibase-sdkJavahttps://github.com/Wikidata/Wikidata-ToolkitPowered By