Package: pandas-datacube
Pandas DatacubeAboutpandas-datacubeis a python package allowing to convert and download adatacubefrom a remote source usingSPARQLqueries and to obtain a pandas dataframe.This module is able to detect the different datasets of an entry point and its different dimensions and measures,
to use the metadata present in the ontology to order the dimensions and to download the dataThis project was realized during an internship atLIGin theGETALPteam under the supervision of Mr Sérasset (Gilles.Serasset@imag.fr)InstallationYou can install pandas-datacube fromPyPi:$ pip install pandas-datacubeHow to useThe module is quite simple to use:get all datasets available:frompandasdatacubeimportget_datasetsimportpandasaspdENDPOINT:str="https://statistics.gov.scot/sparql"datasets:pd.DataFrame=get_datasets(ENDPOINT)datasets.head()datasetcommentaire0http://statistics.gov.scot/data/pupil-attainmentNumber of pupils who attained a given number of qualifications by level and stage.1http://statistics.gov.scot/data/alcohol-related-dischargeNumber and European Age-sex Standardised Rates (EASRs) of general acute inpatient and day case discharges with an alcohol-related diagnosis.2http://statistics.gov.scot/data/business-births-deaths-and-survival-ratesNumber and rate (per 10,000 adults) of VAT/PAYE registrations, de-registrations and business survival rates3http://statistics.gov.scot/data/earningsMean and median gross weekly earnings (£s) by gender, working pattern and workplace/residence measure.4http://statistics.gov.scot/data/economic-inactivityEconomic inactivity level and rate by genderget and transform features of a datasetfrompandasdatacubeimportget_features,transform_featuresimportpandasaspdENDPOINT:str="https://statistics.gov.scot/sparql"DATASET_NAME:str="http://statistics.gov.scot/data/earnings"features:pd.DataFrame=get_features(ENDPOINT,DATASET_NAME)features.head()itemtypeproperty0http://statistics.gov.scot/def/component-specification/earnings/refAreahttp://www.w3.org/1999/02/22-rdf-syntax-ns#typehttp://purl.org/linked-data/cube#ComponentSpecification1http://statistics.gov.scot/def/component-specification/earnings/refAreahttp://purl.org/linked-data/cube#dimensionhttp://purl.org/linked-data/sdmx/2009/dimension#refArea2http://statistics.gov.scot/def/component-specification/earnings/refAreahttp://purl.org/linked-data/cube#order13http://statistics.gov.scot/def/component-specification/earnings/refAreahttp://purl.org/linked-data/cube#codeListhttp://statistics.gov.scot/def/code-list/earnings/refArea4http://statistics.gov.scot/def/component-specification/earnings/refPeriodhttp://www.w3.org/1999/02/22-rdf-syntax-ns#typehttp://purl.org/linked-data/cube#ComponentSpecificationtransformed_features:tuple[list[str]]=transform_features(features)print(transformed_features)Output:(['http://purl.org/linked-data/sdmx/2009/dimension#refArea','http://purl.org/linked-data/sdmx/2009/dimension#refPeriod','http://purl.org/linked-data/cube#measureType','http://statistics.gov.scot/def/dimension/gender','http://statistics.gov.scot/def/dimension/workingPattern','http://statistics.gov.scot/def/dimension/populationGroup'],['http://statistics.gov.scot/def/measure-properties/median','http://statistics.gov.scot/def/measure-properties/mean'])download a datasetfrompandasdatacubeimportdownload_datasetimportpandasaspdENDPOINT:str="https://statistics.gov.scot/sparql"DATASET_NAME:str="http://statistics.gov.scot/data/earnings"DIMENSIONS:list[str]=['http://purl.org/linked-data/sdmx/2009/dimension#refArea','http://purl.org/linked-data/sdmx/2009/dimension#refPeriod','http://purl.org/linked-data/cube#measureType','http://statistics.gov.scot/def/dimension/gender','http://statistics.gov.scot/def/dimension/workingPattern','http://statistics.gov.scot/def/dimension/populationGroup']MEASURES:list[str]=['http://statistics.gov.scot/def/measure-properties/median','http://statistics.gov.scot/def/measure-properties/mean']data:pd.DataFrame=download_dataset(endpoint=ENDPOINT,dataset_name=DATASET_NAME,dimensions=DIMENSIONS,measures=MEASURES)data.head().reset_index()refArearefPeriodmeasureTypegenderworkingPatternpopulationGroupmedianmean0http://statistics.gov.scot/id/statistical-geography/S92000003http://reference.data.gov.uk/id/year/1997http://statistics.gov.scot/def/measure-properties/medianhttp://statistics.gov.scot/def/concept/gender/malehttp://statistics.gov.scot/def/concept/working-pattern/full-timehttp://statistics.gov.scot/def/concept/population-group/workplace-based340.81http://statistics.gov.scot/id/statistical-geography/S92000003http://reference.data.gov.uk/id/year/1997http://statistics.gov.scot/def/measure-properties/meanhttp://statistics.gov.scot/def/concept/gender/malehttp://statistics.gov.scot/def/concept/working-pattern/full-timehttp://statistics.gov.scot/def/concept/population-group/workplace-based387.12http://statistics.gov.scot/id/statistical-geography/S92000003http://reference.data.gov.uk/id/year/1997http://statistics.gov.scot/def/measure-properties/medianhttp://statistics.gov.scot/def/concept/gender/malehttp://statistics.gov.scot/def/concept/working-pattern/part-timehttp://statistics.gov.scot/def/concept/population-group/workplace-based803http://statistics.gov.scot/id/statistical-geography/S92000003http://reference.data.gov.uk/id/year/1997http://statistics.gov.scot/def/measure-properties/meanhttp://statistics.gov.scot/def/concept/gender/malehttp://statistics.gov.scot/def/concept/working-pattern/part-timehttp://statistics.gov.scot/def/concept/population-group/workplace-based110.94http://statistics.gov.scot/id/statistical-geography/S92000003http://reference.data.gov.uk/id/year/1997http://statistics.gov.scot/def/measure-properties/medianhttp://statistics.gov.scot/def/concept/gender/femalehttp://statistics.gov.scot/def/concept/working-pattern/full-timehttp://statistics.gov.scot/def/concept/population-group/workplace-based247do all steps in one linesfrompandasdatacubeimportget_datacubeimportpandasaspdENDPOINT:str="http://kaiko.getalp.org/sparql"PREFIXES:dict[str]={'dbnary':'http://kaiko.getalp.org/dbnary#','dbnstats':'http://kaiko.getalp.org/dbnary/statistics/','lime':'http://www.w3.org/ns/lemon/lime#'}dataset:str="dbnstats:dbnaryStatisticsCube"dimensions:list[str]=['dbnary:observationLanguage','dbnary:wiktionaryDumpVersion']mesures:list[str]=['dbnary:lexicalEntryCount','dbnary:lexicalSenseCount','dbnary:pageCount','dbnary:translationsCount']dtypes:dict[str]={"lexicalEntryCount":int,"translationsCount":int,"lexicalSenseCount":int,"pageCount":int}data:pd.DataFrame=get_datacube(ENDPOINT,dataset,dimensions,mesures,dtypes,PREFIXES)data.head().reset_index()observationLanguagewiktionaryDumpVersionlexicalEntryCountlexicalSenseCountpageCounttranslationsCount0bg20210701186261842027050180861bg20140224188311879827071138882bg20140312188291879627068138953bg20140328188281879527072139094bg2014041518822182942706813920