Package: eurostat
Eurostat Python PackageTools to read data from Eurostat API.FeaturesRead Eurostat data and metadata as list of tuples or as pandas dataframe.Use the new SDMX 2.1 Eurostat web services.Download data from Eurostat, COMEXT, DG COMP, DG ENV, DG GROW.Available from both pip andconda.MIT license.Important!From version 1.0.0, this package implements MAJOR CHANGES.The previous official Eurostat API, used by the older releases of this Python package, is supposed to be decommissioned in January 2023.
It has been replaced with a new dissemination API.
This forced to rewrite this Python package, almost completely.I have done my best to make the new Eurostat Python package compatible with the old releases.
Nevertheless, you may see some differences, also in the output format.The main differences are in the SDMX functions.
In particular, OBS_STATUS is no more provided by the API.
It is replaced by a flag, that has different symbols and meanings.
From version 1.0.0, the SDMX functions of the Eurostat Python package are deprecated, but temporarily kept available.
These functions will be removed from version 2.0.0 of the Eurostat Python package.
They always print an alert message, that can be "muted" by setting the argumentnoalert=True.DocumentationGetting started:Requires Python 3.5+pipinstalleurostatRead the table of contents of the Eurostat database:As a list of tuples:eurostat.get_toc()Read the table of contents and return a list of tuples. The first element of the list contains the header line. Dates are represented as strings.Example:>>>importeurostat>>>toc=eurostat.get_toc()>>>toc[0]('title','code','type','last update of data','last table structure change','data start','data end')>>>toc[12:15][('Employment by NACE Rev. 2 activity and metropolitan typology','MET_10R_3EMP','dataset','2022-05-06T23:00:00+0200','2022-05-06T23:00:00+0200','1995','2020'),('Gross domestic product (GDP) at current market prices by metropolitan regions','MET_10R_3GDP','dataset','2022-04-21T23:00:00+0200','2022-04-21T23:00:00+0200','2000','2020'),('Gross value added at basic prices by metropolitan regions','MET_10R_3GVA','dataset','2022-04-21T23:00:00+0200','2022-04-21T23:00:00+0200','1995','2020')]As a pandas dataframe:eurostat.get_toc_df()Read the table of contents of the main database and return a dataframe.Example:>>>importeurostat>>>toc_df=eurostat.get_toc_df()>>>toc_dftitle...dataend0Roadequipment:numberofroadvehiclesbycat......20181Roadequipment:numberofroadvehiclesbyage...20182Roadequipment:loadcapacityoflorries...20153Roadequipment:newregistrationsbycategories...20154Roadtraffic:roadfreighttransportinvolume...2018.........7225TropicalwoodimportstotheEUfromchapter4......2020-127226Candidatecountriesandpotentialcandidates:......20197227Candidatecountriesandpotentialcandidates:......20147228Candidatecountriesandpotentialcandidates:......20157229Candidatecountriesandpotentialcandidates:......2014You may also want to extract the datasets that pertain a topic. In that case, you can use:eurostat.subset_toc_df(toc_df,keyword)Extract from toc_df the rows where the columntitlecontainskeyword(case-insensitive).Example:>>>f=eurostat.subset_toc_df(toc_df,'fleet')>>>ftitle...dataend4873Fishingfleet,totaltonnage...20214895FishingFleet,NumberofVessels...20216169Commercialaircraftfleetbyageofaircrafta......20206172Commercialaircraftfleetbyageofaircrafta......20206175Commercialaircraftfleetbyaircraftcategory......20206178Commercialaircraftfleetbyaircraftcategory......20206576Commercialaircraftfleetbytypeofaircraft...20207120Fishingfleetbyage,lengthandgrosstonnage...20217121Fishingfleetbytypeofgearandenginepower...2021Get the filter parameters to download a subset of a dataset:eurostat.get_pars(code)Read the parameter names that can be filtered for a given datasetcodeand return them as a list.Example:>>>importeurostat>>>pars=eurostat.get_pars('demo_r_d2jan')>>>pars['freq','unit','sex','age','geo']From the example, you can note thatcodeis generally case-insensitive.To get the parameter values for filtering, you can use:eurostat.get_par_values(code,par)Read the values of a given parameterparthat can be found in a given datasetcode.Example:>>>importeurostat>>>par_values=eurostat.get_par_values('demo_r_d2jan','sex')>>>par_values['T','M','F']Get an Eurostat dictionary:eurostat.get_dic(code,par,[full=True],[frmt="list"],[lang="en"])Read the dictionary of a given parameterparas a list of tuples or as a dictionary.
You must provide also the datasetcode.If you want the full list of possible values ofpar, setfull=True, whilefull=Falsereturns only the values that are in the dataset, output fromget_par_values().
Default isfull=True.frmt="list"makes the function return a list of tuples, where the first element of each tuple is the code value and the second one is its description.
Iffrmt="dict"it returns a dictionary.
Default isfrmt="list".langallows to download the dictionary in one of the following languages:"en"=English,"fr"=French,"de"=German, when provided by Eurostat. Default is English.Example:>>>importeurostat>>>dic=eurostat.get_dic('demo_r_d2jan','sex')>>>dic[('T','Total'),('M','Males'),('F','Females'),('DIFF','Absolute difference between males and females'),('NAP','Not applicable'),('NRP','No response'),('UNK','Unknown')]Read a dataset:As a list of tuples:eurostat.get_data(code,[flags=False],[filter_pars=dict()],[verbose=False],[reverse_time=False])Read an Eurostat dataset and returns it as a list of tuples.
The first element of the list ("the first row") is the data header.To get a subset, setfilter_pars(a dictionary where keys are parameter names, values are the wanted items).To see a rough progress status, setverbose=True.flag=Truedownloads also the flags associated to the data.
Pay attention: the data format changes ifflagsisTrueor not.
Flag meanings can be foundhere.reverse_time=Truereverses the order of the time columns. For compatibility with 0.x.x versions.Example: Full download a dataset without flags as list of tuples>>>importeurostat>>>data=eurostat.get_data('GOV_10DD_SLGD')>>>data[0]('freq','na_item','sector','maturity','unit','geo\\TIME_PERIOD',2018,2019,2020,2021)>>>data[90:95][('A','F3','S11','TOTAL','MIO_EUR','BE',None,None,23.8,39.3),('A','F3','S11','TOTAL','MIO_EUR','ES',None,None,130.0,122.2),('A','F3','S11','TOTAL','MIO_NAC','AT',None,None,0.0,0.0),('A','F3','S11','TOTAL','MIO_NAC','BE',None,None,23.8,39.3),('A','F3','S11','TOTAL','MIO_NAC','ES',None,None,130.0,122.2)]Example: Full download a dataset with flags as list of tuples>>>importeurostat>>>data=eurostat.get_data('GOV_10DD_SLGD',True)>>>data[0]('freq','na_item','sector','maturity','unit','geo\\TIME_PERIOD','2018_value','2018_flag','2019_value','2019_flag','2020_value','2020_flag','2021_value','2021_flag')>>>data[90:95][('A','F3','S11','TOTAL','MIO_EUR','BE',None,':',None,':',23.8,'',39.3,''),('A','F3','S11','TOTAL','MIO_EUR','ES',None,':',None,':',130.0,'',122.2,''),('A','F3','S11','TOTAL','MIO_NAC','AT',None,':',None,':',0.0,'',0.0,''),('A','F3','S11','TOTAL','MIO_NAC','BE',None,':',None,':',23.8,'',39.3,''),('A','F3','S11','TOTAL','MIO_NAC','ES',None,':',None,':',130.0,'',122.2,'')]Example: Download a subset of data from a dataset without flags as list of tuplesTo download a subset, you need to set thefilter_parsdictionary.
Its keys can be:'startPeriod','endPeriod'and any parameter you get witheurostat.get_pars().
Values can be number, string or list and generally are derived byeurostat.get_par_values().>>>importeurostat>>>code='GOV_10DD_SLGD'>>>pars=eurostat.get_pars(code)>>>pars['freq','na_item','sector','maturity','unit','geo']>>>par_values=eurostat.get_par_values(code,'geo')>>>par_values['BE','DE','ES','AT']>>>my_filter_pars={'startPeriod':2019,'geo':['AT','BE']}>>>data=eurostat.get_data(code,filter_pars=my_filter_pars)>>>data[0]('freq','na_item','sector','maturity','unit','geo\\TIME_PERIOD',2019,2020,2021)>>>data[445:447][('A','GD','S1_S2','Y_LT1','PC_TOT','AT',None,1.0,0.9),('A','F22','S1_S2','TOTAL','MIO_EUR','BE',None,0.0,0.0)]Example: Download a subset of data from a dataset with flags as list of tuples>>>importeurostat>>>code='GOV_10DD_SLGD'>>>pars=eurostat.get_pars(code)>>>pars['freq','na_item','sector','maturity','unit','geo']>>>par_values=eurostat.get_par_values(code,'geo')>>>par_values['BE','DE','ES','AT']>>>my_filter_pars={'startPeriod':2019,'geo':['AT','BE']}>>>data=eurostat.get_data(code,True,filter_pars=my_filter_pars)>>>data[0]('freq','na_item','sector','maturity','unit','geo\\TIME_PERIOD','2019_value','2019_flag','2020_value','2020_flag','2021_value','2021_flag')>>>data[446:448][('A','GD','S1_S2','Y_LT1','PC_TOT','AT',None,':',1.0,'',0.9,''),('A','F22','S1_S2','TOTAL','MIO_EUR','BE',None,':',0.0,'',0.0,'')]As a pandas dataframe:eurostat.get_data_df(code,[flags=False],[filter_pars=None],[verbose=False],[reverse_time=False])Read an Eurostat dataset and returns it as pandas dataframe.To get a subset, setfilter_pars(a dictionary where keys are parameter names, values are the wanted items).To see a rough progress status, setverbose=True.flag=Truedownloads also the flags associated to the data.
Pay attention: the data format changes ifflagsisTrueor not.
Flag meanings can be foundhere.reverse_time=Truereverses the order of the time columns. For compatibility with 0.x.x versions.Example: Full download a dataset without flags as pandas dataframe>>>importeurostat>>>data=eurostat.get_data_df('GOV_10DD_SLGD')>>>datafreqna_itemsectormaturity...20182019202020210AF22S1_S2TOTAL...NaNNaN0.00.01AF22S1_S2TOTAL...NaNNaN0.00.02AF22S1_S2TOTAL...0.00.00.00.03AF22S1_S2TOTAL...NaNNaN0.00.04AF22S1_S2TOTAL...NaNNaN0.00.0...........................1181AGDS1_S2Y_LT1...NaNNaN2849.02408.21182AGDS1_S2Y_LT1...NaNNaN1.00.91183AGDS1_S2Y_LT1...NaNNaN6.95.41184AGDS1_S2Y_LT1...4.96.05.95.51185AGDS1_S2Y_LT1...NaNNaN0.90.8Example: Full download a dataset with flags as pandas dataframe>>>importeurostat>>>data=eurostat.get_data_df('GOV_10DD_SLGD',True)>>>datafreqna_itemsectormaturity...2020_value2020_flag2021_value2021_flag0AF22S1_S2TOTAL...0.0:0.01AF22S1_S2TOTAL...0.00.02AF22S1_S2TOTAL...0.0:0.03AF22S1_S2TOTAL...0.00.04AF22S1_S2TOTAL...0.00.0...........................1182AGDS1_S2Y_LT1...1.00.91183AGDS1_S2Y_LT1...6.95.41184AGDS1_S2Y_LT1...5.95.51185AGDS1_S2Y_LT1...0.9:0.81186AGDS1_S2Y_LT1...0.90.8Example: Download a subset of data from a dataset without flags as pandas dataframeTo download a subset, you need to set thefilter_parsdictionary.
Its keys can be:'startPeriod','endPeriod'and any parameter you get witheurostat.get_pars().
Values can be number, string or list and generally are derived byeurostat.get_par_values().>>>importeurostat>>>code='GOV_10DD_SLGD'>>>pars=eurostat.get_pars(code)>>>pars['freq','na_item','sector','maturity','unit','geo']>>>par_values=eurostat.get_par_values(code,'geo')>>>par_values['BE','DE','ES','AT']>>>my_filter_pars={'endPeriod':2020,'geo':['AT','BE']}>>>data=eurostat.get_data_df(code,filter_pars=my_filter_pars)>>>datafreqna_itemsectormaturityunitgeo\TIME_PERIOD2018201920200AF22S1_S2TOTALMIO_EURATNaNNaN0.01AF22S1_S2TOTALMIO_NACATNaNNaN0.02AF22S1_S2Y_LT1MIO_EURATNaNNaN0.03AF22S1_S2Y_LT1MIO_NACATNaNNaN0.04AF29S1_S2TOTALMIO_EURATNaNNaN0.0.............................633AGDS1_S2Y_GT1MIO_NACBENaNNaN72660.0634AGDS1_S2Y_GT1PC_TOTBENaNNaN93.1635AGDS1_S2Y_LT1MIO_EURBENaNNaN5387.0636AGDS1_S2Y_LT1MIO_NACBENaNNaN5387.0637AGDS1_S2Y_LT1PC_TOTBENaNNaN6.9Example: Download a subset of data from a dataset with flags as pandas dataframe>>>importeurostat>>>code='GOV_10DD_SLGD'>>>pars=eurostat.get_pars(code)>>>pars['freq','na_item','sector','maturity','unit','geo']>>>par_values=eurostat.get_par_values(code,'geo')>>>par_values['BE','DE','ES','AT']>>>my_filter_pars={'endPeriod':2020,'geo':['AT','BE']}>>>data=eurostat.get_data_df(code,True,filter_pars=my_filter_pars)>>>datafreqna_itemsectormaturity...2019_value2019_flag2020_value2020_flag0AF22S1_S2TOTAL...NaN:0.01AF22S1_S2TOTAL...NaN:0.02AF22S1_S2Y_LT1...NaN:0.03AF22S1_S2Y_LT1...NaN:0.04AF29S1_S2TOTAL...NaN:0.0.............................635AGDS1_S2Y_GT1...NaN:93.1636AGDS1_S2Y_LT1...NaN:5387.0637AGDS1_S2Y_LT1...NaN:5387.0638AGDS1_S2Y_LT1...NaN:6.9639AGDS1_S2Y_LT1...NaN:6.9In case you need to use a proxy:Before doing anything else, you must configure the https proxy.eurostat.setproxy(proxyinfo)It requires in inputproxyinfo, a dictionary with one key ('https') and value containing the connection parameters in a list.If authentication is not needed, setusernameandpasswordtoNone.Example:>>>importeurostat>>>proxyinfo={'https':['myuser','mypassword','http://url:port']}>>>eurostat.setproxy(proxyinfo)It always returnsNone.Bug reports and feature requests:Pleaseopen an issueor send a message to noemi.cazzaniga [at] polimi.it .Disclaimer:Download and usage of Eurostat data is subject to Eurostat's general copyright notice and licence policy (seePolicies). Please also be aware of the European Commission'sgeneral conditions.Data sources:Eurostat database:online catalog.Eurostat nomenclatures:RAMONmetadata.Eurostat Interactive Data Browser:Data Browser.Eurostat Interactive Tool for Comext Data:Easy Comext.Eurostat PRODCOM website:PRODCOM.Eurostat acronyms:Symbols and abbreviations.Eurostat web services description:Web services.References:R packageeurostat: R Tools for Eurostat Open Data.Python packagepandas: Python Data Analysis Library.History:version 1.0.4 (03 Apr 2023):Changed input of setproxy to read the full address.version 1.0.3 (30 Mar 2023):Bug fix: proxy setting.version 1.0.2 (14 Mar 2023):Bug fix: proxy setting. Deprecated.version 1.0.1 (12 Oct 2022):Bug fix (wheel for conda venv).version 1.0.0 (08 Oct 2022):Adapted to the new Eurostat API (SDMX 2.1).pandasdmx not required anymore.Error messages improved.Compiled also for conda install.Multilingual dictionary.version 0.2.3 (06 Apr 2021):Internal bug fix.version 0.2.2 (31 Mar 2021):Bug fix (sdmx non-annual data). Deprecated.version 0.2.1 (10 Nov 2020):Bug fix (pandasdmx 0.9).version 0.2.0 (22 May 2020):Improved SDMX download capability in case of slow internet connections.version 0.1.5 (08 Jan 2020):Bug fix (proxy info).get_avail_sdmx, get_avail_sdmx_df, subset_avail_sdmx_df added.version 0.1.4 (20 Dec 2019):Added support to proxy.version 0.1.3 (17 Dec 2019):Bug fix (non-annual data headers).version 0.1.2 (25 Nov 2019):Added possibility to download flags.get_toc_df, subset_toc_df added.version 0.1.1 (21 Nov 2019):First official release.