Package: istatapi
istatapiis a Python interface to discover and retrieve data from
ISTAT API (The Italian National Institute of Statistics). The library is
designed to explore ISTAT metadata and to retreive data in different
formats.istatapiis built on top ofISTAT SDMX RESTful
API.Whether you are an existing organization, a curious individual or an
academic researcher,istatapiaims to allow you to easily access ISTAT
databases with just a few lines of code. The library implements
functions to:Explore all available ISTAT datasets (dataflows in SDMX terminology)Search available datasets by keywordsRetrieve information on a specific dataset like: the ID of the
dataflow, the names and available values of the dimensions of the
dataset, available filters.Get data of an available dataset in a pandas DataFrame, csv or json
format.InstallYou can easily install the library by using the pip command:pip install istatapiTutorialFirst, let’s simply import the modules we need:fromistatapiimportdiscovery,retrievalimportmatplotlib.pyplotaspltWithistatapiwe can search through all the available datasets by
simply using the following function:discovery.all_available()df_idversiondf_descriptiondf_structure_id0101_10151.3CropsDCSP_COLTIVAZIONI1101_10301.0PDO, PGI and TSG quality productsDCSP_DOPIGP2101_10331.0slaughteringDCSP_MACELLAZIONI3101_10391.2Agritourism - municipalitiesDCSP_AGRITURISMO_COM4101_10771.0PDO, PGI and TSG products:  operators - municipalities dataDCSP_DOPIGP_COMYou can also search for a specific dataset (in this example, a dataset
on imports), by doing:discovery.search_dataset("import")df_idversiondf_descriptiondf_structure_id10101_9621.0Livestock import exportDCSP_LIVESTIMPEXP47139_1761.0Import and export by country and commodity Nace 2007DCSP_COEIMPEX149143_2221.0Import price index - monthly dataDCSC_PREIMPINDTo retrieve data from a specific dataset, we first need to create an
instance of theDataSetclass. We can usedf_id,df_descriptionordf_structure_idfrom
the above DataFrame to tell to theDataSetclass what dataset we want to retrieve. Here, we are going to use thedf_idvalue. This may take a few seconds to load.# initialize the dataset and get its dimensionsds=discovery.DataSet(dataflow_identifier="139_176")We now want to see what variables are included in the dataset that we
are analysing. Withistatapiwe can easily print its variables
(“dimensions” in ISTAT terminology) and their description.ds.dimensions_info()dimensiondimension_IDdescription0FREQCL_FREQFrequency1MERCE_ATECO_2007CL_ATECO_2007_MERCECommodity Nace 20072PAESE_PARTNERCL_ISOGeopolitics3ITTER107CL_ITTER107Territory4TIPO_DATOCL_TIPO_DATO12Data type 12Now, each dimension can have a few possible values.istatapiprovides
a quick method to analyze these values and print their English
descriptions.dimension="TIPO_DATO"#use "dimension" column from aboveds.get_dimension_values(dimension)values_idsvalues_description0EVexport - value (euro)1TBVtrade balance - value (euro)2ISAVimport - seasonally adjusted value - world based model (millions of euro)3ESAVexport - seasonally adjusted value - world based model (millions of euro)4TBSAVtrade balance - seasonally adjusted value  -world based model (millions of euro)5IVimport - value (euro)If we do not filter any of our variables, the data will just include all
the possible values in the dataset. This could result in too much data
that would slow our code and make it difficult to analyze. Thus, we need
to filter our dataset. To do so, we can simply use thevalues_idsthat
we found using the functionget_dimension_valuesin the cell above.Note: Make sure to pass the names of the dimensions in lower case
letters as arguments of theset_filterfunction. If you want to filter
for multiple values, simply pass them as lists.freq="M"#monthly frequencytipo_dato=["ISAV","ESAV"]#imports and exports seasonally adjusted datapaese_partner="WORLD"#trade with all countriesds.set_filters(freq=freq,tipo_dato=tipo_dato,paese_partner=paese_partner)Having set our filters, we can now finally retrieve the data by simply
passing ourDataSetinstance to the functionget_data.
It will return a pandas DataFrame with all the data that we requested.
The data will be already sorted by datetimetrade_df=retrieval.get_data(ds)trade_df.head()DATAFLOWFREQMERCE_ATECO_2007PAESE_PARTNERITTER107TIPO_DATOTIME_PERIODOBS_VALUEBREAKCONF_STATUSOBS_PRE_BREAKOBS_STATUSBASE_PERUNIT_MEASUNIT_MULTMETADATA_ENMETADATA_IT0IT1:139_176(1.0)M10WORLDITTOTESAV1993-01-0110775NaNNaNNaNNaNNaNNaNNaNNaNNaN360IT1:139_176(1.0)M10WORLDITTOTISAV1993-01-019229NaNNaNNaNNaNNaNNaNNaNNaNNaN364IT1:139_176(1.0)M10WORLDITTOTISAV1993-02-0110014NaNNaNNaNNaNNaNNaNNaNNaNNaN4IT1:139_176(1.0)M10WORLDITTOTESAV1993-02-0110692NaNNaNNaNNaNNaNNaNNaNNaNNaN365IT1:139_176(1.0)M10WORLDITTOTISAV1993-03-019951NaNNaNNaNNaNNaNNaNNaNNaNNaNNow that we have our data, we can do whatever we want with it. For
example, we can plot the data after having it cleaned up a bit. You are
free to make your own analysis!# set matplotlib themesplt.style.use('fivethirtyeight')plt.rcParams['figure.figsize']=[16,5]#fiveThirtyEight palettecolors=['#30a2da','#fc4f30','#e5ae38','#6d904f','#8b8b8b']# calculate moving averages for the plottrade_df["MA_3"]=trade_df.groupby("TIPO_DATO")["OBS_VALUE"].transform(lambdax:x.rolling(window=3).mean())#replace the "TIPO_DATO" column values with more meaningful labelstrade_df["TIPO_DATO"]=trade_df["TIPO_DATO"].replace({"ISAV":"Imports","ESAV":"Exports"})# Plot the dataafter_2013=trade_df["TIME_PERIOD"]>="2013"is_ESAV=trade_df["TIPO_DATO"]=="Exports"is_ISAV=trade_df["TIPO_DATO"]=="Imports"exports=trade_df[is_ESAV&after_2013].rename(columns={"OBS_VALUE":"Exports","MA_3":"Exports - three months moving average"})imports=trade_df[is_ISAV&after_2013].rename(columns={"OBS_VALUE":"Imports","MA_3":"Imports - three months moving average"})plt.plot("TIME_PERIOD","Exports",data=exports,marker="",linestyle="dashed",color=colors[0],linewidth=1)plt.plot("TIME_PERIOD","Imports",data=imports,marker="",linestyle="dashed",color=colors[1],linewidth=1)plt.plot("TIME_PERIOD","Exports - three months moving average",data=exports,color=colors[0],linewidth=2)plt.plot("TIME_PERIOD","Imports - three months moving average",data=imports,marker="",color=colors[1],linewidth=2)# add a titleplt.title("Italy's trade with the world")# add a label to the x axisplt.xlabel("Year")# turn y scale from millions to billions (divide by a 1000), and add a labelplt.ylabel("Value in billions of euros")plt.gca().yaxis.set_major_formatter(plt.FuncFormatter(lambdax,loc:"{:,}".format(int(x/1000))))With just a few lines of code, we have been able to retrieve data from
ISTAT and make a simple plot. This is just a simple example of what you
can do withistatapi. You can find more examples in the_examplesfolder. Enjoy!