Converting multi layered xml files to dataframes in python using xmltree | by Kanishka Narayan | Towards Data Science

I have found that one of the biggest advantages of using any effective programming language is that the language helps in breaking down abstract data structures into simple data points that a programmer can that transform into any convenient structure. Structure of data is merely the form that presents one possible arrangement of an unchangingessence.Thus, when you look at an excel file as a programmer, you realize that excel is merely the ‘form’ of data that presents one possible structure of the ‘essence’ of data which are merely points.
One of the structures of data that allows for large datasets to be presented in a multi-layered format is the Extensible Markup Language or more popularly referred to as XML. Essentially, xml is a markup language that makes data both human and machine readable. Needless to say, this presents tremendous advantages. This also allows for nesting of variables within a dataset i.e. presentation of data as layers. For example, an xml document would look something like below,
#A sample xml file<?xml version="1.0" encoding="UTF-8"?><scenario><world><region name="USA"><AgSupplySector name="Corn"><AgSupplySubsector name="Corn_NelsonR"><AgProductionTechnology name="Corn_NelsonR_IRR_hi"><period year="2015"><agProdChange>0.0043</agProdChange></period></AgProductionTechnology></AgSupplySubsector></AgSupplySector></region></world></scenario>
Thus, you can see that the various attributes of the data are presented as layers. However, making this data human readable can be a challenge especially when working with huge xml files.
The question- In this article, I present an easily modifiable python script that parses through an xml with 6 layers and presents the data in a simple dataframe ideal for analysis. The xml file used in this example is a dataset that presents greenhouse gas emissions by (these are our layers) Country, Crop, Basin, Technology ,Year and Gas type. I have made use of the xmltree package in python for this example. The sample dataset “ag_prodchange_ref_IRR_MGMT.xml” is a part of the data system for the GCAM model which can be accessed here. The file contains data for 31 regions, for approximately 2000 basins, 12 crop types and 15 gas types.
We want to convert the xml file into a simple format that looks like below,
So, the easiest way to approach this question is to consider writing a layered for loop to parse through the xmls to receive our output and then dumping the data into a dataframe in the last section of the for loop. So, let’s get started.
First, let’s import our packages and read in the data using xmltree. We will need pandas as well since we will be working with dataframes. You will see that we are initially parsing the xml object using the parse function within the xml tree and then we are dumping the entire tree to a variable called root.
import xml.etree.cElementTree as etimport pandas as pdtree=et.parse('all_aglu_emissions.xml')root=tree.getroot()
Now, let’s create some empty lists for our layers. As mentioned above, there are 6 layers to consider, so we will create empty lists corresponding to each layer.
Year=[]Gas=[]Value=[]Technology=[]Crop=[]Country=[]
Let the for loop begin! Think of this as a decision tree where you are where you want to arrive at a particular value. So, you want to look through a particular region, in a particular supply sector, in a particular technology type, in a particular year gas to find a particular value. The below diagram describes the for loop.
xml tree conveniently allows the user to parse through elements as iterations. Now, the challenging part is creating an internal memory so that loop remembers what the contents of each section are. Otherwise, we may end up in a situation eloquently described by Ariadne in Inception,
“ Wait, whose subconscious are we going into, exactly?”
So, at each stage of the loop we want to save our iteration to a variable which will be the basis for the loop at the next stage. Our first stage is the ‘region’. We will save the results from this stage to a variable called ‘root 1’. Then we will iterate over this new variable in the second stage and save the results to another temporary variable called ‘root 2’. So, the code will be as follows for the first two levels,
for reg in root.iter('region'):    root1=et.Element('root')    root1=reg    for supply in root1.iter('AgSupplySector'):        root2=et.Element('root')        root2=(supply)
Note that et.Element(‘root’) creates an empty xml object to store our results in. Also, note that the for variables you are declaring are important, so keep track of the ‘x’ and the ‘reg’ in the above. We will need them later to extract our attributes. So, extending this structure for 4 more levels we get,
for x in root.iter('region'):    root1=et.Element('root')    root1=x    for supply in root1.iter('AgSupplySector'):        root2=et.Element('root')        print(supply)        root2=(supply)        for tech in root2.iter('AgProductionTechnology'):            root3 = et.Element('root')            root3=(tech)            for yr in root3.iter('period'):                root4 = et.Element('root')                root4=yr                for gas in root4.iter('Non-CO2'):                    root5 = et.Element('root')                    root5=gas
Now, for the last level, we want the actual values stored within the xml files. These can be extracted in xmltree using the ‘attrib’ function which will extract specific attributes. So, going back to our xml structure, each layer contains specific attributes. For example in the line below, the section region, contains an attribute called ‘name’ that is set to USA.
<region name="USA">
So to extract the name we will write the following line,
reg.attrib['name']
So, by that logic, lets extract all attributes from the above, and let’s assign them to the empty lists we generated in the first step. So, the code becomes,
Technology.append(tech.attrib['name'])Value.append(em.text)Gas.append(gas.attrib['name'])Year.append(yr.attrib['year'])Crop.append(supply.attrib['name'])Country.append(x.attrib['name'])
Note that the value does not have an attribute function like the other lists. We directly called an object called ‘text’. This is because the value is stored within a layer without a specific attribute. So we can directly access the object.
<input-emissions>0.0043</input-emissions>
So, our code with the above chunk becomes,
for x in root.iter('region'):    root1=et.Element('root')    root1=x    for supply in root1.iter('AgSupplySector'):        root2=et.Element('root')        print(supply)        root2=(supply)        for tech in root2.iter('AgProductionTechnology'):            root3 = et.Element('root')            root3=(tech)            for yr in root3.iter('period'):                root4 = et.Element('root')                root4=yr                for gas in root4.iter('Non-CO2'):                    root5 = et.Element('root')                    root5=gas                    for em in root5.iter('input-emissions'):                        Technology.append(tech.attrib['name'])                        Value.append(em.text)                        Gas.append(gas.attrib['name'])                        Year.append(yr.attrib['year'])                        Crop.append(supply.attrib['name'])                        Country.append(x.attrib['name'])
Now, the easy part. Let’s create a dataframe in pandas using our populated lists.
df = pd.DataFrame({'Year': Year,'Gas':Gas,'Value':Value,'Technology':Technology,'Country':Country,'Crop':Crop })
If we print out the heads of this dataframe, we should get,
Year        Gas         Value    Technology Country  Crop0  1975  SO2_1_AWB   3.98749e-05  Corn_NelsonR     USA  Corn1  1975    NOx_AWB   0.000285263  Corn_NelsonR     USA  Corn2  1975     CO_AWB  0.0089533275  Corn_NelsonR     USA  Corn3  1975  NMVOC_AWB  0.0005259348  Corn_NelsonR     USA  Corn4  1975    CH4_AWB  0.0002027923  Corn_NelsonR     USA  Corn
There you go! Just to recap, There are a few principles to keep in mind when converting xmls to dataframe,
Always keep in mind the layers of the xml itself including attributes in each levelWhen writing the for loop, make sure that you are saving the parsed data to temporary variables at each stage.Write out the data to empty lists in the last step of the loop.Keep in mind that the structure is the form of the data. You are modifying the form while maintaining its essence.
Always keep in mind the layers of the xml itself including attributes in each level
When writing the for loop, make sure that you are saving the parsed data to temporary variables at each stage.
Write out the data to empty lists in the last step of the loop.
Keep in mind that the structure is the form of the data. You are modifying the form while maintaining its essence.
The code for the above can be found here- https://github.com/kanishkan91/ConvertXMLtoDataframepy
The base xml used in the example above can be found here- https://github.com/kanishkan91/ConvertXMLtoDataframepy/blob/master/all_aglu_emissions.xml
As always, any and all feedback is welcome.