Parsing XML Data in Python. Storing XML Data in a Pandas Dataframe | by Sadrach Pierre, Ph.D. | Towards Data Science

Extensible Mark up Language (XML) is a mark up language that encodes data in a human and machine readable format. XML is used in a variety of programs for structuring, storing and transmitting data. Python contains several interfaces for processing XML data. In this post, we will discuss how to use the ‘ElementTree’ module in the python ‘xml’ library to parse XML data and store the data in a Pandas data frame.
Let’s get started!
For our purposes we will be using a sample ‘xml’ file, ‘books.xml’ ,which can be found here. The file contains information about a variety of books, such as titles, author names, and prices.
To start, let’s import ‘parse’ from the ‘ElementTree’ module in the python ‘xml’ library:
from xml.etree.ElementTree import parse
Now, let’s take a look at the file tags in ‘books.xml’:
We can define a parsed ‘XML’ document object by passing the file name into the ‘parse()’ method:
document = parse('books.xml')
If we print the object we see that we have an ‘ElementTree’ object at a specified memory address:
print(document)
Let’s take a look at the methods and attributes available to this object using the built-in ‘dir()’ method:
print(dir(document))
Let’s use the method ‘iterfind()’ to return a generator that we can use iterate over in a ‘for-loop’. We will need specify a path argument in the ‘iterfind()’ method. Let’s select the ‘book’ path:
for item in document.iterfind(‘book’):    print(item)
We see that we have several ‘Element book’ objects stored at various memory addresses. We can extract the information from these objects using the ‘findtext()’ method. Let’s extract the information in the ‘author’ tags:
for item in document.iterfind('book'):    print(item.findtext('author'))
We can also extract titles:
for item in document.iterfind('book'):    print(item.findtext('title'))
Let’s also look at prices:
for item in document.iterfind('book'):    print(item.findtext('price'))
Next, we can initialize lists that we can use to store these values:
author = []title = []price = []
And within the for-loop we can append the values:
for item in document.iterfind('book'):    author.append(item.findtext('author'))    title.append(item.findtext('title'))    price.append(item.findtext('price'))
We can then store these lists in a data frame. Let’s first import the Pandas library:
import pandas as pd 
Next, let’s define a data frame that contains the ‘title’, ‘author’ and ‘price’ for each book:
df = pd.DataFrame({'title': title, 'author':author, 'price':price})
Next, let’s print the resulting data frame:
print(df)
We can add the ‘genre’, ‘publish_date’ and ‘description’ to the data frame as well:
genre = []description = []publish_date = []for item in document.iterfind('book'):    ...    genre.append(item.findtext('genre'))    description.append(item.findtext('description'))    publish_date.append(item.findtext('publish_date'))
Now that we have all of the information in a data frame we can do things like convert the ‘price’ strings into ‘float’ and calculate the mean of the ‘price’ column:
df['price'] = df['price'].astype(float)print("Mean price: ", df['price'].mean())
Let’s also convert the ‘publish_date’ into a ‘datetime’ object and pull year, month and day values:
df['publish_date'] = pd.to_datetime(df['publish_date'])df['year'] = df['publish_date'].dt.yeardf['month'] = df['publish_date'].dt.monthdf['day'] = df['publish_date'].dt.dayprint(df.head())
We can also use the ‘Counter()’ method from the collections module to look at the distribution in authors and genres:
from collections import Counterprint(Counter(df['author']))print(Counter(df['genre']))
I’ll stop here but feel free to play around with the data and code yourself.
To summarize, in this post we discussed how to parse XML data using the ‘xml’ library in python. We showed how to use the ‘iterfind()’ method to define a generator object that we can iterate over in a ‘for-loop’. We also showed how to access element tag information using the ‘findtext()’ method. We then stored the XML information in lists which we used to define a Pandas data frame. I hope you found this post useful/interesting. The data and code from this post are available on GitHub. Thank you for reading!