#4 Data Engineering — EXTRACT DATA from JSON and XML files
By: Sakshi Agarwal
Tags: Extract, Data Science, Xml, Data Engineering, Json

EXTRACT XML

What is XML?

XML (Extensible Markup Language) is very similar to HTML at least in terms of formatting. The main difference between the two is that HTML has pre-defined tags that are standardized. In XML, tags can be tailored to the data set. Here is what this same data would look like as XML.

<ENTRY>

<ID>P162228</ID>

<REGIONNAME>Other</REGIONNAME>

<COUNTRYNAME>World;World</COUNTRYNAME>

<PRODLINE>RE</PRODLINE>

<LENDINGINSTR>Investment Project Financing</LENDINGINSTR>

</ENTRY>

<ENTRY>

<ID>P163962</ID>

<REGIONNAME>Africa</REGIONNAME>

<COUNTRYNAME>Democratic Republic of the Congo;Democratic Republic of the Congo</COUNTRYNAME>

<PRODLINE>PE</PRODLINE>

<LENDINGINSTR>Investment Project Financing</LENDINGINSTR>

</ENTRY>

<ENTRY>

<ID>P167672</ID>

<REGIONNAME>South Asia</REGIONNAME>

<COUNTRYNAME>People's Republic of Bangladesh;People's Republic of Bangladesh</COUNTRYNAME>

<PRODLINE>PE</PRODLINE>

<LENDINGINSTR>Investment Project Financing</LENDINGINSTR>

</ENTRY>

XML is falling out of favor especially because JSON tends to be easier to navigate; however, you still might come across XML data. The World Bank API, for example, can return either XML data or JSON data. From a data perspective, the process for handling HTML and XML data is essentially the same.

# print out the first 15 lines of the xml file

print_lines(15, 'population_data.xml')

-----------------------------------------------------------------

OUTPUT: <?xml version="1.0" encoding="utf-8"?>



<Root xmlns:wb="http://www.worldbank.org">



<data>



<record>



<field name="Country or Area" key="ABW">Aruba</field>



<field name="Item" key="SP.POP.TOTL">Population, total</field>



<field name="Year">1960</field>



<field name="Value">54211</field>



</record>



<record>



<field name="Country or Area" key="ABW">Aruba</field>



<field name="Item" key="SP.POP.TOTL">Population, total</field>



<field name="Year">1961</field>



<field name="Value">55438</field>



</record>

XML is formatted with tags having values inside the tags. XML is not as easy to navigate as JSON. Pandas cannot read in XML directly. One reason is that tag names are user-defined. Every XML file might have different formatting. You can imagine why XML has fallen out of favor relative to JSON.

How to read and navigate XML

There is a Python library called BeautifulSoup, which makes reading in and parsing XML data easier. Here is the link to the documentation: Beautiful Soup Documentation

The find() method will find the first place where an XML element occurs. For example, using find(‘record’) will return the first record in the XML file:

<record>

<field name="Country or Area" key="ABW">Aruba</field>

<field name="Item" key="SP.POP.TOTL">Population, total</field>

<field name="Year">1960</field>

<field name="Value">54211</field>

</record>

The find_all() method returns all of the matching tags. So find_all(‘record’) would return all of the elements with the <record> tag.

# import the BeautifulSoup library

from bs4 import BeautifulSoup



# open the population_data.xml file and load into Beautiful Soup

with open("population_data.xml") as fp:

soup = BeautifulSoup(fp, "lxml") # lxml is the Parser type

Navigate through the XML document: