Unzip, update and zip again XMLs in a folder | by Vaclav Dekanovsky | Towards Data Science

In daily office work, you often have to do something with a bunch of files. Unzip, move, rename, update, zip again, and send or load to FTP. All of these things are simple to do, but very time-consuming of you have to repeat them manually on the tens or hundreds of files.
Luckily you can use programming languages to automate these repetitive tasks. Python is an ideal choice because it’s easy to set up on a computer and with little trial and error even the computing newbie can create a program. Python is a versatile programming language and you can tackle basically any problem with it.
Let’s explore quite a common task in many industries. We will:
Unzip a group of filesUpdate their content — in our example we will change the content of the XML nodeZip the result again — using zipfile and shutil libraries
Unzip a group of files
Update their content — in our example we will change the content of the XML node
Zip the result again — using zipfile and shutil libraries
You can follow up this exercise in the github notebook-Unzip_Update_Zip_full.ipynb.
There are several ways how you can set up python on your computer. I personally enjoy using Anaconda and Jupyter notebook. You download and install the conda package meant to make python environment easy to manage with all possible enhancements and run the notebook, which is an essential part of the package.
In the notebook, you can run a separate cell containing one or more lines of code and immediately see the results. Such an approach is ideal for prototyping. With help from google, stackoverflow, and tutorials for example here on medium you can quickly combine the pieces of code required for what you need to do.
Let’s explore how you would approach a task to unzip a file in python. Just go to google and type “unzip python”. It quickly lists results and you pick, let’s say one of the coding biggest Q/A sites — stackoverflow’s Unzipping files in Python. Within 3 lines of code you learn how to do that:
import zipfilewith zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:    zip_ref.extractall(directory_to_extract_to)
You import the library needed for work with .zip files — zipfile. That allows you to use its functions and methods to unpack the archives and zip them again.
First, you open the archive for reading using zipfile.ZipFile(path_to_zip_file, 'r') and then you extract the content of the package to a directory zip_ref.extractall(directory_to_extract_to).
I used the with notation to ensure that the archive is opened, an action performed, and then the archive is closed so its content is released from the memory. It prevents memory leaks in case you process hundreds or thousands of files.
import zipfile

with zipfile.ZipFile(r"to_process/Test_001_20201027.zip", 'r') as zip_ref:
    zip_ref.extractall("temp")

# using with notation ensure that the file is closed after the processing
zip_ref

<zipfile.ZipFile [closed]>
zip_ref = zipfile.ZipFile(r"to_process/Test_001_20201027.zip", 'r')
zip_ref.extractall("temp")

# if you only assign the file to the variable, it remains in the memory after processing
zip_ref

<zipfile.ZipFile filename='to_process/Test_001_20201027.zip' mode='r'>
# unless you intentionally close it
zip_ref.close()
zip_ref

<zipfile.ZipFile [closed]>
To unzip a file in python you use three lines of code. To unzip a bunch of files you add only a few lines more. You have to identify the zip files in a folder and pass them to the procedure we have described above.
The os library lets you both get the content of a folder os.listdir() and combine the folder and file names using os.path.join(folder,file). Listdir returns both the files and the subfolders. You can add file.endswith(".zip") to identify only the archives.
Alternatively you can use os.path.splitext(). Its second output is the file extension.
Knowing all the files in the folder, you can apply the previous code to unzip them. Again you can iterate over the files and unzip each, or wrap the unzipping into a function which is called within the one line comprehensive notation.
def unzip(folder: str, file: str, folder_to_extract: str) -> list:    """unzips a file in a folder into folder_to_extract    returns a list of files in the zip archive"""    with zipfile.ZipFile(os.path.join(folder,file), 'r') as zip_ref:        zip_ref.extractall(folder_to_extract)        return zip_ref.namelist()# applying a function to the output can be squeezed into the list comprehension[unzip(folder, f, "temp") for f in os.listdir(folder) if f.endswith(".zip")]
import os
import zipfile

folder = "to_process"

# os.listdir returns a list with all files and folders contained in a folder
os.listdir(folder)

['a.xml', 'Test_001_20201027.zip', 'xyz_001_20201029.zip']
# to get only .zips let's use the list notation and endswith function
[f for f in os.listdir(folder) if f.endswith(".zip")]

['Test_001_20201027.zip', 'xyz_001_20201029.zip']
# alternativelly you can use os.path's splitext
[f for f in os.listdir(folder) if os.path.splitext(f)[1] == ".zip"]

['Test_001_20201027.zip', 'xyz_001_20201029.zip']
# the list notations is a comprehensive way how to return a list out of a for loop

# initiate an empty list
output = []

# iterate over the files/subfolders in the folder
for f in os.listdir(folder):
    
    # if the file/folder ends with a string ".zip"
    if f.endswith(".zip"):
        
        # append it to the output
        output.append(f)
        
output

['Test_001_20201027.zip', 'xyz_001_20201029.zip']
# iterate over the identified zipfiles and unzip them
for zip_file in [f for f in os.listdir(folder) if f.endswith(".zip")]:
    with zipfile.ZipFile(os.path.join(folder,zip_file), 'r') as zip_ref:
        zip_ref.extractall("temp")

You can wrap the unzipping into a function which would squeeze the final code into a one-liner
def unzip(folder: str, file: str, folder_to_extract: str) -> list:
    """unzips a file in a folder into folder_to_extract
    returns a list of files in the zip archive"""
    with zipfile.ZipFile(os.path.join(folder,file), 'r') as zip_ref:
        zip_ref.extractall(folder_to_extract)
        return zip_ref.namelist()

# applying a function to the output can be squeezed into the list comprehension
[unzip(folder, f, "temp") for f in os.listdir(folder) if f.endswith(".zip")]

[['a.xml'], ['a.xml']]
Once we have extracted the XML to the temp folder we can work with it. Python contains xml.etree library to process the XML files. In our case we know that the zip archive contained one XML file a.xml, though you can always list all the files in the archive using — zip_ref.namelist().
To update the XML file you have to import the tree library and load the XML document.
from xml.etree import ElementTree as ET# parse the XML document into a variabletree = ET.parse(path)# get the root emelement containing all the XML nodesroot = tree.getroot()
Remember that any variable derived from the tree is a reference. When you update the content of these variables, you update the tree as well.
Let’s say we want to update the <id>...</id> node in our XML. We have to first locate it. There are several methods in the etree library for locating XML nodes.
.find() located the first appearance of an XML element
.findall() list all the appearances of the subelement
Both methods accept the name of the element or you can find it using xpath. Xpath is kind of the address inside the XML to locate the desired element. Once the node is found use, it’s .text argument to get the value written between the tags.
# in the <data> find the <id>...</id> node and show its content (.text)id = root.find("id").text
Our Id has three parts split by the “_” underscore:
Prefix
Version
Date
One of the XML has the id — test_001_20201026. .text returns the values as a string and you can apply .split("_") function to break the text into the details.
[In]:   split_id = id.split("_")  print(split_id)[Out]: ["test","001","20201026"]
The output is a python list, which can be easily updated element-wise. First we change the prefix split_id[0] = new_prefix. Then we update the version.
split_id[0] = new_prefixsplit_id[1] = "{:03d}".format(new_version) 
We have used the "{:03d}".format(new_version) to add the leading zeros up to 3 characters, if needed to turn 1 to 001.
Updated values can be combined together into single string by .join("_") and the original XML component can be updated with it:
root.find("id").text = "_".join(split_id)
As I’ve mentioned any update to a variable derived from the tree updates the tree as well. We have achieved our desired update of the XML and we only need to export it using tree.write(output_path).
Let’s look on the complete code of the XML update:
Now the updated a.xml lies in the temp folder. We only need to zip it and place this archive to processed folder. Let’s first create the output folder, if it doesn’t exist. We will use the Path from pathlib library.
# create the output folder if it doesn't existsPath(output_folder).mkdir(parents=True, exist_ok=True)
Then we pack the XML into the zip archive using zipfile lib. Now we have to open the archive for writing 'w'. Then we use the reference to this archive and .write(path_to_file, name_of_this_file).
If you don’t specify the second parameter name_of_the_zipped_file then whole path including the folders and subfolders will be zipped.
If you don’t know the file name, os library can help you with its os.path.basename() function.
with zipfile.ZipFile(output_path, 'w') as myzip:    myzip.write(path_to_processed_xml, os.path.basename(path_to_processed_xml))
Sometimes though you need to pack folders and files as well. In our example Folder_001_20201101.zip contains a file and another files in a folder.
folder_A/a.xml
efg.txt
In such a case it would a bit too completed to use zipfile, but you have an alternative in shutil.make_archive(output_folder, format, folder_to_pack). This method only archives the content of the folder_to_pack and places the result to the output_folder. You can choose from the zip, tar, gztar, bztar, xztar formats.
Now we know all the components of the process:
unzipped multiple archives in a folder into a temp folderupdated each of the unzipped filesand zipped this updated file into an archive again
unzipped multiple archives in a folder into a temp folder
updated each of the unzipped files
and zipped this updated file into an archive again
In this final exercise, we must do a few little tweaks. We will use the name of the original archive, update it as well as we update the content of all the contained XMLs. The reason is that there can be two or more XMLs in the archive each with a different ids and we zip all of them back into the updated zip archive (which cannot have 2 different names).
The complete code, can be found in this notebook — Unzip_Update_Zip_full.ipynb
In this simple tutorial, you have seen the power of python (or any other programming language). Within a few lines of codes, you can automate tasks which would take a lot of time if done manually. I use this principle when deciding whether it’s worth to spend time coding:
If you do something small once, do it manually. If it’s big, you repeat the same operation on many items or you think you will have to do it in the future — automate with a programming language.
The operation in this example can have many variations. You might want to update a different XML node. The zip contains CSV files maybe. You rather update the date and not the prefix and version. You might need to modify the input — e.g. increase the version by 1 or add a week to the date.
All that is possible with only small changes to the code. You can also review the complete example in the jupyter notebook on Github.
I hope you like this tutorial about zipping and XML processing and that you like what you can achieve with python. The possibilities are endless.
Other articles:* Plotly Express complete tutorial* Very illustrative highlighted line chart* Plotly Histogram - Complete Guide* Everything you wanted to know about Kfold train-test split* How to turn a list of addreses into a map