Still Using the OS Module in Python? This Alternative is Remarkably Better | by Dario Radečić | Towards Data Science

File and folder management with Python’s os module is a nightmare. Yet, it’s an essential part of every data science workflow. Saving reports, reading configuration files, you name it — there’s no way around it.
Picture this — you spend weeks building an API around your model, and it works flawlessly, at least on your machine. Once deployed, it’s a whole different story. Your API fails in unexpected places or even won’t run, as absolute paths you’ve hardcoded simply don’t exist.
There’s a no-brainer solution. The pathlib library comes by default with Python 3.4 and above. It’s by far the most humane way to work with files, folder, and their connection in your apps. The best thing is — today you’ll learn all about it. I’m not saying that the os module serves no purpose, just that pathlib is far superior in file and folder management.
To start, create a new Python file and import the pathlib library:
import pathlib
You’ll need to be inside a .py file for some functionalities to work. For example, you won’t have access to __file__ property in Jupyter Notebooks. Everything else should work perfectly.
Here’s what you’ll learn today:
· Get a path to the current Python file· Get a path to the current working directory· Get a first parent folder path· Get an Nth parent folder path· Join paths· Create a directory if it doesn’t exist· Create files· Check if the path is a folder· Check if the path is a file· Get the name of the file· Get the file extension· Iterate over files in a folder
Every so often, you need an absolute path to your working directory with the included Python file name. You can obtain it quickly with pathlib. Keep in mind, this command won’t work in Jupyter Notebooks, as you can’t access the __file__ property there. Skip this section if you’re a Notebook user.
Anyhow, here’s how to get an absolute path + a file name of the Python script:
curr_file = pathlib.Path(__file__)print(curr_file)
Here’s how it looks on my machine:
>>> /Users/dradecic/Desktop/pathlib_guide/python_pathlib.py
Easy, right? Right.
There’s an easier solution if you don’t need the file name.
This one is equivalent to executing pwd in the Unix shell. It will return a path to the directory you’re currently in, or where the running script is located.
Here’s how to use it:
cwd = pathlib.Path.cwd()print(cwd)
And here’s what it prints on my machine:
>>> /Users/dradecic/Desktop/pathlib_guide
No file name, as you can see.
But what if you need to access a file in a parent folder? That would be Desktop in my case. Let’s cover that next.
This one is easy. You only need to access the parent property of a current working directory. Here’s how it’s done:
one_above = pathlib.Path.cwd().parent print(one_above)
Here’s what it prints on my machine:
>>> /Users/dradecic/Desktop
Great! But what if one parent folder isn’t enough? Let’s see what your options are.
You have options. The first one is to call access the parent property multiple times like a crazy person. Here’s an example:
mul_above = pathlib.Path.cwd().parent.parent.parent
The easier option is to access the parents property array and index it. For instance, here’s how you’d get a path to the second parent folder path:
mul_above = pathlib.Path.cwd().parents[1]print(mul_above)
Here are the results:
>>> /Users/dradecic
Array indexing starts at 0, so accessing parents[1] gets you to the second parent folder.
You now have enough knowledge to start joining paths. Let’s see how next.
Let’s say that a folder with sales reports is located two directories above your current location, and the report is called summer-sales.csv. Is there a way to access it with an absolute path?
Of course there is.
You already know how to access the Nth parent folder path. You’ll extend that functionality by calling joinpath() and providing the report name as an argument:
tgt_path = pathlib.Path.cwd().parents[1]tgt_fname = 'summer-sales.csv'print(tgt_path.joinpath(tgt_fname))
Here’s what it prints on my machine:
>>> /Users/dradecic/summer-sales.csv
The joinpath() function is probably the one I use the most. It’s super useful.
If I had a dollar every time something in production failed because I forgot to create a directory... It’s a pretty common mistake, and pathlib allows you to get around it without too much hassle.
Let’s say you want to store sales reports in a reports folder located in your current working directory. You’ll have to create that folder before you can store files in it. You should create it only if it doesn’t exist.
In a nutshell — use mkdir() to create a folder and exists() to check if a folder already exists.
Here’s the complete code snippet:
tgt_path = pathlib.Path.cwd().joinpath('reports')if not tgt_path.exists():    tgt_path.mkdir()
Executing the above code will, you’ve guessed it, create a reports folder:
Neat. Let’s see how you can create files in that folder next.
You’d typically save reports through some third-party libraries. Nevertheless, you can also use pathlib to create empty files of any type.
Here’s how to create both a CSV and TXT file inside the reports folder:
tgt_path = pathlib.Path.cwd().joinpath('reports')tgt_path.joinpath('summer-sales.csv').touch(exist_ok=True)tgt_path.joinpath('winter-sales.txt').touch(exist_ok=True)
The exist_ok=True parameter tells Python to overwrite a file if it already exists.
Let’s see if the files were created:
Works like a charm.
If you want to check if a given path is a folder, look no further than the is_dir() function. It returns a boolean.
The following example uses the mentioned function both on a folder and on a file:
tgt_path = pathlib.Path.cwd().joinpath('reports')print(tgt_path.is_dir())print(tgt_path.joinpath('summer-sales.csv').is_dir())
Here’s what you should see printed:
>>> True>>> False
And that’s all there is to it!
Similarly to the previous example, you can use the is_file() function to check if a given path results in a file.
The example below uses it both on a folder and on a file:
tgt_path = pathlib.Path.cwd().joinpath('reports')print(tgt_path.is_dir())print(tgt_path.joinpath('summer-sales.csv').is_dir())
As you would imagine, you’ll get the exact opposite results this time:
>>> False>>> True
Let’s explore a few more useful functions before calling it a day.
You can access the name property if you need to extract a file name from an absolute path.
Here’s a simple and not so useful example. It prints the file name of our summer-sales.csv file:
tgt_path = pathlib.Path.cwd().joinpath('reports/summer-sales.csv')print(tgt_path.name)
Here’s what you should see in the console:
>>> summer-sales.csv
Not much to it.
Sometimes all you need is a file extension. Maybe you want to process different file types differently, and you don’t care too much about the file name. The suffix property has you covered.
Here’s how you can get the file extension from the same summer-sales.csv file:
tgt_path = pathlib.Path.cwd().joinpath('reports/summer-sales.csv')print(tgt_path.suffix)
Here’s what you should see printed out:
>>> .csv
And finally, let’s cover iteration.
Let’s say you have a bunch of CSV reports in a single directory, and want to process them one by one. The iterdir() function is all you need.
The entire process couldn’t be any simpler:
tgt_path = pathlib.Path.cwd().joinpath('reports')for file in tgt_path.iterdir():    print(file)
Here’s what it prints on my machine:
>>> /Users/dradecic/Desktop/pathlib_guide/reports/summer-sales.csv    /Users/dradecic/Desktop/pathlib_guide/reports/winter-sales.txt
And that does it for today. You now have everything needed to never cause a stupid production mistake again.
What are your thoughts on pathlib? Is it your favourite file and folder management library, or are you a fan of something else? Let me know in the comment section below.
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.
medium.com
Follow me on Medium for more stories like this
Sign up for my newsletter
Connect on LinkedIn