7 Common File System Operations You Can Do With Python | by Frank Andrade | Towards Data Science

One of the coolest things that you can do in Python without installing any third-party library is to perform file system operations such as creating a folder, renaming a file, and working with directories. Although these tasks can be easily done manually, you can automate them with Python code to save some time.
In this article, we’ll see 7 file system operations you can do in Python with the os and Pathlib modules. Each operation includes practical examples so you can understand the difference between these 2 modules.
Knowing the current working directory is fundamental when dealing with paths in a Python script. There are two kinds of paths — relative and absolute paths. An absolute path refers to the same location in a file system relative to the root directory (/), whereas a relative path refers to a specific location in a file system relative to the current working directory.
The relative path gives flexibility to your script, so that’s why it’s sometimes preferred over absolute paths. We can use the current working directory to create a relative path.
For this and all the file system operations listed in this article, we’ll have to import os and Path.
import osIn [1]: os.getcwd()Out [1]: /Users/frank/PycharmProjects/DataScience
As you can see from the code above, I got the directory that contains all my scripts for my data science project. This is my working directory in Pycharm. We can obtain the same directory with the Pathlib library
from pathlib import PathIn [1]: Path.cwd()Out [1]: /Users/frank/PycharmProjects/DataScience
If we print either type(os.getcwd()) or the type(Path.cwd() we can see the main difference between the os and Pathlib modules. The first returns only a string, while the second returns a PosixPath object that can help us do extra operations.
Besides showing the current working directory, we can list all the files that are inside our directory.
In [1]: os.listdir()Out [1]: ['script1.py', 'script2.py', 'script3.py']
In my case, I only have 3 Python scripts in my “Data Science” folder. We can obtain the same results with Path().iterdir(), but we’ll obtain a generator object. To obtain a list directory content from this generator, we use list() as shown in the code below.
In [1]: list(Path().iterdir())Out [1]: [PosixPath('script1.py'), PosixPath('script2.py'), PosixPath('script3.py')]
We can even see the content within a specific folder. For example, my ‘Data Science’ folder has a folder named ‘Dataset.’ We can list the directory content inside the ‘Dataset’ folder with the following code.
# osos.listdir('Dataset')# pathliblist(Path('Dataset').iterdir())
Concatenating strings in Python is as easy as using the “+” sign between two words; however, when it comes to concatenating paths, things get a bit tricky. This happens because Mac OS uses forward slash “/”, while Windows uses backward slash “\” in the directory names, so you need a different approach if you want the Python code to be able to run cross-platform. Fortunately, the os and Pathlib modules can take care of that.
# osos.path.join(os.getcwd(), 'Dataset')# pathlibfrom pathlib import Path, PurePathPurePath.joinpath(Path.cwd(), 'Dataset')
If we print the code written above, we’ll get a path that contains the working directory plus “Dataset.” The Path will look like this /Users/frank/PycharmProjects/DataScience/Dataset. The OS library creates a string, while the Pathlib creates a PosixPath class. However, we haven’t created the “Dataset” folder. To create a directory check the next file system operation.
Although you can create a directory in some seconds, when it comes to task automation, writing some Python code will be the best solution. We can perform this task with both libraries with a slight difference. Let’s create a “Dataset” folder as an example.
# osos.mkdir('Dataset')# pathlibPath('Dataset').mkdir()
If the “Dataset” folder doesn’t exist, the folder will be created. However, Python will throw an error, if we try to create a directory that already exists. That is, the script will break if I already have a folder named Dataset. Fortunately, we can easily control this behavior with Pathlib.
Path('Dataset').mkdir(exist_ok=True)
By adding the exist_ok parameter, we can ignore errors, in case the directory already exists.
Similar to the creation of directories, renaming file names is a task we can automate with a Python script. With the OS library, we can rename files introducing the string name. For example, let’s rename the “Dataset” folder to “Data.”
os.rename('Dataset', 'Data')
We can also do this task with Pathlib. In this case, the target name can be either a string or another path object. Let’s rename the folder to “Dataset”
current_path = Path('Data')target_path = Path('Dataset')Path.rename(current_path, target_path)
You can even rename multiple files with a specific extension using a for loop. For example, I’m going to add the year 2021 in front of each .csv file in my working directory.
for i, file in os.listdir():    if file.endswith('.csv'):        os.rename(file, f'2021_{file}')
If you want to take this one step further, you can use the time module to add the date the script is running to the file name and more!
You can check whether a file/directory exists or not with the OS and Pathlib modules. Let’s see if the folder Dataset exists in my working directory.
# osos.path.exists('Dataset')# pathlibcheck_path = Path('Dataset')check_path.exists()
If you print the code above, Python will return a True value when the folder exists or False when it doesn’t exist. Since I have a folder named “Dataset,” I got True.
Metadata is data that describes other data. For example, the date created, date modified, and file size. Let’s get the absolute path of a test.py script I have in my working directory.
# osos.path.abspath('test.py')# pathlibscript = Path('test.py')script.resolve()
I got the absolute path /Users/frank/PycharmProjects/DataScience/test.py after printing the code above.
With Pathlib we can easily get the stem, suffix of the file, file size, and birthtime.
In [1]: print(script.stem)In [2]: print(script.suffix)In [3]: print(script.stat().st_size)In [4]: print(script.stat().st_birthtime)Out [1]: testOut [2]: .pyOut [3]: 1060Out [4]: 1614190828.6
As you can see the birth time is in timestamp format. In case you want to get a datetime object, run the code below.
from datetime import datetimetimestamp = 1614190828.6dt_object = datetime.fromtimestamp(timestamp)print("dt_object =", dt_object)
That’s it! Now you know 7 file system operations you can do using Python, You can check more operations in the os and pathlib documentations.
Join my email list with 3k+ people to get my Python for Data Science Cheat Sheet I use in all my tutorials (Free PDF)