Automate Data Imports with Python | by Eser Saygın | Towards Data Science

Data import plays a pivotal role in data science. Knowing how to do it prevents the possibility of failure of the model. To achieve this, various methods, which differ depends on the data types (e.g. .csv, .txt, .json), are possible.
Getting the data into the python environment one of the essential steps in building a model. In some circumstances, we have to load data without using API or SQL databases. In this tutorial, I will cover how to automate the data loading in Python. Compiling this might be a bit tricky. Besides, it’s possible to find a straightforward way to do it. But I want to show you my convoluted way. So let’s get started.
Before we start, let’s import the requisite libraries using the following commands. In due course, I will explain why we need these modules.
import pandas as pdimport osfrom os import listdirfrom os.path import isfile,joinimport csv
The below image shows the data folder that contains several data file types. Keep in mind that file names. We will use them to create variable names.
Routinely, I‘d’ rather assign my working directory as a file path. This provides me to combine file path and folder names swiftly.
#assign working directory to variablefile_path=os.getcwd()#setting path for data folderdata_path=file_path+'\\'+'data'
This step is crucial because we’re going to generate variable names from the file names. There are two different codes to extract file names. The first one is with extensions and the other one without extensions. We use “listdir” and “isfile” methods to get the file names. The “listdir” method is to get the list of all files in the specified directory. In our code, the directory is “data_path” that we assigned our working directory shortly before. So, the “isfile” method to check whether the file is existing in that folder.
# extracting file names without extensionsfile_names=[".".join(f.split(".")[:-1]) for f in listdir(data_path) if isfile (join(data_path,f))] # extracting file names with extensionsfull_file_names=[f for f in listdir(data_path) if isfile (join(data_path,f))]#checking file names and extensionsprint(file_names)print(full_file_names)
So far, everything is working fine. Now it is time to detect which delimiter is used in the files as the separator. To do that, we need the “sniff” method provided by the “Sniffer” class. This class is used to deduce the format of a csv file.
#creating a dictionary with file names and their extensionsdict_del={}delimit=[]for i in range(len(file_names)):    delimit.append(i)t=0for s in full_file_names:    with open(data_path+'\\'+s, newline='', encoding="utf8") as csvfile:        dialect = csv.Sniffer().sniff(csvfile.read(1024))        csvfile.seek(0)        reader = csv.reader(csvfile, dialect)        delimit[t]=dialect.delimiter        dict_del[s]=delimit[t]        t+=1
As you notice, we built a dictionary and a list. The list that is named “delimit” is used to store the delimiters. We use a for loop syntax to easily replace the random values with extracted delimiters via indexing. And the dictionary contains file names (as keys) and the delimiters (as values). Let’s check what we have;
#checking dictionary and delimitersprint(dict_del)print(delimit)
It is time to use the “globals()” method to create variables. Some sources (link) say it is dangerous due to it is global. This method can cause confusion, so please read carefully!
#creating global variables depend on file namesz=0files_ready=[]for f,r in zip(file_names,full_file_names):    globals()[f]=pd.read_csv(data_path+'\\'+r,sep=dict_del[r])            files_ready.append(f)    z+=1
If you notice, variables are created separately for each file name. We are done! Let’s make sure it works.
#checking data filesprint(files_ready)bank.head()deacit.head()netflix_titles.head()
Voila! We loaded all files and assigned them a variable.
Lastly, let’s perform a double-check with a new data file. I just saved a .txt file. (PSA)
And I wonder if it works or not.
#checking the new fileprint(files_ready)PSA.head()
Everything seems to be working alright.
As I said before, I don’t claim that these are the only and easy options to achieve this goal. This tutorial is the way I created to automate the process. Feel free to develop the code. :)
Pros:
Just run the code and load several files in a short time.
You don’t need to think about variable names.
You don’t miss out on any new files.
Cons:
I tested for the codes only csv and txt files.
There could be an easier way. Please go check it out :)
Useful Links
To learn how to use split() function (Link)
os.path.join and os.path.isfile (Link)
The data import is grunt work at its essence, but it is still the essential component of building a data model. There are many more methods you can find and use.
Building your method will improve your coding skills. I hope this guide was useful to you. Thanks for reading until the end.