Automate Data Imports with Python
By: Eser Saygın
Tags: Jupyter Notebook, Pandas, Data Import And Export, Data Science, Python

Getting Started

Before we start, let’s import the requisite libraries using the following commands. In due course, I will explain why we need these modules.

import pandas as pd

import os

from os import listdir

from os.path import isfile,join

import csv

The below image shows the data folder that contains several data file types. Keep in mind that file names. We will use them to create variable names.

Routinely, I‘d’ rather assign my working directory as a file path. This provides me to combine file path and folder names swiftly.

#assign working directory to variable

file_path=os.getcwd() #setting path for data folder

data_path=file_path+'\\'+'data'

Extracting File Names

This step is crucial because we’re going to generate variable names from the file names. There are two different codes to extract file names. The first one is with extensions and the other one without extensions. We use “listdir” and “isfile” methods to get the file names. The “listdir” method is to get the list of all files in the specified directory. In our code, the directory is “data_path” that we assigned our working directory shortly before. So, the “isfile” method to check whether the file is existing in that folder.

# extracting file names without extensions

file_names=[".".join(f.split(".")[:-1]) for f in listdir(data_path) if isfile (join(data_path,f))] # extracting file names with extensions

full_file_names=[f for f in listdir(data_path) if isfile (join(data_path,f))] #checking file names and extensions

print(file_names)

print(full_file_names)

Getting Delimiters

So far, everything is working fine. Now it is time to detect which delimiter is used in the files as the separator. To do that, we need the “sniff” method provided by the “Sniffer” class. This class is used to deduce the format of a csv file.

#creating a dictionary with file names and their extensions

dict_del={}

delimit=[]

for i in range(len(file_names)):

delimit.append(i)

t=0

for s in full_file_names:

with open(data_path+'\\'+s, newline='', encoding="utf8") as csvfile:

dialect = csv.Sniffer().sniff(csvfile.read(1024))

csvfile.seek(0)

reader = csv.reader(csvfile, dialect)

delimit[t]=dialect.delimiter

dict_del[s]=delimit[t]

t+=1

As you notice, we built a dictionary and a list. The list that is named “delimit” is used to store the delimiters. We use a for loop syntax to easily replace the random values with extracted delimiters via indexing. And the dictionary contains file names (as keys) and the delimiters (as values). Let’s check what we have;

#checking dictionary and delimiters

print(dict_del)

print(delimit)

A Risky Method: globals()

It is time to use the “globals()” method to create variables. Some sources (link) say it is dangerous due to it is global. This method can cause confusion, so please read carefully!

#creating global variables depend on file names

z=0

files_ready=[]

for f,r in zip(file_names,full_file_names):

globals()[f]=pd.read_csv(data_path+'\\'+r,sep=dict_del[r])

files_ready.append(f)

z+=1

If you notice, variables are created separately for each file name. We are done! Let’s make sure it works.

#checking data files

print(files_ready)

bank.head()

deacit.head()

netflix_titles.head()

Voila! We loaded all files and assigned them a variable.

Lastly, let’s perform a double-check with a new data file. I just saved a .txt file. (PSA)

And I wonder if it works or not.

#checking the new file

print(files_ready)

PSA.head()

Everything seems to be working alright.