Pandas DataFrame: Playing with CSV files | by Deepak K Gupta (DAKSH) | Towards Data Science

CSV stands for Comma Separated Values, A popular way of representing and storing tabular, column oriented data in a persistent storage
Pandas DataFrames is generally used for representing Excel Like Data In-Memory. In all probability, most of the time, we’re going to load the data from a persistent storage, which could be a DataBase or a CSV file.
In this post, we’re going to see how we can load, store and play with CSV files using Pandas DataFrame
I’ve already written a detailed post titled Pandas DataFrame : A Lightweight Intro . If you’re not comfortable with Pandas DataFrame, I’ll highly recommend you to have a look at this post before continuing with this one.
In a nutshell, Pandas DataFrame is nothing but an in-memory representation of excel like data. For example,
This data can also be represented using Python Dictionary as
my_dict = { 'name' : ["a", "b", "c", "d", "e","f", "g"],                   'age' : [20,27, 35, 55, 18, 21, 35],                   'designation': ["VP", "CEO", "CFO", "VP", "VP", "CEO", "MD"]}
And we know that we can create a Pandas DataFrame out of a python dictionary by invoking DataFrame(...) function
df = pd.DataFrame(my_dict)
The resultant DataFrame shall look like
Once we have the DataFrame, we can persist it in a CSV file on the local disk. Let’s first create our own CSV file using the data that is currently present in the DataFrame, we can store the data of this DataFrame in CSV format using the API called to_csv(...) of Pandas DataFrame as
df.to_csv('csv_example')
Now we have the CSV file which contains the data present in the DataFrame above.
Just as we can persist the DataFrame in a CSV file, we can also load the DataFrame from a CSV file.
Let’s go ahead and load the CSV file and create a new DataFrame out of it
df_csv = pd.read_csv('csv_example')
The resultant DataFrame (df_csv) shall look like
Did you notice something unusual?
Well, we can see that the index is generated twice, the first one is loaded from the CSV file, while the second one, i.e Unnamed is generated automatically by Pandas while loading the CSV file.
This problem can be avoided by making sure that the writing of CSV files doesn’t write indexes, because DataFrame will generate it anyway. We can do the same by specifying index = False parameter in to_csv(...) function
df.to_csv('csv_example', index=False)
Now, if we read the file as
df_csv = pd.read_csv('csv_example')
The resultant DataFrame shall look like
Now you can see that the output is similar to what we had earlier, when we created the DataFrame from the python dictionary and this is what we’re expecting.
As we’ve seen that the first row is always considered as column headers, however, it’s possible to have more than one row as column headers by specifying a parameter called header=<integer> in read_csv(...) function.
By default, the value is specified as ‘0’, which means that the top row will be considered as header.
df_csv = pd.read_csv('csv_example', header = 0)
The resultant output will be same as above. However, it open lots of opportunities to play with arranging headers. For example, we can also have more than one row as header as
df_csv = pd.read_csv('csv_example', header=[0,1,2])
The resultant DataFrame shall look like
As we see over here that column 0,1, & 2 are headers now.
It’s also not necessary to have first sequence of row as a header, we can very well skip first few rows and then start looking at the table from a specific row
For example
df_csv = pd.read_csv('csv_example', header=5)
Here, the resultant DataFrame shall look like
The only drawback is that we will have to let go of the data available before the header row number. It can’t be part of the resultant DataFrame.
Even in the case of having multiple rows as header, actual DataFrame data shall start only with rows after the last header rows.
df_csv = pd.read_csv('csv_example', header=[1,2,5])
The resultant DataFrame will start from row ‘6’ and shall look like
Though we’re reading the data from CSV files with Column headers, we can still have our own column names. We can achieve the same by adding a parameter called names in read_csv(...) as
df_csv = pd.read_csv('csv_example', names=['a', 'b', 'c'])
The resultant DataFrame shall look like
However, even though we are successful in adding our own header, the top row still displays header which is a non desired one.
This can be avoided by using the header parameter in read_csv(...)to skip the row depicting the header. In this particular case, we know that first row, i.e. row 0 is header so we can skip it as
df_csv = pd.read_csv('csv_example', names=['a', 'b', 'c'], header=1)
Now, we got the output we were looking for with our customized header.
Another way of doing the same is by skipping the header while writing the CSV files as
df.to_csv('csv_example', index=False, header = False)
And while reading ,we can read without skipping the header as
df_csv = pd.read_csv('csv_example', names=['AGE', 'DESIGNATION', 'NAME'])
Though comma separated values are well known, the read_csv(...) function can identify separators other than comma.
The only difference is that we need to pass the separator explicitly in the function while comma is considered by default
Let’s first create a CSV file using a different separator i.e “:” (A colon)
df.to_csv('csv_example', index=False, sep=":")
This will create a file where the colon (‘:’) instead of comma (‘,’) shall be used as a separator. We can read the file as
df_csv = pd.read_csv('csv_example', sep=":")
The resultant DataFrame shall look like
By default, Pandas DataFrame generates a row index automatically which we can change by setting any column as the Index as
df_csv.set_index('age')
Here is how the resultant DataFrame shall look like
Setting the indexes in this way is a post operation. i.e we already have a DataFrame with pre defined index, but we change it later.
We can do this at the time of loading CSV file by passing a parameter called index_col , which will automatically assign the column depicted by index_col as a row index.
df_csv = pd.read_csv('csv_example', sep=":", index_col=1)
The output of the same shall look like
We can even provide more than one index_col to be treated as index
df_csv = pd.read_csv('csv_example', sep=":", index_col=[0,2])
And the output will look like
Most of the time, the CSV files will have considerable size and you might run into memory constraints while loading them. There is an option of loading only selected few rows from it.
You can do the same by specifying the number of Rows to be loaded by passing an argument nrows in read_csv(...)
# Load Only 3 Rowsdf_csv = pd.read_csv('csv_example', sep=":", nrows=3)
And here is how it will look like
By default, read_csv(...) function skips blank line, i.e it will ignore blank lines while loading the file and constructing the DataFrame.
However, in case you want to load blank line(s) for doing some explicit calculations like counting empty records, you should mark skipping blank lines as False
df_csv = pd.read_csv('csv_example', skip_blank_lines=False, sep=":")
That’s all for this particular post
Thank for Reading...!!!