How to “read_csv” with Pandas. Use read_csv as a versatile tool | by Soner Yıldırım | Towards Data Science

The most time-consuming part of a data science project is data cleaning and preparation. Pandas is a very powerful and versatile Python data analysis library that expedites the preprocessing steps of your project. We usually tend to use functions with default parameters which prevents us from taking full advantage of the function.
One of the most widely used functions of Pandas is read_csv which reads comma-separated values (csv) files and creates a DataFrame. In this post, I will focus on many different parameters of read_csv function and how to efficiently use them.
The basic data structure of Pandas is DataFrame which represents data in tabular form with labeled rows and columns.
As always, we start with importing numpy and pandas.
import numpy as npimport pandas as pd
I created a sample DataFrame aiming to show the effect and usefulness of parameters so the values may not make sense.
Let’s start with using read_csv with no optional parameters:
df = pd.read_csv("SampleDataset.csv")df.head()
The only required parameter is the file path. We need to tell pandas where the file is located. If the csv file is in the same working directory or folder, you can just write the name of the file. If not, we can specify the location as follows:
df = pd.read_csv(r"C:\Users\soner\Downloads\SampleDataset.csv")
An integer index starting from 0 is assigned to the DataFrame by default. We can explicitly define a column to be used as index. For example, we may want to use “ID” column as index:
df = pd.read_csv("SampleDataset.csv", index_col='ID')df.head()
It is a bit strange to have floating point ID numbers. I don’t think any company assigns ID numbers in this way. We can specify the data types of any column in read_csv function using dtype parameter:
df = pd.read_csv("SampleDataset.csv", index_col='ID',                   dtype={'ID':np.int32})df.head()
In some cases, depending on what we plan to do with date, we may not need all of the features (columns). We can drop the unnecessary columns after reading all data. However, a better option would be just reading the columns we need which can easily be done with usecols parameter:
cols = ["ID","Deparment","Salary","StartDate","Location"]df = pd.read_csv("SampleDataset.csv", usecols=cols)df.head()
We can also use indices of columns as argument to usecols parameter. Following code will do the job just fine:
df = pd.read_csv("SampleDataset.csv", usecols=[2,3,4,5,6])
Let’s check the data types of columns:
df = pd.read_csv("SampleDataset.csv")df.dtypesUnnamed: 0      int64RowNumber       int64ID            float64Deparment      objectSalary          int64StartDate      objectLocation       objectdtype: object
The data type of StartDate column is object but we know this column includes dates so we can read the values as date by using parse_dates parameter.
df = pd.read_csv("SampleDataset.csv", parse_dates=[5])df.dtypesUnnamed: 0             int64RowNumber              int64ID                   float64Deparment             objectSalary                 int64StartDate     datetime64[ns]Location              objectdtype: object
The dates are now in a better suited format.
Missing values are sometimes not in a format that can be detected as “missing” by Pandas. For example, in location column, ‘?’ is a missing value but there is no way read_csv function knows this unless we specify it. We can use na_values to indicate additional values to be recognized as “missing”:
df = pd.read_csv("SampleDataset.csv", na_values=['?'])df.head()
If we have a very large DataFrame and want to read only a part of it, we can use nrows parameter and indicate how many rows we want to read and put in the DataFrame:
df = pd.read_csv("SampleDataset.csv")df.shape(30,7)df = pd.read_csv("SampleDataset.csv", nrows=10)df.shape(10,7)
In some cases, we may want to skip some of the rows at the beginning of the file. We can just pass the number of rows to be skipped to skiprows paremeter or pass a list with integers indicating the lines to be skipped:
skiprows=5 : Skip first 5 lines
skiprows=[1,3] : Skip first line and third line
Original DataFrame:
df = pd.read_csv("SampleDataset.csv", skiprows=3)df.head()
The first three rows skipped and the DataFrame starts from the 4th row. However, there is a problem in here. 4th row is used as column names.
We can solve this issue using header parameter. In most cases, the first row in a csv file includes column names and inferred as header. Therefore, if no column names are specified, default behavior of csv file is to take header=0 and column names are inferred from the ,first line of the file.
If header=None , column names are assigned as integer indices and first line of the file is read as first row of the DataFrame:
df = pd.read_csv("SampleDataset.csv", header=None)df.head()
So we can set header=None and use skiprows but keep in mind that the first line includes the column names. If skiprows=3, the first row of DataFrame becomes the person with ID=127.
df = pd.read_csv("SampleDataset.csv", header=None, skiprows=3)df.head()
If we want to change the name of columns or the file does not have a line to include columns names, we can use names parameter:
cols=["Drop1","Drop2","Employee_ID","Department","Salary", "StartDate"]df = pd.read_csv("SampleDataset.csv", names=cols, skiprows=1)df.head()
There are many additional parameters of read_csv function. I tried to explain the ones I commonly use but you can check the whole list of parameters here.
Thank you for reading. Please let me know if you have any feedback.