Error-free import of CSV files using Pandas DataFrame | by Syed Huma Shah | Towards Data Science

Data is at the centre of a Machine Learning pipeline. In order to leverage an algorithm’s full capacity, data must be first cleaned and wrangled properly.
The first step of data cleaning/wrangling is loading the file and then establishing a connection via the path of a file. There are different types of delimited files like tab-separated file, comma-separated file, multi-character delimited file etc. The delimitations indicate how the data is to be separated within columns whether through comma, tab or semicolon etc. The most commonly used files are tab-separated and comma-separated files.
Data wrangling and cleaning accounts for about 50 to 70% of the Data analytics professionals’ time within the whole ML pipeline. The first step is to import the file to a Pandas DataFrame. However, this step constitutes the most encountered errors. People often get stuck in this particular step and come across errors like
EmptyDataError: No columns to parse from file
The common errors occur, mainly, due to :
· Wrong file delimiters mentioned.
· File path not formed properly.
· Wrong syntax or separator used to specify the file path.
· Wrong file directory mentioned.
· File Connection not formed.
Data analytics professionals cannot afford more time being drained into an already time-consuming step. While loading the file, certain important steps must be followed which will save time and cut through the hassle of scouring through a plethora of information to find the solution to your specific problem. Therefore, I have laid out some steps to avoid any error while importing and loading a data file using pandas DataFrame.
Reading and importing the CSV file is not so simple as one may surmise. Here are some tips which must be kept in mind once you start loading your file to build your Machine Learning model.
1. Check your separation type in settings:
For Windows
Go to Control Panel
Click on Regional and Language Options
Click on Regional Options tab
Click on Customize/Additional settings
Type a comma into the ‘List separator’ box (,)
Click ‘OK’ twice to confirm the change
Note: This only works if the ‘Decimal symbol’ is also not a comma.
For MacOS
Go to System Preferences
Click on Language & Region and then go to the Advanced option
Change the ‘Decimal Separator’ to one of the below scenarios
For MacOS, if the Decimal Separator is a period (.) then the CSV separator will be a comma. If the Decimal Separator is a comma (,) then the CSV separator will be a semicolon.
2. Check the preview of the file:
The preview of the file can also be checked and it can be seen how the data is being separated, whether by tab separation or comma separation. One can check the preview either in Jupyter notebook or Microsoft Excel.
3. Specify correctly all the arguments:
Having taken a look at the preview and checking the separation specified for your computer. We now have to fill in the right arguments which need to be mentioned in the “pd.read_csv” function based on the type of file as the type of delimiter( tab-separated etc), blank header (in that case header= none) etc.
Pandas.read.csv has many arguments which need to be taken into account for the file to be read properly.
pandas.read_csv(filepath_or_buffer, sep=<object object>, delimiter=None, header=’infer’, names=None, index_col=None, usecols=None, squeeze=False, prefix=None, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=0, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, cache_dates=True, iterator=False, chunksize=None, compression=’infer’, thousands=None, decimal=’.’, lineterminator=None, quotechar=’”’, quoting=0, doublequote=True, escapechar=None, comment=None, encoding=None, dialect=None, error_bad_lines=True, warn_bad_lines=True, delim_whitespace=False, low_memory=True, memory_map=False, float_precision=None, storage_options=None)
This is the list of all the arguments, but we are most concerned with the following:
sep: This specifies the type of separation between the data values. The default is ‘,’. After checking the preview and the system settings, we know the type of file. The most common type of separator/delimiter is comma, tab and colon. Therefore, it is to be specified as sep= ‘,’ , sep= ‘ ’, sep= ‘;’ This tells the pandas DataFrame how to distribute data into columns
After checking for any required arguments which need to be put, if the issue persists. Then the issue might be with the file path.
4. Check Filepath:
This argument is used to describe the path object or file-like object for the particular data file, basically its location. The Input is a string. A URL can be input as well, the valid schemes are HTTP, FTP, s3, gs, and file.
The file location is to be mentioned correctly. Most often, people are unaware of the working directory and end up mentioning the wrong file path. In that case, we have to check the working directory to ensure that the specified file path is correctly described. Write the code shown below to check the working directory.
This will print the working directory. Then, we have to only specify the location after your working directory.
We can change the working directory also using the below line of code. After specifying the new directory, we have to specify the path.
5. Check separator used to specify file location:
Often times an error occurs while changing the working directory as well. This arises due to not writing the separator according to the proper syntax.
First of all. Check the separator using the below command.
Then use the separator at the beginning of the directory location only and not at the end. Kindly, note that this separator(/) syntax specification is true for MacOS and might not be true for Windows.
Now after specifying the location correctly we have changed it.
Now we have to specify the path. Since we are familiar with the working directory. We have to only specify the location succeeding the working directory.
If your file is in the working directory then only mention the file name as shown below.
But if your file is present in some other folder then you can either specify the succeeding folders after the working directory e.g. your working directory is “/Users/username” and your file is in a folder named ‘huma’ in ‘documents’ then you would write the below code:
path = ‘Documents/huma/filename.csv’
6. Check the file is on the path:
Now check whether your file is present in the described path using the below code. We will get our answer as either ‘true’ or ‘false’.
7. Print the file data to cross-check:
Now, we can check whether our data file has loaded correctly using the below code.
With these tips in hand, you may not face any problem in loading your CSV file using Pandas DataFrame again.