How To Fix pandas.parser.CParserError: Error tokenizing data | Towards Data Science

Importing data from csv files is probably the most commonly used way of instantiating pandas DataFrames. However, many times this could be a little bit tricky especially when the data included in the file are not in the expected form. In these cases, pandas parser may raise an error similar to the one reported below:
pandas.errors.ParserError: Error tokenizing data. C error: Expected 5 fields in line 2, saw 6
In today’s short guide we will discuss why this error is being raised in the first place and additionally, we will discuss a few ways that could eventually help you deal with it.
First, let’s try to reproduce the error using a small dataset I have prepared as part of this tutorial.
Now if we attempt to read in the file using read_csv :
import pandas as pddf = pd.read_csv('test.txt')
we are going to get the following error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 4 fields in line 4, saw 6
The error is pretty clear as it indicates that on the 4th line instead of 4, 6 fields were observed (and by the way, the same issue occurs in the last line as well).
By default, read_csv uses comma (,) as the delimiter but clearly, two lines in the file have more separators than expected. The expected number in this occasion is actually 4, since our headers (i.e. the first line of the file) contains 4 fields separated by commas.
The most obvious solution to the problem, is to fix the data file manually by removing the extra separators in the lines causing us troubles. This is actually the best solution (assuming that you have specified the right delimiters, headers etc. when calling read_csv function). However, this may be quite tricky and painful when you need to deal with large files containing thousands of lines.
Another cause of this error may be related to some carriage returns (i.e. '\r’) in the data. In some occasions this is actually introduced by pandas.to_csv() method. When writing pandas DataFrames to CSV files, a carriage return is added to column names in which the method will then write the subsequent column names to the first column of the pandas DataFrame. And thus we’ll end up with different number of columns in the first rows.
If that’s the case, then you can explicitly specify the line terminator to '\n' using the corresponding parameter when calling read_csv():
import pandas as pd df = pd.read_csv('test.csv', lineterminator='\n')
The error may also be related to the delimiters and/or headers (not) specified when calling read_csv. Make sure to pass both the correct separator and headers.
For example, the arguments below specify that ; is the delimiter used to separate columns (by default commas are used as delimiters) and that the file does not contain any headers at all.
import pandas as pddf = pd.read_csv('test.csv', sep=';', header=None)
Skipping rows that are causing the error should be your last resort and I would personally discourage you from doing so, but I guess there are certain use cases where this may be acceptable.
If that’s the case, then you can do so by setting error_bad_lines to False when calling read_csv function:
import pandas as pd df = pd.read_csv('test.txt', error_bad_lines=False)print(df)   colA colB  colC  colD0     1    A   100   1.01     2    B   121   2.12     4    D   164   3.13     5    E    55   4.5
As you can see from the output above, the lines that they were causing errors were actually skipped and we can now move on with whatever we’d like to do with our pandas DataFrame.
In today’s short guide, we discussed a few cases where pandas.errors.ParserError: Error tokenizing data is raised by the pandas parser when reading csv files into pandas DataFrames.
Additionally, we showcased how to deal with the error by fixing the errors or typos in the data file itself, or by specifying the appropriate line terminator. Finally, we also discussed how to skip lines causing errors but keep in mind that in most of the cases this should be avoided.
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read. You’ll also get full access to every story on Medium.
gmyrianthous.medium.com
You may also like