Explore and Clean: first steps of any data project. | by Shruti Turner | Towards Data Science

It’s easy to be so keen to get something out of your data that you jump right into writing a predictor algorithm or some other form of Machine Learning/AI that you want to try out. As with probably most people out there, I’ve fallen into that trap too and been disappointed at my stunted progress.
In this article, I want to highlight the value of knowing your data before jumping in with the “fun” stuff, with some concrete examples of how to do it, using a simple data set. I’m a Python user, so my code examples will refer to this, but the theory of the methods will apply across languages.
When you think about your data, it might seem logical to explore your data, find out what’s there and which parts need attention before commencing with the cleaning. After all, how can you clean up if you don’t know what the mess is? I do sort of agree with this, but I think it’s an iterative process. You might not be able to explore and find out what you need about the data with it in the wrong format. So, I tend to do some back and forth between exploration and cleaning.
I am a firm believer in the sentiment behind the saying “a picture says a thousand words”, which in the data world means visualising the data you have. In some cases, you might not be able to visualise the data because it might be in the wrong format (your number is a string type rather than a float) or perhaps your time is collected and spread across 4 columns (hours, minutes, seconds, milliseconds), this is particularly important with numerical data.
For this reason, I like to explore my data and then address issues as I see them.
At this point I think it’s worth considering the question:
What does it mean to have clean data? I believe that what constitutes clean data is very much dependent on your task at hand, both the application of your data project and what you want to do with the data. But, generally, it needs to be logical: numbers should be numbers, dates should be dates etc. I’ll explain more about what I mean below.
The first thing I’d recommend once importing your data into a data frame is to just check if the data you have is what you’re expected. For this I check the first five rows using df.head (where df = your DataFrame).
Here you can see that there are 5 columns, some with words or strings, a date field and an integer. All that looks about what I would expect from the column names.
However, what we see might not be what the computer is seeing, so it’s important to check for the types of what is present: df.dtypes will do this for you. For the above DataFrame, we get the following output:
As you can see here, the output is telling me the different column names, with the type of data that sits within them. Something that printing the DataFrame alone does not. From the table, there was no way to know whether the values are strings or floats or other. This is important information, even for just initially creating simple plots — you can’t make a scatter plot with strings! — but when you get further down the line, you might be using a Random Forest Regression that needs numerical data and not strings.
You might be wondering why all of the columns are assigned ‘object’ types. This is where the documentation comes in handy. Strings are assigned the object type, so that is expected for both Name columns and the Sex column. That, however, doesn’t explain enough about the Date of Birth or Age at Death columns. Pandas will also assign object types to columns which have a mix of data, so for the two columns mentioned, we don’t know from this if some or all the data are strings or something else. A hint that the Date of Birth column had a string or mixed data type is that the dates were not presented in a uniform way.
Generally, if we have items we as humans see as numbers or dates in a DataFrame, that’s what we want the computer to see them as too. Here I would suggest converting the Date of Birth column to dates and the Age at Death column to integers using the following code (Where pd is the pandas library):
df['Age at Death'] = df['Age at Death'].astype(int)df['Date of Birth'] = pd.to_datetime(df['Date of Birth'])
When I show the DataFrame again it looks like this:
All the columns look the same, except the Date of Birth column, which has been reformatted so the dates are uniform. It might not be the format that you want, so you can take a look at the documentation for more details on changing that.
However, when we look at the types of data that sit within each column, you’ll see that the two we have converted are shown here:
Exploration can seem a bit of an effort if you have a simple data set, but I still think it’s an important one to go through; both because it will prepare you for when you have complex data sets and also because taking the time to explore can show you so much you don’t see in your raw data or even in a table.
Let’s take another example: sometimes when you source your data you might be given a file which has lots of zeros in it to show that there is nothing present in the cell. Looking at it, that’s fine as long as you have that understanding within your team. However, your computer isn’t able to have that understanding: to the computer zero is a number and therefore there is something present in the cell. We need to tell the computer that the cell is null. Let’s add a couple of rows to the DataFrame we’ve been using:
In the above, we can see that the DataFrame is missing the Date of Birth for Ernest Rutherford — it is clearly seen with NaT or Not a Time (an example of a null value for time). However, without looking carefully at the Age at Death column, you might miss that Marie Skłodowska Curie has an age of zero. With a large DataFrame both these examples might be missed, so it’s important to get the information about the numerical aspects of the DataFrame concisely for you to be able to look at, to do this I use df.describe():
As the only numerical column we have is the ‘Age at Death’ one, it is expected that we will see one column in the output, if you had a DataFrame with more than one numerical column, each would be displayed similarly. We can clearly see that the minimum value is zero, which we can use our human intelligence to realise something is wrong here. In this instance, it might be the case that wherever your data is sourced from replaced the value with a zero to show that there was no data for us. In Python, however, we need to show this as NaN or null, so there really is nothing in that cell. Here you can use df.replace(0, np.NaN) (where np is the numpy library) to give the following:
As you can see, the change to the representation of the data seems minimal when you look at the DataFrame alone, but delving deeper into the types you can see the large difference.
To check for null values in the DataFrame, you can also use df.isna().count() which gives the following output:
The above method is a quick and clear way of identifying how much data is missing in your DataFrame. Again, for this example, it might seem a bit superfluous, but when you get to deal with hundreds of columns and thousands of rows, you’ll appreciate this method (probably even before that too!)
Generally, when I have numerical data, I like to plot it once I have done the initial cleaning. I do this as part of my exploration: can I see an obvious trend? Are there clear anomalies or errors that might skew my data? I find it helps me see what is actually there, rather than what I see in the snippets of DataFrame.
I like to see my data more clearly, for instance like below:
fig, ax = plt.subplots()x = df['Date of Birth']y = df['Age at Death']plt.plot(x, y, 'bo')fig.suptitle('Year of Birth vs Age at Death', fontsize=20)xlabel = 'Year of Birth'ylabel = 'Age at Death'ax.set_xlabel(xlabel, fontsize=10)ax.set_ylabel(ylabel, fontsize='medium')
You might notice here, that there are only six data points on the graph, though there are seven rows of data in the dataframe. The value for Marie Skłodowska Curie has not been plotted, because her Age at Death is NaN. This is a good thing as if we plotted this with the value of zero and continued with analysis, it could lead to misleading results.
Python has got a little confused, however, as it is showing a data point for Ernest Rutherford in about 1680. It seems that due to the NaT being present instead of his Date of Birth, Python has picked the earliest possible date and plotting the Age at Death data point. This is something to watch out for! With a little bit of further manipulation, you can make sure that it is excluded also, by selecting the x values and y values of your graph to be plotted from the rows of the DataFrame where there are no null values with just a small addition of df.dropna():
fig, ax = plt.subplots()df2 = df.dropna()x = df2['Date of Birth']y = df2['Age at Death']plt.plot(x, y, 'bo')fig.suptitle('Year of Birth vs Age at Death', fontsize=20)xlabel = 'Year of Birth'ylabel = 'Age at Death'ax.set_xlabel(xlabel, fontsize=10)ax.set_ylabel(ylabel, fontsize='medium')
If I had tried to plot the data before cleaning, Python would have thrown an error, due to the incorrect data types. Most of the plotting functions need integers, floats or dates and can’t see the numbers we see until they are the correct type. Without the human sense check and the manipulation to remove rows with null values, we would have been left with a misleading graph, though no error was thrown.
Last Words
I could also do some more manipulation of the numerical data if I want to with it now being in integer format, perhaps I want to get the average age of death for men and women so I can compare? I might want to change the Sex column to being a binary type as this could open up more opportunities for analysis.
Naturally, with a more complex data set the possibilities become more and more exciting.
This is a process that I teach to any new programmer as first steps when they’re using data. I use it myself with every new data set I use in my PhD and when I’m just having a play around with new data sets. Naturally, these are more complex datasets that the example I have shown in this article, but the principles are the same.