Data Cleaning Using Python in Data Science
By: Adesoji Alu
Tags: Statistics, Storytelling, Ds

What is Data or rather what is data in Datascience? data are facts and statistics collected together for reference or analysis. data includes numbers, information, quantities, observations, measurements, and pictorial representations in form of graphs to convey a message or messages. in addition, these data are sourced from two main sources

a. Primary Source: These are data collected from its original source. Examples are

Photographs and video footage

Official documents and records

Physical objects

Letter and diaries

Age s of students in a school

b. Secondary Source: This is anything that describes, interprets, evaluates, or analyzes information from primary sources. Common examples include

Article analyzing the novel

An academic book about the musical style

Biography of the historical figure and lots more

What is Data?

2. Types of Data: There are two types of variables you’ll find in your data — numerical and categorical. Numerical data can be divided into continuous or discrete values. And categorical data can be broken down into nominal and ordinal values. A categorical variable (sometimes called a nominal variable) is one that has two or more categories, but there is no intrinsic ordering to the categories. For example, gender is a categorical variable having two categories (male and female) and there is no intrinsic ordering to the categories. Hair color is also a categorical variable having a number of categories (blonde, brown, brunette, red, etc.) and again, there is no agreed way to order these from highest to lowest. A purely categorical variable is one that simply allows you to assign categories but you cannot clearly order the categories. If the variable has a clear ordering, then that variable would be an ordinal variable, as described below. an ordinal variable is similar to a categorical variable. The difference between the two is that there is a clear ordering of the categories. For example, suppose you have a variable, economic status, with three categories (low, medium, and high). In addition to being able to classify people into these three categories, you can order the categories as low, medium, and high. Now consider a variable like educational experience (with values such as elementary school graduate, high school graduate, some college and college graduate). These also can be ordered as elementary school, high school, some college, and college graduate. Even though we can order these from lowest to highest, the spacing between the values may not be the same across the levels of the variables. Say we assign scores 1, 2, 3, and 4 to these four levels of educational experience and we compare the difference in education between categories one and two with the difference in educational experience between categories two and three, or the difference between categories three and four. The difference between categories one and two (elementary and high school) is probably much bigger than the difference between categories two and three (high school and some college). In this example, we can order the people in the level of educational experience but the size of the difference between categories is inconsistent (because the spacing between categories one and two is bigger than categories two and three). If these categories were equally spaced, then the variable would be a numerical variable. A numerical variable is similar to an ordinal variable, except that the intervals between the values of the numerical variable are equally spaced. For example, suppose you have a variable such as annual income that is measured in dollars, and we have three people who make $10,000, $15,000, and $20,000. The second person makes $5,000 more than the first person and $5,000 less than the third person, and the size of these intervals is the same. If there were two other people who make $90,000 and $95,000, the size of that interval between these two people is also the same ($5,000).

Types of data

3. why Clean Data?

why clean if it’s not dirty? if it’s not broken, why fix it? Data cleaning is important in data analysis and also in the field of machine learning. This article reveals the importance of data cleaning and the different data cleaning processes in data science. the why is as a result of our objective or the main goal. I need to represent and quantify without missing quantities and misrepresentations in the data for processing. for example, I want to cook a wholesome food(beans) needed to be digested in the body, this tells me my food must be free from dirt such as stones, glass, and other foreign items which could pose a serious threat to the digestive tract and for the body as a whole. in like manner, our data must be investigated to identify bad data source which means inaccurate, missing part of a data and incorrect data while compilation. in computing since time immemorial, the computer accepts certain data types for computation hence, a corrupted data may hinder an accurate result. in addition if a data quality is high, it means it is “fit for [its] intended uses in operations, decision making and planning “. Moreover, data is deemed of high quality if it correctly represents the real-world construct to which it refers. data cleaning is also important in data science as it helps us make informed decision and to plan accurately.

Why does it matter whether a variable is categorical, ordinal, or numerical?

Statistical computations and analyses assume that the variables have specific levels of measurement. For example, it would not make sense to compute an average hair color. An average of a categorical variable does not make much sense because there is no intrinsic ordering of the levels of the categories. Moreover, if you tried to compute the average of educational experience as defined in the ordinal section above, you would also obtain a nonsensical result. Because the spacing between the four levels of educational experience is very uneven, the meaning of this average would be very questionable. In short, an average requires a variable to be numerical. Sometimes you have variables that are “in-between” ordinal and numerical, for example, a five-point Likert scale with values “strongly agree”, “agree”, “neutral”, “disagree” and “strongly disagree”. If we cannot be sure that the intervals between each of these five values are the same, then we would not be able to say that this is a numerical variable, but we would say that it is an ordinal variable. However, in order to be able to use statistics that assume the variable is numerical, we will assume that the intervals are equally spaced.

4. Environment for Data cleaning in data Science

In this article, you must be proficient in python programming language and also familiar with the python libraries needed for data cleaning. this article focuses on the python idle of the Jupyter notebook from anaconda. see more at https://www.anaconda.com/products/individual which is an important data science toolkit. the dataset for investigation could be downloaded from a website using the appropriate python command or downloaded as a compressed file and extracted to your directory in your local machine and read using the famous python library which works like the Microsoft excel called Pandas. Pandas is a data analysis and manipulation tool in data science. see more at https://pandas.pydata.org/. The data to be read must be read with desktop or laptop machines with high computational processes such as the core i5 8th gen and above or core i7 5th gen and above. here the latter processor is used for data reading and processing. the

I. first we import both python libraries called NumPy and Pandas which are built on the NumPy library or module.

II. Secondly, we read the dataset using the python command to read the dataset which could be in any extension such as excel(.xlsx) or A comma-separated values(.csv). see below for reference. in this article, we use NumPy version 1.18.5 while Pandas version 1.1.3

Jupyter notebook IDLE for running python scripts

iii. thirdly, we need to understand our data structure to check for inconsistencies within the data frame such as missing values, duplicate data, white spaces, bad input shape, or strings or rather do some questioning about the data yourself such as

c. how was the data collected and has it already been processed?

d. do you trust its source?

e. does the data makes sense even though I have domain knowledge on the subject matter?

getting your initial data sources right and making sure you ask the right question is essential to understanding the data so you would know how to remove messy data as no such word “bad data” exists. The image below shows the example of missing values and white space in a data frame

Examples of Missing value, String character, and white space

The dataset loaded is ‘houseprice.csv’ as seen below, we will be considering only 10 entries in our dataset for simplicity. our data is understood when we view and convert our data type where needed, aggregate the data, normalize, transform, and filter data. To start assessing your data, you need the df.head(), df.describe(),df.info(), df.dtypes() to have an overeview of what your data contains. the data cleaning methods are so numerous but we will cover little and it will be explained in the Jupyter notebooks for clarity

The former dataset called df

The new dataset

dtype conversion using astype:

let’s say we want to convert the Mssubclass column datatype from integer to float, we call the astype function to do the conversion data.MSSubClass.astype(float). here the datatype is changed from integer to float but not permanently. To make it permanent, we cast the astype function into the Mssubclass column dataset to make the astype conversion permanent as seen below.

The MSSubclass datatype has been changed successfully

dtype conversion from string to float value:

In order to achieve this we could think of the pd.to_numeric(data[‘LotShape’],errors=’coerce’, downcast=None) to convert our dataype to integer but this returns a column of missing values because the error condition was set to coerce which forcefully converts instead of raising a flag when the error is set as ‘raise’. we need to probe further into machine learning libraries, here we will be considering the scikit learn library. details below in the Jupyter notebook were obtained using the ordinal Encoder family from the sci-kit learn module.

Forceful conversion leads to missing values using pd.to.numeric to convert string to float values

String conversion to float is successful

Common data cleaning attributes:

other data cleaning attributes are seen in the jupyter notebook below