Data Preprocessing & Exploratory Data Analysis (EDA) for Data Science: Tackling the Taarifa Challenge | by Antonio Stark | Towards Data Science

Data preprocessing and exploratory data analysis (EDA) are essential tasks for any data science projects. In this article, we’re going to look at a sample dataset and challenge and apply a few data preprocessing and EDA techniques.
Do note that data preprocessing and EDA are distinct terms, but have many overlapping subtasks and are usually used interchangeably.
The dataset and original code can be accessed through this GitHub link.
The challenge I’ll use for this article is taken from Drivendata.org. You can think of it as a Kaggle for social impact challenges. You still get the same perks for winning and pretty well-formatted datasets, with the additional benefit that you’ll be making a positive impact on the world!
The particular challenge that we’re using for this article is called “Pump it Up: Data Mining the Water Table.” The challenge is to create a model that will predict the condition of a particular water pump (“waterpoint”) given its many attributes.
The data is provided by Taarifa, an open-source API that gathers this data and presents it to the world. If you’re curious about their work and what their datapoints represent, make sure to check out their website (and GitHub!)
The datasets describe over 74,000 data points, which represent a waterpoint in the Taarifa data catalog. 59,400 data points (80% of the entire dataset) are in the training group, while 14,850 data points (20%) are in the testing group.
The training data points have 40 features, one feature being the label for its current functionality. All data points have one feature that is the ID of the water point.
There are three datasets, two for the training group and one for the testing group.
train_values is a dataset of 59400 data points of 40 features train_labels is a dataset of 59400 data points of 2 features test_values is a dataset of 14850 data points of 40 features
The training dataset comes with three labels: ‘functional,’ ‘functional-needs-repair,’’, and ‘non-functional.’ The label indicates the condition of that water point. Our job is to predict the condition of the water points in the testing dataset.
As you can see above, the ‘functional’ takes up 54.3% of the training dataset, ‘functional-needs-repair’ 7.3%, and ‘non-functional’ 38.4%. The output seems to be slightly imbalanced, but we’re not sure (yet) if this imbalance will be significant. We’ll come back to this in the modeling section.
To efficiently preprocess data, and to apply the preprocessing measures to all data points, we combine the training and testing datasets.
# 'status_group' column is assigned the status of each waterpoint, 'test' if unknown (i.e. the waterpoint belongs to the test dataset)train_values['status_group']=train_labels.status_grouptest_values['status_group']=['test']*test_values.shape[0]# 'train' column discerns if the waterpoint belongs to the training group or the testing grouptrain_values['train']=[True]*train_values.shape[0]test_values['train']=[False]*test_values.shape[0]df = pd.concat([train_values,test_values])
Now that we have a combined dataset, we can do some EDA magic on it.
One of the first tasks in any dataset is to find if there are missing values. As this dataset has more features than we’d like, we’re going to drop features that have too many missing values. Let’s see how many missing values each feature has.
nulls = df.isnull().sum()# this will show only features that have nonzero missing valuesnulls[nulls!=0]
There seem to be quite a few features with missing data points! Let’s see what percentage they take up in the whole dataset.
nulls_percentage = nulls[nulls!=0]/df.shape[0]*100print('the percentages of null values per feature:\n')print(round(nulls_percentage,2))
It seems like the features that have any missing values have a significant number of missing values. For ‘scheme_name’, it’s missing almost half of the data! Hence, it is safe to eliminate entire features if they contain any missing values. We can note this by marking how=‘any’ in the dropna function.
features_original = df.shape[1]df.dropna(axis='columns',how='any',inplace=True)features_reduced = df.shape[1]print('number of features reduced from %d to %d'%(features_original,features_reduced))
A dirty way to identify categorical features is to check if its values are non-numerical. But there are some features whose categories are numerical values. This would mean that the magnitude of the numerical value has no innate meaning, i.e. the feature is categorical rather than ordinal (i.e. has an order).
A lot of times, we can’t be sure whether or not a numerical feature is ordinal or categorical. We’d have to ask the people who actually assigned the number to the datapoint to see if they have innate meaning.
But by looking at how many “unique” numbers come up per feature, we can guess whether or not a numerical feature may be categorical.
df.select_dtypes(include='number').nunique()
Here, we can see that there are five features that have an abnormally low number of unique values. That is indicative that those numerical values actually stand for categorical values.
Looking at the dataset documentation, we find that three of the five features: ‘num_private’, ‘region_code’, ‘district_code’ are geographic codes whose numerical values aren’t relevant for our challenge. Let’s convert these into strings so that we can play with them in the next sections.
df['num_private'] = df.num_private.astype(str)df['region_code'] = df.region_code.astype(str)df['district_code'] = df.district_code.astype(str)
Categorical features with too many categories are usually not useful for predictive models. Let’s find how many categories each categorical features have.
uniques = df.select_dtypes(exclude='number').nunique()uniques
We have a few features whose number of categories do not justify them to be a “categorical” variable. Given that we have over 74,000 data points, a feature that has more than 740 categories will have less than 100 data points per category on average. Let’s find which features have more than 740 unique categories.
uniques[uniques>740]
There are only two features that have more than 740 categories: ‘wpt_name’ and ‘ward.’ We can simply drop them with the Pandas drop function.
df.drop(columns=['wpt_name','ward'],inplace=True)
We still have a lot of features left. We want to cut down on categorical features as they will blow up the number of features once we apply one-hot encoding.
Another type of categorical features that are usually useless are features with categories that show up too infrequently. Let’s find such features.
In the above section, we removed features whose “average” data points per category are less than 100 data points. Here, we’ll remove features whose “minimum” data points per category are less than 100.
thresh = 100for col in df.select_dtypes(exclude='number').columns:    value_counts = df[col].value_counts()    if value_counts.min(axis=0) < thresh:        df.drop(columns=col,inplace=True)
There are 10 features whose minimum data points per category was less than 100. Let’s quickly check back to see which nonnumerical features have remained, and how many categories each have:
df.select_dtypes(exclude='number').nunique()
Okay let’s quickly check which features we have left:
print('total number of features: %d\n'%(df.shape[1]))df.dtypes
To be honest, we can’t quite be sure which features will be “irrelevant” when we haven’t run any models (and seen their results) yet. This is especially true when combination of features, not individual features, may be the key to finding the right categorization scheme.
Let’s see if there are “completely uncorrelated” features that, when deleted, might not affect the other features. This still doesn’t guarantee that the feature is useless when combined with other features, but could direct us to a few hopefuls.
As in, let’s draw some correlation heatmaps!
Note that we can draw correlation heatmaps only to numerical features for now. There is a way to draw correlation heatmaps for categorical variables but it’s a lot dirtier.
import seaborn as snssns.heatmap(df.corr(), cmap='RdBu',center=0)plt.show()
There seem to be a few features that may hold no value in relation to other features. Let’s sort the features to find out which ones.
sns.clustermap(df.corr(), cmap='RdBu',center=0)
So the features ‘amount_tsh’ and ‘id’ seems to be relatively uncorrelated with the other features, and may be safely disregarded. To note, ‘amount_tsh’ is the ‘amount of water available to waterpoint’ and ‘id’ is, you know, the ID of the waterpoint. Neither seems particularly valuable so let’s get rid of them.
Looking at the number of categories, we also see that the feature ‘recorded_by’ is a categorical feature with only ONE category. That’s bound to be useless, so we can get rid of that too.
Also recognize that the feature ‘train’ is there for us to just keep track of dataset groupings, so we don’t have to care about that.
df.drop(columns = ['amount_tsh','id','recorded_by'],inplace=True)
Now that we think all the features left are “valuable” ones, we can go ahead and one-hot encode it! Remember this will blow up the total number of features of the dataset.
columns_categorical = ['basin','region','extraction_type_group','extraction_type_class','management','management_group','payment','payment_type','quality_group','quantity','quantity_group','source_type','source_class']df = pd.get_dummies(df,columns=columns_categorical)
After one-hot encoding, we see that our number of features has increased from 20 to 114.
One last thing to do is to provide ordinality to the ‘status_group’ feature. While this is a categorical feature, it has ordinance. I.e., there’s a ordinal relationship of functional > functional needs repair > non functional. A quick and dirty way to do this is by substituting the string values with numerical values.
Pandas does support an in-house ordering feature for Category type series, but it does get a bit hairy, so we’ll not do that here. But to be strict, using the ordinance method given by Pandas is more accurate, especially when the “distance” between the categories may not be equal. As in, by giving the numbers, we’re saying that the magnitude of difference between ‘functional’ and ‘functional needs repair’ is the same as that of ‘functional needs repair’ and ‘non functional’.
status_group_dict = {'functional':2,'functional needs repair':1,'non functional':0}df.status_group = df.status_group.replace(status_group_dict)
That’s it! Now we have a beautifully formatted dataset that we can now put into some models to churn out some predictions. If you were only curious about data preprocessing and EDA, you can end it here. If you want to look at how this cleaned dataset can be used for predictive modeling and multiclass classification, you can check out that article here.