LightGBM on Home Credit Default Risk Prediction
By: Muhammad Ardi
Tags: AI, Feature Engineering, Python, Data Visualization, Machine Learning

Feature engineering

Feature engineering is usually the longest process that we need to do in order to create a machine learning model. Now I would like to start this part with finding out the datatypes of each column which the output can be seen in figure 13.

Figure 13. The datatypes of each column in reduced_df.

Notice that some of the columns in our data frame are still in form of object. In Pandas, object simply means string. This is actually a problem since basically any machine learning algorithm will work only with numerical data. Hence, we need to either label-encode or one-hot-encode all these objects.

To determine whether we should use label encoder or one-hot encoder, we need to find out the number of unique values in each of those columns first. If a column has more than 2 categories, then we need to use one-hot encoder. Otherwise, if the number of categories is exactly 2 we need to use label encoder instead. In order to do that, I decided to create a simple loop which is going to print out the number of unique value in each of the columns in reduced_df.

Figure 13. Code to find out the number of unique values in each column in reduced_df.

Figure 14. The unique values of each reduced_df column.

Pardon the weird indentation :) We can see here that CODE_GENDER, NAME_INCOME_TYPE and NAME_EDUCATION_TYPE are having more than 2 unique categorical values. Hence, we need to convert these into one-hot format. My approach here is to employ get_dummies() function taken from Pandas library.

Figure 15. Code to convert values in the data frame to one-hot representation.

Initially, the shape of reduced_df is (307511, 17), where 17 indicates the number of existing columns. Subsequently, after running the code in figure 15 above, the data frame should now be having the size of (307511, 30). This number of column is taken from 17 (initial) + 3 (code gender) + 8 (income type) + 5 (education type) – 3 (no of columns converted to one-hot) = 30 (final result).

Next, we need to perform label encoding to the columns where the number of unique values are exactly 2. This can be achieved by using LabelEncoder object from Sklearn module. The complete process is done in figure 16. Additionally, notice that here I am using 3 different label encoders for the 3 columns.

Figure 16. Code to perform label encoding.

Alright, up to this point we already created features in which all of those are in form of non-string data. Now if you check the datatypes of all columns in the data frame like what I did in figure 13 above you should see that everything has been in form of either integer or float. And this is exactly what we want.

Figure 17. The number of NaN values in reduced_df.

However though, we still encounter another problem. What’s that? Look at the figure 17. All values displayed here basically denotes the number of missing values in the corresponding column. There are actually 2 options to choose here, either dropping all those NaN values or filling them with a particular number.

In this case, I decided to fill them all using the average value of each column for some reasons. In fact, you may try to use median, zeros, or probably forward fill in order to do so. Yet I feel like taking the mean value is just the right choice.

The implementation itself is pretty simple thanks to fillna() method which can directly be applied to the columns. The detailed steps is shown in figure 18.

Figure 18. Filling missing values with the average value of the corresponding columns.

And that’s the end of feature engineering chapter! Now let’s continue creating the machine learning model in the next section!