Classification on a large and noisy dataset with R | by Gabriel Pierobon | Towards Data Science

Some days ago I wrote an article describing a comprehensive supervised learning workflow in R with multiple modelling using packages caret and caretEnsemble. Back then I mentioned that the I was using was kind of an easy one, in the sense that it was fully numeric, perfectly filled (not a single missing value), no categorical features, no class imbalance (of course since it was a regression problem), and it was a fairly small dataset with just 8 predictors.
It felt almost like cheating but it really helped both for my first article written but also to go through the full workflow without much trouble and finally to do some multiple modelling using caretEnsemble.
So now I’ve decided to take this from easy difficulty to normal difficulty.
In that sense, I came across this dataset in the UCI Machine Learning Repository which I intend to use.
What is the new difficulty we come across with this dataset? Let me summarize it:
Target variable is categorical binomial, with a very high class imbalanceSize of the data set is fairly large. Training set is 60,000 x 171 and test set is 16,000 x 171Huge missing values issuePotential presence of outliers and and multicollinarityThere are specific cost associated to type 1 errors and type 2 errors, which requires that we minimize type 2 errors.
Target variable is categorical binomial, with a very high class imbalance
Size of the data set is fairly large. Training set is 60,000 x 171 and test set is 16,000 x 171
Huge missing values issue
Potential presence of outliers and and multicollinarity
There are specific cost associated to type 1 errors and type 2 errors, which requires that we minimize type 2 errors.
This is the description of the dataset and task included by the owner of the repository:
“The dataset consists of data collected from heavy Scania trucks in everyday usage. The system in focus is the Air Pressure system (APS) which generates pressurized air that are utilized in various functions in a truck, such as braking and gear changes. The datasets’ positive class consists of component failures for a specific component of the APS system. The negative class consists of trucks with failures for components not related to the APS. The data consists of a subset of all available data, selected by experts.
The attribute names of the data have been anonymized for proprietary reasons. It consists of both single numerical counters and histograms consisting of bins with different conditions. (...) In total there are 171 attributes, of which 7 are histogram variables. Missing values are denoted by “na”.
— Challenge metric
Cost-metric of miss-classification:
In this case Cost_1 refers to the cost that an unnecessary check needs to be done by an mechanic at an workshop, while Cost_2 refer to the cost of missing a faulty truck, which may cause a breakdown.
Total_cost = Cost_1 * No_Instances + Cost_2 * No_Instances
We’ll give it a try!
1.1) What are we trying to predict?
We need to predict the type of system failure. It can be either a failing component of the APS or a failing component not related to the APS. This is very important since our prediction errors can result in unnecessary spending by the company. Specifically we want to avoid type 2 errors (cost of missing a faulty truck, which may cause a breakdown).
2.2) What type of problem is it? Supervised or Unsupervised Learning? Classification or Regression? Binary or Multi-class? Uni-variate or Multi-variate?
It is a binary classification problem with multiple features.
2.3) What type of data do we have?
We have two csv files, one for training and one for testing. Both present 20 unimportant text lines before the actual data begins and we we will have to skip those lines when importing the data frame. There is one row for column names and missing values are denoted by “na”, so we will make sure we include that when we read the csv.
2.4) Import the dataset
training_data <- read.csv("aps_failure_traning_set.csv",                          skip = 20,                           na.strings = "na")test_data <- read.csv("aps_failure_test_set.csv",                      skip = 20,                       na.strings = "na")
We check the dimensions of both data sets:
dim(training_data)dim(test_data)[1] 60000   171[1] 16000   171
2.5) Activate packages to be used during the project
library(dplyr)library(caret)library(caretEnsemble)library(mice)library(doParallel)library(car)
2.1) View Data (str or dplyr’s glimpse). First look. Anything strange?
Specifying na.strings=”na" when we imported the sets allows R to recognize each feature as numeric. If we didn’t do that, the presence of na in each column would automatically result in them being categorized as character type.
Using the code below, we can see that other than our response variable class, all the other features are numeric.
glimpse(training_data)glimpse(test_data)
2.2) Is it a “tidy” dataset? Need to “gather” or “spread” it? Is it presented in a way we can work with?
It’s tidy. Each row is an observation, each column is a feature.
2.3) rownames and colnames ok? Should we change them?
I don’t see a reason to change them.
2.4) Check data types. Are they OK? If not, convert
We already assessed that data types are ok. Here you want to also check that the response variable is of the factor type, with the expected two levels.
2.5) What is our response/target variable? Class imbalance? Study it
Our target variable is class, with two levels: neg and pos. Positive class consists of component failures for a specific component of the APS system. The negative class consists of trucks with failures for components not related to the APS.
How many neg and pos do we have in each set?
summary(training_data$class)summary(test_data$class)  neg   pos 59000  1000   neg   pos 15625   375
Let’s check proportions:
options(digits = 2)prop.table(table(training_data$class))prop.table(table(test_data$class))neg   pos 0.983 0.017neg   pos 0.977 0.023
Ok, looks like have a problem here. Class is totally imbalanced.
We could come up with a useless model that classified every observation as neg and get 97.7% accuracy, so let’s be careful with our accuracy score interpretation.
Later we’ll see how we can deal with class imbalance.
2.6) Rest of the features. Summary statistics. Understand your data
We are working with 171 features, so calling summary() on the entire dataset is not going to help a lot in terms of visual interpretation. Instead, we are going to create a data frame out of the summary() function. This will enable us to calculate some new statistics, specifically related to missing values, which as you will see, is another big issue of this data.
With this summary data frame we will also calculate the mean quartiles for all the data. This will also allow us to understand more about the distribution of the features and the average number of missing values.
options(scipen = 999)summary_df <- do.call(cbind, lapply(training_data[,                       2:ncol(training_data)], summary))summary_df_t <- as.data.frame(round(t(summary_df),0))names(summary_df_t)[7] <- paste("Missing_values")summary_df_t_2 <- summary_df_t %>%                       mutate(obs = nrow(training_data),                             Missing_prop = Missing_values / obs)print(summary_df_t_2)
summary_df_t_2 %>% summarise(Min = mean(Min.),                             first_Q = mean(`1st Qu.`),                             Median = median(Median),                             Mean = mean(Mean),                             third_Q = mean(`3rd Qu.`),                             Max = max(Max.),                             mean_MV = mean(Missing_values),                             obs = mean(obs),                             mean_MV_perc = mean_MV / obs)
We can see above that the average number of missing values per feature is 5,000 out of 60,000 samples. This means we have 8.3% o missing values in average in each column. That is a lot!
Let’s check the same for our test set:
Again, we have 8.4% missing values. We’ll have to deal with them and there’s a specific section for that afterwards.
2.7) Categorical data/Factors: create count tables to understand different categories. Check all of them.
We are not working with any other categorical feature other than our response variable.
2.8) Unnecessary columns? Columns we can quickly understand we don’t need. Drop them
So far, we haven’t detected any column that we want to remove at this time. We will leave this for later.
2.9) Check for missing values. How many? Where? Delete them? Impute them?
Our features present more than 8% missing values in average.
Because it’s a lot of information we don’t want to lose, our approach is going to be to impute them. But first, we don’t want to use a technique to impute each of our two sets separately (train and test). It has to be a one time imputation using full information. In order to do this, we are going to combine both sets, work on them and then separate again.
#replicate our setstraining_data_bind <- training_datatest_data_bind <- test_data#create a new column "set" to label the observationstraining_data_bind$set <- "TRAIN"test_data_bind$set <- "TEST"#merge them into 1 single setfull_dataset <- rbind(training_data_bind, test_data_bind)dim(full_dataset)[1] 76000   172
We end up with a single set containing 76,000 samples (16,000 from test set and 60,000 from train set). The number of columns is 172 (171 features + 1 “set” column)
Missing values imputation
We are going to impute missing values using package mice. Here’s a nice explanation of how mice works
The following formula allows us to impute the full dataset using mean imputation:
set.seed(123)imputed_full <- mice(full_dataset,                      m=1,                      maxit = 5,                      method = "mean",                      seed = 500)
Now we store the imputed values:
full_imputed <- complete(imputed_full, 1)
We then check that we still maintain the same dimensions:
dim(full_imputed)[1] 76000   172
And now let’s check that we don’t have missing values:
(na_count_full_imputed <-data.frame(sapply(full_imputed, function(y) sum(length(which(is.na(y)))))))
Wait. If you run that code and look at each row, you will notice there are some features that still have missing values. Specifically there are 9 rows.
Let’s take a look at what is causing them:
issue_columns <- subset(imputed_full$loggedEvents,                         meth == "constant" | meth == "collinear")print(issue_columns)
I filtered the $loggedEvents attribute of the imputed data frame. Here we see that some features were tagged as constant or collinear. One of them is our set column (the one we used to combine the two sets into one), so we don’t worry about that one. The rest of them are mainly collinear variables and one constant variable. mice automatically skips those columns and lets us know of the issue. We will want to drop those features so we have a full dataset without missing values.
To do that, we first store the column names being careful not to store the set column name (we need it)
#create vector of column namesissue_columns_names <- as.character(issue_columns[, "out"])issue_columns_names <- issue_columns_names[-2]print(issue_columns_names)[1] "cd_000" "bt_000" "ah_000" "bu_000" "bv_000" "cq_000" "cf_000" "co_000"
We then use the stored vector to remove those columns from the data frame and store it as our final imputed data frame:
full_imputed_filtered <- full_imputed[ , !(names(full_imputed) %in%                                       issue_columns_names)]dim(full_imputed_filtered)[1] 76000   164
Notice the number of columns reduced from 172 to 164.
Now we don’t have missing values any more! (check it!)
Finally, it’s time to separate our full imputed dataset into train and test sets again, and we need to separate it into the exact same samples that it was splitted before. In order to do that we just filter the data frame using our set column.
#subset the full_imputed_filtered datasettraining_data_imp <- subset(full_imputed_filtered, set == "TRAIN")test_data_imp <- subset(full_imputed_filtered, set == "TEST")#drop the "set" column, we don't need it anymoretraining_data_imp$set <- NULLtest_data_imp$set <- NULL#check dimensionsdim(training_data_imp)dim(test_data_imp)[1] 60000   163[1] 16000   163
Great!, we have our training and test set splitted and with no missing values!
2.10) Check for outliers and other inconsistent data points. Box-plots. Cooks’ distance. DBSCAN?
For this dataset we are going to use cook’s distance:
cooksd <- cooks.distance(glm(class ~ .,                              family = "binomial",                              data = training_data_imp))
Let’s plot the results:
plot(cooksd,      pch="*",      cex=2,      main="Influential Obs by Cooks distance")  abline(h = 4*mean(cooksd, na.rm=T), col="red")
In this plot, what seems to be a dark thick black line is actually all our data points. In the right-top corner we see also what seems to be 1 outlier, or a bunch of them grouped.
Let’s check how many are there:
outliers <- rownames(training_data_imp[cooksd > 4*mean(cooksd, na.rm=T), ])print(outliers)[1] "617"   "3993"  "5349"  "10383" "10829" "18764" "19301" "21138" "22787" "24360" "24975" "29146" "30633" "33684"[15] "38785" "45978" "50283" "51003" "52573" "53283" "54957" "57186"
This is a total of 22 points that are considered outliers according to cook’s distance test.
Since this is a very low number of observations compared to our total 60,000 I decided not to remove them. This just seems to be variability in the measurement rather than experimental error.
2.11) Check for multicollinearity in numeric data
Last step ouf our EDA phase is to check for multicollinearity. We have already removed some collumns which mice detected as collinear. Here we will just do some analysis of how many bivariate relations have high correlation and if it’s a significant high proportion:
sum((correlation > 0.5 | correlation < -0.5) & correlation < 1) / (162*162)[1] 0.12sum((correlation > 0.7 | correlation < -0.7) & correlation < 1) / (162*162)[1] 0.047sum((correlation > 0.9 | correlation < -0.9) & correlation < 1) / (162*162)[1] 0.0055
This means that out of 26,244 possible variable correlations:
12% present scores higher than 0.5
4.7% higher than 0.7
and 0.55% higher than 0.9
I won’t go any further than this and consider multicollinearity not to be a big issue at this point.
We won’t also spend time engineering features in this already heavy feature loaded dataset. There is no additional information we could use.
We have now both our training and testing data sets ready for modelling. We have imputed missing values, removed collinear features and verified that outliers and multicollinearity is not a big deal that we should be concerned about.
Due to the size of the data, we will train a Logistic Regression model and a Naive Bayes model. Both are very fast compared to more advanced algorithms such as random forests, SVMs or gradient boosting models.
I’m not looking here to win the contest but to have an acceptable scoring just to demonstrate.
I’ll use caretEnsemble’s caretList()to train both at the same time and with the same resampling .
Notice I’m specifying “up”-sampling within trainControl(). This will take care of class imbalance. More about upsampling.
registerDoParallel(3)getDoParWorkers()set.seed(123)my_ctrl <- trainControl(method = "cv",                         number = 5,                        classProbs = TRUE,                        savePredictions = "final",                        index =                         createResample(training_data_imp$class, 3),                        sampling = "up",                        allowParallel = TRUE)model_list <- caretList(class ~ .,                        data = training_data_imp,                        methodList = c("glm", "nb"),                        metric = "Kappa",                        tuneList = NULL,                        continue_on_fail = FALSE,                          preProcess = c("center", "scale"),                        trControl = my_ctrl)
Our results:
Notice that our accuracy scores are lower than our usless predict-all-neg model that had 97,7%. The reason we don’t score higher is because we upsampled the data, thus generated new data points to fix the class imbalance. We did it as a parameter set within caret’s trainControl so I’m not showing any details of that, but by doing it, we have improved our ability to predict the positive class (in this case “neg”) in detriment of just looking to maximize accuracy.
Let’s quickly check the confusion matrices:
#Logistic Regression modelconfusionMatrix(predict(model_list$glm,test_data_imp, type = "raw"), test_data_imp$class)#Naive Bayes modelconfusionMatrix(predict(model_list$nb,test_data_imp, type = "raw"), test_data_imp$class)
According to the cost matrix for each type of error, our scoring is:
Logistic Regression model: 52 x 500 + 834 x 10 = 34,340
Naive Bayes Model: 42 x 500 + 498 x 10 = 25,980
This is fine-ish performance for this quick of a modelling. I bet that with some more work we can get very close to the best 3 contestants: