Crafting a Machine Learning Model to Predict Student Retention Using R | by Luciano Vilas Boas | Towards Data Science

First and foremost, let’s start by defining what student retention is, at least in the scope of this article. We’ll define it, as the indicator that tells us if a student that started in College for the first time in a particular Fall semester, came back to the following next Fall (or not). For instance, let’s say a student started on particular University for the first time in the Fall of 2018. If this student enrolled himself/herself for Fall 2019, then this student was retained.
Another common names for retention are, persistence, and/or drop out. Here, these names mean the same thing: one year undergraduate retention. The reader may ask at this point, “Why is student retention important anyways?” That’s a fair question, and without the pretension of exhausting this answer, we can say that is important for a myriad of reasons, starting with the financial impact, the ranks and prestige that schools can get and the list may go on and on. By the way, when we say “school” here, we are specifically referring to Universities and Colleges (Higher Education).
Finding a dataset that goes at the student level is very hard — if you have one and want to send me, please do so, but it must be anonymized. Meaning, no student’s name or ID, or any other information that allows the researcher to identify the student — particularly, when we have regulations that rightfully, protect student’s data, such as FERPA. This is a big deal, and we must be very careful when handling sensitive information.
That is to say, locating the right dataset, explicitly for this experiment, really imposes a challenging. However, and on the positive side, we managed to find one dataset that we could use, and that is the one we’ll be manipulating here. You can find it at UCI Machine Learning Repository, just click here.
To use this data, it’s requested that we do a proper citation, and please refer toe the “Reference” section of this article to check that out. On an additional note, a big shout out for these researchers that made this dataset available. Thank you.
Although we found this dataset, the truth is that it’s really a small dataset for a Machine Learning (ML) project, and on top of that, we’ll see later, that the variables are not really relevant in the big scheme of things. So, ultimately, the ML model can potentially have a poor performance. We’ll cover more on that as we go.
Another important point to emphasize is that, originally, this dataset was used to predict student performance [1], and NOT retention. Funny enough, the dataset has interesting features, but with no relevant significance when predicting the performance [1], and the retention. What I mean here, is that despite the fact that we have several variables, basically, just very few of them tell us the history.
We are going to use a lot of creativity here. Note that the original data is about students in High School. But for here, let’s imagine that those students are College students. Additionally, we had to create and incorporate a new feature called “retention”, which was randomly created to simulate whether or not a student was retained. Apparently, this had a negative impact on the model, since this is actually our predictor variable — and it’s random.
In spite of that, the main goal of performing this experiment is to make available a machine learning pipeline that can be used, especially with the right dataset, to leverage student retention in Higher Ed. I hope this ML project can help administrators and researches on that regard. Here is how we broke down our task into four main phases as following:
Phase 1: Data Munging
In this phase, we’ll look for missing data, and get familiar with the dataset. Curiously, we have not found any missing data here. While one may think that this is good, which ultimately is, it also makes us think if some imputation was done prior to us working with this dataset. Here is a tip: whenever you decide to input missing values, please document that by leaving the original column with the missing values, and the new column with the imputation on the dataset. Additionally, state which technique was used to input values of if you just dropped. By reading the original article that was used with this dataset, we came to the conclusion that the missing data was dropped out, and what we have was the actual responses, meaning we have no missing values.
Phase 2: Exploratory Analysis
I can never get my head around of people that perform “Exploratory Analysis” without knowing which variables are really relevant in a particular dataset. Don’t go around plotting charts randomly. It’s okay to have some initial assumptions, but you should always try to figure out what features matter for what you are trying to study. So, some of our plots were done based on what we found to be relevant and what we wanted to see, but mostly, based on scientific understanding of the dataset.
Phase 3: Dimensionality Reduction
In my opinion, one of the most important steps. Particularly, because this dataset is really small and it has a considerable number of variables. Here, we’ll look for multicollinearity, and how to eliminate variables. We’ll use Logistic Regression to get rid of irrelevant variables. On top of that, we’ll also performed PCA (Principal Component Analysis) with the goal of figuring out relevant features.
Phase 4: Machine Learning
That’s the part we perform a ML modeling and get awesome results, right!? Well, it turns out that this phase wast quite challenging, and we’ll get to that later. But some of the techniques we used here was to balance the predictor using ROSE [2] library, assessing the AUC (Area Under to Curve) and ROC curve, and checking accuracy and kappa scores of multiple algorithms in order to check each one works best with this data.
Small datasets tend to impose challenges for ML algorithms. Especially when features are not really significant and the predictor is artificially created, random, and imbalanced. What a recipe for a disaster, huh?!
With this in mind, and even before actually starting to work with this dataset, we came up with these following assumptions, but not limited to:
Above we can see the six main assumptions and some of their causes. We won’t go through all of them here, for the sake of time, but we tried to mitigate the effects of them as much as we could when coding and developing the model. Note that the Imbalanced and Random Classifier issues were created for this project, and they are not part of the original dataset.
Remember, the reason we were anticipating these problems has to do with the fact we artificially created a new feature called retention, which sorted the binary values (1 or 0) randomly, according to the weight that was given to them.
To make it clear, the total students in the dataset we are using is just 649, and when creating this artificial predictor — retention — we said that ~85% of those students would be flagged as “retained” (binary indicator is 1) and ~15% “not retained” (binary indicator 0). That’s pretty much the average retention rate for a public College/University.
We used the statistical and open-source application called R to perform this task. And, to make this more visually appealing (that’s our hope), we won’t be showing all codes here, but you’ll find everything on the link as shown below.
All codes and files can be found here on my GitHub page.
Likewise, we’ll break down this hands-on section into the same four phases we saw previously. Without further ado, let’s start from the beginning:
Again, you can find the original dataset and paper on UCI ML Repository. But, here is a snapshot of all variables for you:
Keep in mind that the retention variable that we talked before, was inserted into this dataset set artificially in order to simulate predictive modeling for student’s retention. You don’t see it here in this table, but in the R codes. There are also two datasets available, one for Math and one for Portuguese. We’ll only use the one for Portuguese simply because this was the one with more students (649) on it. The article [1] combined Math and Portuguese into one single set, but that has an even smaller sample size.
One of the main goals in this phase 1 was to create the artificial variable, get familiarized with this dataset, and check for missing values.
As we can see in the plot 1, there were no missing values in this dataset. As a matter of fact, we have stated that previously, but here is the proof. Note that we added the retention variable in this plot.
Additionally, I had to create dummy variables for some of the variables in the dataset that was not dummy-coded. At the end of this processing, a new csv file, just with the dummy codes, numeric values, and the predictor, can be exported using the R codes provided.
Now that we have the green light, and the dataset ready for the initial assessments, we can move to the next step.
This is where people go crazy on plotting histograms and box plots, isn’t it?! They are necessary, don’t get me wrong. We’ll see some of those in a sec. Also, you can check things like skewness, kurtosis and other basic statistical checklist on the R codes.
In this simple bar chart, we can see the counts for the predictor variable, according to the weight we set for them when creating this artificial predictor.
As we stated previously, students that were retained are flagged with “1” (538) and students not retained “0” (111).
Ultimately, it also shows the imbalance between the classifier, and we’ll take care of later.
In the original article [1], the variable “G1” was a very strong predictor for student’s performance. The following scatterplot tells us a little bit of this expected positive correlation between the first grade (G1), and the final (G3). We can also spot some potential outliers.
Continuing with this analysis, we were interested to see what could impact student’s performance on G1. For that, we focused on checking the studytime, and the freetime students have available, and we broke that down by gender.
Okay, I surrender! Here is where I come up with a box plot for all variables. We can easily see here where we have presence of outliers, and quickly think about some option to whether or not deal with this issue. For this testing, we are not going to exclude the outliers.
Once more, I want to give some attention to the variable G1, so let’s check it’s distribution as following below:
Considering this is a real world data, its distribution it’s not too bad, really. But, it’s not perfect — obviously.
In conclusion of this Phase 3, we just want to check one more thing. That is, how G1 and G2 variables interact when adding the students into a bivariate plot, like the one as following below:
Think about this bivariate plot as being an ordinary box plot, but mixing together two variables instead of just one. Specifically, the numbers in blue could be the students ID (that’s not the case here because we don’t have students IDs in the dataset). Any student outside of the bigger ellipse would be an outlier. This could be helpful when trying to identify which students we would reach out in order to take some proactive actions towards to, let’s say, improve their grades on G3.
This is where the fun starts. A good practice is to always check for collinearity/multicollinearity among the dataset. The easiest way to do that is by checking the correlation that independent and dependent variables may have. If two or more variables have a very strong or almost perfect correlation, that might be an indication of collinearity/multicollinearity.
When facing a scenario where two or more variables have a strong correlation, we want to eliminate one of them. Why? Because in essence, you have two variables telling you the same story. But being a little be more technical, collinearity can mess up with the variance of at least one estimated regression coefficient, and this can lead to some regressor to come up with the wrong sign. Which can deteriorate your analysis and lead you the wrong way.
Let’s check if we can spot some of this issue using the following Correlation Matrix:
There are some interesting correlations here, but mainly about the grades (G1, G2, and G3). They seem to have a very strong correlation, indicating some potential collinearity. We also see a strong negative correlation between grades and failures, as well as some other features. You can always check the actual numbers using a correlation table, or by simply adding them directly on the plot itself. For now, we just want to get a big picture of how the variables may be correlated.
The Principal Component Analysis (PCA) is used to shape the original variables into a new set of features, which are linear combinations of the original variables. PCA’s main goal is to reduce the number of variables, but considering the original variation as much as possible.
In the first component, or dimension, you’ll see a combination of the original variables whose sample variance is greatest amongst all possible linear combinations. In the second dimension, we account for the maximal proportion of the rest of the remaining variance, which can be uncorrelated with the first dimension. The other components follows a similar approach.
When analyzing the output from running a PCA, we want to see a “Cumulative Proportion”, between dimension one and dimension two, that is significative. Generally speaking, a cumulative proportion equal or greater than 0.7 is considered to be very good. Meaning, the reduction from multiple variables into two dimensions can still have a strong representation of the entire dataset.
One important thing prior to performing a PCA is to make sure the dataset is standardized. Deciding whether or not to remove outliers is also a good practice at this point.
To kick off our analysis using PCA, we’ll start with a bar chart that displays all the cumulative proportions among the components as well as their variances.
In essence, we are particularly interested in shrinking down our analysis up to the first two dimensions. We want to make sure that the cumulative proportion between them both are significant enough to “explain” the rest of the dataset.
Unfortunately, as you’ll see in the next plot, the cumulative proportion is just about 21% (12.9% + 8.1%). So reducing into two dimensions would not be recommended, hence, we would be losing information and end up with two dimensions that don’t “explain” the dataset much.
From the biplot, we can also see the correlation among the variables e.g., grades (G1, G2 and G3) and failures pointing on the opposite direction. Although the cumulative proportion of the two dimensions are not really useful when boils down to dimensionality reduction of this dataset, we want to check which are the contributions of the top ten variables for each dimension. This might help us to understand a lit bit more about this dataset.
We can see that in Dim 1, grades are the variables that contributes the most in this dimension. But, it’s important to stress that we still have that assumption of collinearity present here. So this is one more clue that, indeed, we may have collinearity here because we see all grades with very similar contribution levels. On the other hand, in the Dim 2 the Walc feature has the highest contribution.
In summary, if one asks what Dim 1 and Dim 2 is about, we can say that Dim 1 is about student grades, and Dim 2 student’s alcohol consumption. Meaning that we should expect some of these variables to have statistical significance when predicting the retention.
All of this testing would be extremely helpful when guiding our decision about which variables to keep and which ones to disregard moving further in our analysis. Unfortunately, we won’t be using PCA to take any decisions just yet.
Circling back, the hypothesis that we have collinearity is still alive, but how to measure that, and how to decide each variable to get rid of?
In the light of acknowledging this issue, and with this in mind, we’ll take advantage of VIF (Variable Inflation Factors) to detect and support our decision towards to the best approach when dealing with multicollinearity. To make it short, VIF detects how strong the correlation between variable is, and its score guide us on each variable(s) to drop and mitigate collinearity.
As a general rule of thumb, VIF scores equal to or greater than 5 suggests collinearity. We will use 5 as our threshold here as well. As we can see below, the VIF output, which was based on a Logistic Regression, tell us that G2 and G3 are the variables we should disregard moving further on our analysis.
Usually, you should get rid of one variable at the time and run the analysis again. That is, G2 would be the first to leave, and G3 the second. Here is a graphical way to visualize the before and after of the removal of these two variables (G2 and G3):
As a result of all this wrangling, we just got rid of two variables. That was not really productive in some sense. We still have a considerable large set of features that doesn’t tell much about what we trying to do. So, our shortcut will be using the Logistic Regression to not only help us in finding the predictors but also to eliminate non statistically significant variables.
As has been noted from the Logit Regression output — on the left side — we have five variables that could work as predictors.
However, only three of them has meaningful significance at 0.05 - we marked them down with a red arrow.
The R-squared is a common indicator that some researchers may use when picking a model over others. Here, we would just use the AIC (Akaike Information Criterion) score to compare the output with 5 variable, against the output with 3 variables.
As you can see below, eliminating two of the less significant variables doesn’t affect the AIC score much.
All five statistically significant variables: AIC: 589.84
All three statistically significant variables at 0.05: AIC: 589.67M
Yes, the AIC for the whole dataset is bigger (622.4), but because it’s full of not really useful features, we are taking this curve here and deciding to work with only these three variables in the next phase.
Considering all the challenges we have been facing so far, what could possibly go wrong when modeling the ML model? A poor performance model is what can go wrong. But, we want to get the best outcome possible, and at least a fair model.
That being said, we gotta be real here, ML is really suitable for large datasets. But with some tweaks we can make it work. In this case, we’ll take advantage of a library called ROSE to balance the predictor and improve the model’s performance.
Originally, our predictor was split this way: 1= 538, and 0= 111. As you can clearly see, we have an imbalanced classifier. We’ll artificially balance it to improve the outcome and mitigate some of the statistical issues that comes along with it. However, we’ll do that over the quantities for the trainSplit. Thus, our train set has a total of 553 students, where: 1= 458, and 0= 95. In other words, we’ll be balancing the training data and not the entire dataset for now.
ROSE offers some different options to balance the classifier. We’ll cover 4 different approaches and pick the one that gives us the best result. Below are the codes we borrowed from the paper “ROSE: A Package for Binary Imbalanced Learning” [2].
### Balancing the data:# Resampling Option 1 (over):data.bal.ov <- ovun.sample(retention ~ ., data = trainSplit, method = "over", p=0.5, seed = 2)$datatable(data.bal.ov$retention)# Resampling Option 2 (under):data.bal.un <- ovun.sample(retention ~ ., data = trainSplit, method = "under", p = 0.5, seed = 2)$data                           table(data.bal.un$retention)# Resampling Option 3 (both):data.bal.ou <- ovun.sample(retention ~ ., data = trainSplit, method = "both", N = 553, p = 0.5, seed = 2)$data                           table(data.bal.ou$retention)# Resampling Option 4 (ROSE):data.rose <- ROSE(retention ~ ., data = trainSplit, seed = 1)$datatable(data.rose$retention)
Resampling Option 1 (over): Here we oversampling the “0” until it’s leveled with the counts of “ones”. As a result, we’ve balanced the classifier as following: 1= 458, and 0=453.
Resampling Option 2 (under): By undersampling the majority class (1) to match the minority, we get this result: 1=97, and 0=95.
Resampling Option 3 (both): In this case we use the entire train set (553) and split, such as: 1=281, and 0=272.
Resampling Option 4 (ROSE): Finally, we use ROSE to split the data and we got this outcome: 1=293, and 0=260.
Next, we train the classifiers, run the test set using Classification Trees, and plot the ROC Curve.
# Training the Classifiers and run test set using classification trees:library(rpart)tree.ov <- rpart(retention ~ ., data = data.bal.ov)tree.un <- rpart(retention ~ ., data = data.bal.un)tree.ou <- rpart(retention ~ ., data = data.bal.ou)tree.rose <- rpart(retention ~ ., data = data.rose)# Predict in the new data (test):pred.tree.ov <- predict(tree.ov, newdata = testSplit)pred.tree.un <- predict(tree.un, newdata = testSplit)pred.tree.ou <- predict(tree.un, newdata = testSplit)pred.tree.rose <- predict(tree.rose, newdata = testSplit)# Plot ROC Curve - Model Evaluation:roc.curve(testSplit$retention, pred.tree.rose[,2], col = 0, main= "AUC: 0.75", lty = 1)roc.curve(testSplit$retention, pred.tree.ov[,2], add.roc = TRUE, col = 12, lty = 2) roc.curve(testSplit$retention, pred.tree.un[,2], add.roc = TRUE, col = 0, lty = 3) roc.curve(testSplit$retention, pred.tree.ou[,2], add.roc = TRUE, col = 0, lty = 4)
Resampling Option 2 (under) has given us the best AUC (Area Under the Curve) output. See below:
The AUC is an algorithm, which is widely used to measure the trade-off between the True Positive (TP) and False Positive (FP) rates of a said ML classifier.
Likewise, a model that fails to predict will have a score of 0%, while a model that is able to make “perfect” predictions will get a score of 100%.
Accordingly, our model scored 75%, which is not too bad, but a number ≥80% would be preferable.
On the positive side, using the ROSE package gave us a huge improvement over the imbalanced scenario. So, we’ll celebrate this as a big victory. YAY!
Another very common way to measure classifiers is by getting a Decision Tree, which would give us more intuition on how the model is performing. We’ll skip that here, and move to the next and final piece of this ML analysis.
A lot of people rely on accuracy to pick a ML model. That alone can be misleading, but since this is a very “quick and dirty” type of project, I’ll allow myself to experiment with that.
Accuracy: it basically pinpoints the proportion of “1” and “0” that were flagged correctly.
Kappa: tells us the performance of the classifier in comparison with values assigned randomly.
Without delay, here is how we’ll handle this part of the analysis: we’ll again use ROSE to balance the entire dataset now (649 rows). This time we won’t split between train and test, and that is because the set is very small. We are going to use everything we have.
Similarly, we’ll use the following code to balance the predictor, but as you’ll see, we are artificially duplicating the dataset up to 1298 observations, and using the “both” method here.
# Create balanced training data for LR:dataC <- ovun.sample(retention ~ ., data = dataC, method = "both",                        N = 1298, seed =1)$data # 1298 (649x2)table(dataC$retention)
Therefore, we get the new balanced classifier such as: 1=652, and 0=646. Now, we will use the “mlbench” library to run a benchmark comparison between few different algorithms to see each one(s) has the best performance.
As we can see above, the “best” algorithm for this analysis was the svm (Support Vector Machines), with an accuracy of almost 70%, and kappa of 39%. Those are fair results, but definitely not optimal.
Keep in mind that when choosing each model to actually use in production, many other factors has to be taken in consideration, and not only the accuracy scores alone. For the purpose of what we were trying to accomplish, our ML analysis ends here.
In conclusion, you saw that the three main predictors were Medu, famsup_yes, and Dalc. A next step would be getting the odds ratio out of the Logit Regression and make the analysis more interpretable. I’ll leave that for future improvements and save our time here.
As a reminder, our goal here was to provide some guidelines of how to use multiple techniques to work with a small dataset and get a fair to good results. There are many other ways to tackle this problem, and here we are providing you a solid one - using a real world dataset.
With a more well-designed dataset that is suited to predict retention, and with more observations, we can certainly expect to have better results. This analysis was created for that purpose.
Ultimately, retention rates is one of the most important indicator in Higher Education, and this pipeline can be useful to further research on the subject, and for administrators to make better decisions based on data.
‘Till next time!
References:
[1] P. Cortez and A. Silva. Using Data Mining to Predict Secondary School Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5–12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978–9077381–39–7.
[2] Nicola Lunardon, Giovanna Menardi, and Nicola Torelli. ROSE: A Package for Binay Imbalanced Learning. The R Journal Vol. 6/1, June 2014. ISSN 2073-4859.
[3] Whitlock, Joshua Lee, "Using Data Science and Predictive Analytics to Understand 4-Year University Student Churn" (2018). Electronic Theses and Dissertations. Paper 3356. https://dc.etsu.edu/etd/3356