Practical Explainable AI: Loan Approval Use Case | by Raheel Ahmad | Towards Data Science

Explainability is extremely crucial in systems that are responsible for carrying out mission-critical tasks. For example, in healthcare, if scientists are relying on AI models to help them figure out whether the patient will have cancer or not, they need to be 100% sure of their diagnoses otherwise this can result in death, a lot of lawsuits and a lot of damage done in trust. Nature is this problem is so intense that explainability sits at the core of this problem: the data scientists and the human operators, in this case, the doctors need to understand how the machine learning system is behaving and how did it come to a decision.
Explainable AI is also important in finance or fintech in particular due to the growing adoption of machine learning solutions for credit scoring, loan approval, insurance, investment decisions and so on. Here again, there is a cost associated with the wrong decisions by the machine learning system: so there is a huge need to understand how the model actually works.
Using black-box AI increases business risk and exposes the businesses to a deep downside — from credit card applications to determining disease to criminal justice.
The reason why black-box models are not desirable becomes more clear when we look at how the business functions as a whole:
For the business decision-maker, data scientists need to answer the question of why they can trust our models, for IT & Operation, data scientists need to tell them how can they monitor and debug if an error occurs, for the data scientist, they need to know how they can further improve the accuracy of their models and finally, for regulators and auditors, they need to be able to get an answer to whether our AI system is fair or not?
Explainable AI aims at providing clear and transparent predictions. An end-to-end system that provides decisions and explanations to the user and ultimately provides automated feedback to constantly improve the AI system. Remember, xAI is highly driven by the feedback so is a two-way interaction between the human and the AI system.
In the end, we should understand why behind the model, the impact of the model, where the model fails and what recommendations to provide.
explainx is an open-source explainable AI platform created by explainX.ai. Explainx, written in python, aims to help data scientists explain, monitor and debug black-box AI model — the purpose is to help build robust, unbiased and transparent AI applications.
Within the explainx architecture, explainx provides access to state of the art interpretability techniques in just a single line of code within our Jupyter notebook.
For this example, we will use the HELOC dataset provided by FICO. The customers in this dataset have requested a credit line in the range of USD 5,000–150,000. Our job is to predict RiskPerformance: whether they will make timely payments over a two year period. The prediction can then be used to decide whether the homeowner qualifies for a line of credit.
For this example, we will train CatBoost classifier model. After the training is done, we will use the explainx xAI module to explain our model and build a narrative for a business user to understand!
Let’s start with opening up our Jupyter notebooks and installing the explainx library. You can also clone the repository from the link below:
https://github.com/explainX/explainx
pip install explainx
Let’s import relevant packages:
from explainx import *import catboostfrom sklearn.model_selection import train_test_split
Let’s load and pre-process our dataset for model building. The dataset is already available in the explainx library.
X, y = explainx.dataset_heloc()#split data into train and testX_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.2, random_state=0
Begin training. For the sake of this tutorial, we are going to keep it simple!
# Run catboost modelmodel = CatBoostClassifier(iterations=500,                          learning_rate=.3,                          depth=2)# Fit modelmodel.fit(X_train.to_numpy(), y_train)
After the training is done, we can simply pass the test data into the explainx function and get our explanations!
explainx.ai(X_test, y_test, model, model_name="catboost")
Once explainx and running, all you need to do is point your browser to http://127.0.0.1:8050 and you’ll see a very nice user interface called explainX.Dashboard.
App running on https://127.0.0.1:8050/
Note: If you want to view it inline, simple pass mode="inline” argument into the explainx function.
For this tutorial, we will not go in the nitty-gritty of model building, model metrics and evaluation. Instead, we will dive right into the explanation part which is the main aim of this tutorial. So let’s start opening up the black-box!
We will tackle the model at four levels:
Global Level Explanation
Local Prediction Explanation
Scenario Analysis
Feature Interaction & Distributions
We will be using the overall feature importance and overall feature impact graphs to give us a basic underlying logic of the model.
Interpretation: This tells that according to the CatBoost model, ExternalRiskEstimate, MSinceMostRecentInq and PercentTradesNeverDelqare the top three variables with the largest impact on RiskPerformance. These three represents Risk Estimate, Credit Inquiries and Debt Level information: extremely important categories when assessing risk.
This information gives us a general idea of feature contribution but to understand whether each of these features has a positive or a negative impact on RiskPerformancewe need to consult the feature impact graph.
This graph gives us even more insight into the model logic. We can clearly observe that ExternalRiskEstimate impacts the predicting variable positively — pushes it towards “Good Credit Risk Performance” and that matches our intuition as well. We have to use a little bit of our domain knowledge in finance for this: ExternalRiskEstimateis a consolidated version of some risk markers (higher is better) so automatically we learn that this variable will always positively affect the prediction. Then we have NumSatisfactoryTrades:the number of “satisfactory” accounts (“trades”) has a significant positive effect on the predicted probability of good credit.
However, in the lower end, delinquency variables push the overall prediction towards 0 (in this case, application denied). This is interesting and we can even dig deeper and see when does this negative effect of delinquency wears off? (Something to try on your own!)
Now as we have an idea of how each feature affects the prediction, we can move on to explaining a single prediction for a specific customer. For that, we will be using an impact graph or a decision plot that will help us get attribution scores for that particular prediction. To further support our analysis, we will calculate similar profiles that most closely resemble the one we are trying to predict.
So let’s explain for customer on Row # 9 in our data. For this specific customer, the application was approved because the RiskPerformance was “Good” and our model also classified it correctly!
Let’s explore the model logic:
So this graph clearly shows the top three positive and top three negatively impacting variables. According to the model, MSinceMostRecentInq had the most positive impact on the variable. So this tells us that the higher value of this variable means that there is no penalty for having more than one month since the most recent inquiry. Then we have ExternalRiskEstimate that again plays a positive role in pushing the prediction towards “good credit behaviour”. However, PercentTradesNeverDelq affected the prediction negatively: this might be the case if the value of this variable is extremely small because the smaller value for this variable lowers the probability of good credit score.
To keep it short, these findings match our mental-models as the attribution score of each variable is correctly assigned. To further support our analysis, we will find similar customers!
Explainx comes with an in-built prototypical analysis function which provides a much more well rounded and comprehensive view of why the decision for the applicant may be justifiable.
The above table depicts the five closest user profiles to the chosen applicant. Based on importance weight assigned to each profile by the method, we see that the prototype under column zero is the most representative user profile by far. This is (intuitively) confirmed from the feature similarity where more than 50% of the features (12 out of 23) of this prototype are identical to that of the chosen user whose prediction we want to explain. Also, the bank employee looking at the prototypical users and their features surmise that the approved applicant belongs to a group with high values of ExternalRiskEstimate. This justification gives the bank employees more confidence in approving the user’s application.
Now, let’s explore different scenarios and see how the model performs. We can apply the data filters within the data table (no need to write SQL queries to filter your data) and explain multiple instances and scenarios very easily.
This is extremely useful when you are trying to understand behaviour on a specific cluster or group of data. For example, we want to see if the model attributes the same weights to the users when the ExternalRiskEstimate is > 60 and MSinceOldestTradeOpen is greater than 200.
We can clearly see the dominance of ExternalRiskEstimate from the chart below.
When the ExternalRiskEstimate is greater than 60, it is seen as a positive sign which matches our internal mental model as well: ExternalRiskEstimate is actually monotonically decreasing which means that as the value of ExternalRiskEstimate increases, the probability of Bad Credit Score decreases! So in this cluster where the RiskEstimate is greater than 60, we will have more Good customers who were extended the credit line. We can confirm this by using a feature interaction plot, specifically a partial dependence plot:
The partial dependence plot validates our assumption. In the plot below, Red = Good RiskPerformance (credit line extended) and Silver = Bad RiskPerformance (credit line denied).
We can clearly see that pattern that as ExternalRiskEstimate value increases, its impacts on the output also increases and we have more instances of good risk performance. We see more concentration of red dots as the ExternalRiskEstimate value goes up and that makes a lot of sense!
Data Scientists can use explainx to further explore patterns by looking at interactions between different variables and how they impact the overall prediction. So let’s end by summarizing our findings for a business user to understand:
According to the CatBoost model, ExternalRiskEstimate, MSinceMostRecentInq and PercentTradesNeverDelqare the top three variables with the largest impact on RiskPerformance.
ExternalRiskEstimate impacts the predicting variable positively — pushes it towards “Good Credit Risk” but delinquency variables push the overall prediction towards “Bad Credit Risk” (in this case, application denied).
For our customer # 9, RiskPerformance = Good, ExternalRiskEstimate played a positive role in pushing the prediction towards “good credit behaviour”. However, PercentTradesNeverDelq affected the prediction negatively: this might be the case if the value of this variable is extremely small because the smaller value for this variable lowers the probability of good credit score.
We found customers that had very similar behaviour and variable values to our customer # 9: strengthening our hypothesis even more.
We were able to validate the model’s logic by further exploring the PDP that clearly showed us how increasing the value of ExternalRiskEstimate increases the probability of loan approval.
Data Scientist can further explore a similar process and build an even more comprehensive data narrative that anyone can easily understand.
I hope you all enjoyed this case study. Explainability is extremely crucial and more than relevant today — so the ability to present a narrative that shows your understanding of how AI works is a vital skill for a data scientist. This is the real essence of human-AI understanding and democratization of AI.
Download explainx: https://github.com/explainX/explainxDocumentation: https://explainx-documentation.netlify.app/