3 ways to interpretate your NLP model to management and customer | by Edward Ma | Towards Data Science

Machine Learning (ML) and Artificial Intelligence (AI) changed the landscape in doing business. Lots of company hire data scientist (I am one of them:)) to deliver a data product.
It is absolute easy to deliver a engine with “high” accuracy (Here is the reason why I double quoted). If somebody ask about reason, we can say that the model is black box or it is a statistical calculation few year ago but we cannot say that anymore nowadays. We also need to convince product team and management to trust our model but we cannot simply say that the engine “learnt” from data or it is a black box which cannot be explained as well.
Of course, we may not need to explain model in some special scenario. For example, you do not want to explain the fraud detection model to public. Otherwise, public can find a way to “hack” the system.
After reading this article, you will understand:
What is model interpretation?
Why do we need to interpretate model?
Interpreting NLP model
Take Away
Reference
Model interpretation means providing reason and the logic behind in order to enable the accountability and transparency on model. There are many types of model interpretations which including:
During exploratory data analysis (EDA), we may apply principal component analysis (PCA) or t-Distributed Stochastic Neighbor Embedding (t-SNE) to understand about the feature
After built model, we will use various metric to measure classification and regression model. Accuracy, precision and recall used in classification while mean squared error (MSE) used in regression. This kind of metric help us to understand the model performance
Besides performance, static feature summary such as feature importance can be retrieved from model. However, it only exist in those decision tree base algorithm such as Random Forest and XGBoost.
When evaluating model, we want to know why we predict it wrongly. We can leverage library to explain it so that we can fix it.
Before explaining the model, we can first understand the criteria of model interpretation:
Interpretability
Intrinsic: We do not need to train another model to explain the target. For example, it is using decision tree or liner model
Post hoc: The model belongs to black-box model which we need to use another model to interpret it. We will focus on this area in the following part
Approach
Model-specific: Some tools are limited to specific model such as liner model and neural network model.
Model-agnostic: On the other hand, some tools able to explain any model by building write-box model.
Level
Global: Explain the overall model such as feature weight. This one give you a in general model behavior
Local: Explain the specific prediction result.
As mentioned before, we should explain in most of the scenario because:
Accountability. Model should has a accountability to provide a correct (relatively) answer to consumer. As a model author, we should validate model features to guarantee it help on make a correct decision but not including as much feature as we can.
Transparency. Machine learning is not a black box and we should provide the model transparency to customer and management. So that they know why you are doing. Just like open source, people are willing to use open source much more as they know what you are doing.
Trustability. It is the basic requirement if consumer want to use the classification or prediction result. Most of the time, use has some domain knowledge such as trader need to understand why you provide the decision for buy/ sell particular stock.
As you know, I mainly focus on NLP. Therefore, I only demonstrate the capability of interpreting NLP model although the following library can also explain other problems such as regression and image.
For the following demonstration, I did NOT do any preprocessing for the sake of keeping it easy to understand. In real life business problem, we should do the preprocessing all the time. Otherwise, garbage in garbage out.
Several libraries helps to explain model which including Explain Like I am Five (ELI5), Local Interpretable Model-agnostic Explanations (LIME) [1] and Skater.
Library: I will demonstrate how we can use ELI5 and LIME. How about Skater? Stay tuned, will explain reason why I do not use it in NLP.
Algorithm: In the following demonstration, it will include linear model (Logistic Regression in scikit-learn), ensemble model (Random Forest in scikit-learn, XGBoost) and deep learning model(self build word embedding in keras). For word embedding, you can check out my previous post for detail.
ELI5
ELI5 provides both global model interpretation and local model interpretation. You may simply consider the global model interpretation as a feature importance but it not only support decision tree algorithm such as Radom Forest and XGBoost but also all sci-kit learn estimators.
ELI5 author introduces it as Permutation Importance for global interpretation. To calculate the score, feature (a word) will be replaced by other word (noise) and predicting it. The idea is that feature importance [2]can be deduced by getting how much score decrease when a particular word is not provided. For example, “I like apple”. It will may changed to “I like orange” and then it will classify the newly created record to understand how “apple” is important. Of course, we need to assume the replaced word (e.g. orange) is noise and it should not provide major change on score.
for pipeline in pipelines:    print('Estimator: %s' % (pipeline['name']))    labels = pipeline['pipeline'].classes_.tolist()        if pipeline['name'] in ['Logistic Regression', 'Random Forest']:        estimator = pipeline['pipeline']    elif pipeline['name'] == 'XGBoost Classifier':        estimator = pipeline['pipeline'].steps[1][1].get_booster()#     Not support Keras#     elif pipeline['name'] == 'keras':#         estimator = pipeline['pipeline']    else:        continue        IPython.display.display(        eli5.show_weights(estimator=estimator, top=10,        target_names=labels, vec=vec))
The above figure means that, if input includes “keith”, then score of y=0 increase 2.138. Another case is “the”, it will decrease score -0.731 and -1.387 in y=1 and y=2 respectively.
For local model interpretation, ELI5 use LIME algorithm. The display format is different from LIME but using same idea.
number_of_sample = 1sample_ids = [random.randint(0, len(x_test) -1 ) for p in range(0, number_of_sample)]for idx in sample_ids:    print('Index: %d' % (idx))#     print('Index: %d, Feature: %s' % (idx, x_test[idx]))    for pipeline in pipelines:        if pipeline['name'] in ['Logistic Regression', 'Random Forest']:            estimator = pipeline['pipeline'].steps[1][1]        elif pipeline['name'] == 'XGBoost Classifier':            estimator = pipeline['pipeline'].steps[1][1].get_booster()        #     Not support Keras#         elif pipeline['name'] == 'Keras':#             estimator = pipeline['pipeline'].model        else:            continueIPython.display.display(            eli5.show_prediction(estimator, x_test[idx], top=10, vec=vec, target_names=labels))
ELI5 explains the test data and deciding that it may be y=0 and the probabilty is 0.031. It also highlight some high positive score and negative score words.
LIME
As the name mentioned, LIME focus on local model interpretable and model-agnostic part only.
Passing trained model and target record to LIME library. A liner bag-of-words model will be created and providing lots of generated record for training a white box model. The while box model work as a binary classifier indicating the impact of word existence. For example, “I like apple”. It will may changed to “I like” and then it will classify the newly created record to understand how “apple” is important.
So no matter what is the provided model is, it can also explain the model by generating records. No matter it is scikit-learn library, XGBoost or even Keras word embedding model. In my sample code, I implemented Keras in sci-kit learn API interface so that I can also use LIME to explain Keras’s word embedding model.
number_of_sample = 1sample_ids = [random.randint(0, len(x_test) -1 ) for p in range(0, number_of_sample)]for idx in sample_ids:    print('Index: %d' % (idx))#     print('Index: %d, Feature: %s' % (idx, x_test[idx]))    for pipeline in pipelines:        print('-' * 50)        print('Estimator: %s' % (pipeline['name']))                print('True Label: %s, Predicted Label: %s' % (y_test[idx], pipeline['pipeline'].predict([x_test[idx]])[0]))        labels = pipeline['pipeline'].classes_.tolist()                explainer = LimeTextExplainer(class_names=labels)        exp = explainer.explain_instance(x_test[idx], pipeline['pipeline'].predict_proba, num_features=6, top_labels=5)        IPython.display.display(exp.show_in_notebook(text=True))        #         break#     break
LIME builds binary classifiers so we can see that it represent whether it is 0, 9, 15 classes with weight and highlighting high positive and negative score words.
Skater
For global interpretation, Skater provides feature importance and partial dependence approaches.
Tried several time and cannot find a way to provide sci-kit learn pipeline package. Have to convert raw text to numpy format and passing the model separately. Second thing is that it takes about 2 minutes to prepare the feature importance when there is about 800 unique words.
%matplotlib inlineimport warningswarnings.filterwarnings('ignore')import matplotlib.pyplot as pltfrom skater.model import InMemoryModelfrom skater.core.explanations import Interpretationtransfromed_x_test = vec.transform(x_test[:2]).toarray()interpreter = Interpretation(transfromed_x_test, feature_names=vec.get_feature_names())for pipeline in pipelines:    print('-' * 50)    print('Estimator: %s' % (pipeline['name']))        if pipeline['name'] in ['Logistic Regression', 'Random Forest']:        estimator = pipeline['pipeline'].steps[1][1]    else:        continue            print(estimator)            pyint_model = InMemoryModel(estimator.predict_proba, examples=transfromed_x_test)        f, axes = plt.subplots(1, 1, figsize = (26, 18))    ax = axes    interpreter.feature_importance.plot_feature_importance(pyint_model, ascending=False, ax=ax)    plt.show()
Skater provides the feature importance per word.
Partial dependence [3]is a bunch of figures to represent the relationship (linear, monotonic or complex) between target and features one by one. Providing a visualization, we can visualize the impact between one feature and target. However, we may not use partial dependency if there is NLP.
For local interpretation. Skater wraps LIME module to perform local Text Interpretation. So I will prefer to use native LIME rather than a pure wrapper.
You can access code via my github repo
When you read Christoph’s Blog, you can refer to above code for Local Surrogate Models (LIME) part.
As a data scientist, we have to explain the model behavior so that product team and management trust the deliverable and debug your model. Otherwise you may not know what you build.
For NLP problem, feature importance or permutation feature importance are demanding as too much features (word). Not quite useful/ recommend to use it in NLP aspect.
LIME do not able to explain the model correctly on some scenario. For example, the score can be different every time as it generates samples randomly. Will have another article to introduce how to resolve it.
For ELI5, sci-kit learn pipeline is supported in show_weight (Global Interpretation) but it is not supported in show_prediction (Local Interpretation) in ELI5. Reason is mentioned in here. To overcome it, you can simply use LIME directly.
There is bug when using ELI5 (version: 0.8) if the classifier is XGBoost Classifier. If you use XGBoost classifier, have to perform some workaround due to ELI5 bug (xgb_classifier.get_booster())
For Keras model, implemented sci-kit learn API interface but still cannot use ELI5. ELI5 will check the object type to prevent unexpected library.
Special thank to Dipanjan (DJ) Sarkar supplement on Skater part.
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related.
Visit my blog from http://medium.com/@makcedward/
Get connection from https://www.linkedin.com/in/edwardma1026
Explore my code from https://github.com/makcedward
Check my kernal from https://www.kaggle.com/makcedward
[1] Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. “Why Should I Trust You?” Explaining the Predictions of Any Classifier. Sep 2016. https://arxiv.org/pdf/1602.04938.pdf
[2] Breiman Leo. “ Random Forests”.Jan 2001. https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf
[3] Friedman, Jerome H. 2001. “Greedy Function Approximation: A Gradient Boosting Machine.” Apr 2001. https://statweb.stanford.edu/~jhf/ftp/trebst.pdf