Explainable Artificial Intelligence (XAI). But, for Whom? | by Carlos Mougan | Towards Data Science

Attempting to explain predictions or build interpretable machine learning models is a hot topic that is quickly expanding. When applying Machine Learning in the decision process in many areas such as medicine, nuclear energy, terrorism, healthcare, or finance, one cannot blindly trust the predictions and leave the algorithm “in the wild,” as that might have disastrous consequences.
Predictive performance of models is evaluated through a wide portfolio of metrics such as accuracy, recall, F1, AUC, RMSE, MAPE... but high predictive performance might not be the only requirement.
There is a huge trend in the field of Explainable AI and Interpretable ML to build trust in the predictions of the model. Many papers, blogs, and software tools present explainability and interpretability in a very defined way, but...
Accountability for whom? Explainability for whom?
I guess that we can all agree on the fact that we don't all need the same type of explanation for how a model behaves. Artificial Intelligence experts such as Andrew NG, Geoffrey Hinton, Ian Goodfellow... won't need the same explanations of why a model is doing what and for which reason, compared to the average data scientist. Moreover, the layperson user probably needs a completely different explanation since they probably have no knowledge of the AI field (nor do they probably wants to).
So, explanations in Artificial Intelligence have different audiences.
Andrew NG and a randomly selected user of the Netflix recommendation system have different needs of understanding “why was this TV series recommended?” A random user might find it sufficient to get suggested a certain action movie because they have been watching action movies every night for the last two weeks, but then Andrew NG (replace name with any other famous AI person) might require a different type of explanation.
Machine Learning explanations are done for a specific audience. Current xAI trends seem to be building explanations that are made for Data Scientists that have a good knowledge of Machine Learning... But should xAI focus on explaining AI to the Data Scientist or to average users?
One typical error in software development is that developers end up developing software for themselves and it turns out to be poorly designed for their target audience [3].
Explainable AI might risk a similar fate as poor software development. Explanations of AI might be developed for data scientists and machine learning researchers themselves and not for the everyday user.
One technique that has proved useful is the usage of Local Surrogate models. Probably the most famous is Local Interpretable Model-agnostic Explanations (LIME). The idea is quite simple: the goal is to understand why the machine learning model made an individual prediction; this is called individual explanations. LIME tests what happens to the predictions when you give variations of your data into the machine learning model [5].
In the figure below we can see a local approximation to a certain instance of the well-known wine quality dataset. No matter what the original model is (model agnostic), we can make a local approximation to see which features contribute and for how much.
With this local approximation, we have a linear model that works well just for this data instance. We have perturbed the input data space, defined a local approximation, and fitted a model there.
To whom can we explain this?
Someone can say that this explanation is intuitive for them since it’s built as a linear model estimator with few features. The contribution of each feature is linear. One could even argue that this kind of explanation will be good for a person that has a bit of knowledge in the field, but no previous experience in ML. Also, this can be useful to a Data Scientist that has no time and is going fast through the same interpretations samples for some reason, so a Data Scientist with no time.
But, what are we really doing? Are we using a black-box explainer to predict a black-box algorithm? Only a fraction of the Data Scientists are able to actually understand how a Random Forest or Gradient Boosted Decision Tree behaves, but the fraction of Data Scientists who actually know what LIME does exactly when built out of a Random Forest is even smaller. What is being represented in the LIME local explanation graph? How stable is this explanation? What about its contestability?
Even though the rise of the interest in explainable AI is evolving really fast, some questions remain. For whom are we building the explanations? Are we targeting researchers/data scientists? Or are we targeting the general audience?
This blog is a scientific divulgation from the following published paper(link) at the ECMLPKDD workshop on Bias. Please, cite as:
@misc{mougan2021desiderataECB, title={Desiderata for Explainable AI in statistical production systems of the European Central Bank},  author={Carlos Mougan Navarro and Georgios Kanellos and Thomas Gottron}, year={2021}, eprint={2107.08045}, archivePrefix={arXiv}, primaryClass={cs.CY}}
1- Desiderata for Explainable AI in statistical production systems of the European Central Bank
2- May Edition: Questions on Explainable AI
3- Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences
4- Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead
5- “Why Should I Trust You?”: Explaining the Predictions of Any Classifier
6- Explanation in Artificial Intelligence: Insights from the Social Sciences
7- Interpretable Machine Learning: A Guide for Making Black Box Models Explainable.
Many thanks to Laura State and Xuan Zhao for the input and discussions.
European Commission, NoBIAS — H2020-MSCA-ITN-2019 project GA No860630.