TITLE: Systematic review of user experience and trust improvement evaluations on healthcare oriented explainable artificial intelligence
AUTHORS: Iñaki Soto-Rey, Samantha Cramer

[DESCRIPTION]
Many different AI systems have been developed in recent years. The field of explainable AI is designed to shed light on the black box modles of the various AIs. However, the explainable AI is rarely developed in relation to the needs of the users, but is based on the technical side that the AI structure provides. Our goal is to explore the various scientific researches that have been conducted on explainable AI in terms of user satisfaction. This goal is to be achieved through a systematic review based on the PRISMA guidelines.




 


[BEFORE_START]
Before starting the Systematic Review we studied the PRISMA guidlines, in order to proceed and complete the Systematic Review according to these guidlines.


[GUIDELINES]
PRISMA guidlines.

    


[STEPS]
SECTION: Background
1. The reviews subjective is to identify the amount of evaluations done on the usability, user satisfaction, user experience and trust of XAI. The domain of interest is the medical field.
SECTION: Methodology - Literature Search
2. General search query:  ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence") AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND ("Evaluation")

Search restrictions: publication between 01.01.2017-31.12.2022     
4. Extracted Literature was imported into Civati (version 6), excluding Duplicates during import process.
 

5. Fulltexts were collected for results imported into Citavi, using the "find Fulltext"-function of Citavi as well as manual searches in the web. The searches resulted in:
- 3869 results including fulltext
- 395 no fulltext
SECTION: Methodology - Screening
6. Inclusion:
- health care information systems
- scientific evaluation in terms of usability, user satisfaction, user experience or trust and XAI
- scienticfic publication (poster/abstract/article/book)

Exclusion:
- non health care publications
- non english publications
- non scientific publications
3.4. Google Scholar: 
Website: 
using the Website for a quick overview and the Query: find article... with all words: “Evaluation” ; …and one of the words: “XAI” “X AI” “Explainable AI” “Explainable artificial intelligence” 
Words contained in Title.
Results: 83

Problem: articles can’t be extracted from google scholar, therefor the Literature Search for this databank was done using the following software: 


 
Due to Publish and Perish set up extraction of max. 1000 results possible.
Publish or Perish search:
Query in Title words: (“XAI" OR “X-AI” OR “Explainable AI” OR "Explainable artificial intelligence”) AND (“Evaluation”)
Citation records included
Result: 80 records 
Formats saved: csv ; bibtex

filter out the results included in the last part of the query: AND (“Usability” OR “satisfaction” OR “experience” OR “trust”) by looking for the keywords in the .csv file 
Results: 6 


3.3. Science Direct: 
Query: ("XAI" OR "X-AI" OR ("Explainable AI") OR ("Explainable artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND ("Evaluation")
Formats saved: RIS
Results: 1296
3.2. PubMed Central (PMC): 
Query:  ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND "Evaluation"
Formats saved: MEDLINE
Results: 2996
3.1. PubMed: 
Query: ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND "Evaluation"
Formats saved: PubMed
Results: 17
3. Data extraction from the following data bases on 02.05.2023, with the following results:
SECTION: Methodology - Screening
7. The screening process was done individually by two seperate reviewers. Individual screening results were then compared.

Disclaimer: The results are at risk of bias due to the subjective opinion of the reviewers, which we aim to avoid by reviewing the papers individually. Furthermore, risk of bias exists, due to the fact we might not find all fitting publications with our search query, even though we have developed it to the best of our knowledge.