## 1 Introduction  
Overcrowding in emergency departments (EDs) has become an international issue of overriding importance. Several studies from different countries have provided indisputable evidence to show that overcrowded EDs can lead to a decrease in quality of care in general, and unintended harm to patients in particular [1, 2, 3]. In turn, these issues influence patient management; patients admitted to the wrong ward may experience higher care costs [4], a longer hospital stay [4], a greater risk of morbidity [5] and mortality [6, 7, 8, 9], and a greater likelihood of leaving the ED without being examined [10]. Accordingly, a variety of solutions have been developed with the goal of limiting ED overcrowding, including pre-hospital dispatch before attending the ED (to encourage attendance at other healthcare facilities), improved bed coordination, effective triage with front-loading investigations, fast tracks, optimized transfer to the destination ward (even if the bed is not ready), and an increase in the number of available beds [11, 12, 13].  
Furthermore, machine learning (ML) and artificial intelligence (AI) have become promising tools in medicine; by learning from large amounts of data, these tools can help clinicians to detect disease patterns or improve the patient care pathway, for example. In 2012, the results of two studies confirmed that the timely prediction of patient outcomes helps to reduce the length of stay in the ED [1, 14]. Thus, the use of AI in the ED has emerged as an active topic of research for the prediction of patient outcomes (discharge or admission) using triage data [15], mortality prediction [16], prioritization [17], prediction of adverse event (on the basis of symptoms) [18], and diagnosis prediction [19]. The literature data cover a variety of types of ML algorithms, e.g., logistic regression (LR), support vector machines, decision trees, ensemble methods, fuzzy logic, naive Bayes classifiers, and deep learning models [15, 16, 17].  
Nevertheless, the credibility of predictive models is still a concern for clinicians [20, 21]. Greater clinical uptake will require more readily accessible and interpretable results [22]. Explainable artificial intelligence (XAI) has been developed to meet these requirements [23] and thus allow healthcare experts to confidently make use of data-driven decisions and provide more personalized, trustworthy treatments and diagnoses [24]. In the ED, an XAI model might help (i) emergency physicians to understand how data are considered internally in the model and (ii) managers to gain insight into the model's logic and justify its deployment in the hospital. Furthermore, XAI might allow patients to understand how their data are used, which is in line with the terms of the European Union's General Data Protection Regulation and the \"right to an explanation\" for people whose data are processed by an AI algorithm [25].  
In the context of overcrowding in the ED, Amiens Picardy University Hospital (APUH, located in the city of Amiens in northern France, with 1.9 million inhabitants in the surrounding region) [26] has developed an AI model called \"Prediction of the Patient Pathway in the Emergency Department\" (3P-U, U for \"Urgence\" which is the French word for ED). This model has been designed to improve patient flow by predicting the final medical decision [27, 28]. However, to ensure wider adoption by ward specialists, the 3P-U model was refined and made more explainable (especially for the outputs generated by the algorithms) while maintaining the level of classification performance for each specific medical issue. This balance between accuracy and explainability is especially important in the case ofneural networks and deep learning models, due to the complexity of the internal layers and the \"black box\" behavior.