The current review addresses emerging issues that arise from the creation of safe, beneﬁcial, and trusted artiﬁcial intelligence– air trafﬁc controller (AI-ATCO) systems for air trafﬁc management (ATM). These issues concern trust between the human user and automated or AI tools of interest, resilience, safety, and transparency. To tackle these issues, we advocate the development of practical AI-ATCO teaming frameworks by bringing together concepts and theories from neuroscience and explainable AI (XAI). By pooling together knowledge from both ATCO and AI perspectives, we seek to establish conﬁdence in AI-enabled technologies for ATCOs. In this review, we present an overview of the extant studies that shed light on the research and development of trusted human-AI systems, and discuss the prospects of extending such works to building better trusted ATCO-AI systems. This paper contains three sections elucidating trust-related human performance, AI and explainable AI (XAI), and human-AI teaming. Index Terms Trusted ATCO-AI Systems, Neuroscience, Neuroimaging, Neuroscience-inspired AI, Machine Learning, Explainable AI (XAI), Human-AI Teaming I. I NTRODUCTION Contemporary technology advances in automation, artiﬁcial intelligence (AI), and computational algorithms have created great beneﬁts for human operators, as well as increases in adverse or unexpected consequences [1], [2], [3]. To ensure safe and seamless human-machine interaction (HMI), trust between humans and machines (in the form of AI or automated tools) has become a widely discussed topic over the past three decades [4], [5], [6]. In particular, within the air trafﬁc management (ATM) domain, human-machine trust has long been recognized as essential for complex ATM systems to deliver the proposed capacity and safety beneﬁts [7], [8]. Irrespective of the types of interactive agents involved, trust, in general terms, is a psychological state that can manifest itself as an intervening variable between particular external situations or experiences (e.g., social interactions) and human behaviors inherent to such situations [7]. More speciﬁc and operational deﬁnitions of human-machine trust that has been widely accepted in the current human factors literature concern the perception of trust as: (i) " the extent to which a user is willing to act on the basis of the information, recommendations, actions, and decisions of a computer-based tool or decision aid " (p. 11 in [7], adapted from p. 1 in [9])) or (ii) " the attitude that an agent will help achieve an individual’s goals in a situation characterized by uncertainty and vulnerability " (p. 6 in [10]). Here, it is worth noting that according to the second operational deﬁnition, trust is characterized as decision-making heuristic that machine operators use in situations that are too uncertain or changing rapidly [3], [2]. Together with these two popular operational deﬁnitions of trust, the concept of calibrated trust must also be mentioned. Calibrated trust refers to the condition in which the anticipated or perceived trust of a machine, automation or AI matches the experienced trust associated with the actual use of the tool [10], [1], [3]. Seen in this light, trust becomes an evolving process that gets updated with increased information of the operating machine or system [1]. Following a decision-making framework, this means that a dynamic interaction between two variables occurs: (i) an initial perception or biased opinion of the machine or system before its actual use, (ii) updated perception and knowledge of the machine or system during and after usage [1], [11], [12]. With all these concepts of trust in mind, the direction of our current research aims to study the operational nature of trust and afﬁliated human factors (e.g., cognitive bias and constraints) in the ATM domain and apply such knowledge to the development of human-centered AI (HAI) systems for the purpose of achieving trusted human-AI teaming. As shown in Figure 1, our approach covers three primary components: (i) Human performance of air trafﬁc controllers (ATCOs), (ii) HAI tools, and (iii) "human-in-the-loop" techniques and procedures, which incorporates trust as a major component. Currently, it is well known that ATCOs are using state-of-the-art technology to deliver safe and efﬁcient air trafﬁc services. A huge gap remains, however, in understanding "human-in-the-loop" processes and mechanisms tied to optimal cooperation between ATCOs and the automated or AI tools they are using [4]. Henceforth, in the sections below, we shall review recent works related to: (a) trust-related behavioral and neuroscience research in the HMI and ATM domains, (b) AI and Explainable (XAI) systems – and (c) discuss how knowledge gained from both human experimentation and AI development can be harnessed for building better trusted ATCO-AI systems. Figure 1: A proposed research framework for developing trusted HAI systems. COPYRIGHT CC BY-NC-ND 4.0, S. K. GOH & J. Y . ZHONG, 2020, NANYANG TECHNOLOGICAL UNIVERSITY II. T RUST -RELATED HUMAN PERFORMANCE Over the past three decades, numerous studies in the human factors and ergonomics domain had investigated trust between human users and automated tools, which is simply termed as human-automation trust [8], [7], [5], [6]. As AI tools and systems are currently undergoing rapid development and reﬁnements in the ATM domain, we have yet to witness a published ATM study that made a systematic investigation of human-AI trust. Henceforth, this section will focus exclusively on human-automation trust in both ATM and non-ATM domains. Note that by "automation," we refer to the "machine execution of functions" (p. 286 [13]) and "technology that actively selects data, transforms information, makes decisions, or controls processes" (p. 50 [10]). Trust has been conceptualized as an essential mechanism for automated ATM systems to deliver the proposed capacity and safety beneﬁts [14]. However, research on trust in ATM systems has been scarce and and has almost all been done using questionnaires that gave a subjective measure of trust [8], [7], [5], [6]. For instance, a popular questionnaire that has been used widely in the ATM domain is the SHAPE Automation Trust Index (SATI) [15], which contains rating scales measuring ATCOs’ overall level of trust and the constituents of trust such as reliability, predictability, and understandability. While SATI has been found to be easy to administer, it has been criticized for being unable to provide a multidimensional measure of trust [16]. This means that personality- and individual history-related aspects of trust cannot be assessed by SATI. Apart from knowing that trust is commonly measured through self-report questionnaires, it is worth noting that a common trust-related issue in the use of automated ATM tools or systems relates to the phenomenon of complacency , which occurs when an ATCO places a high level of trust in an automated tool to the negligence or failure of monitoring the "raw" sources of information that provide inputs to the automated system [8], [5]. In the event of over-complacency or over-conﬁdence in the functionalities of the automated system, system errors may go about undetected [17]. As for the measurement of complacency, the presence of complacency is usually inferred with respect to human performance data reﬂecting lower levels of attention and shorter monitoring when using automation [18], [19], [20]. For instance, operators have been shown to scan raw information sources less often when using automation than when performing the task manually and when automation reliability is higher rather than lower [19], [20]. In the same vein, when operators were given a tool that contains automated settings for extracting raw information sources, they used it much less often – indicative of faster data extraction – under automation than under manual control [18]. On another end, mistrust of an automated system, the polar opposite of complacency, can also pose signiﬁcant risks for ATCO performance. Mistrust can either lead an ATCO to "over-monitor" the system in the event of errors [8] or ignore error warnings given by the system entirely [21]. The latter phenomenon has been dubbed the "cry wolf effect" [22] and has led to users of automated tools taking longer to to re-engage the use of automated services after experiencing system failures [23]. Moreover, mistrust is a negative psychological phenomenon that can persist even in the presence of a perfectly reliable automated tool [8]. For instance, using an automated conﬂict detection advisory in a simulated trajectory-based operation (TBO) airspace, Metzger and Parasumaran (2005) [19] showed that trust ratings of the automated tool were not high even though it was 100% reliable. In addition to the aforementioned studies, current state-of-the-art studies of human-automation trust have incorporated neuroimaging technologies and techniques. Unlike questionnaires that provide subjective measures of trust, neuroimaging provide objective and real-time measurement of trust and trust-related psychophysiological signals emanating from the human brain and peripheral nervous system [24]. Speciﬁcally, these techniques encompass: (i) electroencephalogram (EEG), (ii) functional magnetic resonance imaging (fMRI), (iii) functional near-infrared spectroscopy (fNIRS), (iv) electrocardiogram (ECG), (v) electrodermal activity (EDA) recording, (vi) electromyography (EMG), (vii) electrooculography (EOG), (viii) eye- tracking, and (ix) photoplethysmography (PPG) [25], [24]. Among these, the ﬁrst three techniques relate to the recording of signals emanating from the brain reﬂective of changes in underlying neural activity while the latter ﬁve techniques involve the recording of signals from the peripheral or autonomic nervous system (i.e., from the heart–ECG, PPG; from the skin and musculature–EDA, EMG; from the eyes–EOG, eye-tracking). In this paper, our main focus is on understanding the brain mechanisms of human-automation trust in ATM, and hence we shall only provide a review of notable brain imaging studies. In recognition of the fact that EEG has been the overwhelming favorite brain imaging technique used by researchers of human-automation trust and that we plan to harness fMRI for future projects on human-automation/AI trust in ATM, extant human-automation studies that utilized these two types of brain imaging techniques shall be the focus of this review. We elucidate the contributions of these two brain imaging technologies to human- automation interaction and trust in the subsections below. Speciﬁcally, because we aimed at learning and adopting the newest state-of-the-art experimental methodologies in our future projects, we concentrated our efforts on a review of notable human- automation trust studies utilizing EEG and fMRI studies over the ﬁve years only [2016 – 2021 (present)]. A. EEG Studies 1) General Introduction EEG is a non-invasive, electrophysiological technique used for recording electrical signals emanating from brain activity. Multiple electrodes, either wet or dry, are placed on an individual’s scalp to record the brain’s spontaneous activity over a period of time. The instruments for conducting EEG are relatively inexpensive (compared with those for conducting fMRI and fNIRS) and highly portable. Unlike fMRI, which requires subjects to lie supine during the brain scan, subjects sit or stand in their normal upright position during EEG recording. Moreover, EEG offers higher temporal resolution than both fMRI and fNIRS, being able to record ﬂuctuation in brain waves or rhythms on the scale of milliseconds [25]. Due to all these these factors, EEG has been the most preferred brain imaging technique for investigating neurocognitive mechanisms in the ATM domain [25]. 2) Mental Workload With respect to the cognitive factors investigated, a preponderance of ATM EEG studies focused on the cognitive factor of mental workload [25], [26]. In view of the prominence of such studies and the practical implications they convey, notable ﬁndings and concepts from these studies shall be presented ﬁrst before discussing trust-related EEG studies. In layman’s terms, mental workload can be conceived as the amount of information processing an ATCO has to perform in response to the operational demands of the system(s) under his/her control [27]. EEG power spectrum analysis, used to identify changes in the EEG spectra reﬂecting different levels of mental workload, has consistently shown the correlation of certain EEG spectra with task difﬁculty or complexity. Most notably, numerous studies have shown that an increase of the theta band spectral power (4-7 Hz) over the frontal cortex and a decrease in the alpha band spectral power (8-12 Hz) over the parietal cortex correlated positively with an increase in mental workload or task complexity [28], [29], [30], [31]. For instance, Borghini et al. (2017) [29], through the use of machine learning-based automatic stop stepwise linear discriminant analysis (asSWLDA) of EEG spectral features, showed that frontal theta power spectral density (PSD) increased with ATCOs’ operational task complexity while frontal alpha PSD decreased more when automatic actions are performed (e.g., opening up a pie menu to select an option for an aircraft) than when more intentional or deliberative activities were performed (e.g., when conﬂicts and ﬂight plan deviations were spotted). In contrast to the changes in the frontal alpha PSD, theta PSD in the parietal cortex increased more when conscious planning and decision making were called for than when effortless or automatic actions were performed. With reference to a skill, rule, and knowledge (SRK) model, increases in frontal alpha, frontal theta, and parietal theta PSDs were proposed to reﬂect rule-based and knowledge-based behaviors, with the latter requiring a higher degree of synchronization of all three types of EEG rhythms than the former (Figure I). Speciﬁcally, rule-based behavior relate to following a learned rule or procedure during problem solving or planning while knowledge-based behavior is more cognitively demanding, requiring a person to engage in stored and newly acquired memories or knowledge in order to reach a desired goal. On the other hand, a decrease in frontal alpha PSD (or alpha desynchronization) was proposed to reﬂect skill-based behavior, automatic actions that occur without mental exertion. Table I: Hypothesized EEG rhythms assigned to the cognitive control behaviors of skill, rule, and knowledge (SRK). The SRK behaviours were described in terms of variation from a reference condition: synch — synchronization; desynch — desynchronization. [Source: Table 1, Borghini et al. (2017)] Due to the fact that workload can be inﬂuenced by other cognitive factors such as attention, vigilance, mental fatigue, and environmental factors such as operational task demands and complexity, the establishment of valid and reliable methods for measuring mental workload is of top priority for human factors researchers in ATM [25]. Regarding this, numerous recent ATM studies have employed machine learning algorithms and models such as linear discriminant analysis (LDA) and support vector machine (SVM) to classify EEG-based spectral features (obtained from ATCO participants) into different categories of workload during real-time air trafﬁc simulations [32], [33], [29], [34]. These EEG-based workload indices provide objective and rapid online assessment of task engagement and perceived workload on the scale of seconds or minutes, and can be readily applied to trigger adaptive automation (AA) solutions for brain-computer interface (BCI) systems [33], [25]. For instance, Aricò et al. (2016) [33] computed a real-time EEG-based workload index through asSWLDA of frontal theta and parietal alpha bands. When this workload index was classiﬁed online as high, AA solutions were triggered (e.g., highlighting a calling aircraft on the radar to facilitate an ATCO’s pinpointing of its location). By contrast, no AA solutions were triggered when the workload index was classiﬁed as low. By triggering AA solutions in response to the online workload status of the ATCO in this way, a reduction in ATCOs’ workload co-occurred with an enhancement in operational performance (i.e., faster responses to operational demands), particularly under simulated air trafﬁc scenarios with high task difﬁculty. In addition, a recent EEG study by Radüntz et al. (2020) [34] showed that an a workload index computed through Dual Frequency Head Maps (DFHM) method – which involved a SVM-based classiﬁcation algorithm – increased in a stepwise fashion as the number of aircraft to be handled by each ATCO per hour increased. Interestingly, relatively high correlations were observed between this objective workload measure and subjective, questionnaire-based reports of workload (.67 r.81) under scenarios with equal volumes of air trafﬁc. 3) Trust Over the past ﬁve years, there has only been a limited set of EEG studies that investigated human-related trust. We found six noteworthy EEG studies and tabulated their key features – experimental task(s) and design, data analysis technique(s), main ﬁndings and implications – in Table II below. A phenomenon common to all these studies pertains to the use of experimental design or tasks that involved the modulation of reliability (or dependability) offered by an automated tool (Table II, column 2). Reliability refers to how consistent an automation can perform in providing accurate information and was largely manipulated through the programming and presentation of automated agents or advisories that can provide information with different probabilities of accuracy [35], [1], [36], [3] or risk-taking tendencies [37], [38]. More importantly, with respect to the brain regions that were activated during human-automation trust-related decision making, these recent EEG studies pinpointed anterior regions such as the (i) lateral prefrontal cortex [38], (ii) anterior cingulate cortex (ACC) [1], [3], and posterior regions such as the (iii) occipital cortex [38] and (iv) fusiform gyrus (also known as the occipitotemporal gyrus) [3]. Two studies further showed that power increases or variation in the beta frequency band (12 Hz – 35 Hz) were associated with increased levels of trust [36] and making discriminatory judgments between trustworthy and untrustworthy stimuli [35]. Figure 2 shows the brain regions from which trust-related decision making EEG signals were recorded. Figure 2A shows the effective connectivity network uncovered by Sanders et al. (2019) [3] during a participant’s detection of mistakes made by the automated advisory while Figure 2B shows the locations of the EEG electrodes, mapped out by Wang et al. (2018) [38], at which signiﬁcant trust-related activations were found. B. fMRI Studies 1) General Introduction Functional Magnetic Resonance Imaging (fMRI) began in the early 1990s with its initial development generally credited to Seiji Ogawa and Ken Kwong [39]. Today, it is regarded as one of the most well-known neuroimaging techniques. fMRI works by inferring brain activity from changes in the blood ﬂow in the brain. Changes in cerebral blood ﬂow translated to changes in the magnetization levels of oxygenated and deoxygenated blood. A higher inﬂow of oxygenated blood into a particular brain region usually occurs when a person is thinking and performing a particular task, and this will lead to a rise in the magnetic signal that surrounds the cerebral capillaries in that particular region. This signal is recorded by the fMRI scanner and is called as "blood-oxygen-level-dependent (BOLD)", meaning that there is a coupling between cerebral blood ﬂow into a brain region and the neuronal activation in that region [40]. In other words, when an area of brain is “activated” by some mental processes, the ﬂow of oxygenated blood into it increases. Due to the application of blood ﬂow and oxygen metabolism to infer brain activity, fMRI offers several signiﬁcant capabilities. It is non-invasive, does not involve radiation, and is generally safe for most people. It offers good spatial resolution of the brain to an accuracy of up to 1.0 mm and temporal resolution of up to 1.0 second [40]. It renders structural or anatomical images of the brain, as well its functional images, which represent hemodynamic responses to decisions and/or responses made over the period of the brain scan [40]. Based on statistical analyses that commonly operate on the basis of General Linear Models, functional brain images are normally always color-coded to convey the regions of signiﬁcant brain activity [41]. This makes it easy for any reader to discern the location of different brain regions. 2) Trust Over the past ﬁve years, there has only been three notable studies on human-automation trust [42], [43], [44]. Two investigated brain activations and effective connectivity between regions of interest through the use of a X-ray luggage screening task that presented automated advisories with varying proportion of false alarms [42] and misses [43]. [Note that a false alarm refer an emergency alarm (or alert) that is activated under an situation that does not require it [42], [7] while a miss refer to the absence of an alarm under an emergency situation that truly requires it [43].] The remaining and more recent study was a proof-of-concept study that investigated human-automation trust in the form of a quantum-inspired mathematical model [44] – to date, it was the only published trust-related study in the ATM domain. Table III lists the experimental tasks and design, data analysis techniques, main ﬁndings and implications tied to these three studies. From a quick overview of the brain-related ﬁndings, several facts are worth highlighting: (i) Even though the behavioral response associated with advice utilization can be the same for both human and machine agents, the brain activation patterns in response to the advice given by these agents can be different [43]. (ii) Greater neural resources, as shown through brain activations observed from contrast analyses, during decision making in response to advice given by a human agent than to advice given by an automated agent – and these activation patterns differ based on the proportion of false alarms (Figures 3 A and B) and misses (Figures. 3 C and D) experienced by the user [42], [43]. (iii) Human-to-machine-agent contrasts done by Goodyear et al. (2016, 2017) [42], [43] showed activations in the ACC and insula; the same ﬁndings were uncovered by Pushparaj et al. (2019) [44], who postulated ACC activation to reﬂect reciprocal trust and insular activation to reﬂect pure distrust. (iv) ACC activation has also been shown in human-human (interpersonal) trust under using an voluntary trust game that require mutual trust between two participants in deciding the amount of ﬁnancial payoffs [45]. This shows that the ACC is vital for representing or inferring the mental states of another agent, irrespective of whether it is a human or an automated tool. With these facts in mind, it is worth mentioning that there is current movement toward the treatment of human-automation trust as a decision-making process [2]. Speciﬁcally, this process frames trust in the form of value-based decisions that required the human user to continuously weigh the expected personal value of letting the automation to complete the task versus performing it manually [2], [46], [47]. In this way, any trust-related assessment must be made against a backdrop of risk and/or reward that may come with human-automation cooperation. Through a critical review of the decision neuroscience literature, Drnec et al. (2016) [2] identiﬁed an assortment of brain regions, located in the prefrontal, parietal, and limbic regions that can be involved in trust-related value-based decisions. As shown in Table IV, these hypothesized value-based regions involve the amygdala, ventral striatum, and ventral medial prefrontal cortex (vmPFC). Currently, to our knowledge, it is unknown as to: (i) how many of these regions will be activated during human-automation trust-related decision making, (ii) the extent to which they are activated and the scale of functional connectivity between them, and (iii) how these different brain activation and functional connectivity patterns change over time – under different physical or social environments. These issues show that there are many unknowns concerning the neural correlates of human-automation trust and that further investigations are deﬁnitely needed. Table II: Summary of Key EEG Studies of Human-Automation Trust done over the past Five Years (2016-2020) Figure 2: ( A) EEG-based effective connectivity ﬁndings taken from one participant. There was strong ﬂow of time-frequency information (as indicated by the bright red color) from the source node of fusiform gyrus (FG) [source node] to the dorsal anterior cingulate cortex (dACC) and posterior cingulate cortex (dPCC) [terminal nodes] when the automation was observed to be working successfully under the high reliability condition. Lighter colors in the low reliability condition indicate relatively weaker ﬂow of time-frequency information. These ﬁndings were time-locked to the occurrence of the observational error positivity (oPE, see Table 1, row no. 4) 200 milliseconds after the onset of the automated event. [Source: Figure 3, Sanders et al. (2019) [3]] ( B) Map of EEG electrodes that recorded signiﬁcant human-automation trust-related signals from the frontal and occipital cortices. [Source: Figure 5, Wang et al., 2018 [38]] Table III: Summary of Key fMRI Studies of Human-Automation Trust done over the past Five Years (2016-2019) Table IV: Brain regions with putative roles in decision making. [Source: Table 1, Drnec et al. (2016)] Figure 3: ( A) Brain regions activated during the decision phase in a X-ray luggage scanning task with 40% false alarms based on human-to-machine agent contrast analysis. ( B) Effective connectivity network during the decision phase in a X-ray luggage scanning task with 40% false alarms based on human-to-machine agent contrast analysis. [Source for panels A and B: Figures. 5 and 7, Goodyear et al. (2016) [42]]. ( C) Brain regions activated during the decision phase in a X-ray luggage scanning task with 40% misses based on human-to-machine agent contrast analysis. ( D) Effective connectivity network during the decision phase in a X-ray luggage scanning task with 40% misses based on human-to-machine agent contrast analysis. [Source for panels C and D: Figures. 3 and 5, Goodyear et al. (2017) [43]] III. AI AND E XPLAINABLE AI (XAI) In contemporary times, AI has achieved unprecedented success in solving increasingly complex real-world problems from many sectors – ranging from facial recognition, natural language processing, and ﬁnance to safety-critical self-driving and medical diagnosis [48]. Currently, AI is powering cutting-edge research in the sciences and interface development between AI and the the sciences. While AI demonstrates tremendous beneﬁts, it cannot explain how they make speciﬁc decisions and plan for actions in a comprehensible way to human users (e.g., air trafﬁc controllers, medical doctors). Henceforth, the Defense Advanced Research Projects Agency (DARPA) in the USA created an ambitious XAI Program [49] that commenced in 2017 for exploring the theories for create AI systems whose learned models and decisions can be made comprehensible and trusted by end users. As shown in Figure 4, there has been a yearly growth in the number of AI-focused research works over the past decade, especially in the realm of XAI, which showed marked increases in research outputs from 2018 onwards. Figure 4: The trend of explainable AI research, progressing from interpretability toward explainability. Source: Figure 1 in [50]. Note that "XAI" and "Explainable Artiﬁcial Intelligence" refer to the same thing and are search terms used in the literature search. A. Opening the Black Box using XAI Since the second half of the last century, AI has evolved from a simple rule-based algorithm to a machine being able to learn on its own [51]. In the 1950s, Alan Turing explored the mathematical possibility of Artiﬁcial intelligence. He came up with the framework in which he discussed how to build intelligent machines and how to test their intelligence [52]. However, during that time, AI was built upon a rule-based system in which a machine follows a set of rules to come up with the desired output(s) [53]. Due to the fact that a rule-based system does not have learning capability, it cannot adapt to changes when a different input is fed into it. As time passed, computer scientists created algorithms to allow the machine to learn based on the features of the input in order to deliver an appropriate output. By doing so, this covered the weakness entailed by the rigidity of a rule-based system, and rendered learning and adaptability functions to the machine. During the digital revolution that started in the 1980s with the internet, big data has become easily accessible and obtainable, and AI started to develop expertise in speciﬁc domains that extend beyond the capability of humans. A prime example is AlphaGO, which was created by Google DeepMind using an artiﬁcial neural net architecture. In 2016, AlphaGo defeated 18-time Go (Weiqi) World Champion, Lee Sedol, four games to one [54]. Following on the footsteps of AlphaGo, DeepMind developed AlphaZero, the most powerful Chess engine to date, with the same type of neural net architecture built for AlphaGo combined with two types of advanced algorithms: Monte Carlo tree search and self-play reinforcement learning [55], [56]. By feeding information from two types of neural networks – (i) a policy network that computes the possibility of future moves and (ii) a value network that considers the positional outcome of a particular chess variation – into a Monte Carlo tree search algorithm that selects moves and outcome positions with the highest odds of winning, AlphaZero was able to widen its search for optimal chess moves with much greater speed and efﬁciency compared to extant chess engines like Stockﬁsh that relies on rule-based alpha-beta tree search (pruning) algorithms [55], [56]. The superiority of the AlphaZero’s AI architeciture was demonstrated when it beat Stockﬁsh 8 in a 1,000 game match in 2018 with a score of 155 wins, 6 losses, and 839 draws [57]. Even though a rule-based system is rigid, it is transparent and comprehensible as to how its outputs are generated compared with neural networks that power AlphaGo and AlphaZero. While a machine learning (ML)-based architecture or model – which a neural network is built upon – has the ability to adapt to task demands and learn rapidly through the ﬁne-tuning of parameters built as part of its functional system, how it made its decisions and generated its outputs can remain opaque even to its own programmers [55]. As such, a conventional ML system can be seen as a "black box" in terms of the interpretability of output generation. As far as to how AI can mimic humans, it cannot totally replace humans. This is because AI lacks the elements of creativity, ingenuity, and human nature that humans possess. Unless AI start to possess these human traits, humans and AI will have to work hand-in-hand with each other. For human machine operators, it is important to embrace AI as AI can generally provide advice that are much more accurate than a human’s, leading to the most appropriate solution to a question [58]. Despite the advantages of using AI, AI has also introduced elements of uncertainty into the decisions it made. As mentioned earlier, an AI model is analogous to a black box that cannot explain how it comes up with a decision. This creates concerns among AI users that they will not be able to make the most accurate decisions due an imperfect understanding of the algorithms. Such a lapse in AI comprehension among users can lead to unfair, biased, or erroneous decision-making [59]. Figure 5: The Explainable AI (XAI) program in DARPA aims to create a suite of machine learning (ML) techniques that: produce more explainable models, while maintaining a high level of learning performance (prediction accuracy); and enable human users to understand, appropriately trust, and effectively manage the emerging generation of artiﬁcially intelligent partners. Source: Extracted from [49]. This is where Explainable AI (XAI) comes in. XAI provides the next stage of human-machine collaboration as it explains why AI has reached a speciﬁc decision [50]. XAI can help human machine operators to make faster and more accurate decisions and uncover biases in the use of any existing ML system. By doing so, XAI can build "bridges of trust" between humans and machines. Being able to understand how an AI system works is essential to its improvement and practical utilization [50] and Figure 9 shows the promises the XAI of tomorrow can bring, as conceptualized by DARPA [49]. XAI development requires an inclusive design of AI systems, which takes the needs of different end-users into consideration during the AI development. As shown in Fig 6, different stakeholders of AI in ATM (e.g., ATCOs, pilots, passengers, ATMRI, CAAS) will harness XAI over a range of diverse purposes. In Fig 7, Professor Sargur Srihari summarized the reasons for why XAI is needed for system justiﬁcation and improvement, compliance to legislation, and scientiﬁc discovery. Figure 6: Different audiences and users of AI systems require AI explainability for different purposes (e.g., trust, understanding, legal compliance). We also adopt these concepts for the stakeholders of ATM. Source: Extracted from [60]. Figure 7: Summary of needs for explainable AI. Explanation is required for justiﬁcation, improvement, control and discovery of knowledge learned from data. Source: Extracted from lecture slides of Prof. Sargur Srihari. The main challenges of XAI, depending on the operational relevance of its application, can be broadly categorized into data analytics and autonomy problems, as illustrated in Fig 8. These two problems cover the prediction of events of interests and the decision-making policies of autonomous systems. In ATM, the ﬁrst problem is related to prediction systems (e.g., runway exit and occupancy time), while the second problem is relevant to recommended systems (e.g., conﬂict detection and resolution, surface movement optimization). As these two problems concern AI-enabled tools for ATCOs, XAI is important for helping ATCOs to manage their AI "partners" effectively and appropriately. There are a number of ways to achieve XAI, which can be broadly categorized into ante-hoc, post-hoc and transparent models (see Figure 10, details to be provided in the next section). Building upon the categorization, DARPA also lays out new approaches to create a suite of machine learning techniques that aim at producing explanation while maintaining a high level Figure 8: Challenging XAI problem research areas. Source: Extracted from [49]. Figure 9: Illustration of different approaches to achieve explainable AI, by developing ante-hoc, post-hoc and transparent models. Source: Extracted from lecture slides of Prof. Sargur Srihari. of learning performance. Figure 10 shows a notional Venn diagram that clusters machine learning techniques with trade-offs in explainability and learning performance. In this ﬁgure, DARPA also outlines three main strategies to achieve XAI, which include deep explanation, interpretable models and model induction. B. Taxonomy of XAI Techniques As shown in Figure 11, there are different ways to get explanations from a ML model. It is either that the model itself is transparent or that there is a need to use post-hoc explainability techniques on an already developed model to generate understandable information with respect to its prediction for any given input. Under post-hoc explainability techniques, there are model-agnostic and model-speciﬁc techniques. Model-speciﬁc techniques are designed with the purpose for extracting Figure 10: DARPA’s strategies to develop XAI. Source: Extracted from [49]. information related to any model from its prediction procedure. By contrast, model-agnostic techniques are dependent on model simpliﬁcation, feature relevance explanation, local and visual explanation techniques. The subsections below present an overview of the most popular model-agnostic and model-speciﬁc techniques. This overview is intended to be brief and comprehensible for a general audience with elementary computer science knowledge. For detailed explanations of these techniques, the reader is advised to refer to the systematic review done by Arrieta et al. (2019) [50]. Starting with model-agnostic techniques, we describe three techniques of interest (in bold) in the following paragraphs. We follow up on these information with a description of some model-speciﬁc techniques in the next subsection. 1) Model-agnostic Techniques Explanation by simpliﬁcation is based on rule extraction techniques, where simpler explanations of what a model does is provided based on relevant ML rules. One way is to use LIME. LIME is a local interpretable model that focuses on training a local surrogate model to explain individual predictions [61]. Basically, a local surrogate model is an interpretable model that is used to explain individual predictions of "black box" ML learning model [62]. With reference to Figure 11, LIME falls under local explanation techniques. A brief mathematical description of how LIME works is provided in the subsection "Two popular model-agnostic XAI techniques" below. Feature relevance explanation techniques explain the model by ranking or measuring the signiﬁcance of each feature in the prediction output. One of the main contributors to this technique is SHAP. The explanation model of SHAP assigns gain and loss values (i.e., "payout" values) to each input feature, in the form of Shapley values, as shown in Figure 14. These values represent and explain three desirable properties of the SHAP model – local accuracy, missingness, consistency [62]. A brief mathematical description of how SHAP works is provided in the subsection "Two popular model-agnostic XAI techniques" below. Visual explanation techniques use visualization to help explain a black box model. However, visual explanations are not common because they must ﬁt nicely into a ML model without regards for its inner structure. Hence, visual explanation is usually tagged with feature relevance techniques which provide information that is eventually displayed to the end user. 2) Model-speciﬁc Techniques Ensembles and multiple classiﬁer systems include tree ensembles (also known as "random forests"). Tree ensembles offer one of the highest accuracy among ML models. With a single decision tree, the decision ﬂow can be visualized and interpreted Figure 11: Taxonomy of XAI techniques. Source: Figure 6 in [50].