Deliverable 5.2 : report on human-machine distributed situation awareness
Creators: name: Häusler Hermann, Ruth
name: Vetter, Celina
name: Samardžić, Kristina
name: Antolović, Dorea
name: Tukarić, Ivan
name: Roth, Manuel
name: Tensfeldt, Luca
name: Bazina, Mia
name: Burkhalter, Jennifer
Publisher: name: AISA Consortium
Year: 2022
Subjects: AI
Situation awareness
Knowledge graph
Air traffic control
Monitoring task
Eye-tracking
380: Verkehr

description: This report presents the results of two simulation experiments performed with an AI-based situation awareness system (AI SA system) developed in the AISA project to check the accuracy of the AI SA system’s estimations and predictions and its capability to contribute to human-machine team situation awareness. It represents the AISA project deliverable 5.2 Report on Human-Machine Distributed Situation Awareness and contains four topical sections – described below – that address requirements to fulfil the project tasks 5.1 Comparison of SA between AI and ATCO and task 5.3 Human performance in distributed SA. The task 5.2 Risk assessment of AISA is covered in a separate deliverable D5.1 Risk assessment report.

• Topical section 1: Measurement of ATCO situation awareness and scanning behaviour
• Topical section 2: Comparison of human and machine situation awareness
• Topical section 3: Exploration of human-machine team situation awareness and its impact on human performance
• Topical section 4: Accuracy of AI SA system’s estimations and predictions and its level of situation awareness

Two simulations were conducted with licensed Air Traffic Controllers working as radar executive. Situation awareness was assessed with multiple methods. The probe technique was applied to compare compare human and artificial situation awareness. ATCOs’ experience with AI-based machine situation awareness (receiving “AI SA inputs”) and its impact on performance were explored. Post hoc simulations with data collected in experiment 1 were conducted to assess the accuracy of AI SA systems’ estimations and predictions.

Main findings per topical section are:

1. ATCOs with preserved situation awareness have characteristic scanning behaviour: Their gaze is less fixed on aircraft or conflicts, and they filter out more effectively non-critical information than ATCOs with degraded situation awareness do.
2. Partial agreement of human and machine situation awareness on conflict detection. Both human and AI SA system missed conflicts (false negative) and named conflicts that were not present (false positive). The AI SA system is better at monitoring non-obvious/unexpected aspects (e.g., non-conformances).
3. ATCOs detected some conflicts earlier and solved them faster when they received AI SA inputs compared to working without AI SA inputs. Input modality (oral messages) was inadequate due to distraction and additional workload.
4. Successful automation of 46 out of 57 en-route air traffic monitoring tasks. Accuracy of Machine Learning module predictions for CD tested (70%): Partly results were inaccurate, and predictions were partly inconsistent. Plausibility checks on CD module’s inputs and outputs were successful.

Limitations reduce the validity of situation awareness measurement for ATCOs (use of an unfamiliar simulation tool), the significance of the results (exploration of human-machine team situation awareness was done with early-stage implementation of the AI SA system and with inadequate design of HMI inducing additional workload on ATCOs).

The results generally support the proof-of-concept system of the AISA project in its ability to accomplish en-route air traffic management tasks. Further improvement of accuracy is needed for machine learning modules. Accuracy per se is not sufficient, considerable effort needs to be spent on solutions on how to integrate machine situation awareness. A long anticipation span is desirable for optimisation but does not comply with ATCOs’ need for prioritization of tasks and information. The HMI of the future AI SA system will need distinctive ways of informing ATCOs about aspects of higher or lower urgency. Half of the participating ATCOs were willing to trust future AI-based tools – even after partially unfavourable experiences with an AI-based SA system in the experiment, about one third is neutral and a fifth is negative about including AI in tools.