# Bias Influence on AI Accuracy: The Case of Air Traffic Controllers' Experience

Aurelie Peuaud, Anthony Clerquin, Antoine Alaverdov

► To cite this version:

Aurelie Peuaud, Anthony Clerquin, Antoine Alaverdov. Bias Influence on AI Accuracy: The Case of Air Traffic Controllers' Experience. HCII 2025, Jun 2025, Gothenburg, Sweden. pp.120-139, 10.1007/978-3-031-93721-7_10. hal-05362989

HAL Id: hal-05362989

https://hal.science/hal-05362989v1

Submitted on 13 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Bias Influence on AI Accuracy : the case of air traffic controllers' experience

Aurelie Peuaud¹[0009-0003-0572-071X], Anthony Clerquin², and Antoine Alaverdov¹[0009-0004-7775-5214]

¹ Ecole Nationale de l'Aviation Civile, Toulouse, France

² Airbus Defense & Space, Labège, France

**Abstract.** This study examines the effect of a biased dataset in the Artificial Intelligence (AI) learning process. Biases are almost inevitable and history has shown that it may significantly influence AI models, leading potentially to erroneous results and even dangerous consequences. We propose, through the case of air traffic control, an evaluation of the influence of bias in AI model training and performance. Air traffic control requires years of operational experience to master the environment and develop effective conflict resolution strategies. The level of experience is known to influence significantly the aircraft collision avoidance strategy. Our study is based upon a dataset collected with controllers of various levels of experience, leading to the creation of a bias in their avoidance strategy, the validation of this bias through statistics and the AI model development and simulation. Our approach compares two iterations of the model; one without any action to handle the bias and one by integrating it as a feature. The main findings show that declaring the bias as a feature does not necessarily impact AI model learning and overall accuracy, but can clearly influence the results over specific classes. The results of the second iteration contributing to higher alignment with specific preferences of the experienced controllers, versus the novice ones.

**Keywords:** Digital Assistant · Artificial Intelligence Learning · Dataset Bias.

## 1 Introduction

Data is crucial to feed research and innovation projects and developments. This is even more true with Artificial Intelligence (AI)-based products. All aspects of data management are relevant, in particular in our study, availability and accuracy.

For decades, this topic is a struggle for many researchers and innovation stakeholders, especially when dealing with restricted data in secured domains like air traffic control.

Various initiatives have emerged to counterbalance the lack of data, such as the development of frugal AI algorithms and the generation of synthetic data.

The synthetic data generation poses major drawbacks that could prevent using them massively. Most of classical techniques like SMOTE or ADASYN, as
well as more complex techniques such as GAN and deep learning can lead to over-representation of minor classes and alter the original distribution. In addition, the synthetic data evaluation is challenging, subjective and very difficult to generalize across studies, often requiring combination of several techniques, such as descriptive statistics, graphical representations and machine learning efficiency [1].

In the context of the SESAR 3 JARVIS Project [2], Airbus DS and ENAC teams are developing a digital assistant for confirmed air traffic controllers. The tool under development is an AI advisory assistant, that can suggest up to four conflict resolution strategies, ranked by order of preference, to support air traffic controllers. The AI component is in charge of the ranking based on past air traffic situations. To train the digital assistant learning, historical data are needed to provide samples including both air traffic conflict situations and separation strategy chosen by the air traffic controller. These data are not recorded in operational systems, as it is not a necessity for the operations and could jeopardize security. As a result, we created a specific dataset.

Synthetic data generation was discarded early due to drawbacks mentioned above and as we had the possibility to create our own dataset. However, confirmed air traffic controllers have limited availability, we thus include also novice air traffic controllers, easier to mobilize.

The level of experience of air traffic controllers, namely the difference between confirmed and novice, is of major importance with regards to separation task and it has already been demonstrated that it influences the conflict resolution strategy [3, 4]. For this reason, a priori assumption was that the dataset collected integrated a bias. If confirmed, this bias would need to be addressed during the AI algorithm development. At this stage, we choose to launch a dedicated study in parallel with the work performed for JARVIS project to quantify the influence of a bias in AI algorithm learning. So, we choose to develop two iterations of the algorithm: one without managing the bias and one that incorporated it as a feature.

This paper presents valuable insights of this parallel study and aims at enriching knowledge in AI domain: how does a bias in dataset may influence AI learning and performance and which approaches to identify and address them?

The remainder of the paper is structured as follows: section 2 presents the case study and its characteristics to provide essential insights into air traffic control specificity, especially to non-specialists in air traffic management (ATM). Section 3 presents the dataset collection approach and describes the potential bias. Section 4 presents the first AI algorithm iteration, without managing the bias related to the level of experience. Section 5 presents the statistical study to assess the bias. Section 6 presents the second iteration of the algorithm integrating the bias as a feature. Finally, section 7 discusses results and proposes recommendations for further studies.
## 2 Case Study

The case study of this paper is the development of a digital assistant for civil air traffic controllers. This chapter provides essential notions for any reader to comprehend the rest of the study. The first section explains key terminologies and basics while the second one focuses on the digital assistant.

### 2.1 Air Traffic Control Basics and Key Terminology

Air traffic controllers' main task is to ensure separation between aircraft and physical obstacles [8]. Separation for air traffic control means that the minimum standard lateral and vertical distances are guaranteed, as defined by the International Civil Aviation Organization: these minima are in general of five nautical miles in lateral and 1000 feet in vertical [5]. These values have been retained for our study.

To perform separation, the air traffic controller is equipped with a radar position with visual display of all civil aircraft' positions, relevant information such as speed, heading and a set of tools to ease detections of a loss of separation. Visual monitoring is complemented by radio contact with the pilots to exchange information and provide instructions called *clearances* [11]. Clearances can be issued for various reasons. In this paper, we will focus on clearances provided for separation reason only, namely, to ensure the minimum separation distances are guaranteed. A loss of separation is usually detected a few minutes ahead of time, either by a tool or by the controller, and is called a *conflict* [9].

Clearances issued for separation reason are called *conflict resolution clearance*, or *resolution clearance* in this paper as they aim at solving a conflict. They aim at instructing a specific maneuver to the pilot, resulting in a change of trajectory and can be of different types. It is important in the context of this paper, to bear in mind that a conflict can be avoided through many different strategies, some being more efficient than others in terms of fuel consumption but all reaching the same objective, i.e.: avoiding the conflict.

The term *separation strategy* used in this paper, refers to the choice of conflict resolution clearances retained by an air traffic controller to solve a conflict. There could be several clearances for a same conflict, however in this study, as stated in later sections, we choose to retain cases where only one clearance is issued for one aircraft to solve a given conflict.

### 2.2 Digital Assistant

The scope of the digital assistant under development in this paper is to provide support in decision-making to issue separation strategy. The aim of the AI component is to rank the preferred option, i.e.: predict the most likely used clearance type by a controller for a given conflict.

There are several types of conflict resolution clearances, linked to physical capabilities of the aircraft [10]. In this paper, four have been retained: provision
of a direct (DIRECT), change of heading (HEADING), change of flight level (LEVEL) and change of speed (SPEED) as illustrated in Fig. 1 and 2.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Illustration of an aircraft trajectory DIRECT and HEADING type of clearances.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2: Illustration of a flight vertical profile of the LEVEL type of clearance.</figcaption></figure>

The SPEED clearance consists in a small reduction or increase of the aircraft speed.

This paper concerns the case of a conflict between two aircraft in stable altitude and the use of only one resolution clearance type issued to one of the two aircraft. It is important to highlight that the targeted end-users for this digital assistant are *confirmed* air traffic controllers working in en-route, in other words with stable traffic mostly, and in operational environment, as opposed to a simulated one. In this paper, a *confirmed* air traffic controller owns a valid radar license and has thus completed initial course and validated the qualification on
at least one air traffic control area. This qualification gives the controller the authorization to perform air traffic control on the qualified area.

# 3 Presentation of the Dataset Collected

## 3.1 Dataset Collection Method

From May to July 2024, volunteers among ENAC<fnref n="3" /> [7] students and staff controllers were invited to play scenarios on the simulation platform used for the JARVIS [2] experimentation. Each volunteer was first briefly trained on the simulation platform to ensure proper usage and then tasked to perform typical controller job on at least one scenario. Scenario duration was about 30 to 45 minutes and volunteers were free to play multiple scenarios which included various situations of conflicts. More information can be found in section 5.3.

Situations of conflict and separation strategy were recorded throughout all simulation runs. Separation strategy could be composed of one of the four following clearance types: provision of a direct (DIRECT), change of heading (HEAD-ING), change of flight level (LEVEL) and change of speed (SPEED). Approximately 50 hours of simulation have been recorded, with 837 entries. An entry is composed of a conflict, a resolution clearance and the corresponding aircraft context. Each component can be found in separate log files and linked by a common timestamp column, representing the computer internal clock.

## 3.2 Description of the A Priori Bias in the Dataset

Volunteers originated among students and staff controllers and represented on the Fig.s 3 which split the dataset population per level of experience as follows:

- confirmed: 8 volunteers having completed initial course and holding a radar license. They all had at least 10 years of experience.

- novice that can be eventually split into two sub-categories.

* trainees: 18 volunteers having completed initial course and training for a year in their assigned operational centers. As a general rule, obtaining a radar license requires three years of training.

* students: 16 volunteers having completed basics controller modules in en-route context and still studying to complete initial course at school. Initial course requires in general two years. For most of the volunteers, they were mid-course and had completed one year at school.

<footnote n="3">³ Ecole Nationale de l'Aviation Civile - French Civil Aviation University</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 3: Distribution of level of experience among the volunteers</figcaption></figure>

Authors were informed by peers at the beginning of the data collection of a potential bias in the methodology. Previous work conducted at ENAC [3] reported of a significant difference in air traffic control performances and strategy between *novice* and *confirmed* controllers and a tendency for the later to rely on conservative strategies in complex situations, which was aligned with previous findings [4].

To further investigate, authors performed their own literature review starting with A.Bisseret paper [4]. A.Bisseret studied the effect of professional experience on air traffic control and focus on the potential causes, such as the information processing method used (judgment versus calculation). This finding seems aligned with eye movement comparison between *confirmed* and *novice* controllers [6], with a significant difference in the eye movement when analysing an air traffic situation, which can be related to the information processing method.

As a conclusion, it is likely that significant differences in the separation strategy among the groups of volunteers, due to the level of professional experience and in particular the information processing method, will be observed. This bias, if confirmed, may influence the training of the AI digital assistant, leading to incorrect proposals for *confirmed* controllers. As this group is also underrepresented in our data set (19 %), the impact is expected to be high.

# 4 AI Algorithm Development Approach Without Level of Experience Feature

## 4.1 Dataset Exploration and Analysis

During this step, the dataset collected has been analysed to ensure data quality and completeness of the values. The parameters retained for analysis do not consider any distinction between volunteers nor scenario played in this iteration.
Several usual actions have been performed as listed below. These are very
typical data preprocessing techniques used before training AI algorithms and are
only briefly presented.

- Cleaning:

* clearances not related to a conflict through timestamp comparison were removed.

* conflicts between aircraft and boundaries, as focus is on two aircraft type of conflicts only, were removed.

- Formatting: spatial coordinates of the involved aircraft are transformed from absolute values to relative values to allow comparison between aircraft in conflict.

- Exploration:

* analysis of the data distribution across clearance types, referred to as *Target classes* as the clearance type is the objective of the AI.

* identification of outliers: Correlation matrix and pair plots to gain insight on dimensionality reduction.

The exploring step highlighted two elements. First, the distribution among
classes was very uneven as shown on Fig. 4, which would require balancing
techniques as described in section 4.2.

<figure><img src="image_5.png" /><figcaption>Fig. 4: Target Classes Distribution</figcaption></figure>

Secondly, no extreme or correlated data has been identified, as shown on Fig. 5. The figure does not offer a proper view on this paper but is presented to show:

- there is a great majority of red and black boxes on Fig. 5b, which represents the absence of correlation.

- there is no significant extreme value as shown on boxplots 5a

All parameters and data have thus been kept for next steps, resulting in 837 entries in total.
<figure><img src="image_6.png" /><figcaption>Fig. 5: Identification of outliers</figcaption></figure>

## 4.2 Dataset Preparation and Models Evaluation Method

The 70/30 train-test split was selected after several trials as the optimal ratio based on a trade-off between accuracy, confidence in the result and generalisation ability. Due to the limited entries of SPEED clearances (14 over 837), there is a risk that all entries of SPEED clearance could be assigned exclusively to the test set during the split. That may skew the model, as there would be no exposure to SPEED clearance during training, rendering it incapable of predicting this class. To prevent this kind of problem, we used the stratify parameter during the dataset split. This parameter enforces the same representation ratio of each target class in both the train and test split.

We also applied class-weighting based on the number of entries of each target class to address the under-representation of the SPEED and LEVEL clearances. However, the balance remained imperfect due to the significant difference in representation between HEADING and DIRECT clearances, compared to LEVEL and SPEED.

Four indicators have been selected to score performance of the algorithm predictions, namely:

- The accuracy, which provides the percentage of correctly predicted instances.

- The F1-score, which provides the reliability by combining both precision and recall scores, good to use when dealing with imbalanced datasets.

- The AUC-ROC score, which evaluates the learning capability of the model. When AUC-ROC = 0.5, then the model is random. The higher the score is, the higher the model is able to learn.

- Confusion Matrix to identify correct predictions % per class type.

After the training, we performed a cross-validation to fine-tune the hyperparameter of the models tested. The cross-validation technique allows to fine-tune the parameters of the model, for example the number of trees or branches for a random forest. This technique consists of splitting a small part of the train set
to evaluate various combinations of a same algorithm and ensures the optimum combination is found.

## 4.3 Models Results

Several types of models have been tested :

- Random Forest

- XGBoost

- LightGBM

- (Histogram) Gradient Boosting Classifier

- Extra Tree Classifier

- Voting Classifier

- KNN, SVM

- Logistic Regression

The best performing models were the Random Forest and the XGBoost. The comparison is provided in the table 1. Values indicated are the average value obtained after 10 iterations on the test set. As a reminder, at each iteration, the 70/30 train-test split has been randomly performed (ensuring however that each type of class is represented in both train and test sets).

<table><caption>Table 1: Random Forest and XGBoost Results Comparison.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Scores</th><th colspan="3">Correct prediction %</th></tr><tr><th>Accuracy</th><th>F1-score</th><th>AUC-ROC</th><th>SPEED</th><th>LEVEL</th><th>HEADING</th><th>DIRECT</th></tr></thead><tbody><tr><td>Random Forest</td><td>61.5%</td><td>60.9%</td><td>0.81</td><td>0</td><td>34.62</td><td>69.23</td><td>60.67</td></tr><tr><td>XGBoost</td><td>65.8%</td><td>64.5%</td><td>0.82</td><td>0</td><td>38.46</td><td>76.92</td><td>62.92</td></tr></tbody></table>

The results of XGBoost outperform the RandomForest for each type of clearance prediction with very good results on HEADING predictions. In the test set, when for a given situation the controller would have selected a HEADING clearance to solve the conflict, then the AI Algorithm predicts indeed the use of HEADING in more than 76% of cases. However both models consistently failed to predict the SPEED type, which is correctly predicted in 0 case, confirming the unbalance of the data, despite the class weight technique.

Overall, the results were evaluated satisfying. However, the AI model would need further development to increase accuracy and align better with operational needs where far higher accuracy is expected (at least above 90%).

## 5 Dataset Statistical Study

In parallel with the first AI model iteration development described in section 4, a statistical analysis was conducted to characterize the bias presented in section 3.2.
**5.1 Overview**

The aim of this statistical study is to search for significant differences between the two groups of volunteers, *confirmed* and *novice* and further study evolution of strategy pattern with level of experience, by splitting the *novice* group into the two sub-categories, *student* and *trainee*. The statistical analysis has been performed on the same log files as for the AI algorithm development. However, the detection has been simplified as we decided to focus on stable conflicts only, and the most complex situations where several clearances have been used to solve a unique conflict. This resulted in 538 retained entries out of a total set of 837. In addition, data from the volunteer file have been extracted. This file provided the level of experience for each volunteer and the scenario played. First, the dataset distribution has been explored to determine the most suitable statistical approach. Then, other influences than level of experience have been explored. Finally, analysis has been performed to find correlation between separation strategy and level of experience.

**5.2 Statistical Approach Method**

Among the 538 entries, all scenarios, all groups considered, 35% were of DIRECT type, 55% of HEADING type, 9% of LEVEL type and 1% of SPEED type, as shown on Fig. 6.

<figure><img src="image_7.png" /><figcaption>Fig. 6: Number of clearances types.</figcaption></figure>

The distribution was studied for the two main categories, DIRECT and HEADING, with Q-Q plots method. SPEED and LEVEL were too poorly represented to provide reliable results.
<figure><img src="image_8.png" /><figcaption>Fig. 7: Q-Q Plots</figcaption></figure>

As shown on Fig. 7, the Q-Q plots indicate a normal gaussian distribution, except with right extreme values, which will allow us nonetheless to perform F-Fisher test with satisfying confidence in results.

## 5.3 Influence of Other Factors

This section aims at exploring if other factors that the level of experience may influence significantly the variety of strategy resolution that could be observed among the groups.

**Scenario Variation** Seven different scenarios were designed to collect data, built in collaboration with a confirmed controller. These scenario were of various complexity. The most complex ones were often played by the *confirmed* volunteers. We need thus to ensure that the scenario played has no real significance in the resolution strategy. We first computed the number of runs for each scenario, i.e. the number of times a given volunteer played a given scenario, and then the average number of clearance types for each scenario to compare with the average number of clearance types all scenario mixed.

The table 2 summarizes the number of runs for each scenario and the average number of conflicts encountered which may vary due to actions of volunteer, avoiding or creating unforeseen conflicts.

Fig. 8 shows that the 4 main used scenarios are presenting a similar profile, thus the scenario influence is considered negligible for this dataset.
<table><caption>Table 2: Distribution of runs per scenario</caption><thead><tr><th rowspan="2">Scenario ID</th><th colspan="3">Nb. of runs</th><th rowspan="2">Average nb. of conflicts</th></tr><tr><th>ALL</th><th>Novice</th><th>Confirmed</th></tr></thead><tbody><tr><td>MEDIUM_1</td><td>19</td><td>16</td><td>3</td><td>6.5</td></tr><tr><td>MEDIUM_2</td><td>10</td><td>10</td><td>0</td><td>6.6</td></tr><tr><td>MEDIUM_3</td><td>7</td><td>7</td><td>0</td><td>7.4</td></tr><tr><td>DENSE_1</td><td>15</td><td>11</td><td>4</td><td>7.6</td></tr><tr><td>DENSE_2</td><td>8</td><td>7</td><td>1</td><td>7.1</td></tr><tr><td>DENSE_3</td><td>15</td><td>10</td><td>5</td><td>7.4</td></tr><tr><td>VERY_DENSE_1</td><td>1</td><td>1</td><td>0</td><td>11</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Fig. 8: Average number of clearances per type and scenario</figcaption></figure>

**Cursus Variation Among Novice Group** Among the *novice* group, 4 various promotions have participated from the controller's cursus called MCTA for "Master en Contrôle du Traffic Aérien" which trains French controllers. It is relevant to distinguish the operational experience among *trainee* and *student* as the first group has a year of experience in operational centre, while the second has only experience on school simulators.

The statistics as shown in Fig. 9 tend to show a variant in their separation strategy pattern and a slight shift of preferred clearance type from massive usage of HEADING among students to more DIRECT among trainees.
<figure><img src="image_10.png" /><figcaption>Fig. 9: Comparison of clearances types distribution among two sub-groups of Novice</figcaption></figure>

Both groups include a similar number of volunteers (respectively 16 for student and 18 for trainee), we may conclude that the year experience in operational centres influences the separation strategy. In the next section, we will thus propose an analysis over both two classes, confirmed and novice and three, student, trainee and confirmed.

## 5.4 ANOVA Results

The previous section, 5.3, showed that the level of experience is most likely to have an influence on the separation strategy and thus, a risk to lead to a bias, especially as population among novice and confirmed is alike (respectively 36 and 8 individuals).

The Fisher test has been performed on dataset to determine correlation between level of experience and separation strategy.

Let's denote H0 and H0' the hypothesis to reject or not:

H0: DIRECT strategy and level of experience are independent.

H0': HEADING strategy and level of experience are independent.

**ANOVA on Two Classes, Novice and Confirmed** For each run (i.e.: a scenario played once by a given volunteer), the number of DIRECT clearances have been summed-up. When none was used, 0 was indicated. A similar approach was used for HEADING clearances. We thus obtain 62 entries for novice class and 13 for confirmed class for each type of clearance.

F was computed following the equation 1 and compared to the theoretic F at 5%. F at 5% indicates that we have 5% risk of being wrong with the dependence.

$$F = \frac{\frac{\text{SCEinter}}{\text{DDLinter}}}{\frac{\text{SCEintra}}{\text{DDLintra}}} \qquad (1)$$
where

$$SCEinter = \sum_{i=1}^{k} n * (ave_i - \overline{ave})^2$$

$$SCEintra = \sum_{j=1}^{n} (y^j - ave_i)^2$$

$$DDLinter = k - 1$$

$$DDLintra = n - k$$

and

- k is the number of classes, in extenso 2 in this case

- n is the number of elements

- $y^j$ is the $j^{th}$

- $ave_i$ is the average of values for class i

- $\overline{ave}$ is the average of the average of each class

The results presented in table 3 show that H0 is not rejected while H0' is rejected, though intensity is quite low, which can be due to the imbalance between the two classes. This would seem that the HEADING strategy is linked to the level of experience, while no conclusion could be drawn for the DIRECT strategy.

<table><caption>Table 3: ANOVA results on 2 classes : Novice , Confirmed</caption><thead><tr><th></th><th>DIRECT</th><th>HEADING</th></tr></thead><tbody><tr><td>F <span style="text-decoration: overline;">computed</span></td><td>0.34</td><td>4.1</td></tr><tr><td>F Theoric at 5%</td><td>3.9</td><td>3.9</td></tr><tr><td>Intensity</td><td>N/A</td><td>0.05</td></tr></tbody></table>

**ANOVA on Three Classes: Student, Trainee and Confirmed** The same test was performed by splitting the dataset into three classes, namely *student*, *trainee* and *confirmed*. The number of entries for each class was slightly more balanced with respectively 37, 25 and 13 entries.

The results presented in table 4 show that H0 is not rejected while H0' is rejected with a higher intensity. This would tend to confirm that the HEADING strategy is linked to the level of experience.

<table><caption>Table 4: ANOVA results on 3 classes : Student, Trainee , Confirmed</caption><thead><tr><th></th><th>DIRECT</th><th>HEADING</th></tr></thead><tbody><tr><td>F <span style="text-decoration: overline;">computed</span></td><td>0.38</td><td>5.47</td></tr><tr><td>F Theoric at 5%</td><td>3.12</td><td>3.12</td></tr><tr><td>Intensity</td><td>N/A</td><td>0.13</td></tr></tbody></table>
**Conclusions on Statistical Study** HEADING type, and thus part of the separation strategy is linked to the level of experience. No solid conclusion could be drawn for the DIRECT type. The main explanation relies with the dataset composition. HEADING use decreases significantly with the level of experience. The number of *confirmed* volunteers being quite low compared to *novice*, it is likely that the dependence could not be seen with such methods.

# 6 AI Algorithm Development Approach With Level of Experience Feature

## 6.1 Implementation of the Level of Experience Feature

To improve our analysis, we investigated the impact of incorporating the *experience level* feature into our dataset on our model performance. To add this feature to our dataset, each experience session was linked to the corresponding level of experience of the test subject, categorized into one of three labels: *student* for experience <1 year, *trainee* for experience <1 year and having spent their last year in their future assigned operational center, identified by their cursus promotion, and *confirmed* for experience >1 year.

Once the feature has been added, we encoded it using a label encoder, and the updated dataset is then used to train an XGBoost model with the same hyperparameter as the XGBoost model that provided the best results in section 4.3.

## 6.2 Analysis of the feature importance

To assess the importance of the *experience level* feature during the training of the XGBoost algorithm, we have used 2 methods :

**Permutation importance** is a technique used to assess the importance of individual features in a machine learning model. The method evaluates how much a model's performance metric (the accuracy in our case) decreases when the values of a particular feature are randomly shuffled. By permuting the values of the feature, their relationship with the target are disrupted, effectively removing the information they provides to the model. A significant drop in performance indicates that the feature is important for the model's predictions, while a negligible change suggests the feature has little to no contribution. We applied this technique and presented the results in Fig. 10.

We performed the permutation importance analysis 10 times, recording the decrease in accuracy at each iteration. Each point in Fig. 10 represents the accuracy reduction resulting from a single shuffle of the specified feature. When focusing on the experience feature, the permutation importance analysis revealed no significant decrease in model accuracy. For further validation, we repeated the shuffling process 10,000 times, consistently observing accuracy decreases close to
<figure><img src="image_11.png" /><figcaption>Fig. 10: Permutation importance results indicating no decrease in accuracy when the experience values are shuffled.</figcaption></figure>

zero. Using this technique, we can say that the *level of experience* feature has no significant impact on the learning process of our algorithm.

**Random baseline feature** is another technique to estimate the importance of a feature. The principle of this technique is to add a random feature to the dataset as a benchmark to evaluate the importance of actual features. If a feature's importance is similar to or lower than the random feature, it suggests the feature may not be meaningful for the model. In this paper, the random feature is made up of random integer between 1 and 10000. To estimate the feature importance, we used the gain. In XGBoost, gain is a measure of feature importance that quantifies the improvement in the model's performance brought by a feature when it is used for splitting a decision tree. The results are presented in Fig. 11

The gain feature importance of the *level of experience* is very close to the random feature. This strengthens our first analysis with permutation technique and confirms that the *level of experience* does not impact the accuracy of the gradient boosting model.
<figure><img src="image_12.png" /><figcaption>Fig. 11: Feature importances in gain of the XGBoost algorithm.</figcaption></figure>

## 6.3 Impact on models prediction

As observed earlier, there is no change in the model's overall accuracy. However, we can refine our analysis by examining the model's performance across specific classes. It is possible that, by adding the level of experience, the model improves its prediction of a certain target class while showing a decline in the prediction of another target class, resulting in stable overall accuracy but with more biased clearance decisions. We can see this kind of information on the confusion matrix of the model in Fig. 12.

<figure><img src="image_13.png" /><figcaption>(a) Without experience level feature</figcaption></figure>

<caption>(b) With experience level feature</caption>

<caption>Fig. 12: Comparison of confusion matrices for test dataset (252 entries) of the XGBoost algorithms</caption>
It is interesting to note that the prediction of DIRECT clearances improves, while the prediction of HEADING clearances declines in accuracy. Several factors could explain this outcome. One reason could be that in multi-classes classification, the model balances predictions across all classes, and introducing a new feature may enhance its ability to distinguish one class, while potentially diminishing its performance for another. This could occur if the new feature provides more relevant information for one class, but introduces noise or ambiguity for another. Alternatively, the new feature could interact with existing features in ways that improve the separability of one class but confuse the model for another.

In this section, we have seen that the introduction of the *level of experience* feature does not enhance the overall accuracy of our model, it even decreases it a little from 65.8% to 61.1% but we believe that the overall accuracies could match after fine-tuning the hyperparameters of the model with the experience level feature. However, it does improve the prediction of DIRECT clearances. Given that our goal is to develop a digital assistant for experienced air traffic controllers, and as we have seen in the statistical analysis, experienced controllers tend to favor DIRECT clearances, so retaining the *level of experience* feature in our dataset may still be valuable. Although it does not contribute to a higher overall accuracy, it aligns with the specific preferences of experienced controllers, making it a worthwhile inclusion.

# 7 Interpretation and Discussion

The statistical study detailed in section 5 confirms findings from [3] with a significant difference observed in the separation strategy pattern correlated with the level of experience. It confirms the dataset collected is biased. Based upon previous experience [3], it was assumed at the beginning of the study that it would likely lead to impacts in the AI learning and accuracy. Although the data collection has been conducted for another use, namely to feed the AI model developed under the scope of JARVIS project [2], it was a great opportunity to conduct the parallel study described in the present paper, providing insights to bias influence in AI model.

The present section gathers first initial answers to the research questions outlined in the section 1 and opens then on further research needs.

## 7.1 How does a bias in dataset may influence AI learning and performance?

The section 4 indicates that AI model overall accuracy is satisfying (65.8% of overall accuracy for the best model), considering no tuning nor optimisation have been applied. However it is interesting to note, that when term *Correct Predictions* is used in table 2, it means that, for a given situation, the type of clearance predicted would indeed be used to solve a conflict. However that would be by a controller from the dataset and not by a *confirmed* controller, who is
the targeted end-user. As detailed in the paper, the targeted end-user is bound
to rely different separation strategy. Does that mean that the dataset provokes
a too important bias and is thus not relevant, or not fitted to develop the AI
assistant as intended? The study does not allow to provide solid conclusion on
this question, although the scores are already a good basis to perform model
evaluation with human in the loop, as suggested for further studies in section
7.3.

## 7.2 How to identify and handle a bias in dataset?

The study proposes a method to identify the bias through statistical approach and to manage the bias with the feature integration strategy. This strategy seems a good approach as it allows to obtain better predictions scores for the DIRECT type of clearance (+2 points for DIRECT good predictions as visible in Fig. 12) without degrading significantly the overall accuracy of the model (-4.7 points, from 65.8% to 61.1%). As for the previous question, no solid conclusion could yet be drawn on the effectiveness of the method, but provides already a good basis to perform model evaluation with human in the loop, as suggested for further studies in section 7.3.

## 7.3 Further Research Recommendations

The current study does not involve a validation of the AI model with human
in the loop. This is thus our primary and first recommendation : set-up an
experimentation with *confirmed* controllers on the two iterations of the model
to capture their feedback on the clearances type predictions and conclude on
their relevance. This would allow to have better insights on the bias influence
as well as on the feature integration strategy. In case predictions are deemed
not accurate for both iterations, then it would mean that the bias may have
too much influence for this application and that a proper dataset is needed, or
else other bias management techniques would be required. In case predictions
are deemed better for the second iteration, then the bias management technique
effectiveness could be evaluated. To provide further elements on bias influence, a
second recommendation would be to collect a dataset from exclusively *confirmed*
controllers and feed it on the AI models to compare accuracy. As this might be
very difficult to obtain due to scarce availability of the *confirmed* controllers, a
bigger and more balanced dataset, including higher representation of SPEED and
LEVEL types of clearances and more volunteers from the *confirmed* population,
would also be an interesting option to pursue the work.

**Acknowledgments.** This study is part of a project that has received funding from the SESAR 3 Joint Undertaking under grant agreement No 101114692 under European Union's Horizon Europe research and innovation programme.

**Disclosure of Interests.** The authors have no competing interests to declare that are relevant to the content of this article.
References

1. Figueira, A., Vaz, B. Survey on Synthetic Data Generation, Evaluation Methods and GANs. Mathematics 2022, 10, 2733. https://doi.org/10.3390/math10152733

2. JARVIS SESAR Project Official Website, https://research.dblue.it/jarvis/

3. Durand, N., Gotteland, J-B., Matton, N., Bortolotti, L., Sandt, M. : Understanding and Overcoming Horizontal Separation Complexity in Air Traffic Control: An expert/novice comparison (2020) https://doi.org/10.1007/s1011-020-00634-z

4. Bisseret, A. : Application of signal detection theory to decision making in super-
visory control The effect of the operator's experience. Ergonomics, 24(2), 81–94.
(1981) https://doi.org/10.1080/00140138108924833

5. ICAO Document 4444, https://www.icao.int/EURNAT

6. Yanjun, W., Liwei, W., Siyuan, L., Wei, C., Jianfei, X., Washington, O. :Ef-
fect of Working Experience on Air Traffic Controller Eye Movement (2021)
https://doi.org/10.1016/j.eng.2020.11.006

7. ENAC Web site, https://www.enac.fr/en

8. Air Traffic Controller (ATCO) activities, https://atco.eurocontrol.int

9. Loss of separation definitions SKYbrary, https://skybrary.aero/operational-issues/loss-separation

10. Conflict solving definitions SKYbrary, https://skybrary.aero/articles/conflict-solving

11. Clearance definition SKYbrary, https://skybrary.aero/articles/atc-clearance