# From Human-Human Computer Mediated Communication to Human-Automation Collaboration in the light of Large Civil Aircraft Workplace

Célia Martinie, Elodie Bouzekri, Philippe Palanque

► To cite this version:

Célia Martinie, Elodie Bouzekri, Philippe Palanque. From Human-Human Computer Mediated Communication to Human-Automation Collaboration in the light of Large Civil Aircraft Workplace. Workshop on Automation Experience at the Workplace, AutomationXP 2021, co-located with the ACM Conference on Human Factors in Computing Systems (CHI 2021), May 2021, Yokohama, Japan. hal-03376276

HAL Id: hal-03376276

https://hal.science/hal-03376276

Submitted on 13 Oct 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
From Human-Human Computer Mediated Communication to Human-Automation Collaboration in the light of Large Civil Aircraft Workplace

Elodie Bouzekri, Célia Martinie &Philippe Palanque

ICS-IRIT, Université Paul Sabatier - Toulouse III, Toulouse, France, palanque@irit.fr

This position paper proposes the use of human-human collaboration models to study human-automation collaboration. We first present some of most prominent models from psychology and HCI and project their content to identify design rules that could be used to design and evaluate human-automation collaborations. We apply these principles to the workplace of cockpits of large civil aircrafts.

• Human-centered computing → Human computer interaction (HCI); *Interactive systems and tools*;

**Additional Keywords and Phrases:** Automation, collaboration, task modelling, aircraft cockpits

**ACM Reference Format:**

Palanque, P., Bouzekri, E., Martinie, C. 2021. From Human-Human Computer Mediated Communication to Human-Automation Collaboration in the light of Large Civil Aircraft Workplace. In *Proceedings of AutomationXP'21: Workshop on Automation Experience at the Workplace*. In conjunction with CHI'21, May 07, 2021.

# 1 INTRODUCTION

The evolution of computer use from one computer for several persons to many computers for one person could have been the end of multi-user computing. However, the widespread of internet and the rise of social computing has demonstrated that dealing with single user applications is nowadays part of history. Designing interactive systems thus requires, most of the time, to address the needs of groups of users involved in common tasks for which the communication, coordination and production is mediated by computers. Despite this undeniable situation, most of the research contributions in the area of interactive systems engineering still focus on single user applications. This is easily understandable as multi-users application are far more difficult to build than single user ones. This difficulty comes from different sources:

• The difficulty to gather and understand the requirements as well as the need of the users;

• The difficulties to address the required communication infrastructures in order to allow both synchronous and asynchronous communication between the users;

• The difficulty to ensure usability of these applications that are used jointly by different users (with different characteristics and needs) and under different environmental conditions (time zones, seasons, light, sound, ...);

• The difficulty to ensure the reliability of these computing systems involving underlying communication mechanisms, networks and the fact that their testing and validation is even more complex as they involve multiple, diverse software and hardware entities.

These problems are even more prominent when it comes to largely distributed worldwide teamwork such as international collision avoidance systems for satellites [7].
When it comes to automation, there is a very important difference to highlight, the fact that automation can be considered as co-located with the user thus removing the issue of latency, time zones and reliability of communication means. However, if the human-automation team is controlling a remote entity (e. g. the Automated Transfer Vehicles for refueling the International Space Station or satellite ground segments) some of the problems become valid again [5].

This position paper aims a positioning human-human computer mediated communication knowledge and principles in the perspective of human-automation collaboration. On the other side, we will also revisit the work on human-automation teaming such as [1]. Beyond, as the workshop focuses on automation, experience and workplace we will concentrate on the special context of work of large civil aircrafts.

## 2 HUMAN-HUMAN COLLABORATION

Aircraft cockpits are workplaces where several persons, names crewmembers collaborate to achieve a common goal: to bring passengers of the aircraft from one location to another.

### 2.1 Collaboration in large civil aircraft cockpit workplace

To pilot the large civil aircraft, the crewmembers operate the avionics systems through human-machine interfaces called command and control systems and located in the cockpit. Avionics system includes all the physical systems (engines, sensors, flaps ...), electrical, electronic and computer systems (autopilot, digital flight controls, digital flight controls ...), embedded in the aircraft. The cockpit embed screens that support the display of information coming from several avionics systems on the same screen. To control the avionics systems, the crewmembers use many physical buttons, analog cockpits, physical buttons placed at the disposal beside the displays.

The main goals of the crewmembers, by order of importance, are to fly, navigate, communicate and manage platforms. Nowadays, flying crews are composed of two crewmembers, which can have one of the two following roles: Pilot Flying (PF) and Pilot Monitoring (PM). PF is in charge of flying the aircraft and of stabilizing its pitch, altitude, bank angle, vertical and horizontal flight plans. Pilot Monitoring is here to back up PF by monitoring and making call outs. In the early days of commercial aviation, a third crewmember (see example in Figure 1) was in charge of managing avionics system and of recalculating flight plan in case of adverse events, such as clearance from the air traffic controller [11].

<figure><img src="image_2.png" /><figcaption>Figure 1. Cockpit of an airbus A300 with pilot flying, pilot monitoring and flight engineer</figcaption></figure>
## 2.2 Main principles of description of human-human collaboration

The description of human-human collaboration requires the explicit identification of the tasks that each human has to perform, in order to understand and allocate the work between them. It also requires the identification of specific aspects of the collaboration. Cooperative work may be dedicated to one or more of the following type of collaborative activities: production, coordination, communication. It is then possible to associate one or more properties amongst this set. For example, Figure 4 a) shows that one task is dedicated to coordination whereas Figure 4 b) shows that the task is dedicated to both coordination and communication.

<figure><img src="image_3.png" /><figcaption>Figure 2. Example of cooperative task properties from a “functional clover” [2]</figcaption></figure>

Cooperative tasks may be performed within various space-time constraints (local/distant, synchronous/asynchronous) [4]. Table 1 presents these different space-time constraints and illustrates the different possibilities with an example.

<table><caption>Table 1. Time Space Matrix from [4]</caption><thead><tr><th></th><th>Same Time</th><th>Different Times</th></tr></thead><tbody><tr><th>Same Place</th><td>Face to face interaction<br>Ex: Collaboration between pilots in the cockpit</td><td>Asynchronous interaction<br>Ex: Post-it</td></tr><tr><th>Different Places</th><td>Synchronous distributed interaction<br>Ex: Collaboration between pilot and air traffic controller</td><td>Asynchronous distributed interaction<br>Ex: Mail</td></tr></tbody></table>

## 2.3 Notation for the description of human-human collaboration

To apply these main principles of description, we use the notation HAMSTERS-XL [7] for describing collaborative tasks i.e. tasks having group of users trying to achieve common goals. HAMSTERS-XL provides elements to describe abstract group tasks as well as individual cooperative tasks. Several human persons are involved in collaborative work, each one having a role in the achievement of common goals. Collaborative work can be described at different abstraction levels: at the group level and at the individual level. A group task is a set of tasks that a group has to carry out in order to achieve a common goal [5], whereas a cooperative task is an individual task performed by a person in order to contribute to the achievement of the common goal [9]. Table 2 presents the main types of tasks for describing individual and collaborative tasks in task models.
<table><caption>Table 2. Individual and human-human computer mediated collaboration task types in HAMSTERS-XL</caption><thead><tr><th></th><th>Abstract</th><th>Input</th><th>Output</th><th>I/O</th><th>Processing</th><th>Group</th></tr></thead><tbody><tr><td>Abstract</td><td></td><td>Not applicable</td><td>Not applicable</td><td>Not applicable</td><td>Not applicable</td><td></td></tr><tr><td rowspan="2">User</td><td>Indiv.</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Coop.</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="2">Interactive</td><td>Indiv.</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Coop.</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>System</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

To identify time-space constraints, dedicated notation elements are available for cooperative tasks, as illustrated in Table 3.

<table><caption>Table 3. Time and space properties for cooperative tasks</caption><thead><tr><th></th><th>Local</th><th>Distant</th></tr></thead><tbody><tr><td>Synchronous</td><td>S<sub>1</sub> L<sub>1</sub><br>Interactive input<br>cooperative task</td><td>S<sub>2</sub> D<sub>2</sub><br>Interactive input<br>cooperative task</td></tr><tr><td>Asynchronous</td><td>A<sub>1</sub> L<sub>1</sub><br>Interactive input<br>cooperative task</td><td>A<sub>2</sub> D<sub>2</sub><br>Interactive input<br>cooperative task</td></tr></tbody></table>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

## 2.4 Example: 'Modify flight plan' group task with three crewmembers

Figure 3 presents an example of the usage of this notation through tasks models. These tasks models describe a group task: "Modify flight plan". This group task describes the recalculation of the flight plan by the flight engineer on pilot's request due to weather adverse events. In this section, we focus on two roles of the crew: pilot flying and flight engineer.

After the detection of weather adverse, the pilot flying asks for a new flight plan to the flight engineer. "Ask for a new flight plan" (P1) is a cooperative motor task with "Listen to the request for a new flight plan" (FE1) cooperative perception task of the flight engineer. This cooperation is local (in the cockpit) and synchronous (oral communication). These tasks describe a coordination between the pilot flying and the flight engineer. The pilot allocates a new next task (calculate a new flight plan) to the flight engineer. Then, the pilot flying and the flight engineer discuss and choose a new flight plan until one of the options is confirmed. The flight engineer communicates his progression (P2 and FE2) while calculating a new flight plan. Afterwards, the flight engineer proposes a new flight plan (P3 and FE3). After a validation (PE5 and FE5 user cooperative task) or a rejection (P4 and FE4 user cooperative task) of this new flight plan, the flight engineer calculates another flight plan or applies the new flight plan. All these cooperative tasks are local to the cockpit and synchronous.
These tasks describe communication between the pilot flying and the flight engineer in order to produce a new flight plan.

The Table 4 presents associations of cooperative tasks of the pilot and flight engineer roles.

<figure><img src="image_42.png" /><figcaption>Figure 3. Recalculation of the flight plan by the flight engineer on pilot flying's request due to weather adverse events</figcaption></figure>

<table><caption>Table 4. Cooperative tasks table.</caption><thead><tr><td>Cooperative task</td><td>Linked to</td></tr></thead><tbody><tr><td>"Ask for a new flight plan" (P1)</td><td>"Listen to the request for a new flight plan" (FE1)</td></tr><tr><td>"Communicate progression" (FE2)</td><td>"Listen to flight engineer's report on his progression" (P2)</td></tr><tr><td>"Communicate a new flight plan" (FE3)</td><td>"Listen to flight engineer's recommendation" (P3)</td></tr><tr><td>"Request for another route" (P4)</td><td>"Hear and acknowledge other route request" (FE4)</td></tr><tr><td>"Confirm the route" (P5)</td><td>"Hear and acknowledge confirmation of the route" (FE5)</td></tr></tbody></table>

# 3 HUMAN AUTOMATION TEAMING

Aircraft cockpits are nowadays workplaces where crewmembers interact between each other and with automation.

## 3.1 Automation of the flight management

Several decades ago, the two pilots' forward facing cockpit was introduced, such as A300¹, and reduced the crew to two members: Pilot Flying and Pilot Monitoring. The "Manage systems" tasks was partly automated with an alerting system and the Pilot Monitoring is now in charge of monitoring this alerting system to manage systems. The flight path calculation has also been automated using the Flight Management System (FMS) and an auto-piloting system has been introduced to reduce workload during critical phases of flight. Figure 4 depicts the three loops of automation that include: the direct interaction between Pilot Flying and flight commands (no automation), the interaction between Pilot Flying and auto-pilot (automation of task "fly"), and the interaction with the FMS (automation of the task "Navigate").

<footnote n="1">¹ https://www.airbus.com/company/history/aircraft-history/1970-1972/a300.html</footnote>
<figure><img src="image_43.png" /><figcaption>Figure 4. Three levels of automation of the flight management task</figcaption></figure>

## 3.2 Main principles of description of human-automation collaboration

When dealing with human-automation collaboration, it becomes insufficient to describe only the program of the system. Indeed, human-automation teaming requires transparency (mutual understanding), bi-directional communication and operator directed execution (responsibility of the final decision to the human) [1]. Description of human-automation teaming requires to describe [10]: actors (both human and automation), role allocations of actors and their relationships. In addition, it is required to identify the cooperative tasks of the ‘automation’ actor that are actually implemented by one or several functions, as well as the type of collaborative tasks between the ‘automation’ actor and the human actor. Cummings and Bruni [2] defined a taxonomy of human-automation collaboration based on the tasks that the human and the automation perform: data acquisition tasks, analysis tasks, decision tasks and motor (action) tasks (see Figure 5). For example, the Decider is the actor(s) that makes the final decision.

<figure><img src="image_44.png" /><figcaption>Figure 5. High-level view of the three collaborative decision-making process roles: moderator, generator, and decider from [2]</figcaption></figure>

## 3.3 Notation for the description of human-automation collaboration

To apply these main principles of description, we use the HAMSTERS-XL notation [7] that provide elements of notation for describing cooperative tasks of both human actors and ‘automation’ actors. The cooperative human tasks are the same as the ones presented in Table 2. The Figure 6 presents the system task types. The input and output tasks are cooperative tasks that receive or send information.
<figure><img src="image_45.png" /><figcaption>Figure 6. System task types</figcaption></figure>

A separate model describes the tasks performed by a role of an ‘automation’ actor. In order to describe the associations between automation cooperative tasks and human cooperative tasks and the flow of information between them, we define cooperation protocols. A cooperation protocol describes information sharing between several cooperative tasks, which contributes to transparency. Five attributes composed a cooperation protocol: (main) type of collaborative activities (see Figure 2), the localization and time (see Table 3), cardinality (broadcast, unicast, groupcast or anycast) and the information shared.

**3.4 Example: ‘Modify flight plan’ group task with a two-pilots crew and automation of flight management**

Figure 7 presents an example of the usage of this notation. These tasks models describe the same group task as before. However, the flight engineer role is automated. We consider an envisioned FMS. This envisioned FMS can recommend flight plans to the pilots. This FMS performs similar tasks as the flight engineer but automated: receive the request (CP1), calculate a possible new flight plan and display progression (CP2), then decide a flight plan to propose to the pilots (CP3) and iterates (if rejection by pilots (CP4)) until the proposed flight plan is confirmed (CP5). Finally, the FMS displays the new flight plan to pilots (CP6).

<figure><img src="image_46.png" /><figcaption>Figure 7. Recalculation of the flight plan due to weather adverse events with the envisioned FMS</figcaption></figure>

Figure 8 presents the cooperation protocols between the pilot and FMS roles. For example, the CP4 and CP5 are cooperation protocols describing the communication of the decision made by pilots on the proposed flight plan (reject or confirm). The pilots are the deciders of the flight plan selection. These cooperative tasks are synchronous, local to the cockpit and unicast (from the pilot to the FMS).
<figure><img src="image_47.png" /><figcaption>Figure 8. Cooperation protocols properties</figcaption></figure>

Despite the fact that the envisioned FMS aims to implement some of the tasks that the flight engineer used to perform, the collaboration will be of a different nature. For instance, the flight engineer might take time between receiving a request and processing it and might even forget it. The automation at the opposite will process information immediately (at least this will be perceived as such by the pilot). Indeed, the interaction with automation might look like a standard interaction on a button: the user presses the mouse button while the cursor is on it (this corresponds to sending a request) and the immediate graphical feedback showing the button pressed corresponds to the acknowledgement. This fine grained interaction will have to be designed, standardized and implemented for any kind of cooperation or the interaction with automation will not be transparent [12]. This might be even more complex due to the multiple objectives of automation [14]. Even though some design rules have been proposed we are far from having standards defining interaction [13].

## 4 CONCLUSION AND TAKE AWAY MESSAGE

Integration of automation in the workplace started several decades ago. This paper highlights how automation has been integrated in commercial aircraft cockpits to automate part of the flight management tasks. Nowadays, there is a raising interest in AI technologies to automate additional human tasks. This questions the way the humans will interact with this new kind of automation. Should this interaction be limited as it can be interpreted by exploring a Google car (see Figure 9 a))? Or should this interaction be very complete in order to provide humans information about what the system is collecting, doing, proposing and to provide humans ways to control this intelligence, as in command and control rooms (see Figure 9 b))? The design and development of workplaces that exploit automation and AI will still require notations, techniques and tools to describe precisely the human automation collaboration. Furthermore, in workplaces using critical systems, such descriptions will be mandatory to support the certification process of these systems.
<figure><img src="image_48.png" /><figcaption>Figure 9. Cockpit of the a) Google car b) control room of the Large Hadron Collider at CERN</figcaption></figure>

ACKNOWLEDGMENTS

This work is partly funded by IKKY research project grant from Airbus.

REFERENCES

[1] Battiste V., Lachter J., Brandt S., Alvarez A., Strybel T.Z., Vu K.P.L. (2018) Human-Automation Teaming: Lessons Learned and Future Directions. In: Yamamoto S., Mori H. (eds) Human Interface and the Management of Information. Information in Applications and Services. HIMI 2018. Lecture Notes in Computer Science, vol 10905. Springer, Cham. https://doi.org/10.1007/978-3-319-92046-7_40

[2] Mary Cummings and Sylvain Bruni. (2009). Collaborative Human–Automation Decision Making. In Springer Handbook of Automation, edited by Shimon Y. Nof, 437-47. Springer Handbooks. Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-540-78831-7_26.

[3] Calvary, G., Coutaz, J., Nigay, L. From single-user architectural design to PAC*: a generic software architecture model for CSCW. In Proc. of CHI '97. ACM, NY, USA, 242-249.

[4] Ellis, C. A., Gibbs, S. J., & Rein, G. (1991). Groupware: Some issues and experiences. Communications of the ACM, 34(1), 39–58. https://doi.org/10.1145/99977.99987

[5] Michael Feary, Célia Martinie, Philippe A. Palanque, Manfred Tscheligi. Multiple Views on Safety-Critical Automation: Aircrafts, Autonomous Vehicles, Air Traffic Management and Satellite Ground Segments Perspectives. CHI Extended Abstracts 2016: 1069-1072

[6] McGrath J. E. Groups: Interaction and Performance. Prentice Hall, Inc., Englewood Cliffs, 1984.

[7] Célia Martinie, Eric Barboni, David Navarre, Philippe A. Palanque, Racim Fahssi, Erwann Poupart, Eliane Cubero-Castan. Multi-models-based engineering of collaborative systems: application to collision avoidance operations for spacecraft. EICS 2014: 85-94

[8] Célia Martinie, Philippe Palanque, Elodie Bouzekri, Andy Cockburn, Alexandre Canny, and Eric Barboni. 2019. Analysing and Demonstrating Tool-Supported Customizable Task Notations. Proc. ACM Hum.-Comput. Interact. 3, EICS, Article 12 (June 2019), 26 pages. DOI: https://doi.org/10.1145/3331154

[9] Roschelle, J., and Teasley, S. D. (1995). The construction of shared knowledge in collaborative problem solving. In C. E. O'Malley (Ed.), Computer-supported collaborative learning (pp. 69-197).

[10] Axel Schulte, and Diana Donath. (2018). A Design and Description Method for Human-Autonomy Teaming Systems. In *Intelligent Human Systems Integration*, édité par Waldemar Karwowski et Tareq Ahram, 3-9. Advances in Intelligent Systems and Computing. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-73888-8_1.

[11] Wiener, Earl L. 1989. « Human factors of advanced technology (glass cockpit) transport aircraft ». NASA

[12] R. Bernhaupt, M. Cronel, F. Manciet, C. Martinie, and P. Palanque. 2015. Transparent Automation for Assessing and Designing better Interactions between Operators and Partly-Autonomous Interactive Systems. In Proceedings of the 5th International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS '15). Association for Computing Machinery, New York, NY, USA, 129–139. DOI:https://doi.org/10.1145/2899361.2899375

[13] Philippe Palanque. 2020. Ten Objectives and Ten Rules for Designing Automations in Interaction Techniques, User Interfaces and Interactive Systems. In Proceedings of the International Conference on Advanced Visual Interfaces (AVI '20). Association for Computing Machinery, New York, NY, USA, Article 2, 1–10. DOI:https://doi.org/10.1145/3399715.3400872

[14] Philippe Palanque. 2018. Engineering Automations: From a Human Factor Perspective to Design, Implementation and Validation Challenges. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '18). Association for Computing Machinery, New York, NY, USA, Article 2, 1–2. DOI:https://doi.org/10.1145/3220134.3223044