# From Human Automation Interactions to Social Human Autonomy Machine Teaming in Maritime Transportation

Lakshmi Vuddaraju, Carine Dominguez-Péry

► To cite this version:

Lakshmi Vuddaraju, Carine Dominguez-Péry. From Human Automation Interactions to Social Human Autonomy Machine Teaming in Maritime Transportation. International Working Conference on Transfer and Diffusion of IT (TDIT), Dec 2020, Tiruchirappalli, India. pp.45-56, 10.1007/978-3-030-64861-9_5. hal-03744804

HAL Id: hal-03744804

https://inria.hal.science/hal-03744804v1

Submitted on 3 Aug 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# From Human Automation Interactions to Social Human Autonomy Machine Teaming in maritime transportation

First Author<fnref n="1" /> Carine DOMINGUEZ-PÉRY and Second Author<fnref n="2" /> Lakshmi Narasimha
Raju VUDDARAJU

**Abstract.** Recent technological advances in the field of Artificial intelligence (AI) and machine learning led to the creation of smart AI-enabled automation systems that are drastically changing maritime transportation. We developed a systematic literature review to understand how automation, based on Information Technologies (IT), has tackled the challenges related to human and machine interactions. We notably discuss the conceptual evolution from Human-Automation Interaction (HAI) to Human Autonomy Teaming (HAT) and present the risks of high levels of automation and the importance of teamwork in safety critical systems. Our results lie on a map of five clusters that highlight the importance of trust in the interactions between humans and machines, the risks related to automation, the human errors that are arising from these interactions, the effects of automation on situational awareness and the social norms in human-computer interactions. This literature show that human-machines interactions have mainly been studied from the computer/information systems' (IS) point of view, hence neglecting the social dimensions of humans. Building on the difference between the concepts of automation and autonomy, we suggest the development of the concept of Social Human Autonomy Machine Teaming (SHAMT) to better consider the social dimensions of humans in these new interactions. Future research should focus on the right equilibrium between social needs, social interactions among humans and with autonomous machines with AI to optimize the global autonomy of the human-machine teammates in a whole ecosystem.

**Keywords:** Social Human Autonomy Machine Teaming (SHAMT), Unmanned ship, Artificial intelligence, Maritime transportation, Human Automation Interactions

## 1 Introduction

Recent technological advances in the field of Artificial intelligence (AI) and machine learning led to the creation of smart AI-enabled automation systems, that are capable of processing huge amounts of data, evaluate alternatives and execute those decisions [1]. Even though the aviation industry spearheaded the introduction of these automation technologies, the maritime transportation industry is fast catching up in the implementation of these automation technologies [2] leading to increased opportunities to use Information Technologies (IT) and Information Systems (IS) related to lower risk

<footnote n="1">¹Univ. Grenoble Alpes, Grenoble INP*, CERAG, 38000 Grenoble, France
*carine.dominguez@gmail.com*</footnote>

<footnote n="2">²Univ. Grenoble Alpes, Grenoble INP*, CERAG, 38000 Grenoble, France
*vamsi.vuddaraju@gmail.com*</footnote>
accidents. The major drivers behind the increasing trend of automation are economic benefits or safety [3]. Technological innovations in AI and machine learning create a possibility of autonomous ship operations [4] leading new perspectives on human-machine interactions such as the use of big data and predictive analytics on decision-making and improved social and environmental sustainability related to maritime accidents[5]. The prospects of operationalizing unmanned ships have been gaining traction during the past few years. For instance, the EU funded MUNIN (Maritime Unmanned Navigation through Intelligence in Networks) developed the concept of the unmanned bulk carrier to carry economic, technological, and legal improvements [6]. The project of launching "Yara Birkeland" was the world's first fully autonomous electric-powered container ship that was expected to operate with complete autonomy by 2022 [7]. As of now, most of these unmanned ships are expected to be operated with a high level of supervisory control from the Shore-based Control Centers (SCC) [8]. In the future, most of these ships will be operated with high levels of IT/IS automation with limited human interventions both for operational maneuvers [6] and for supervision tasks. These evolutions drive several changes in terms of working modes between the crew and automated systems, leading to new challenges and risks. This new configuration of human on boards lead to a reorganization of tasks questioning how humans and autonomous technologies can work together in these safety-critical systems. The task of navigating the ship can be defined as teamwork as it requires coordination and interactions between members of the team [9]. The importance of teamwork and the skills required by the teammates has been highlighted in similar safety-critical environments like nuclear power plant operations [10]. Several research call for further research in order to better understand how humans and machines can act as partners in performing the maritime transportation tasks, the success of such a system depends upon the ability of the system and human operators to understand each other's state and functioning [11]. Even though current concepts have improved our understanding, we argue that too much focus has been paid to machines rather than humans for two main reasons. Firstly, there is still a poor understanding of the importance human social interactions on board, its benefits and the risks generated with the current reduction of crew members. Secondly, little research has been developed to understand how humans on ships should socially interact with IT/IS machines not only with actors on board but also more largely with their ecosystem (notably with the SCC) and hence form a team with increased autonomous machines.

To do so, we suggest to introduce the concept of Social Human Autonomy Machine Teaming (SHAMT) while answering the two following research questions:

RQ1: What are the various concepts that have tackled the interactions between humans and machines in the literature review on maritime transportation industry?

RQ2: What are the future key questions related to this field?

This paper is structured as follows: in section 2, we present a literature review on the different concepts related to the interactions between humans and machines in the maritime transportation industry. In section 3, we present the methodology used for the
systematic literature review. In section 4, we present the results structured around the co-citation analysis. Section 5 summarizes the main results, the limits and perspectives for further development.

## 2. Literature Review

### 2.1 From Human-Automation Interaction (HAI) to Human Autonomy Teaming (HAT)

The equipment on-board ships became more complex since the 1960s with technological advances and the pace of change has accelerated rapidly from the 1990s with widespread utilization of computers and other navigational technologies like GPS, increasing the trend of automation on ships [2] due to rapid technological advances and economic benefits [12].

Even though historically the term ‘autonomy’ appeared before the word ‘automation’, the literature most of the time uses these terms interchangeably [13,14]. Automation can be defined as “technology that actively selects data, transforms information, makes decisions or controls processes” [15,p.50]. Automation refers to a system that can only perform tasks for which it was programmed by the designer, without any degree of freedom to adapt itself to the changing dynamic situation at hand [13]. In this paper, we define automation as technology or system that will augment, substitute or replace functions, which human operators were carrying out before. Autonomy is a more refined and well-defined form of automation that can better adapt itself to a wide array of situations [16].

The early work on human-automation interaction was carried by Bainbridge (1983). In his seminal work “Ironies of automation”, he shed light into the impact of automation on human cognitive skill degradation and on the role of the human operator in decision making in human-computer collaboration environment [17]. More recently, some research introduced the necessity of forming a team between humans and machines with the concept of Human Autonomy Teaming (HAT), machine as teammates [11], symbiotic teamwork [14] raising questions about “turning automated systems into effective team players” [11,p.95].

### 2.2 The limits of higher Levels of Automation on HAI and related risks

The Level of Automation (LoA) will drastically change human-machine cooperation at various levels [16, 18]. There are multiple taxonomies that explain the levels of automation in complex dynamic systems [13]. These taxonomies fall somewhere between manual operation to fully autonomous systems [18]. There are two development areas in the maritime domain that can be classified as autonomous operations: self-navigating vessels and remotely operated vessels [19]. Both these systems require some sort of
human involvement, either from SCCs or human operators on board to take control if needed [8]. For that reason, a fully autonomous vessel without supervision is not going to happen in the near future [20]. Due to the advances in technology, a huge amount of information is siphoned to the human operator, which he needs to process. Hence “information isn’t the scarce resource; human time and attention are the scarce resources” [21,p.5]. The Human-Autonomy teams should act as complimentary in which they should compensate for each other’s weakness [22]. Humans have the rare ability to learn and adapt dynamically based on the tasks, environment, situations [1]. Unlike human’s, automation systems are trained with data, when the situation emerges out of the limited trained data, the automation systems cannot adopt dynamically, thus human intervention is needed when a system encounters an off-normal situation [11]. Norman et al. (1990) stated that the human errors in safety-critical industries were not due to ‘over-automation’ but rather to “inappropriate feedback and integration” [23]. Too much reliance on the automated system will induce ‘automation bias’, in which the humans will rely on automated alarms or warning systems without performing any manual checks [24].

Finally, looking at the extreme case of unmanned ships, new risks are appearing that are still understudied [25]. Some risks are related to the lack of social interactions for the reduced crew on board or for those piloting the ships: boredom and related risk of distraction that would lower situation awareness; other risks are related to skill degradation due to overreliance on machines; finally some risks are related to remote operation and monitoring such as diminished ship sense (for instance, no bodily feeling of the ship rocking), information overload (with ships equipped with multiple sensors), latency and cognitive horizon not adapted to the real conditions, mishaps during changeovers and handoffs (when a pilot is managing several ships remotely at the same time).

## 2.3 The importance of teamwork in safety critical systems

A significant portion of operational tasks in safety-critical industries ([14], aerospace or in nuclear disaster [22]) were performed by teams rather than individuals working in isolation [10]. A team can be defined as “a small number of people with complementary skills who are committed to a common purpose, set of performance goals and approach for which they hold themselves mutually accountable.” [26, p.121]. Hutchins, (1995) described navigation (navigating a vessel) as a collective cognitive process which cannot be accomplished by a single brain and that requires other people, equipment as component [27]. Usually the bridge team in a ship contains a team of individuals having a degree of shared knowledge and experience, each member of the team contributing a distinct part (plotting the course, providing coordination, operating the helm) in successfully maneuvering the vessel [9]. Since navigational decisions and actions are highly time critical, the bridge functions acts as a variant of team-based organization in which tasks and decisions are carried out interactionally [28]. Even though IT/IS
improve the performance of the automated systems; yet, humans are still vital for the
operation of automation systems for the years to come [12].

"Man-computer symbiosis" is a subset of "man-machine systems", when both the man and the machine are tightly coupled. Symbiosis can be defined as "the living together in more or less intimate association or close union of two dissimilar organisms" [31, p.4]. It is important to understand how humans and machines work together as a team, especially taking into consideration that the limits of human minds have not been taken into account and may lead to system's failure [21]. The concept of "Human-Autonomy Teams" (HAT) is then valuable because it suggests that both humans and machines may develop autonomy skills to adapt to different contexts with the ability to learn and improve performance [3].

**3. Methodology**

We used a similar 4-stage systematic review literature review model based on [31], [32].First, we created a search string ("Human-machine*" or "Man-machine*" or "Hu-man-automation" or "man-automation" or "Human-AI" or "Human-Artificial intelli-gence" or *machine* or *automation* or AI or *Artificial intelligence* or "Autono-mous*) and ("*collaboration" or "*symbiosis" or "*team*" or "interaction") with the relevant keywords like ‘Human-Autonomy Interaction’, ‘Human-Autonomy collabo-ration’. We used the Web of Science-core collection, due to its extensive and well cu-rated collection of articles from multiple domains. For the initial search we got around n=79,191 results. Since our primary motive was understanding Human factors and other ergonomic issues related to the Human and autonomy teaming, we narrowed our research [32] to the Ergonomics domain in Web of Science (WoS) that published most articles related to maritime transportation, knowing that some papers could be included in other categories (for instance Business for research that include cost-arguments in relation to maritime safety) Reading the abstracts, we excluded all the articles that were unrelated to the topic leading to n=789 results.. As the topic of HAT is multidisciplinary in nature, similar to [33], we included all the articles found without considering the journal or discipline they belong to.

In step 2 we downloaded all the records from the WoS portal to conduct a bibliographic
analysis and performed a co-citation analysis (CCA) in the VosViewer software, as co-
citation analysis provides the theoretical background for the intended topic [32]. "Co-
citation is defined as the frequency with which two documents are cited together" [34,
p.269].

Before using the records for creating CCA analysis it is important to ensure that the
documents included for conducting the analysis are relevant to our study and match the
criteria. First, we looked for any duplicate records in the filtered n=789 documents and
we found two duplicate records and removed them. Second, we looked for false posi-
tives, for example the articles might include keywords mentioned in the search string
in their titles, abstract and keywords but were not related to the domain [32]. To do so, we checked the abstracts and introductions of the articles to exclude the articles that were relevant to the study following a protocol that determines the inclusion and exclusion criteria [35]. We found several false positives that we removed from the records: for instance Davis, (1989) user acceptance model [36]; Mayer et al., (1995) model for organizational trust [37] and some articles related to psychology (Hockey, (1997) [38] or to human performance under stress and workload (Wickens & Hollands, 2000) [39]engineering psychology and human performance (Rempel et al., 1985) [40]; finally, trust in human relations and some books in behavioral economics (Kahneman, 1973) [41].

Finally, we created the CCA and adjusted in VosViewer the hyperparameters to 10 minimum citations. 144 documents met the criteria gathered into 5 clusters that are described in details in the following section 4.

## 4. Results

Fig 1. shows the bibliographic map of five clusters based on CCA. We made the interpretation of these clusters based on the top-five cited articles.

<figure><img src="image_2.png" /><figcaption>Fig. 1. CCA based on the keywords in VosViewer</figcaption></figure>
## 4.1 Cluster 1: The influence of trust in human-automation interaction

Cluster 1 is the largest with 40 items. It primarily deals with the influence of trust in Human-automation interaction; the trust in automation will determine the use or misuse of automation [15], [42], [43]. Too much or little reliance on the system, would result in automation and complacency bias [44], [45]. This cluster is closed to Cluster 2, as trust in a system with higher level of automation affects the skill of human operators and may result in out of the loop performance [46] Cluster 1 is also closed to the Cluster 3 as there are close associations between trust in automation and potential human errors. This Cluster is located at the center, with linkages to all the other clusters, showing the central role of trust in HAI.

## 4.2 Cluster 2: The effects of Automation levels and reliability on human performance

Cluster 2 has 39 items, most of them presenting classification of automation levels and related taxonomies [47], [48]. Parasuraman, (2000) classified the automation levels based on four information processing levels [49]. This cluster also includes some papers that emphasize on the importance of functional allocation, describing the functional superiority of humans or machines in performing various tasks [50], [51]. Other papers discuss the issue of complacency at higher automation levels and its effects on performance under different workload conditions [52], [53]. This cluster is in closed to Cluster 1 and Cluster 4 highlighting the effects of higher levels of automation on [54].

## 4.3 Cluster 3: The human errors arising from man-machine interactions

Cluster 3 has 33 items that mostly deal with human errors that may arise with automation. Human errors are mainly explained according to two different perspectives: firstly, from the cognitive engineering [55], [56] and secondly, from the human factors' perspective [23], [57]–[60]. Finally, the research of [27], [61] develop the interactions between human and technology in their natural settings. This cluster is in proximity with Cluster 1 and Cluster 4 justifying the links between human performance, trust and situation awareness

## 4.4 Cluster 4: The effects of automation on situational awareness

Cluster 4 has 23 items that deals with increased automation and its effect on situational awareness [17], [62]. When most of the tasks are automated, the human operator is generally restricted to monitoring, a task for which he is not well suited [63]. Cluster 4 is situated at the top of the overall map with links with all other clusters showing that the concept of situational awareness has been central in research related to human
errors, levels of automation and performance. It is noticeable that Cluster 4 is the only cluster with connections to Cluster 5, hence considering the social dimension of the situational awareness, notably in case of decision-making within groups.

## 4.5 Cluster 5: The social norms in Human-Computer Interactions

Cluster is the less developed with only 9 items. It is situated at the top right part of the overall map, isolated and widespread compared to the other four clusters. This cluster primarily deals about the perceptions of humans about their machine counterparts, the perceptions of humans on computers as teammates; humans' perceptions about the quality of information provided by computers and finally how humans apply stereotypes of gender to computers [64], [65]. This cluster has strong linkages to Cluster 1 as trust is one of the most important social norms.

The social dimensions related to HAI are still understudied and mainly developed around the perceptions of computers or on their interactions with humans, while neglecting the social human part. Both looking at the literature review and based on the conclusion of the CCA, we call in the next section for further research based on the new concept of “Social Human Autonomy Machine Teaming” (SHAMT).

# 5. Conclusion: an agenda for research based on the Social Human Autonomy Machine Teaming (SHAMT) concept

The outcome of the literature review provides a solid foundation on which to build an agenda for future research. Here, we propose the concept of Social Human Autonomy Machine Teaming (SHAMT), which we define as a team consisting of both humans and machines, working together in a complementary way to optimize decision-making, while incorporating the social dimension of humans within the whole ecosystem. SHAMT will focus researchers' attention on the automation and the social within a sectoral ecosystem.

To date, automation has been mostly considered as a way to fulfill prescribed tasks, with minimal or no human interventions [16] with the ultimate objective of minimizing costs for the ship owner and other stakeholders regardless of the potential economic, human and ecological damage. Advances in automation technology, has led to smart machines that can potentially work with human teams [30], but the literature tends to highlight how increased autonomy of the machine can lead to changing human capabilities and skills in the workplace. For instance, reduced skills, less attention and memory span leading to reduced situational awareness or over reliance on the machine. There is a significant and widening gap in the understanding of how human and autonomous machines can interact with each other and effectively work as part of a team [14]. Humans are better at tasks requiring judgement, whereas machines are better at performing routine and repetitive tasks [2], [10] and if these human and machine skills
are combined, they can potentially form a smarter entity, a “Supermind” [66]. If humans combine their unique ‘general intelligence’ (the capability to achieve different goals effectively in diverse environments) with machine’s ‘specialized intelligence’ and capabilities that humans don’t possess, this new combination can mean improved decision-making in certain stress and emergency situations.

Future research should also take into consideration the social aspect from two perspectives. Firstly, the reduction of crew members, as well as the increased number of situations where humans are monitoring or piloting ships alone on board or remotely, raise questions about situations of loneliness. For instance, with different temporalities – such as being alone at the sea in an unmanned ship for long periods of time, or even monitoring several ships at home for months during pandemic or other lockdown periods) - will human behaviors and interactions with machines change? There is still limited research that considers navigation within a whole ecosystem of actors (the ship plus all the information arriving from the outside, notably the SCC) neither does it take into account how these social interactions, both on the ship and with remote actors, may change the way humans acquire and interpret information and how they make decisions.

One of the limitation of this study is the breadth of our search which is based on the Ergonomics domain in Web of Science (WoS) [32]. Broadening the search to include other domains, might bring to light studies that might have not been published in our selected database. Other scholars could develop a bibliographic coupling analysis (BCA) in order to highlight future research trends to cross it with the SHAMT concept.

All in all, our study recommends that future research should focus on all the elements of a maritime ecosystem to optimize the effectiveness of autonomous human-machine teammates incorporating environmental, social and human needs for interactions with autonomous machines and AI. The ideas developed in this research may also be useful in other transportation industries such as aviation [16] space exploration, SAR (Search and Rescue) operations, UAV (Unmanned Aerial Vehicles) UGV(Unmanned Ground Vehicles) enabling both researchers and practitioners to perform tasks in extreme environments and thus improve disaster management without putting human operators at risk [67].

## References

[1] Y. K. Dwivedi et al., "Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy," *Int. J. Inf. Manag.*, p. 101994, Aug. 2019, doi: 10.1016/j.ijinfomgt.2019.08.002.

[2] M. R. Grech, T. Horberry, and A. Smith, "Human Error in Maritime Operations: Analyses of Accident Reports Using the Leximancer Tool," *Proc. Hum. Factors Ergon. Soc. Annu. Meet.*, vol. 46, no. 19, pp. 1718-1721, Sep. 2002, doi: 10.1177/154193120204601906.
[3] K. Pazouki, N. Forbes, R. A. Norman, and M. D. Woodward, "Investigation on the impact of human-automation interaction in maritime operations," *Ocean Eng.*, vol. 153, pp. 297-304, Apr. 2018, doi: 10.1016/j.oceaneng.2018.01.103.

[4] B.-M. Batalden, P. Leikanger, and P. Wide, "Towards autonomous maritime operations," in *2017 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA)*, Annecy, France, Jun. 2017, pp. 1-6, doi: 10.1109/CIVEMSA.2017.7995339.

[5] R. Dubey et al., "Can big data and predictive analytics improve social and environmental sustainability?," *Technol. Forecast. Soc. Change*, vol. 144, pp. 534-545, Jul. 2019, doi: 10.1016/j.techfore.2017.06.020.

[6] H.-C. Burmeister, W. Bruhn, Ø. J. Rødseth, and T. Porathe, "Autonomous Unmanned Merchant Vessel and its Contribution towards the e-Navigation Implementation: The MUNIN Perspective," *Int. J. E-Navig. Marit. Econ.*, vol. 1, pp. 1-13, Dec. 2014, doi: 10.1016/j.enavi.2014.12.002.

[7] Yara Birkeland, "Autonomous ship project, key facts about YARA Birkeland," 2020. https://www.kongsberg.com/maritime/support.themes/autonomous-ship-project-key-facts-about-yara-birkeland/ (accessed Jul. 31, 2020).

[8] I. B. Utne, B. Rokseth, A. J. Sørensen, and J. E. Vinnem, "Towards supervisory risk control of autonomous ships," *Reliab. Eng. Syst. Saf.*, vol. 196, p. 106757, Apr. 2020, doi: 10.1016/j.ress.2019.106757.

[9] N. Bailey, W. Housley, and P. Belcher, "Navigation, interaction and bridge team work," *Sociol. Rev.*, vol. 54, no. 2, pp. 342-362, 2006, doi: 10.1111/j.1467-954X.2006.00617.x.

[10] P. O'Connor, A. O'Dea, R. Flin, and S. Belton, "Identifying the team skills required by nuclear power plant operations personnel," *Int. J. Ind. Ergon.*, vol. 38, no. 11-12, pp. 1028-1037, Nov. 2008, doi: 10.1016/j.ergon.2008.01.014.

[11] C. P. Janssen, S. F. Donker, D. P. Brumby, and A. L. Kun, "History and future of human-automation interaction," *Int. J. Hum.-Comput. Stud.*, vol. 131, pp. 99-107, Nov. 2019, doi: 10.1016/j.ijhcs.2019.05.006.

[12] R. Parasuraman and C. D. Wickens, "Humans: Still Vital After All These Years of Automation," *Hum. Factors*, p. 10, 2008.

[13] M. Vagia, A. A. Transeth, and S. A. Fjerdingen, "A literature review on the levels of automation during the years. What are the different taxonomies that have been proposed?," *Appl. Ergon.*, vol. 53, pp. 190-202, Mar. 2016, doi: 10.1016/j.apergo.2015.09.013.

[14] M. D. McNeese and N. J. McNeese, "Humans interacting with intelligent machines: at the crossroads of symbiotic teamwork," in *Living with Robots*, Elsevier, 2020, pp. 165-197.

[15] J. D. Lee and N. Moray, "Trust, self-confidence, and operators' adaptation to automation," *Int. J. Hum.-Comput. Stud.*, vol. 40, no. 1, pp. 153-184, Jan. 1994, doi: 10.1006/ijhc.1994.1007.

[16] M. R. Endsley, "From Here to Autonomy: Lessons Learned From Human-Automation Research," *Hum. Factors J. Hum. Factors Ergon. Soc.*, vol. 59, no. 1, pp. 5-27, Feb. 2017, doi: 10.1177/0018720816681350.

[17] L. Bainbridge, "Ironies of automation," *Automatica*, vol. 19, no. 6, pp. 775-779, Nov. 1983, doi: 10.1016/0005-1098(83)90046-8.

[18] D. B. Kaber, E. Onal, and M. R. Endsley, "Design of automation for telerobots and the effect on performance, operator situation awareness, and subjective workload," *Hum. Factors Ergon. Manuf. Serv. Ind.*, vol. 10, no. 4, pp. 409-430, 2000, doi: 10.1002/1520-6564(200023)10:4<409::AID-HFM4>3.0.CO;2-V.
[19] T. Relling, M. Lützhöft, R. Ostnes, and H. P. Hildre, "A Human Perspective on Maritime Autonomy," in *Augmented Cognition: Users and Contexts*, Cham, 2018, pp. 350–362, doi: 10.1007/978-3-319-91467-1_27.

[20] M. A. Ramos, C. A. Thieme, I. B. Utne, and A. Mosleh, "Human-system concurrent task analysis for maritime autonomous surface ship operation and safety," *Reliab. Eng. Syst. Saf.*, vol. 195, p. 106697, Mar. 2020, doi: 10.1016/j.ress.2019.106697.

[21] H. A. Simon, "The future of information systems," *Ann. Oper. Res.*, vol. 71, no. 0, pp. 3–14, Jan. 1997, doi: 10.1023/A:1018975616482.

[22] K. T. Wynne and J. B. Lyons, "An integrative model of autonomous agent teammate-likeness," *Theor. Issues Ergon. Sci.*, vol. 19, no. 3, pp. 353–374, May 2018, doi: 10.1080/1463922X.2016.1260181.

[23] D. A. Norman, D. E. Broadbent, A. D. Baddeley, and J. Reason, "The 'problem' with automation: inappropriate feedback and interaction, not 'over-automation,'" *Philos. Trans. R. Soc. Lond. B Biol. Sci.*, vol. 327, no. 1241, pp. 585–593, Apr. 1990, doi: 10.1098/rstb.1990.0101.

[24] M. H. Lützhöft and S. W. A. Dekker, "On Your Watch: Automation on the Bridge," *J. Navig.*, vol. 55, no. 1, pp. 83–96, Jan. 2002, doi: 10.1017/S0373463301001588.

[25] R. Jalonen, R. Tuominen, and M. Wahlström, *Safety of Unmanned Ships - Safe Shipping with Autonomous and Remote Controlled Ships*. Aalto University, 2017.

[26] N. B. Sarter and D. D. Woods, "Team Play with a Powerful and Independent Agent: Operational Experiences and Automation Surprises on the Airbus A-320;," *Hum. Factors*, 1997, doi: 10.1518/001872097778667997.

[27] E. Hutchins, *Cognition in the Wild*. MIT Press, 1995.

[28] W. Housley, *Interaction in Multidisciplinary Teams*. 2003.

[29] J. C. R. Licklider, "Man-Computer Symbiosis," *IRE Trans. Hum. Factors Electron.*, vol. HFE-1, no. 1, pp. 4–11, Mar. 1960, doi: 10.1109/THFE2.1960.4503259.

[30] N. J. McNeese, M. Demir, N. J. Cooke, and C. Myers, "Teaming With a Synthetic Teammate: Insights into Human-Autonomy Teaming;," *Hum. Factors*, Nov. 2017, doi: 10.1177/0018720817743223.

[31] A. Kovács, B. Van Looy, and B. Cassiman, "Exploring the scope of open innovation: a bibliometric review of a decade of research," *Scientometrics*, vol. 104, no. 3, pp. 951–983, Sep. 2015, doi: 10.1007/s11192-015-1628-0.

[32] J. A. W. H. van Oorschot, E. Hofman, and J. I. M. Halman, "A bibliometric review of the innovation adoption literature," *Technol. Forecast. Soc. Change*, vol. 134, pp. 1–21, Sep. 2018, doi: 10.1016/j.techfore.2018.04.032.

[33] K. Guirguis, "From Big Data to Big Performance – Exploring the Potential of Big Data for Enhancing Public Organizations' Performance. A Systematic Literature Review.," *Swiss Yearb. Adm. Sci.*, vol. 11, no. 1, pp. 55–65, Mar. 2020, doi: 10.5334/ssas.140.

[34] H. Small, "Co-citation in the scientific literature: A new measure of the relationship between two documents," *J. Am. Soc. Inf. Sci.*, vol. 24, no. 4, pp. 265–269, 1973, doi: 10.1002/asi.4630240406.

[35] P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, "Lessons from applying the systematic literature review process within the software engineering domain," *J. Syst. Softw.*, vol. 80, no. 4, pp. 571–583, Apr. 2007, doi: 10.1016/j.jss.2006.07.009.

[36] F. D. Davis, "Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology," *MIS Q*, 1989, doi: 10.2307/249008.
[37] R. C. Mayer, J. H. Davis, and F. D. Schoorman, "An Integrative Model of Organizational Trust," *Acad. Manage. Rev.*, vol. 20, no. 3, pp. 709-734, 1995, doi: 10.2307/258792.

[38] G. Robert J. Hockey, "Compensatory control in the regulation of human performance under stress and high workload: A cognitive-energetical framework," *Biol. Psychol.*, vol. 45, no. 1, pp. 73-93, Mar. 1997, doi: 10.1016/S0301-0511(96)05223-4.

[39] C. D. Wickens and J. G. Hollands, *Engineering Psychology and Human Performance*. Prentice Hall, 2000.

[40] J. K. Rempel, J. G. Holmes, and M. P. Zanna, "Trust in close relationships," *J. Pers. Soc. Psychol.*, vol. 49, no. 1, pp. 95-112, 1985, doi: 10.1037/0022-3514.49.1.95.

[41] D. Kahneman, *Attention and effort*. Englewood Cliffs, N.J.: Prentice-Hall, 1973.

[42] S. Lewandowsky, M. Mundy, and G. P. Tan, "The dynamics of trust: comparing humans to automation.," *J. Exp. Psychol. Appl.*, 2000, doi: 10.1037/1076-898X.6.2.104.

[43] B. M. Muir, "Trust between humans and machines, and the design of decision aids," *Int. J. Man-Mach. Stud.*, vol. 27, no. 5, pp. 527-539, Nov. 1987, doi: 10.1016/S0020-7373(87)80013-5.

[44] J. D. Lee and K. A. See, "Trust in Automation: Designing for Appropriate Reliance," *Hum. Factors*, p. 31, 2004.

[45] M. T. Dzindolet, S. A. Peterson, R. A. Pomranky, L. G. Pierce, and H. P. Beck, "The role of trust in automation reliance," *Int. J. Hum.-Comput. Stud.*, vol. 58, no. 6, pp. 697-718, Jun. 2003, doi: 10.1016/S1071-5819(03)00038-7.

[46] M. R. Endsley and E. O. Kiris, "The Out-of-the-Loop Performance Problem and Level of Control in Automation," *Hum. Factors J. Hum. Factors Ergon. Soc.*, vol. 37, no. 2, pp. 381-394, Jun. 1995, doi: 10.1518/001872095779064555.

[47] T. B. Sheridan, *Telerobotics, automation, and human supervisory control*. Cambridge, MA, US: The MIT Press, 1992, pp. xx, 393.

[48] M. R. Endsley, "Level of automation effects on performance, situation awareness and workload in a dynamic control task," *Ergonomics*, vol. 42, no. 3, pp. 462-492, Mar. 1999, doi: 10.1080/001401399185595.

[49] R. Parasuraman, "Designing automation for human use: empirical studies and quantitative models," *Ergonomics*, vol. 43, no. 7, pp. 931-951, Jul. 2000, doi: 10.1080/001401300409125.

[50] S. W. A. Dekker and D. D. Woods, "MABA-MABA or Abracadabra? Progress on Human-Automa-tion Co-ordination," *Cogn. Technol. Work*, vol. 4, no. 4, pp. 240-244, Nov. 2002, doi: 10.1007/s101110200022.

[51] P. M. Fitts, *Human engineering for an effective air-navigation and traffic-control system*. National Research Council, 1951.

[52] R. Parasuraman and D. H. Manzey, "Complacency and Bias in Human Use of Automation: An At-tentional Integration:," *Hum. Factors*, Oct. 2010, doi: 10.1177/0018720810376055.

[53] D. B. Kaber and M. R. Endsley, "The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task," *Theor. Issues Ergon. Sci.*, vol. 5, no. 2, pp. 113-153, Mar. 2004, doi: 10.1080/1463922021000054335.

[54] L. Onnasch, C. D. Wickens, H. Li, and D. Manzey, "Human Performance Consequences of Stages and Levels of Automation: An Integrated Meta-Analysis," *Hum. Factors*, Aug. 2013, doi: 10.1177/0018720813501549.

[55] E. Hollnagel and D. D. Woods, "Cognitive Systems Engineering: New wine in new bottles," *Int. J. Man-Mach. Stud.*, vol. 18, no. 6, pp. 583-600, Jun. 1983, doi: 10.1016/S0020-7373(83)80034-0.

[56] E. Hollnagel, *Human Reliability Analysis: Context and Control*. Academic Press, 1993.
[57] J. Reason, *Human Error*. Cambridge University Press, 1990.

[58] J. Rasmussen, "Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models," *IEEE Trans. Syst. Man Cybern.*, vol. SMC-13, no. 3, pp. 257-266, May 1983, doi: 10.1109/TSMC.1983.6313160.

[59] J. Rasmussen, *Information processing and human-machine interaction: an approach to cognitive engineering*. North-Holland, 1986.

[60] K. J. Vicente and J. Rasmussen, "The Ecology of Human-Machine Systems II: Mediating 'Direct Perception' in Complex Work Domains," *Ecol. Psychol.*, vol. 2, no. 3, pp. 207-249, Sep. 1990, doi: 10.1207/s15326969eco0203_2.

[61] E. Hutchins, "How a Cockpit Remembers Its Speeds," *Cogn. Sci.*, vol. 19, no. 3, pp. 265-288, 1995, doi: 10.1207/s15516709cog1903_1.

[62] M. R. Endsley, "Measurement of Situation Awareness in Dynamic Systems," *Hum. Factors J. Hum. Factors Ergon. Soc.*, vol. 37, no. 1, pp. 65-84, Mar. 1995, doi: 10.1518/001872095779049499.

[63] C. D. Wickens, "Situation Awareness: Review of Mica Endsley's 1995 Articles on Situation Awareness Theory and Measurement;," *Hum. Factors*, Jun. 2008, doi: 10.1518/001872008X288420.

[64] C. Nass and Y. Moon, "Machines and Mindlessness: Social Responses to Computers," *J. Soc. Issues*, vol. 56, no. 1, pp. 81-103, 2000, doi: 10.1111/0022-4537.00153.

[65] C. Nass, B. J. Fogg, and Y. Moon, "Can computers be teammates?," *Int. J. Hum.-Comput. Stud.*, vol. 45, no. 6, pp. 669-678, Dec. 1996, doi: 10.1006/ijhc.1996.0073.

[66] T. W. Malone, *Superminds: The Surprising Power of People and Computers Thinking Together*. Little, Brown, 2018.

[67] M. S. Prewett, R. C. Johnson, K. N. Saboe, L. R. Elliott, and M. D. Coovert, "Managing workload in human-robot interaction: A review of empirical studies," *Comput. Hum. Behav.*, vol. 26, no. 5, pp. 840-856, Sep. 2010, doi: 10.1016/j.chb.2010.03.010.