<figure><img src="image_1.png" /></figure>

HAL
open science

# Agent's Cooperation Levels to Enhance Human-Robot Teaming

Sridath Tula, Marie-Pierre Pacaux-Lemoine, Emmanuelle Grislin-Le Strugeon, Paulo Eduardo Santos, Anna Ma-Wyatt, Jean-Philippe Diguet

<figure><img src="image_2.png" /></figure>

## To cite this version:

Sridath Tula, Marie-Pierre Pacaux-Lemoine, Emmanuelle Grislin-Le Strugeon, Paulo Eduardo Santos, Anna Ma-Wyatt, et al.. Agent's Cooperation Levels to Enhance Human-Robot Teaming. Workshop ARMS (Autonomous Robots and Multirobot Systems), 23rd Inter. Conference on Autonomous Agents and Multiagent Systems (AAMAS'24), May 2024, Auckland, New Zealand. hal-04571571

HAL Id: hal-04571571

https://hal.science/hal-04571571

Submitted on 8 May 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Agent's Cooperation Levels to Enhance Human-Robot Teaming

Sridath Tula, Marie-Pierre Pacaux-Lemoine, Emmanuelle Grislin-Le Strugeon, Paulo Eduardo Santos, Anna Ma-Wyatt, Jean-Philippe Diguet

► To cite this version:

Sridath Tula, Marie-Pierre Pacaux-Lemoine, Emmanuelle Grislin-Le Strugeon, Paulo Eduardo Santos, Anna Ma-Wyatt, et al.. Agent's Cooperation Levels to Enhance Human-Robot Teaming. Workshop ARMS (Autonomous Robots and Multirobot Systems), 23rd Inter. Conference on Autonomous Agents and Multiagent Systems (AAMAS'24), May 2024, Auckland, New Zealand. hal-04571571

HAL Id: hal-04571571

https://hal.science/hal-04571571v1

Submitted on 8 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Agent's Cooperation Levels to Enhance Human-Robot Teaming

Sridath Tula

UPHF, CNRS, UMR 8201-LAMIH,
CNRS, IRL 2010 CROSSING
Valenciennes, France
sridath.tula@uphf.fr

Marie-Pierre Pacaux-Lemoine

UPHF, CNRS, UMR 8201-LAMIH
Valenciennes, France
marie-pierre.lemoine@uphf.fr

Emmanuelle

Grislin-Le Strugeon

INSA HdF, CNRS, UMR 8201-LAMIH
Valenciennes, France
emanuelle.grislin@uphf.fr

Paulo Eduardo Santos

Flinders University,
CNRS, IRL 2010 CROSSING
Adelaide, Australia

paulo.santos@flinders.edu.au

Anna Ma-Wyatt

The University of Adelaide,
CNRS, IRL 2010 CROSSING
Adelaide, Australia

anna.mawyatt@adelaide.edu.au

Jean-Philippe Diguet

CNRS, IRL 2010 CROSSING
Adelaide, Australia

jean-philippe.diguet@cnrs.fr

## ABSTRACT

Effective cooperation between human operators and robots is essential for the accomplishment of crucial tasks in tough and dangerous contexts, such as search and rescue operations. While autonomous robots excel at moving through dangerous conditions, human decisions made on the basis of global knowledge and experience, are still required to adapt to unforeseen events. We used a Human-Machine Cooperation model to address this topic. The model proposes the definitions to analyze the Know-How-to-Operate and Know-How-to-Cooperate competencies of the teaming agents, in relation to the different phases of a shared task. On this basis, we designed four cooperation levels for the team with the addition of an intelligent assistant system (IAS) agent. As a proof-of-concept, these levels have been implemented for evaluation purposes in a team composed of one human, one IAS, and two robots. In our study we demonstrated the practical implementation of the proposed cooperation levels within a human-robot team, emphasizing the Intelligent Assistance System (IAS) agent's ability to improve cooperation and task efficiency in hazardous environments.

## KEYWORDS

Agent, Cooperation, Human-Robot Cooperation, Intelligent Assistance System (IAS), Know-how-to-operate (KHO), Know-how-to-cooperate (KHC)

### ACM Reference Format:

Sridath Tula, Marie-Pierre Pacaux-Lemoine, Emmanuelle Grislin-Le Strugeon, Paulo Eduardo Santos, Anna Ma-Wyatt, and Jean-Philippe Diguet. 2024. Agent's Cooperation Levels to Enhance Human-Robot Teaming. In *Proc. of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024)*, Auckland, New Zealand, May 6 – 10, 2024, IFAAMAS, 9 pages.

## 1 INTRODUCTION

Human operators and robots must work together effectively, especially in crucial circumstances such as search and rescue [35] and disaster response[5]. For example, in search and rescue efforts after a natural catastrophe, time is critical, and effective cooperation

between humans and robots may mean the difference between life and death for those in need. Similarly, in disaster response activities, such as firefighting [35] in hazardous situations, cooperation among human firefighters and robotic systems equipped with specific sensors and equipment may considerably improve operational performance and safety.

The synergy between humans and robots in hazardous settings is explored against the backdrop of existing research in the field. Projects like the DARPA Robotics Challenge (DRC) [30] and the Robotic Firefighter (RFF) [11] have significantly contributed to advancing the capabilities of robots in disaster response scenarios. Despite the advantages of fully autonomous robots [31], they usually struggle to handle complex tasks, such as navigating uncertain environments, which frequently require human intervention for accurate and efficient completion. In such cases, agents may request human intervention when the environmental circumstances or unanticipated difficulties exceed the agent's capabilities, when they are not aware of the current global situation, or if it is the case where the agents did not receive enough training to handle unexpected events. The issues posed by hazardous situations require a fundamental change in the way humans and robots cooperate [20].

Our goal is thus to improve cooperation between human operators and robots, especially in situations when seamless coordination is essential. The idea is to elevate cooperation between human operators and robots in situations where effective teamwork is not just advantageous but pivotal.

With this aim in mind, we examined both the cooperation and the autonomy aspects of human-robot teaming. In addition to Human-Robot Interaction studies, the Human-Machine Systems field provides models that can offer valuable insights to human-robot cooperation, such as the model in [27] that proposes analysis grids based on the Humans' and Machines' skills, regarding not only the task performance but also the interactions inside the team. The notion of levels of autonomy is also used in different fields, including multi-agent systems. These levels are intended to define the degree of independence and decision-making ability of agents or systems [23]. Nevertheless, there is a recognized gap in seamlessly merging these models with multi-agent principles to achieve a comprehensive Human-Robot Cooperation framework.

<footnote>Proc. of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024), N. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.), May 6 – 10, 2024, Auckland, New Zealand. © 2024 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). This work is licenced under the Creative Commons Attribution 4.0 International (CC-BY 4.0) licence.</footnote>
# Agent's Cooperation Levels to Enhance Human-Robot Teaming

Sridath Tula

UPHF, CNRS, UMR 8201-LAMIH,
CNRS, IRL 2010 CROSSING
Valenciennes, France
sridath.tula@uphf.fr

Marie-Pierre Pacaux-Lemoine

UPHF, CNRS, UMR 8201-LAMIH
Valenciennes, France
marie-pierre.lemoine@uphf.fr

Emmanuelle

Grislin-Le Strugeon

INSA HdF, CNRS, UMR 8201-LAMIH
Valenciennes, France
emanuelle.grislin@uphf.fr

Paulo Eduardo Santos

Flinders University,
CNRS, IRL 2010 CROSSING
Adelaide, Australia

paulo.santos@flinders.edu.au

Anna Ma-Wyatt

The University of Adelaide,
CNRS, IRL 2010 CROSSING
Adelaide, Australia

anna.mawyatt@adelaide.edu.au

Jean-Philippe Diguet

CNRS, IRL 2010 CROSSING
Adelaide, Australia

jean-philippe.diguet@cnrs.fr

## ABSTRACT

Effective cooperation between human operators and robots is essential for the accomplishment of crucial tasks in tough and dangerous contexts, such as search and rescue operations. While autonomous robots excel at moving through dangerous conditions, human decisions made on the basis of global knowledge and experience, are still required to adapt to unforeseen events. We used a Human-Machine Cooperation model to address this topic. The model proposes the definitions to analyze the Know-How-to-Operate and Know-How-to-Cooperate competencies of the teaming agents, in relation to the different phases of a shared task. On this basis, we designed four cooperation levels for the team with the addition of an intelligent assistant system (IAS) agent. As a proof-of-concept, these levels have been implemented for evaluation purposes in a team composed of one human, one IAS, and two robots. In our study we demonstrated the practical implementation of the proposed cooperation levels within a human-robot team, emphasizing the Intelligent Assistance System (IAS) agent's ability to improve cooperation and task efficiency in hazardous environments.

## KEYWORDS

Agent, Cooperation, Human-Robot Cooperation, Intelligent Assistance System (IAS), Know-how-to-operate (KHO), Know-how-to-cooperate(KHC)

### ACM Reference Format:

Sridath Tula, Marie-Pierre Pacaux-Lemoine, Emmanuelle Grislin-Le Strugeon, Paulo Eduardo Santos, Anna Ma-Wyatt, and Jean-Philippe Diguet. 2024. Agent's Cooperation Levels to Enhance Human-Robot Teaming. In *Proc. of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024)*, Auckland, New Zealand, May 6 – 10, 2024, IFAAMAS, 9 pages.

## 1 INTRODUCTION

Human operators and robots must work together effectively, especially in crucial circumstances such as search and rescue [35] and disaster response[5]. For example, in search and rescue efforts after a natural catastrophe, time is critical, and effective cooperation

between humans and robots may mean the difference between life and death for those in need. Similarly, in disaster response activities, such as firefighting [35] in hazardous situations, cooperation among human firefighters and robotic systems equipped with specific sensors and equipment may considerably improve operational performance and safety.

The synergy between humans and robots in hazardous settings is explored against the backdrop of existing research in the field. Projects like the DARPA Robotics Challenge (DRC) [30] and the Robotic Firefighter (RFF) [11] have significantly contributed to advancing the capabilities of robots in disaster response scenarios. Despite the advantages of fully autonomous robots [31], they usually struggle to handle complex tasks, such as navigating uncertain environments, which frequently require human intervention for accurate and efficient completion. In such cases, agents may request human intervention when the environmental circumstances or unanticipated difficulties exceed the agent's capabilities, when they are not aware of the current global situation, or if it is the case where the agents did not receive enough training to handle unexpected events. The issues posed by hazardous situations require a fundamental change in the way humans and robots cooperate [20].

Our goal is thus to improve cooperation between human operators and robots, especially in situations when seamless coordination is essential. The idea is to elevate cooperation between human operators and robots in situations where effective teamwork is not just advantageous but pivotal.

With this aim in mind, we examined both the cooperation and the autonomy aspects of human-robot teaming. In addition to Human-Robot Interaction studies, the Human-Machine Systems field provides models that can offer valuable insights to human-robot cooperation, such as the model in [27] that proposes analysis grids based on the Humans' and Machines' skills, regarding not only the task performance but also the interactions inside the team. The notion of levels of autonomy is also used in different fields, including multi-agent systems. These levels are intended to define the degree of independence and decision-making ability of agents or systems [23]. Nevertheless, there is a recognized gap in seamlessly merging these models with multi-agent principles to achieve a comprehensive Human-Robot Cooperation framework.

<footnote>Proc. of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024), N. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.), May 6 – 10, 2024, Auckland, New Zealand. © 2024 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). This work is licenced under the Creative Commons Attribution 4.0 International (CC-BY 4.0) licence.</footnote>
After positioning the problem regarding the related literature in Section 2, we present the proposed model made of four cooperation levels in Section 3, and its application to a team composed of one human, one assistant system and two robots in Section 4. The planned experiments and evaluation metrics are described in Section 5, before the conclusion and future works.

## 2 PROBLEM STATEMENT AND LITERATURE GAP

In the area of difficult and hazardous circumstances, maximizing co-operation among human operators and autonomous robots emerges as a significant task. While autonomous robots excel at moving through dangerous conditions, human decisions made on the basis of global knowledge and experience are still required to adapt to unforeseen events. Current approaches focus heavily on fully self-governing robots [19], sometimes missing the distinct advantages that human operators bring to the table.

In the field of human-robot interaction (HRI) and Human-robot teaming, successful collaboration between human operators and autonomous robots is critical for attaining common goals [28] in challenging circumstances. HRI involves numerous aspects of interaction, such as communication, coordination [21], and collaboration, to enable humans and robots to work together. Human-robot teaming combines human experience with autonomous robots to improve work performance and adaptability [7].

In addition, since our context includes interacting entities with autonomy-related characteristics, we considered the approaches provided by the Multi-Agent System (MAS) field. Indeed, MAS proposes systematic techniques for defining and implementing artificial agents' competencies and skills [36], allowing them to complete individual and cooperative tasks, in some cases using predefined or adaptive organizations [13]. MAS approaches also make it easier to integrate agents' learning capabilities, with the help of models and algorithms common to the robotics field. In MAS literature, the term "agent" refers to both human and artificial beings, such as robots, virtual agents, and AI-based systems, with distinctions drawn between the agent's embodiment and its software components. In the agent systems where organizations are modeled (at the difference from emerging organization, as in swarm for instance [6, 29]), the agents have roles in the group and follow rules that define the way they interact. Organizational structures within multi-agent systems vary from completely decentralized to a variety of hierarchical structures [3, 12], including mixed structures like the holonic organizations [1, 2]. In these systems, the autonomy of robot agents primarily relates to motion planning and navigation, with higher-level decisions involving task scheduling or conflict resolution. Additionally, agents in MAS often exhibit various behaviors and communication abilities, further enhancing their cooperative capabilities.

Despite the advancement of current human-robot interaction approaches, multi-agent systems have certain limitations as mentioned below. These approaches frequently rely on completely autonomous robotic systems, ignoring the particular problem-solving talents and situational awareness of human operators. To address these constraints and improve human-robot collaboration in complex contexts, we propose integrating an Intelligent Assistance

System (IAS) agent and using Human-Machine collaboration models.

**Identifying the shortcomings in the current methodologies, we pinpoint three key challenges:**

**Lack of Swift Human Decisions:**

In rapidly changing situations, autonomous robots struggle to make swift decisions. Human operators, with their cognitive capabilities and field-related knowledge, possess the agility to respond promptly to unforeseen events. In most cases, autonomous robots do not necessarily consider human opinions or actions to finish the goal.

**Autonomous Robots' Struggle with Complex Sensor Data:**

The robot's analysis of combined sensor data can be an issue for autonomous robots. Humans, on the other hand, excel in analyzing and understanding global information, at a cognitive level, therefore their presence is critical in navigating complex circumstances.

**Communication Weakness Between Humans and Robots:**

Effective communication between humans and robots is pivotal for successful cooperation. Current approaches often exhibit weaknesses in sharing high-level information, related to the team organization or plan changes due to new context for instance.

In the context of shared tasks, a large number of studies consider the physical human-robot interaction [17], while our focus is on the cognitive aspects of Human-Robot interaction: how to let the Human take control when the robot shows difficulties answering to unforeseen events or assess the current context, how to share high-level information.

Existing research on human-robot cooperation and multi-agent systems provides valuable insights, but a noticeable gap persists in integrating Human-Machine Cooperation models with multi-agent principles. This gap hinders the achievement of seamless teamwork between humans and robots, a critical aspect in complex and unpredictable environments.

To address these challenges, and elevate Human-Robot Cooperation (HRC), we introduce an innovative assistance system. This assistance system serves as an intelligent bridge, empowering humans and robots to communicate, observe, and make real-time decisions effectively. With features such as real-time monitoring, decision support, workload optimization, bi-directional communication, and instant alerts, the assistance system fills the existing gaps, fostering a new era of enhanced cooperation in hazardous environments. The subsequent sections describe the architecture, functionality, and evaluation metrics of our proposed assistance system, arguing for its innovative potential to improve Human-Robot Cooperation.

## 3 COOPERATION MODEL

The proposed cooperation model of the studied team is adapted from human-machine cooperation models, based on Know-How-to-Operate (KHO) and Know-How-to-Cooperate (KHC) competencies, proposed in [25].

The paradigm uses the cooperation model definition as shown in Figure 1) to identify the competencies of each agent, including humans and robots. The agent's interactions are analyzed based on the agent's competencies, i.e. what the agents are able to do regarding the process and about the cooperation, and the management
After positioning the problem regarding the related literature in Section 2, we present the proposed model made of four cooperation levels in Section 3, and its application to a team composed of one human, one assistant system and two robots in Section 4. The planned experiments and evaluation metrics are described in Section 5, before the conclusion and future works.

## 2 PROBLEM STATEMENT AND LITERATURE GAP

In the area of difficult and hazardous circumstances, maximizing co-operation among human operators and autonomous robots emerges as a significant task. While autonomous robots excel at moving through dangerous conditions, human decisions made on the basis of global knowledge and experience are still required to adapt to unforeseen events. Current approaches focus heavily on fully self-governing robots [19], sometimes missing the distinct advantages that human operators bring to the table.

In the field of human-robot interaction (HRI) and Human-robot teaming, successful collaboration between human operators and autonomous robots is critical for attaining common goals [28] in challenging circumstances. HRI involves numerous aspects of interaction, such as communication, coordination [21], and collaboration, to enable humans and robots to work together. Human-robot teaming combines human experience with autonomous robots to improve work performance and adaptability [7].

In addition, since our context includes interacting entities with autonomy-related characteristics, we considered the approaches provided by the Multi-Agent System (MAS) field. Indeed, MAS proposes systematic techniques for defining and implementing artificial agents' competencies and skills [36], allowing them to complete individual and cooperative tasks, in some cases using predefined or adaptive organizations [13]. MAS approaches also make it easier to integrate agents' learning capabilities, with the help of models and algorithms common to the robotics field. In MAS literature, the term "agent" refers to both human and artificial beings, such as robots, virtual agents, and AI-based systems, with distinctions drawn between the agent's embodiment and its software components. In the agent systems where organizations are modeled (at the difference from emerging organization, as in swarm for instance [6, 29]), the agents have roles in the group and follow rules that define the way they interact. Organizational structures within multi-agent systems vary from completely decentralized to a variety of hierarchical structures [3, 12], including mixed structures like the holonic organizations [1, 2]. In these systems, the autonomy of robot agents primarily relates to motion planning and navigation, with higher-level decisions involving task scheduling or conflict resolution. Additionally, agents in MAS often exhibit various behaviors and communication abilities, further enhancing their cooperative capabilities.

Despite the advancement of current human-robot interaction approaches, multi-agent systems have certain limitations as mentioned below. These approaches frequently rely on completely autonomous robotic systems, ignoring the particular problem-solving talents and situational awareness of human operators. To address these constraints and improve human-robot collaboration in complex contexts, we propose integrating an Intelligent Assistance

System (IAS) agent and using Human-Machine collaboration models.

**Identifying the shortcomings in the current methodologies, we pinpoint three key challenges:**

**Lack of Swift Human Decisions:**

In rapidly changing situations, autonomous robots struggle to make swift decisions. Human operators, with their cognitive capabilities and field-related knowledge, possess the agility to respond promptly to unforeseen events. In most cases, autonomous robots do not necessarily consider human opinions or actions to finish the goal.

**Autonomous Robots' Struggle with Complex Sensor Data:**

The robot's analysis of combined sensor data can be an issue for autonomous robots. Humans, on the other hand, excel in analyzing and understanding global information, at a cognitive level, therefore their presence is critical in navigating complex circumstances.

**Communication Weakness Between Humans and Robots:**

Effective communication between humans and robots is pivotal for successful cooperation. Current approaches often exhibit weaknesses in sharing high-level information, related to the team organization or plan changes due to new context for instance.

In the context of shared tasks, a large number of studies consider the physical human-robot interaction [17], while our focus is on the cognitive aspects of Human-Robot interaction: how to let the Human take control when the robot shows difficulties answering to unforeseen events or assess the current context, how to share high-level information.

Existing research on human-robot cooperation and multi-agent systems provides valuable insights, but a noticeable gap persists in integrating Human-Machine Cooperation models with multi-agent principles. This gap hinders the achievement of seamless teamwork between humans and robots, a critical aspect in complex and unpredictable environments.

To address these challenges, and elevate Human-Robot Cooperation (HRC), we introduce an innovative assistance system. This assistance system serves as an intelligent bridge, empowering humans and robots to communicate, observe, and make real-time decisions effectively. With features such as real-time monitoring, decision support, workload optimization, bi-directional communication, and instant alerts, the assistance system fills the existing gaps, fostering a new era of enhanced cooperation in hazardous environments. The subsequent sections describe the architecture, functionality, and evaluation metrics of our proposed assistance system, arguing for its innovative potential to improve Human-Robot Cooperation.

## 3 COOPERATION MODEL

The proposed cooperation model of the studied team is adapted from human-machine cooperation models, based on Know-How-to-Operate (KHO) and Know-How-to-Cooperate (KHC) competencies, proposed in [25].

The paradigm uses the cooperation model definition as shown in Figure 1) to identify the competencies of each agent, including humans and robots. The agent's interactions are analyzed based on the agent's competencies, i.e. what the agents are able to do regarding the process and about the cooperation, and the management
of their interactions, for instance how the control on a process is
shared in the team. This comprehensive methodology enables the
evaluation of each agent's contribution as well as the efficiency of
the cooperation.

<figure><img src="image_4.png" /><figcaption>Figure 1: Cooperation Model [26]</figcaption></figure>

## 3.1 Know-How-to-Operate and Know-How-to-Cooperate

The system leverages the Know-How-to-Operate (KHO) model,
defining the internal and external competencies of each agent in-
volved in completing a task. The internal KHO focuses on an agent's
capacity to control the process and problem-solving abilities, while
the external KHO pertains to obtaining information from and acting
upon the process.

KHO is presented in four steps, each corresponding to the process control :

(1) **Information Gathering (IG):** agents acquire data from the environment, which essential for understanding the current state and making informed decisions.

(2) **Information Analysis (IA):** the collected information undergoes analysis to discern patterns, anomalies, or trends, aiding in determining the optimal course of action.

(3) **Decision Selection (DS):** based on the analyzed information, agents make decisions regarding process control, determining the most effective strategies to achieve desired goals.

(4) **Action Implementation (AI):** once decisions are made, agents execute the chosen actions.

This categorization establishes the task allocation to agents in pur-
suit of common goals using the Know-How-to-Cooperate functions.

The Know-How-to-Cooperate (KHC) model is divided into in-
ternal and external components. The internal component enables
an agent to construct models of other agents, facilitating smoother
cooperation. Meanwhile, the external KHC involves the exchange
of information, contributing to mutual understanding and control.
This dual model governs interactions and supports a nuanced un-
derstanding of Human-Robot Cooperation.

Similarly, KHC is also presented in four steps:

(1) **Information Gathering on/from the other (IGO):** Agents gather information on other agents, crucial for understanding their capabilities and intentions and for effective cooperation.

(2) **Interference Detection (ID):** agents detect potential interferences, either positive interferences when the actions of one agent can help in achieving another agent's goals, or negative interferences when the actions of one agent may obstruct another agent's plans [14].

(3) **Interference Management (IM):** agents manage interference by negotiating, imposing or accepting the other agent's function, ensuring smooth coordination and cooperation.

(4) **Function Allocation (FA):** agents manage function allocation to define the organization between themselves by distributing tasks to optimize cooperation and performance.

We used these definitions to analyze and design the Know-how-
to-Operate and Know-how-to-Cooperate competencies of the team-
ing agents in relation to the different phases of the activity.

## 4 APPLICATION OF THE COOPERATION MODEL TO THE HUMAN-ROBOTS TEAMING

In our application domain, a team composed of a human operator and two robots is set to explore various cooperative scenarios in which we demonstrate how to handle tasks using the KHO and KHC terminologies. To support smooth cooperation and knowledge exchange within the team, we introduced an additional agent assuming the function of an Intelligent Assistance System (IAS). This customized IAS agent takes on the vital role of monitoring and facilitating cooperation between the human operator and the robots. In the next paragraphs, we are going to look at how to use the cooperation model in practice, using the KHO and KHC functions to optimize the cooperation between humans, IAS, and robots. Furthermore, we use the cooperation model competence definitions [26] (in Figure 1) to define agent competencies and skills, setting the groundwork for improved cooperation and synergies between humans and robots.

The KHO functions (IG, IA, DM, AI) and KHC functions (IGO, ID, IM, FA) presented in 3, form the operational pillars of the system. These functions orchestrate the flow of information and actions between agents, ensuring a comprehensive understanding of each agent's role in the cooperation process.

### 4.1 The IAS agent

Direct communication protocols [22] between humans and robots are frequently used in the majority of human-robot teaming concepts. Nevertheless, several drawbacks to this approach may affect the workload of human operators as we discussed a few of them below, namely: real-time monitoring, decision support, and bidirectional communication. Furthermore, it might be difficult for human operators to keep an eye on several robots at once, which could result in performance oversight [28]. Additionally, making decisions [9] in changing surroundings might be difficult, which can cause delays in quickly replying to the needs of robots. With these constraints in mind, we have developed the Intelligent Assistance System (IAS) agent to improve human-robot cooperation efficiency.
of their interactions, for instance how the control on a process is
shared in the team. This comprehensive methodology enables the
evaluation of each agent's contribution as well as the efficiency of
the cooperation.

<figure><img src="image_2.png" /><figcaption>Figure 1: Cooperation Model [26]</figcaption></figure>

**3.1 Know-How-to-Operate and**
**Know-How-to-Cooperate**

The system leverages the Know-How-to-Operate (KHO) model,
defining the internal and external competencies of each agent in-
volved in completing a task. The internal KHO focuses on an agent's
capacity to control the process and problem-solving abilities, while
the external KHO pertains to obtaining information from and acting
upon the process.

KHO is presented in four steps, each corresponding to the process control :

(1) **Information Gathering (IG):** agents acquire data from the environment, which essential for understanding the current state and making informed decisions.

(2) **Information Analysis (IA):** the collected information undergoes analysis to discern patterns, anomalies, or trends, aiding in determining the optimal course of action.

(3) **Decision Selection (DS):** based on the analyzed information, agents make decisions regarding process control, determining the most effective strategies to achieve desired goals.

(4) **Action Implementation (AI):** once decisions are made, agents execute the chosen actions.

This categorization establishes the task allocation to agents in pur-
suit of common goals using the Know-How-to-Cooperate functions.

The Know-How-to-Cooperate (KHC) model is divided into in-
ternal and external components. The internal component enables
an agent to construct models of other agents, facilitating smoother
cooperation. Meanwhile, the external KHC involves the exchange
of information, contributing to mutual understanding and control.
This dual model governs interactions and supports a nuanced un-
derstanding of Human-Robot Cooperation.

Similarly, KHC is also presented in four steps:

(1) **Information Gathering on/from the other (IGO):** Agents gather information on other agents, crucial for understanding their capabilities and intentions and for effective cooperation.

(2) **Interference Detection (ID):** agents detect potential interferences, either positive interferences when the actions of one agent can help in achieving another agent's goals, or negative interferences when the actions of one agent may obstruct another agent's plans [14].

(3) **Interference Management (IM):** agents manage interference by negotiating, imposing or accepting the other agent's function, ensuring smooth coordination and cooperation.

(4) **Function Allocation (FA):** agents manage function allocation to define the organization between themselves by distributing tasks to optimize cooperation and performance.

We used these definitions to analyze and design the Know-how-
to-Operate and Know-how-to-Cooperate competencies of the team-
ing agents in relation to the different phases of the activity.

4 APPLICATION OF THE COOPERATION MODEL TO THE HUMAN-ROBOTS TEAMING

In our application domain, a team composed of a human operator and two robots is set to explore various cooperative scenarios in which we demonstrate how to handle tasks using the KHO and KHC terminologies. To support smooth cooperation and knowledge exchange within the team, we introduced an additional agent assuming the function of an Intelligent Assistance System (IAS). This customized IAS agent takes on the vital role of monitoring and facilitating cooperation between the human operator and the robots. In the next paragraphs, we are going to look at how to use the cooperation model in practice, using the KHO and KHC functions to optimize the cooperation between humans, IAS, and robots. Furthermore, we use the cooperation model competence definitions [26] (in Figure 1) to define agent competencies and skills, setting the groundwork for improved cooperation and synergies between humans and robots.

The KHO functions (IG, IA, DM, AI) and KHC functions (IGO, ID, IM, FA) presented in 3, form the operational pillars of the system. These functions orchestrate the flow of information and actions between agents, ensuring a comprehensive understanding of each agent's role in the cooperation process.

**4.1 The IAS agent**

Direct communication protocols [22] between humans and robots are frequently used in the majority of human-robot teaming concepts. Nevertheless, several drawbacks to this approach may affect the workload of human operators as we discussed a few of them below, namely: real-time monitoring, decision support, and bidirectional communication. Furthermore, it might be difficult for human operators to keep an eye on several robots at once, which could result in performance oversight [28]. Additionally, making decisions [9] in changing surroundings might be difficult, which can cause delays in quickly replying to the needs of robots. With these constraints in mind, we have developed the Intelligent Assistance System (IAS) agent to improve human-robot cooperation efficiency.
At its core, the IAS agent prioritizes real-time monitoring, decision support, and bidirectional communication. The KHO & KHC functions provide the foundation for analyzing sensor data, facilitating informed decision-making, and ensuring optimal agent cooperation.

**Real-time Monitoring:** The IAS continuously monitors sensor data from robots and the environment, providing up-to-date information on the operational status, and surroundings, to display alerts to notify the human if an urgent decision is required and potential hazards. This real-time monitoring ensures that human operators and the IAS agent are equipped with timely information to make informed decisions.

**Decision Support:** Drawing from the KHO and KHC functions, the IAS agent offers decision support by analyzing sensor data, evaluating situational factors, and presenting actionable insights to human operators. By providing contextual information and suggesting optimal plans of action, the system enhances the decision-making capabilities of human operators, leading to more efficient task execution.

**Bidirectional Communication:** The system facilitates seamless bidirectional communication channels between human operators, the IAS, and robots. Through clear and efficient communication protocols, human operators can convey instructions, receive real-time updates, and collaborate with the IAS agent and robots effectively. Likewise, the IAS agent can provide feedback, request clarification, and coordinate actions with human operators and robots, ensuring synchronized cooperation.

## 4.2 Levels of Cooperation

In various fields like Robotics, automation, and multi-agent systems, different scales are used to measure the levels of autonomy of systems, agents, or robots. However, these levels of autonomy may not necessarily align with the levels of cooperation we designed for our needs because it is very rare to find a similar application with the respective levels according to the needs of the application proposed. In our approach, we identified the levels of cooperation according to the competencies provided to the agents, based on the model of cooperation 3. The more functions of the agent's Know-How-to-Operate and Know-How-to-Cooperate are chosen, the higher the autonomy of the agents and the more the agents are able to cooperate. The difficulty is then to identify the best balance between high autonomy with the human as a supervisor and less autonomy with the human-in-the-loop as a team member. Cooperation must not be the main task that could overload human and decrease performance. Inspired by Pacaux-Lemoine et al's work [24] and adapted from the automation domain, our model identifies four degrees of human-robot cooperation.

Drawing on automation and human-machine cooperation principles, these stages evolve from manual teleoperation to supervised autonomy, to optimize cooperation and autonomy in complex situations. By combining insights from pre-existing concepts and modifying them to the specific needs of human-robot cooperation, we built structured levels that allow for efficient cooperation and task performance in numerous scenarios.

The Intelligent Assistance System (IAS) is an important support part of the human-robot cooperation framework, acting similarly to

a cooperative agent in a Multi-Agent System (MAS) context. The IAS allows for seamless communication as well as cooperation between human operators and robots, which is crucial to accomplishing tasks successfully.

The IAS functions on four unique levels, each having its own hierarchical structure based on the levels of autonomy obtained from human-machine cooperation and the automation domain.

At each level, as shown in Figures 2 and 3, we illustrated the steps of Know-How-to-Operate (KHO) by solid blue arrows representing the flow of knowledge and information within the system; the dashed green lines represent the steps of Know-How-to-Cooperate (KHC), indicating the management of cooperative interactions and the allocation of functions among the agents. This color-coded distinction visually distinguishes between the internal competencies of individual agents (KHO) and the processes involved in cooperation (KHC).

### Level 1: Teleoperation Interface

Figure 2 represents the multi-agent system composed of four agents: two robots, one IAS, and one human. The diagram shows how these agents cooperate at an initial level (Level 1), where the IAS acts as an interface for teleoperation between the human and the robots. The information gathered (IG) by the robots is made available to the Human via the IAS. The Human performs the Information Analysis (IA), makes decisions (DS), and controls the robots through the IAS mediation that translates the operation into a format understandable by the robots (AI) and transmits them to Robot 1 and Robot 2.

### Level 2: Decision Support and Plan Presentation

In Level 2 (see Fig. 2), the Intelligent Assistance System (IAS) enhances the capabilities of the human operator by providing specified strategies tailored to the task at hand. Through the KHO steps, represented by solid blue arrows, the IAS offers various plans, a step-by-step proposal outlining actions for the agent to achieve the respective goal and recommendations based on real-time data analysis and environmental factors. These plans are communicated to the human operator through solid blue arrows, facilitating informed decision-making (DS) and action implementation (AI). Concurrently, the KHC steps, illustrated by dashed green lines, govern the allocation of functions among the agents, ensuring coordinated efforts. The IAS leverages its Interference Management (IM) function to address any conflicts or discrepancies in the proposed plans, while Function Allocation (FA) organizes the distribution of tasks among the agents.

### Level 3: Autonomous Decision-Making

In Level 3 (see Fig. 3), characterized by increased autonomy, the IAS gains additional authority which refers to its capability to make decisions autonomously [10], as depicted by the solid blue arrows representing KHO steps. While the IAS continues to present pre-defined plans to the human operator, now it can execute actions independently if the situation aligns with the available plans. This autonomous decision-making process, indicated by solid blue arrows, enables the IAS to analyze real-time data and environmental cues to determine the most suitable actions for task execution. However, in scenarios where actions deviate from the predefined plans, the IAS seeks permission from the human operator, as indicated by dashed green lines representing KHC steps. This cooperative approach ensures that the human operator remains actively engaged
At its core, the IAS agent prioritizes real-time monitoring, decision support, and bidirectional communication. The KHO & KHC functions provide the foundation for analyzing sensor data, facilitating informed decision-making, and ensuring optimal agent cooperation.

**Real-time Monitoring:** The IAS continuously monitors sensor data from robots and the environment, providing up-to-date information on the operational status, and surroundings, to display alerts to notify the human if an urgent decision is required and potential hazards. This real-time monitoring ensures that human operators and the IAS agent are equipped with timely information to make informed decisions.

**Decision Support:** Drawing from the KHO and KHC functions, the IAS agent offers decision support by analyzing sensor data, evaluating situational factors, and presenting actionable insights to human operators. By providing contextual information and suggesting optimal plans of action, the system enhances the decision-making capabilities of human operators, leading to more efficient task execution.

**Bidirectional Communication:** The system facilitates seamless bidirectional communication channels between human operators, the IAS, and robots. Through clear and efficient communication protocols, human operators can convey instructions, receive real-time updates, and collaborate with the IAS agent and robots effectively. Likewise, the IAS agent can provide feedback, request clarification, and coordinate actions with human operators and robots, ensuring synchronized cooperation.

## 4.2 Levels of Cooperation

In various fields like Robotics, automation, and multi-agent systems, different scales are used to measure the levels of autonomy of systems, agents, or robots. However, these levels of autonomy may not necessarily align with the levels of cooperation we designed for our needs because it is very rare to find a similar application with the respective levels according to the needs of the application proposed. In our approach, we identified the levels of cooperation according to the competencies provided to the agents, based on the model of cooperation 3. The more functions of the agent's Know-How-to-Operate and Know-How-to-Cooperate are chosen, the higher the autonomy of the agents and the more the agents are able to cooperate. The difficulty is then to identify the best balance between high autonomy with the human as a supervisor and less autonomy with the human-in-the-loop as a team member. Cooperation must not be the main task that could overload human and decrease performance. Inspired by Pacaux-Lemoine et al's work [24] and adapted from the automation domain, our model identifies four degrees of human-robot cooperation.

Drawing on automation and human-machine cooperation principles, these stages evolve from manual teleoperation to supervised autonomy, to optimize cooperation and autonomy in complex situations. By combining insights from pre-existing concepts and modifying them to the specific needs of human-robot cooperation, we built structured levels that allow for efficient cooperation and task performance in numerous scenarios.

The Intelligent Assistance System (IAS) is an important support part of the human-robot cooperation framework, acting similarly to

a cooperative agent in a Multi-Agent System (MAS) context. The IAS allows for seamless communication as well as cooperation between human operators and robots, which is crucial to accomplishing tasks successfully.

The IAS functions on four unique levels, each having its own hierarchical structure based on the levels of autonomy obtained from human-machine cooperation and the automation domain.

At each level, as shown in Figures 2 and 3, we illustrated the steps of Know-How-to-Operate (KHO) by solid blue arrows representing the flow of knowledge and information within the system; the dashed green lines represent the steps of Know-How-to-Cooperate (KHC), indicating the management of cooperative interactions and the allocation of functions among the agents. This color-coded distinction visually distinguishes between the internal competencies of individual agents (KHO) and the processes involved in cooperation (KHC).

### Level 1: Teleoperation Interface

Figure 2 represents the multi-agent system composed of four agents: two robots, one IAS, and one human. The diagram shows how these agents cooperate at an initial level (Level 1), where the IAS acts as an interface for teleoperation between the human and the robots. The information gathered (IG) by the robots is made available to the Human via the IAS. The Human performs the Information Analysis (IA), makes decisions (DS), and controls the robots through the IAS mediation that translates the operation into a format understandable by the robots (AI) and transmits them to Robot 1 and Robot 2.

### Level 2: Decision Support and Plan Presentation

In Level 2 (see Fig. 2), the Intelligent Assistance System (IAS) enhances the capabilities of the human operator by providing specified strategies tailored to the task at hand. Through the KHO steps, represented by solid blue arrows, the IAS offers various plans, a step-by-step proposal outlining actions for the agent to achieve the respective goal and recommendations based on real-time data analysis and environmental factors. These plans are communicated to the human operator through solid blue arrows, facilitating informed decision-making (DS) and action implementation (AI). Concurrently, the KHC steps, illustrated by dashed green lines, govern the allocation of functions among the agents, ensuring coordinated efforts. The IAS leverages its Interference Management (IM) function to address any conflicts or discrepancies in the proposed plans, while Function Allocation (FA) organizes the distribution of tasks among the agents.

### Level 3: Autonomous Decision-Making

In Level 3 (see Fig. 3), characterized by increased autonomy, the IAS gains additional authority which refers to its capability to make decisions autonomously [10], as depicted by the solid blue arrows representing KHO steps. While the IAS continues to present pre-defined plans to the human operator, now it can execute actions independently if the situation aligns with the available plans. This autonomous decision-making process, indicated by solid blue arrows, enables the IAS to analyze real-time data and environmental cues to determine the most suitable actions for task execution. However, in scenarios where actions deviate from the predefined plans, the IAS seeks permission from the human operator, as indicated by dashed green lines representing KHC steps. This cooperative approach ensures that the human operator remains actively engaged
<figure><img src="image_5.png" /><figcaption>Figure 2: Level 1 & 2 cooperation. The flow of information and knowledge is represented by solid arrows and the flow of control is represented by dashed arrows.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 3: Level 3 & 4 cooperation. The flow of information and knowledge is represented by solid arrows; the flow of control is represented by dashed arrows.</figcaption></figure>

in the decision-making process and retains oversight over the actions performed by the IAS. Through the Interference Decision (ID) function, the IAS evaluates deviations from predefined plans and cooperates with the human operator to resolve any conflicts or deviations from the plans. Additionally, the Function Allocation (FA) step ensures the equitable distribution of tasks between the IAS and the human operator, fostering a balanced and cooperative interaction. Overall, Level 3 cooperation reflects a transition towards increased autonomy for the IAS while maintaining a cooperative partnership with the human operator to achieve common goals.

**Level 4: Supervised Autonomy**

In Level 4 (see Fig. 3), the IAS has the highest level of autonomy, assuming full control over the robots, illustrated by solid blue arrows representing KHO steps. At this stage, the IAS operates independently without constant guidance from the human operator, enabling efficient task execution based on real-time data and environment. The IAS manages tasks autonomously, as indicated by

solid blue arrows, leveraging its decision-making capabilities to determine optimal actions for achieving predefined objectives. While the human operator transitions into a supervisory role, denoted by solid red lines representing KHC steps, they retain the ultimate authority and oversight over the IAS's actions. As a supervisor, the human operator remains on standby to intervene in emergencies or unforeseen circumstances, ensuring the safety and efficiency of task execution. Through the Interference Management (IM) function, the IAS addresses any conflicts or challenges that may arise during autonomous operation and cooperates with the human operator to resolve issues effectively. Additionally, the Function Allocation (FA) step ensures a balanced distribution of responsibilities between the IAS and the human operator, promoting effective cooperation and task management.

This hierarchical design improves the overall flexibility and effectiveness of the human-robot cooperation model by ensuring an
<figure><img src="image_3.png" /><figcaption>Figure 2: Level 1 & 2 cooperation. The flow of information and knowledge is represented by solid arrows and the flow of control is represented by dashed arrows.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3: Level 3 & 4 cooperation. The flow of information and knowledge is represented by solid arrows; the flow of control is represented by dashed arrows.</figcaption></figure>

in the decision-making process and retains oversight over the actions performed by the IAS. Through the Interference Decision (ID) function, the IAS evaluates deviations from predefined plans and cooperates with the human operator to resolve any conflicts or deviations from the plans. Additionally, the Function Allocation (FA) step ensures the equitable distribution of tasks between the IAS and the human operator, fostering a balanced and cooperative interaction. Overall, Level 3 cooperation reflects a transition towards increased autonomy for the IAS while maintaining a cooperative partnership with the human operator to achieve common goals.

**Level 4: Supervised Autonomy**

In Level 4 (see Fig. 3), the IAS has the highest level of autonomy, assuming full control over the robots, illustrated by solid blue arrows representing KHO steps. At this stage, the IAS operates independently without constant guidance from the human operator, enabling efficient task execution based on real-time data and environment. The IAS manages tasks autonomously, as indicated by

solid blue arrows, leveraging its decision-making capabilities to determine optimal actions for achieving predefined objectives. While the human operator transitions into a supervisory role, denoted by solid red lines representing KHC steps, they retain the ultimate authority and oversight over the IAS's actions. As a supervisor, the human operator remains on standby to intervene in emergencies or unforeseen circumstances, ensuring the safety and efficiency of task execution. Through the Interference Management (IM) function, the IAS addresses any conflicts or challenges that may arise during autonomous operation and cooperates with the human operator to resolve issues effectively. Additionally, the Function Allocation (FA) step ensures a balanced distribution of responsibilities between the IAS and the human operator, promoting effective cooperation and task management.

This hierarchical design improves the overall flexibility and effectiveness of the human-robot cooperation model by ensuring an
increasing integration of autonomy and balancing human oversight and robot capabilities.

# 5 EXPERIMENTAL SETUP

The main goal is to improve the cooperation between human and robots in a crisis management scenario. In order to achieve it we decided to mimic a fire accident situation in a controlled setting, as seen in Figure 4, where cooperation between humans and robots is essential. The experimental setup aims to validate and evaluate the proposed cooperation levels, particularly aimed at improving human-robot cooperation in challenging and risky situations such as fire accidents or post-earthquake scenarios. To simulate these challenging scenarios, we have devised an experiment focusing on various cooperative tasks similar to real-life scenarios. These tasks include some basic activities like line following for robot navigation along predetermined paths, a pick-and-place operation involving two robots, and cooperative tasks demonstrating the support provided by the IAS agent to both human and robot agents. With the help of an IAS agent, we aim to address challenges such as real-time monitoring, decision support, and bidirectional communication to enhance the effectiveness of cooperation.

<figure><img src="image_7.png" /><figcaption>Figure 4: Environment setup inside a room</figcaption></figure>

In the experimental setup, two RoboMaster EP robots, equipped with infrared sensors for collision avoidance, a camera for line following, and object detection, and programmed to navigate challenging situations, are used in the simplified experiment. Among the cooperative tasks are pick-and-place tasks between two robots, object detection in the surrounding environment, and optimal route planning from point A to point B.

## 5.1 Experiment levels

The experiment intends to illustrate the actual implementation of the proposed cooperative organization at various levels of autonomy, with an emphasis on a pick-and-place operation. While several tasks may be completed cooperatively, the pick-and-place operation is a fine example of how each agent's authority varies over levels.

On the basis of the four levels described in Section 3, we designed four configurations that differ according to the roles assigned to the Human, the Robots, and the IAS, see Table 1. A role synthesizes a specific set of task-related Know-How and interaction-related Know-How-to-Cooperate abilities and responsibilities.

At Level 1, the configuration is a **teleoperation** setup. A Human Operator utilizes the IAS interface to guide the robots to designated areas with objects. The robots follow teleoperation commands from the human operator to move and perform actions like detection of objects and the use of the robot arm.

At Level 2, the configuration focuses on **decision support**. The IAS displays available plans for pick and place based on object size and shape and receives sensor information (e.g., camera feedback, object detection, obstacle avoidance) from robots. The Human operator selects one of the plans based on the presented options. The robots follow the selected plan.

At Level 3, the configuration allows **autonomous decision-making**. The IAS displays available plans and proposes the best plan according to the object and situation. The Human operator provides only high-level guidance and approves deviations from the predefined plans. The robots communicate with each other and with the IAS for coordination while achieving the plan.

At Level 4, the configuration includes a higher level of **autonomy**. The Human Operator's role consists of supervising the IAS's autonomous operation and intervening if an emergency occurs. The robots communicate with each other for cooperative decision-making and task execution. The IAS operates autonomously to control the robots based on the current situation and communicates with them for task assignment and coordination. The Human operator can communicate through the IAS interface for supervision and intervention if necessary.

<figure><img src="image_8.png" /><figcaption>Figure 5: IAS agent</figcaption></figure>

Figure 5 represents an IAS agent that acts according to the mentioned levels of cooperation, to improve cooperation among human operators and robots. Its capabilities include real-time monitoring, decision assistance, and bidirectional communication to handle issues like prompt decision-making and information exchange. By considering the criteria for objective and subjective metrics evaluation with the respective tasks (described below in Section 5.2), our aim will be to attempt to improve cooperation efficiency and task performance using the IAS agent.
increasing integration of autonomy and balancing human oversight and robot capabilities.

# 5 EXPERIMENTAL SETUP

The main goal is to improve the cooperation between human and robots in a crisis management scenario. In order to achieve it we decided to mimic a fire accident situation in a controlled setting, as seen in Figure 4, where cooperation between humans and robots is essential. The experimental setup aims to validate and evaluate the proposed cooperation levels, particularly aimed at improving human-robot cooperation in challenging and risky situations such as fire accidents or post-earthquake scenarios. To simulate these challenging scenarios, we have devised an experiment focusing on various cooperative tasks similar to real-life scenarios. These tasks include some basic activities like line following for robot navigation along predetermined paths, a pick-and-place operation involving two robots, and cooperative tasks demonstrating the support provided by the IAS agent to both human and robot agents. With the help of an IAS agent, we aim to address challenges such as real-time monitoring, decision support, and bidirectional communication to enhance the effectiveness of cooperation.

<figure><img src="image_5.png" /><figcaption>Figure 4: Environment setup inside a room</figcaption></figure>

In the experimental setup, two RoboMaster EP robots, equipped with infrared sensors for collision avoidance, a camera for line following, and object detection, and programmed to navigate challenging situations, are used in the simplified experiment. Among the cooperative tasks are pick-and-place tasks between two robots, object detection in the surrounding environment, and optimal route planning from point A to point B.

## 5.1 Experiment levels

The experiment intends to illustrate the actual implementation of the proposed cooperative organization at various levels of autonomy, with an emphasis on a pick-and-place operation. While several tasks may be completed cooperatively, the pick-and-place operation is a fine example of how each agent's authority varies over levels.

On the basis of the four levels described in Section 3, we designed four configurations that differ according to the roles assigned to the Human, the Robots, and the IAS, see Table 1. A role synthesizes a specific set of task-related Know-How and interaction-related Know-How-to-Cooperate abilities and responsibilities.

At Level 1, the configuration is a **teleoperation** setup. A Human Operator utilizes the IAS interface to guide the robots to designated areas with objects. The robots follow teleoperation commands from the human operator to move and perform actions like detection of objects and the use of the robot arm.

At Level 2, the configuration focuses on **decision support**. The IAS displays available plans for pick and place based on object size and shape and receives sensor information (e.g., camera feedback, object detection, obstacle avoidance) from robots. The Human operator selects one of the plans based on the presented options. The robots follow the selected plan.

At Level 3, the configuration allows **autonomous decision-making**. The IAS displays available plans and proposes the best plan according to the object and situation. The Human operator provides only high-level guidance and approves deviations from the predefined plans. The robots communicate with each other and with the IAS for coordination while achieving the plan.

At Level 4, the configuration includes a higher level of **autonomy**. The Human Operator's role consists of supervising the IAS's autonomous operation and intervening if an emergency occurs. The robots communicate with each other for cooperative decision-making and task execution. The IAS operates autonomously to control the robots based on the current situation and communicates with them for task assignment and coordination. The Human operator can communicate through the IAS interface for supervision and intervention if necessary.

<figure><img src="image_6.png" /><figcaption>Figure 5: IAS agent</figcaption></figure>

Figure 5 represents an IAS agent that acts according to the mentioned levels of cooperation, to improve cooperation among human operators and robots. Its capabilities include real-time monitoring, decision assistance, and bidirectional communication to handle issues like prompt decision-making and information exchange. By considering the criteria for objective and subjective metrics evaluation with the respective tasks (described below in Section 5.2), our aim will be to attempt to improve cooperation efficiency and task performance using the IAS agent.
<table><caption>Table 1: Levels of cooperation developed for the experiments</caption><thead><tr><td>Cooperation level</td><td>Human's role</td><td>IAS' role</td><td>Robots' roles</td><td>Communication</td></tr></thead><tbody><tr><td>Teleoperation</td><td>teleoperates the robots via the IAS</td><td>control interface</td><td>follow Human's commands through the IAS interface</td><td>from Human to Robots through the IAS agent interface</td></tr><tr><td>Decision Support</td><td>selects the plan for the robot's actions</td><td>Displays the available plans and data from robots' sensors</td><td>follow the selected plan</td><td>IAS and robots can inform each other about the plan situation</td></tr><tr><td>Autonomous Decision-Making</td><td>high-level guidance</td><td>presents pre-selected available plans to the human operator</td><td>follow the selected plan and manage interferences</td><td>robot share plans for coordinated pick and place; IAS helps the robots to coordinate</td></tr><tr><td>Supervised Autonomy</td><td>supervises IAS agent, intervenes in emergency situations</td><td>operates autonomously, assigns tasks to Robots and helps in coordination</td><td>follow IAS' plans</td><td>Direct communication among robots for cooperative decision-making in real-time</td></tr></tbody></table>

## 5.2 Evaluation Metrics

It is important to note that, while the current experimental setup offers a fundamental illustration of the proposed principles, no formal testing has been conducted at this time. This initial step acts as a proof-of-concept, establishing the framework for future experiments. As part of our ongoing research schedule, we intend to systematically evaluate the effectiveness and performance of the proposed approach.

The evaluation of the proposed IAS agent impact on human-robot cooperation encompasses a comprehensive set of metrics, combining both objective measurements and subjective user feedback for a nuanced understanding of performance [15].

**Objective Metrics:**

**Response Time:** Measures the system's speed in delivering real-time monitoring and feedback to human operators, ensuring swift and efficient information provision during the tasks.

**Event Detection Precision:** Assesses the system's accuracy in recognizing and reporting crucial occurrences such as robot behavior, objects, and obstructions, revealing information about the IAS agent's reliability.

**Navigation Accuracy:** Assesses the system's proficiency in translating human directives into precise robot movements, determining the accuracy and effectiveness of the cooperative navigation.

**Workload Management:** Determines how well the IAS agent supports decision-making while controlling the cognitive load of human operators to guarantee the best possible task performance under pressure.

**Subjective Metrics:**

**HRI Questionnaires:** Gathers human operator feedback through Human-Robot Interaction (HRI) questionnaires [34], offering insights into their perceptions of the cooperation, ease of system use, and workload experienced during operations.

**User Experience and Safety:** Captures qualitative feedback on user experience, comfort, trust in robot behaviors, and safety considerations in hazardous environments.

This subjective assessment provides a holistic understanding of the system's usability and safety features.

**Performance Comparison:** Analyzes scenarios both with and without the IAS agent to determine its added value in event detection, navigation, and operator workload. This comparative analysis elucidates the system's tangible contributions to Human-Robot Cooperation.

**Ethics and Safety Considerations:** Addresses ethical and safety concerns inherent in the system, emphasizing human well-being, adherence to ethical norms, and the confidentiality of sensitive data. This critical evaluation ensures the system's alignment with ethical standards and safety protocols.

The utilization of this diverse set of objective and subjective metrics contributes to a comprehensive assessment of the proposed IAS agent's efficacy in enhancing Human-Robot Cooperation within challenging and hazardous environments.

## 5.3 Discussion

Since the proposed application of the KHO-KHC terminologies has not been evaluated yet, its efficiency is still to be demonstrated. However, the model we based on has already proven its applicability, for instance, [24] that provides a practical framework to design cooperative interaction behaviors in a system made of such heterogeneous agents as humans and robots.

The decomposition in four levels is a first attempt that will be refined, considering other propositions that are provided in the Human-Robot Interaction [16] and MAS fields [4, 8].

The Intelligent Assistance System agent is another proposition that has to be refined. For instance, the decision part of it may be implemented directly in the agents-robots' behaviors. Finally, the objective is to get relevant indicators regarding the situation to make dynamic changes in the autonomy level. The literature about situation awareness, mode awareness [32], and competence-awareness [18, 33] will make the ground for this development.
<table><caption>Table 1: Levels of cooperation developed for the experiments</caption><thead><tr><td>Cooperation level</td><td>Human's role</td><td>IAS' role</td><td>Robots' roles</td><td>Communication</td></tr></thead><tbody><tr><td>Teleoperation</td><td>teleoperates the robots via the IAS</td><td>control interface</td><td>follow Human's commands through the IAS interface</td><td>from Human to Robots through the IAS agent interface</td></tr><tr><td>Decision Support</td><td>selects the plan for the robot's actions</td><td>Displays the available plans and data from robots' sensors</td><td>follow the selected plan</td><td>IAS and robots can inform each other about the plan situation</td></tr><tr><td>Autonomous Decision-Making</td><td>high-level guidance</td><td>presents pre-selected available plans to the human operator</td><td>follow the selected plan and manage interferences</td><td>robot share plans for coordinated pick and place; IAS helps the robots to coordinate</td></tr><tr><td>Supervised Autonomy</td><td>supervises IAS agent, intervenes in emergency situations</td><td>operates autonomously, assigns tasks to Robots and helps in coordination</td><td>follow IAS' plans</td><td>Direct communication among robots for cooperative decision-making in real-time</td></tr></tbody></table>

## 5.2 Evaluation Metrics

It is important to note that, while the current experimental setup offers a fundamental illustration of the proposed principles, no formal testing has been conducted at this time. This initial step acts as a proof-of-concept, establishing the framework for future experiments. As part of our ongoing research schedule, we intend to systematically evaluate the effectiveness and performance of the proposed approach.

The evaluation of the proposed IAS agent impact on human-robot cooperation encompasses a comprehensive set of metrics, combining both objective measurements and subjective user feedback for a nuanced understanding of performance [15].

**Objective Metrics:**

**Response Time:** Measures the system's speed in delivering real-time monitoring and feedback to human operators, ensuring swift and efficient information provision during the tasks.

**Event Detection Precision:** Assesses the system's accuracy in recognizing and reporting crucial occurrences such as robot behavior, objects, and obstructions, revealing information about the IAS agent's reliability.

**Navigation Accuracy:** Assesses the system's proficiency in translating human directives into precise robot movements, determining the accuracy and effectiveness of the cooperative navigation.

**Workload Management:** Determines how well the IAS agent supports decision-making while controlling the cognitive load of human operators to guarantee the best possible task performance under pressure.

**Subjective Metrics:**

**HRI Questionnaires:** Gathers human operator feedback through Human-Robot Interaction (HRI) questionnaires [34], offering insights into their perceptions of the cooperation, ease of system use, and workload experienced during operations.

**User Experience and Safety:** Captures qualitative feedback on user experience, comfort, trust in robot behaviors, and safety

considerations in hazardous environments. This subjective assessment provides a holistic understanding of the system's usability and safety features.

**Performance Comparison:** Analyzes scenarios both with and without the IAS agent to determine its added value in event detection, navigation, and operator workload. This comparative analysis elucidates the system's tangible contributions to Human-Robot Cooperation.

**Ethics and Safety Considerations:** Addresses ethical and safety concerns inherent in the system, emphasizing human well-being, adherence to ethical norms, and the confidentiality of sensitive data. This critical evaluation ensures the system's alignment with ethical standards and safety protocols.

The utilization of this diverse set of objective and subjective metrics contributes to a comprehensive assessment of the proposed IAS agent's efficacy in enhancing Human-Robot Cooperation within challenging and hazardous environments.

## 5.3 Discussion

Since the proposed application of the KHO-KHC terminologies has not been evaluated yet, its efficiency is still to be demonstrated. However, the model we based on has already proven its applicability, for instance, [24] that provides a practical framework to design cooperative interaction behaviors in a system made of such heterogeneous agents as humans and robots.

The decomposition in four levels is a first attempt that will be refined, considering other propositions that are provided in the Human-Robot Interaction [16] and MAS fields [4, 8].

The Intelligent Assistance System agent is another proposition that has to be refined. For instance, the decision part of it may be implemented directly in the agents-robots' behaviors. Finally, the objective is to get relevant indicators regarding the situation to make dynamic changes in the autonomy level. The literature about situation awareness, mode awareness [32], and competence-awareness [18, 33] will make the ground for this development.
# 6 CONCLUSION

This paper described an early investigation of an intelligent assistance agent intended to improve cooperation among humans and robots in demanding circumstances. We have presented a proof-of-concept with the aim to improve cooperation between Human and robots, with the help of an IAS agent, with implementation of four levels demonstrating degrees of autonomy and cooperation, with respective to Know-how-to-Operate and Know-how-to-Cooperate model.

The experimental setup serves as a starting point for future tests and experimental evaluation, with the help of several objective and subjective metrics. Notably, the aim is to show the system's adaptation to a variety of cooperative tasks, such as line following, object manipulation, and collaborative navigation, to demonstrate its flexibility and utility. Moving ahead, efforts will be focused on improving the cooperation model and IAS agent to suit different operator competency levels and expand the scope of cooperative activities. Ongoing improvements in algorithm development, sensor integration, and real-world testing are intended to position this system as a useful tool for tackling difficult issues while supporting enhanced human-robot cooperation across several domains.

# 7 FUTURE WORK

Further research and development of the suggested cooperation model as well as the IAS agent are planned. The study trajectory involves investigating the model's adaptability across a wide range of users and testing it with a variety of human operators with varying competency levels. To provide adaptability in a wide range of applications cooperative tasks inside the system will be varied beyond pick-and-place and line following. To systematically evaluate the scalability and flexibility of the system, the complexity of the scenarios will be gradually enhanced. To validate the system in real-world scenarios and correlate its performance to real-world needs in disaster response, search and rescue, or hazardous situations, field trials will be planned. The development of algorithms, decision-making procedures, real-time monitoring, agent modeling, and human-robot cooperation will be the main areas of ongoing work. The research into the use of extra sensors aims to increase system adaptability by integrating cutting-edge technology for better situational awareness and environmental perception. The goal of this extensive future effort is to improve the system's efficacy by expanding its capabilities for a wider range of real-world circumstances and user profiles.

# ACKNOWLEDGMENTS

The present research work is supported by the CNRS and CROSSING: the French-Australian Laboratory for Humans / Autonomous Agents Teaming.

# REFERENCES

[1] C.V. Almagro, G. Lunghi, M.D. Castro, D.C. Beltrán, R.M. Prade, A. Masi, and P.J. Sanz. 2020. Cooperative and multimodal capabilities enhancement in the CERNTAURO human-robot interface for hazardous and underwater scenarios. *Applied Sciences (Switzerland)* 10, 17 (2020). https://doi.org/10.3390/app10176144

[2] José Barbosa, Paulo Leitão, Emmanuel Adam, and Damien Trentesaux. 2015. Dynamic self-organization in holonic multi-agent manufacturing systems: The ADACOR evolution. *Computers in Industry* 66 (2015), 99-111. https://doi.org/10.1016/j.compind.2014.10.011

[3] C. Bensaci, Y. Zennir, D. Pomorski, F. Innal, Y. Liu, and C. Tolba. 2020. STPA and Bowtie risk analysis study for centralized and hierarchical control architectures comparison. *Alexandria Engineering Journal* 59, 5 (2020), 3799-3816. https://doi.org/10.1016/j.aej.2020.06.036

[4] Chaima Bensaci, Youcef Zennir, Denis Pomorski, Fares Innal, Yiliu Liu, and Cherif Tolba. 2020. STPA and Bowtie risk analysis study for centralized and hierarchical control architectures comparison. *Alexandria Engineering Journal* 59, 5 (2020), 3799-3816.

[5] Juan Bravo-Arrabal, Manuel Toscano-Moreno, JJ Fernandez-Lozano, Anthony Mandow, Jose Antonio Gomez-Ruiz, and Alfonso García-Cerezo. 2021. The internet of cooperative agents architecture (X-ioca) for robots, hybrid sensor networks, and mec centers in complex environments: A search and rescue case study. *Sensors* 21, 23 (2021), 7843.

[6] Daniel S Brown, Michael A Goodrich, Shin-Young Jung, and Sean Kerman. 2016. Two invariants of human-swarm interaction. *Journal of Human-Robot Interaction* 5, 1 (2016), 1-31.

[7] Antonio Chella, Francesco Lanza, Arianna Pipitone, and Valeria Seidita. 2018. Knowledge acquisition through introspection in human-robot cooperation. *Biologically inspired cognitive architectures* 25 (2018), 1-7.

[8] Puwadol Oak Dusadeerungsikul, Xiang He, Maitreya Sreeram, and Shimon Y Nof. 2022. Multi-agent system optimisation in factories of the future: cyber collaborative warehouse study. *International Journal of Production Research* 60, 20 (2022), 6072-6086.

[9] Florin Gheorghe Filip. 2022. Collaborative Decision-making: concepts and supporting information and communication technology tools and systems. *International Journal of Computers Communications & Control* 17, 2 (2022). https://doi.org/10.1007/978-3-031-44373-2_5

[10] Frank Flemisch, Matthias Heesen, Tobias Hesse, Johann Kelsch, Anna Schieben, and Johannes Beller. 2012. Towards a dynamic balance between humans and automation: authority, ability, responsibility and control in shared and cooperative control situations. *Cognition, Technology & Work* 14 (2012), 3-18.

[11] Björn Gernert, Sebastian Schildt, Lars Wolf, Björn Zeise, Paul Fritsche, Bernardo Wagner, Maksims Fiosins, Ramin Safar Manesh, and Jörg P. Müller. 2014. An interdisciplinary approach to autonomous team-based exploration in disaster scenarios. In *2014 IEEE International Symposium on Safety, Security, and Rescue Robotics (2014)*. 1-8. https://doi.org/10.1109/SSRR.2014.7017655

[12] Emmanuelle Grislin-Le Strugeon. 2012. A Typology of Multi-agent Reorganisation Approaches. In *Highlights on Practical Applications of Agents and Multi-Agent Systems (Advances in Intelligent and Soft Computing, Vol. 156)*. Springer, 19-27. https://doi.org/10.1007/978-3-642-28762-6_3

[13] E. Grislin-Le Strugeon, K. Marcal de Oliveira, M. Thilliez, and D. Petit. 2022. A systematic mapping study on agent mining. *Journal of Experimental & Theoretical Artificial Intelligence* 34, 2 (2022), 189-214. https://doi.org/10.1080/0952813X.2020.1864784

[14] Jean-Michel Hoc. 2001. Towards a cognitive approach to human-machine cooperation in dynamic situations. *International journal of human-computer studies* 54, 4 (2001), 509-540.

[15] Guy Hoffman. 2019. Evaluating fluency in human-robot collaboration. *IEEE Transactions on Human-Machine Systems* 49, 3 (2019), 209-218.

[16] Roohollah Jahanmahin, Sara Masoud, Jeremy Rickli, and Ana Djuric. 2022. Human-robot interactions in manufacturing: A survey of human behavior modeling. *Robotics and Computer-Integrated Manufacturing* 78 (2022), 102404.

[17] Shu Jiang and Ronald C. Arkin. 2015. Mixed-Initiative Human-Robot Interaction: Definition, Taxonomy, and Survey. In *2015 IEEE Int. Conf. on Systems, Man, and Cybernetics*. 954-961. https://doi.org/10.1109/SMC.2015.174

[18] Arodh Lal Karn, Sudhakar Sengan, Ketan Kotecha, Irina V Pustokhina, Denis A Pustokhin, V Subramaniyaswamy, and Dharam Buddhi. 2022. ICACIA: An Intelligent Context-Aware framework for COBOT in defense industry using ontological and deep learning models. *Robotics and Autonomous Systems* 157 (2022), 104234.

[19] Shufei Li, Pai Zheng, Sichao Liu, Zuoxu Wang, Xi Vincent Wang, Lianyu Zheng, and Lihui Wang. 2023. Proactive human-robot collaboration: Mutual-cognitive, predictable, and self-organising perspectives. *Robotics and Computer-Integrated Manufacturing* 81 (2023), 102510.

[20] Giacomo Lunghi, Raul Marin, Mario Di Castro, Alessandro Masi, and Pedro J Sanz. 2019. Multimodal human-robot interface for accessible remote robotic interventions in hazardous environments. *IEEE Access* 7 (2019), 127290-127319.

[21] Dor Mizrahi, Inon Zuckerman, and Ilan Laufer. 2020. Using a stochastic agent model to optimize performance in divergent interest tacit coordination games. *Sensors* 20, 24 (2020), 7026.

[22] Arsham Mostaani, Thang X. Vu, Shree Krishna Sharma, Van-Dinh Nguyen, Qi Liao, and Symeon Chatzinotas. 2022. Task-Oriented Communication Design in Cyber-Physical Systems: A Survey on Theory and Applications. *IEEE Access* 10 (2022), 133842-133868. https://doi.org/10.1109/ACCESS.2022.3231039

[23] Thomas O'Neill, Nathan McNeese, Amy Barron, and Beau Schelble. 2022. Human-autonomy teaming: A review and analysis of the empirical literature. *Human factors* 64, 5 (2022), 904-938.
# 6 CONCLUSION

This paper described an early investigation of an intelligent assistance agent intended to improve cooperation among humans and robots in demanding circumstances. We have presented a proof-of-concept with the aim to improve cooperation between Human and robots, with the help of an IAS agent, with implementation of four levels demonstrating degrees of autonomy and cooperation, with respective to Know-how-to-Operate and Know-how-to-Cooperate model.

The experimental setup serves as a starting point for future tests and experimental evaluation, with the help of several objective and subjective metrics. Notably, the aim is to show the system's adaptation to a variety of cooperative tasks, such as line following, object manipulation, and collaborative navigation, to demonstrate its flexibility and utility. Moving ahead, efforts will be focused on improving the cooperation model and IAS agent to suit different operator competency levels and expand the scope of cooperative activities. Ongoing improvements in algorithm development, sensor integration, and real-world testing are intended to position this system as a useful tool for tackling difficult issues while supporting enhanced human-robot cooperation across several domains.

# 7 FUTURE WORK

Further research and development of the suggested cooperation model as well as the IAS agent are planned. The study trajectory involves investigating the model's adaptability across a wide range of users and testing it with a variety of human operators with varying competency levels. To provide adaptability in a wide range of applications cooperative tasks inside the system will be varied beyond pick-and-place and line following. To systematically evaluate the scalability and flexibility of the system, the complexity of the scenarios will be gradually enhanced. To validate the system in real-world scenarios and correlate its performance to real-world needs in disaster response, search and rescue, or hazardous situations, field trials will be planned. The development of algorithms, decision-making procedures, real-time monitoring, agent modeling, and human-robot cooperation will be the main areas of ongoing work. The research into the use of extra sensors aims to increase system adaptability by integrating cutting-edge technology for better situational awareness and environmental perception. The goal of this extensive future effort is to improve the system's efficacy by expanding its capabilities for a wider range of real-world circumstances and user profiles.

# ACKNOWLEDGMENTS

The present research work is supported by the CNRS and CROSSING: the French-Australian Laboratory for Humans / Autonomous Agents Teaming.

# REFERENCES

[1] C.V. Almagro, G. Lunghi, M.D. Castro, D.C. Beltrán, R.M. Prade, A. Masi, and P.J. Sanz. 2020. Cooperative and multimodal capabilities enhancement in the CERNTAURO human-robot interface for hazardous and underwater scenarios. *Applied Sciences (Switzerland)* 10, 17 (2020). https://doi.org/10.3390/app10176144

[2] José Barbosa, Paulo Leitão, Emmanuel Adam, and Damien Trentesaux. 2015. Dynamic self-organization in holonic multi-agent manufacturing systems: The ADACOR evolution. *Computers in Industry* 66 (2015), 99-111. https://doi.org/10.1016/j.compind.2014.10.011

[3] C. Bensaci, Y. Zennir, D. Pomorski, F. Innal, Y. Liu, and C. Tolba. 2020. STPA and Bowtie risk analysis study for centralized and hierarchical control architectures comparison. *Alexandria Engineering Journal* 59, 5 (2020), 3799-3816. https://doi.org/10.1016/j.aej.2020.06.036

[4] Chaima Bensaci, Youcef Zennir, Denis Pomorski, Fares Innal, Yiliu Liu, and Cherif Tolba. 2020. STPA and Bowtie risk analysis study for centralized and hierarchical control architectures comparison. *Alexandria Engineering Journal* 59, 5 (2020), 3799-3816.

[5] Juan Bravo-Arrabal, Manuel Toscano-Moreno, JJ Fernandez-Lozano, Anthony Mandow, Jose Antonio Gomez-Ruiz, and Alfonso García-Cerezo. 2021. The internet of cooperative agents architecture (X-ioca) for robots, hybrid sensor networks, and mec centers in complex environments: A search and rescue case study. *Sensors* 21, 23 (2021), 7843.

[6] Daniel S Brown, Michael A Goodrich, Shin-Young Jung, and Sean Kerman. 2016. Two invariants of human-swarm interaction. *Journal of Human-Robot Interaction* 5, 1 (2016), 1-31.

[7] Antonio Chella, Francesco Lanza, Arianna Pipitone, and Valeria Seidita. 2018. Knowledge acquisition through introspection in human-robot cooperation. *Biologically inspired cognitive architectures* 25 (2018), 1-7.

[8] Puwadol Oak Dusadeerungsikul, Xiang He, Maitreya Sreeram, and Shimon Y Nof. 2022. Multi-agent system optimisation in factories of the future: cyber collaborative warehouse study. *International Journal of Production Research* 60, 20 (2022), 6072-6086.

[9] Florin Gheorghe Filip. 2022. Collaborative Decision-making: concepts and supporting information and communication technology tools and systems. *International Journal of Computers Communications & Control* 17, 2 (2022). https://doi.org/10.1007/978-3-031-44373-2_5

[10] Frank Flemisch, Matthias Heesen, Tobias Hesse, Johann Kelsch, Anna Schieben, and Johannes Beller. 2012. Towards a dynamic balance between humans and automation: authority, ability, responsibility and control in shared and cooperative control situations. *Cognition, Technology & Work* 14 (2012), 3-18.

[11] Björn Gernert, Sebastian Schildt, Lars Wolf, Björn Zeise, Paul Fritsche, Bernardo Wagner, Maksims Fiosins, Ramin Safar Manesh, and Jörg P. Müller. 2014. An interdisciplinary approach to autonomous team-based exploration in disaster scenarios. In *2014 IEEE International Symposium on Safety, Security, and Rescue Robotics (2014)*. 1-8. https://doi.org/10.1109/SSRR.2014.7017655

[12] Emmanuelle Grislin-Le Strugeon. 2012. A Typology of Multi-agent Reorganisation Approaches. In *Highlights on Practical Applications of Agents and Multi-Agent Systems (Advances in Intelligent and Soft Computing, Vol. 156)*. Springer, 19-27. https://doi.org/10.1007/978-3-642-28762-6_3

[13] E. Grislin-Le Strugeon, K. Marcal de Oliveira, M. Thilliez, and D. Petit. 2022. A systematic mapping study on agent mining. *Journal of Experimental & Theoretical Artificial Intelligence* 34, 2 (2022), 189-214. https://doi.org/10.1080/0952813X.2020.1864784

[14] Jean-Michel Hoc. 2001. Towards a cognitive approach to human-machine cooperation in dynamic situations. *International journal of human-computer studies* 54, 4 (2001), 509-540.

[15] Guy Hoffman. 2019. Evaluating fluency in human-robot collaboration. *IEEE Transactions on Human-Machine Systems* 49, 3 (2019), 209-218.

[16] Roohollah Jahanmahin, Sara Masoud, Jeremy Rickli, and Ana Djuric. 2022. Human-robot interactions in manufacturing: A survey of human behavior modeling. *Robotics and Computer-Integrated Manufacturing* 78 (2022), 102404.

[17] Shu Jiang and Ronald C. Arkin. 2015. Mixed-Initiative Human-Robot Interaction: Definition, Taxonomy, and Survey. In *2015 IEEE Int. Conf. on Systems, Man, and Cybernetics*. 954-961. https://doi.org/10.1109/SMC.2015.174

[18] Arodh Lal Karn, Sudhakar Sengan, Ketan Kotecha, Irina V Pustokhina, Denis A Pustokhin, V Subramaniyaswamy, and Dharam Buddhi. 2022. ICACIA: An Intelligent Context-Aware framework for COBOT in defense industry using ontological and deep learning models. *Robotics and Autonomous Systems* 157 (2022), 104234.

[19] Shufei Li, Pai Zheng, Sichao Liu, Zuoxu Wang, Xi Vincent Wang, Lianyu Zheng, and Lihui Wang. 2023. Proactive human-robot collaboration: Mutual-cognitive, predictable, and self-organising perspectives. *Robotics and Computer-Integrated Manufacturing* 81 (2023), 102510.

[20] Giacomo Lunghi, Raul Marin, Mario Di Castro, Alessandro Masi, and Pedro J Sanz. 2019. Multimodal human-robot interface for accessible remote robotic interventions in hazardous environments. *IEEE Access* 7 (2019), 127290-127319.

[21] Dor Mizrahi, Inon Zuckerman, and Ilan Laufer. 2020. Using a stochastic agent model to optimize performance in divergent interest tacit coordination games. *Sensors* 20, 24 (2020), 7026.

[22] Arsham Mostaani, Thang X. Vu, Shree Krishna Sharma, Van-Dinh Nguyen, Qi Liao, and Symeon Chatzinotas. 2022. Task-Oriented Communication Design in Cyber-Physical Systems: A Survey on Theory and Applications. *IEEE Access* 10 (2022), 133842-133868. https://doi.org/10.1109/ACCESS.2022.3231039

[23] Thomas O'Neill, Nathan McNeese, Amy Barron, and Beau Schelble. 2022. Human-autonomy teaming: A review and analysis of the empirical literature. *Human factors* 64, 5 (2022), 904-938.
[24] M-P Pacaux, SA Debernard Godin, B Rajaonah, Françoise Anceaux, and Frédéric Vanderhaegen. 2011. Levels of automation and human-machine cooperation: Application to human-robot interaction. *IFAC Proceedings Volumes* 44, 1 (2011), 6484–6492.

[25] Marie-Pierre Pacaux-Lemoine and Frank Flemisch. 2019. Layers of shared and cooperative control, assistance, and automation. *Cognition, Technology & Work* 21 (2019), 579–591.

[26] Marie-Pierre Pacaux-Lemoine and Patrick Millot. 2016. Adaptive Level of Automation for risk management. *IFAC-PapersOnLine* 49, 19 (2016), 48–53.

[27] Marie-Pierre Pacaux-Lemoine and Frédéric Vanderhaegen. 2013. Towards levels of cooperation. In *2013 IEEE International Conference on Systems, Man, and Cybernetics*. IEEE, 291–296.

[28] Mateusz Paliga. 2022. Human-cobot interaction fluency and cobot operators' job performance. The mediating role of work engagement: A survey. *Robotics and Autonomous Systems* 155 (2022), 104191.

[29] H. V. D. Parunak and S. A. Brueckner. 2004. *Engineering swarming systems*. Kluwer, 341–376.

[30] João Ramos, Albert Wang, and Sangbae Kim. 2019. The brain in the machine: MIT is building robots that use full-body teleoperation to move with greater agility.

*IEEE Spectrum* 56, 6 (2019), 22–27. https://doi.org/10.1109/MSPEC.2019.8727142

[31] Tatsuya Sakai, Kazuki Miyazawa, Takato Horii, and Takayuki Nagai. 2021. A framework of explanation generation toward reliable autonomous robots. *Advanced Robotics* 35, 17 (2021), 1054–1067.

[32] Nadine B Sarter and David D Woods. 1995. How in the world did we ever get into that mode? Mode error and awareness in supervisory control. *Human factors* 37, 1 (1995), 5–19.

[33] Abbas Sidaoui, Naseem Daher, and Daniel Asmar. 2022. Human-robot interaction via a joint-initiative supervised autonomy (jisa) framework. *Journal of Intelligent & Robotic Systems* 104, 3 (2022), 51.

[34] Nicolas Spatola, Barbara Kühnlenz, and Gordon Cheng. 2021. Perception and evaluation in human-robot interaction: The Human-Robot Interaction Evaluation Scale (HRIES)—A multicomponent approach of anthropomorphism. *International Journal of Social Robotics* 13, 7 (2021), 1517–1539.

[35] Pablo Vera-Ortega, Ricardo Vázquez-Martín, Juan Jesús Fernández-Lozano, Alfonso García-Cerezo, and Anthony Mandow. 2022. Enabling Remote Responder Bio-Signal Monitoring in a Cooperative Human-Robot Architecture for Search and Rescue. *Sensors* 23, 1 (2022), 49.

[36] M. Wooldridge. 2009. *An Introduction to MultiAgent Systems, 2nd Edition*. Wiley.
[24] M-P Pacaux, SA Debernard Godin, B Rajaonah, Françoise Anceaux, and Frédéric Vanderhaegen. 2011. Levels of automation and human-machine cooperation: Application to human-robot interaction. *IFAC Proceedings Volumes* 44, 1 (2011), 6484–6492.

[25] Marie-Pierre Pacaux-Lemoine and Frank Flemisch. 2019. Layers of shared and cooperative control, assistance, and automation. *Cognition, Technology & Work* 21 (2019), 579–591.

[26] Marie-Pierre Pacaux-Lemoine and Patrick Millot. 2016. Adaptive Level of Automation for risk management. *IFAC-PapersOnLine* 49, 19 (2016), 48–53.

[27] Marie-Pierre Pacaux-Lemoine and Frédéric Vanderhaegen. 2013. Towards levels of cooperation. In *2013 IEEE International Conference on Systems, Man, and Cybernetics*. IEEE, 291–296.

[28] Mateusz Paliga. 2022. Human-cobot interaction fluency and cobot operators' job performance. The mediating role of work engagement: A survey. *Robotics and Autonomous Systems* 155 (2022), 104191.

[29] H. V. D. Parunak and S. A. Brueckner. 2004. *Engineering swarming systems*. Kluwer, 341–376.

[30] João Ramos, Albert Wang, and Sangbae Kim. 2019. The brain in the machine: MIT is building robots that use full-body teleoperation to move with greater agility.

*IEEE Spectrum* 56, 6 (2019), 22–27. https://doi.org/10.1109/MSPEC.2019.8727142

[31] Tatsuya Sakai, Kazuki Miyazawa, Takato Horii, and Takayuki Nagai. 2021. A framework of explanation generation toward reliable autonomous robots. *Advanced Robotics* 35, 17 (2021), 1054–1067.

[32] Nadine B Sarter and David D Woods. 1995. How in the world did we ever get into that mode? Mode error and awareness in supervisory control. *Human factors* 37, 1 (1995), 5–19.

[33] Abbas Sidaoui, Naseem Daher, and Daniel Asmar. 2022. Human-robot interaction via a joint-initiative supervised autonomy (jisa) framework. *Journal of Intelligent & Robotic Systems* 104, 3 (2022), 51.

[34] Nicolas Spatola, Barbara Kühnlenz, and Gordon Cheng. 2021. Perception and evaluation in human-robot interaction: The Human-Robot Interaction Evaluation Scale (HRIES)—A multicomponent approach of anthropomorphism. *International Journal of Social Robotics* 13, 7 (2021), 1517–1539.

[35] Pablo Vera-Ortega, Ricardo Vázquez-Martín, Juan Jesús Fernández-Lozano, Alfonso García-Cerezo, and Anthony Mandow. 2022. Enabling Remote Responder Bio-Signal Monitoring in a Cooperative Human-Robot Architecture for Search and Rescue. *Sensors* 23, 1 (2022), 49.

[36] M. Wooldridge. 2009. *An Introduction to MultiAgent Systems, 2nd Edition*. Wiley.