<figure><img src="image_41.png" /><figcaption>Fig. 43 Experimental environment to study human-robot cooperation in reconnaissance mission</figcaption></figure>

Evaluating workload and situation awareness in a real environment is of course more problematic than on a
simulator, unless evaluating their workload and awareness are part of participants' tasks. Concerning workload
assessment, Durso et al. proposed an over-the-shoulder subjective assessment by a subject matter expert
(SME)(Durso et al., 1998). They also used a query technique that did not have a memory component, SPAM
(Situation Present Assessment Method). The measurement consists in experimenters' detecting participants'
errors concerning specific events integrated into the scenarios. Regarding situation awareness assessment in
real situations, Jones and Endsley proposed the "real time probes" method. It consists in asking a series of
periodic questions related to the three SA levels (but without blanking the displays) and using participants'
response time to the questions as a complementary index of the SA quality. They partly validated the method
by making comparisons with SAGAT (D. G. Jones & Endsley, 2004).

Online records provide objective and subjective data that reflect human and technical agents’ activities. Focusing on their tasks, humans normally had no time to reconstruct their understanding of situation, or to imagine more than what they actually remembered doing during the experiments. However, they prevent experimenters from going deeper into the analysis to understand reasons for participants’ behavior. Offline records can mitigate such shortfalls, and various methods help aggregate online and offline data. The following sections detail these capabilities.

4.3.2.3 Offline data records

After the end of each experimental session, experimenters can again assess workload and situation awareness, however questionnaires constitute the main tool for the offline records.

One of the main methods to evaluate participants’ workload is NASA TLX (Task Load indeX). This method poses questions to participants in two phases (Hart S. G. & Staveland L. E., 1988). After each experimental session or each experimental condition, experimenters ask participants to give their feelings about six descriptors by drawing a cross on a 10cm line. Descriptors are mental demand, physical demand, temporal demand, own performance, effort and frustration level (cf. Fig. 44).
<figure><img src="image_42.png" /><figcaption>Fig. 44 Descriptors of NASA TLX</figcaption></figure>

Participants then specify between two descriptors which one is the most important for them when they complete tasks (cf. Fig. 45). These selections define a weight for each descriptor.

<table><caption>Fig. 45 Selection of descriptors according to importance perceived by participants</caption><tr><td>Physical demand / Mental demand</td><td>Temporal demand / Physical demand</td><td>Temporal demand / Frustration level</td></tr><tr><td>Temporal demand / Mental demand</td><td>Own performance / Physical demand</td><td>Temporal demand / Effort</td></tr><tr><td>Own performance / Mental demand</td><td>Frustration level / Physical demand</td><td>Own performance / Frustration level</td></tr><tr><td>Frustration level / Mental demand</td><td>Effort / Physical demand</td><td>Own performance / Effort</td></tr><tr><td>Effort / Mental demand</td><td>Temporal demand / Own performance</td><td>Effort / Frustration level</td></tr></table>

After these two steps, experimenters calculate the workload for each participant for each experimental condition using the formula presented on the equation 1.

$$WL = \sum_{n=1}^{6} (\alpha_n * R_n)$$

Equation 1: Workload calculation according to descriptors “R” and their weight “α”

We used this method for several experiments and we found that it could be useful with participants who are professional human operators, like air traffic controllers, aircraft pilots, teleoperators of robots. Indeed, they give a good representation of their work and tasks, and they are used to evaluating themselves with similar questions, especially when they perform risky tasks. However, these questions are too difficult for non-professional human operators like drivers or students.

Nevertheless, questionnaires remain the main tool we use to evaluate the quality of the interaction between participants and assistance systems, but also how participants perceive experiments, the experimental environment, procedures and evaluations. For experiments conducted in the automotive domain, we designed specific questionnaires with the help of colleagues who are experts in psychology. However, in order to have a generic questionnaire dealing with human-machine cooperation, we proposed to use the model of a cooperative agent to deduce questions. The grid presented in Fig. 33, and summarized by Fig. 46 may support the proposition of questions.
<figure><img src="image_43.png" /><figcaption>Fig. 46 Grid based on the cooperative agent model, a tool for design and evaluation of human-machine cooperation (IG: Information Gathering, IA: Information Analysis, DS: Decision Selection, AI: Action Implementation)</figcaption></figure>

In the case of the study of cooperation between driver and ADAS, participants answer by drawing a cross on a 10cm line, from low to high, in response to the following questions. The questions address participants feelings regarding their own KH, KH and KHC of assistant system. Each square of the grid on Fig. 46 can be associated to a question. Some examples are provided below:

* To what extent do you think the assistance system is able to perceive the situation?

* To what extent do you think the assistance system is able to make a good decision?

* To what extent do you think you are able to detect conflict with the assistance system (decision, action...)?

* To what extent do you feel in control of the assistance system?

* To what extent do you think the assistance system is able to evaluate your decision making?

* To what extent do you think the assistance system is able to manage conflicts with your activity?

* To what extent do you think the assistance system is able to manage authority?

* To what extent do you think you can negotiate with the assistance system?

As a conclusion for this section, we can write that experimenters have to adapt data recordings according to type of experimental environment (real vs. simulated), type of participants (professional vs. non-professional) and type of assistance system (very simple vs. complex or complicated one able to complete the four functions of the KH and the four functions of the KHC). The next section deals with the presentation of iterative and gradual ways to evaluate several states of development of the assistance system (maturity), as well as its effectiveness and resilience according to unexpected events of the process or the environment.
### 4.3.3 Assistance systems implementation

The way assistance systems are implemented in the real or simulated experimental environment constitutes the 10th step of the design and evaluation method (cf. Fig. 32, p. 64). In system engineering, designers are used to testing assistance systems in terms of their operating envelope, their robustness and technical performances. They usually refer to TRL (Technological Readiness Level). They use a typical numerical database or randomized data for assistance systems to be independent of inputs, such as input from a human operator. However, sometimes, participants have “a particular ability” for finding breaches in assistance system functioning and crashing it. Therefore, such technical tests are, of course, necessary, but tests of usability and acceptability must be the next steps. Boy deals with technology maturity, maturity of practice (at the individual level), and organizational maturity (Guy André Boy, 2018). Therefore, assistance systems can be evaluated at different levels and for different kinds of maturity. The following sections provide details about such steps and their complementarity. The descending phase of HMS design provides functions that human and technical agents have to perform to reach objectives. Designers may fully program these functions in order to test them with human participants. Nevertheless, experimenters may face several technical problems that may disturb scenarios they had prepared. Moreover, even if participants do not find breaches in the systems, some of them may be able to anticipate events that experimenters hoped were unpredictable. Several experimental techniques make it possible to avoid, or at least mitigate such problems.

The authors proposed the theater-system technique, which consists in letting both designer and participant share the same experimental environment (Schieben, Heesen, Schindler, Kelsch, & Flemisch, 2009). Each has a similar workstation that can communicate with the others, and participants can comment or answer experimenter’s questions when they are modifying assistance systems. This technique is useful when designers want to adapt an assistance system to participants. However, there is a risk of having too many individual differences between participants and not being able to design a system suitable for several human characteristics. The following techniques focus on experimental design. The first one deals with online control of scenarios in order to avoid the problems mentioned above. The second technique explains how it is possible to simulate an assistance system.

#### 4.3.3.1 Online control of scenario

We felt the need to have an online control of the scenario when we had to conduct experiments in the automotive domain to evaluate cooperation between drivers and ADAS. We carried out two main improvements to the UPHF driving simulator by adding a module that gave us the ability to have a bird eye’s view of traffic around the ego car, i.e. the car driven by the participant. The driving simulator software supported the definition of scenario according to number of cars, their type, speed and running time. However, it was not possible to modify traffic during experiments to adapt the scenario to participants’ behavior. Therefore, we added a second module to be able to select a car in the traffic and to control it through a specific interface. This new module aims to remotely control the selected car (steering wheel at the right of the experimenter on Fig. 47), or to select predefined rules for this car to have specific behavior (Marie-Pierre Pacaux-Lemoine, Simon, & Popieul, 2003). These new modules helped conduct experiments in the two projects, ST2 and ARCOS (cf. §7.2.1, p.125). However, today the driving simulator software, based on SCANeR (software of OKTAL Company) offers such abilities in the latest version.
<figure><img src="image_44.png" /><figcaption>Fig. 47 Control room of UPHF's car driving simulator</figcaption></figure>

### 4.3.3.2 Online control of assistance systems

In a research domain like the study of Human-Machine Cooperation, it would be useful to evaluate concepts of assistance systems (i.e. their main rules) in order to know if it would be interesting to design and program such types of assistance systems. How many designers programmed and implemented systems that were never used? We had the objective of simulating an assistance system when we proposed new assistance systems to fighter aircraft crews. We used the Wizard of Oz techniques, based on the children's story from 1900, in which the wizard of the wonderful land of Oz is an impostor, because he used trickery to give the illusion of power. Therefore, with these techniques, experimenters are temporally the “imposters”. Indeed, they act as if assistance systems are completely programmed systems, and so participants believe they are fully implemented.

In the project about the domain of fighting aircraft, we had to simulate several assistance systems (cf. §7.3, p.133). Two experimenters had to simulate two types of tactical assistance systems and voice recognition (M P Pacaux-Lemoine & Loiselet, 2002), i.e. KH and KHC of assistance systems. Three workplaces made up the experimental environment: one for the pilot, one for the weapon system officer and one for experimenters who controlled experiments and simulated the assistance systems (cf. Fig. 48).

<figure><img src="image_45.png" /><figcaption>Fig. 48 Wizard of Oz platform to conduct experiments with fighter aircraft crews</figcaption></figure>
Designers only programmed the KHC of the assistance systems (visual display for communication, analysis of the current participants' tasks for mutual control). They also had to program specific tools to simulate their KH (analysis of mission, updating information about missions) (cf. Fig. 49).

<figure><img src="image_46.png" /><figcaption>Fig. 49 Workstations to control experiments and play Wizard of Oz roles</figcaption></figure>

We conducted a similar type of experiment in robotics domain with the TAROT project (cf. §7.4.2, p.137). Experimenters simulated assistance systems in the first phase of the project, but tested programmed assistance systems in the second phase. Designers programmed assistance systems according to results stemming from experiments conducted in the first phase. Experimenters simulated the KH of the ground robot. First, a hidden experimenter (number 1 in Fig. 50) simulated, with teleoperation, the operational KH to move between two points using algorithms like edge and object tracking, and obstacle avoidance. This experimenter was also in charge of robot safety.

<figure><img src="image_47.png" /><figcaption>Fig. 50 Wizard of Oz platform for experiments in robotics</figcaption></figure>
Experimenters 2 and 3 in Fig. 50 simulated respectively updating a tactical assistance system with the mission (KH of robot to complete mission at the tactical level), messages from assistance system to participant (KHC of robot), and feedback about edge/object tracking (KHC of robot at the operational level). This experiment split experimenters into two areas. Therefore, two more experimenters were necessary to orchestrate scenarios (experimenters 4 and 5 in Fig. 50). We had to write scripts for each experimenter in order to synchronize their activities, and to be sure they repeated same behavior for each task and each participant (cf. Fig. 51).

<figure><img src="image_48.png" /><figcaption>Fig. 51 Example of script for experimenters who play the Wizard of Oz role for the tactical assistance system (experimenter 2 on Fig. 50)</figcaption></figure>

This section dealt with implementation of assistance systems within the experimental framework with scenario. Thanks to the implementation or simulation of assistance systems in the experimental environment, experimenters and participants can conduct experiments. Predefined data recordings provide a lot of information about several aspects of the human-machine system. The following section deals with data processing and especially the tools we designed to take advantage of all the types of data, online/offline, objective/subjective, quantitative/qualitative data.

### 4.3.4 Data processing

Data processing is the 11th step of the method (cf. Fig. 32, p.64). Many methods exist to analyze data, e.g. factorial analysis, data mining. However, few methods exist that are able to confront subjective, objective, quantitative and qualitative data. Moreover, few methods help experimenters to understand data and provide explanations or justification of results. The model of a cooperative agent may support such an analysis. The idea is to use the coding of agents' activity done according the model of cooperation to highlight interactions
With the help of projects for engineers and master's students, we programmed a piece of software on similar principles as Mac SHAPA, but adapted to allow synchronization with the driving simulator of UPHF during the ST2 project (cf. Fig. 53) (cf. §7.2.1, p.125).

<figure><img src="image_49.png" /><figcaption>Fig. 53 Software designed to code drivers and assistance systems activities</figcaption></figure>

We tried to reach the same objective with a piece of software we designed using RTMaps environment, when we conducted experiments during the SARI project. During this project, we had to conduct experiments in the real world, directly on the road. As explained in section 4.3.2.2, we needed to put hidden sensors in the road and cameras at several places, in order to record cars and drivers behaviors in specific situations, with or without assistance systems (dynamic signs). The software designed supported the synchronization of quantitative and qualitative data (cf. Fig. 54).

<figure><img src="image_50.png" /><figcaption>Fig. 54 RTMaps for SARI: synchronization of video records with vehicles data along three geographical points (position in the lane, speed, distance from other vehicles, size of vehicle...).</figcaption></figure>
The SARI project, and in particular the PhD of Ludovic Duquesne (Duquesne, 2009), addressed another critical use of data, with the confrontation of data which came from different disciplines highlighting different types of system acceptability: Individual (cognitive), social, economic and legal acceptability. The fact that what is acceptable from one point of view is not obviously acceptable from another means that the decisions are taken according to a multitude of criteria. We initially established one model of each acceptability by direct discussions with experts. All of these models were then synthesized in an inter-acceptability global model which considers all the acceptabilities together. Based on a multi-criterion decision aid methodology, some classification must be made, leading to a reading grid which will help a decision maker to choose an action (Duquesne, Millot, Pacaux-Lemoine, & Crevits, 2009).

### 4.3.5 Experiments results and their impacts on the Human-Machine Systems

The 12th step of the method (cf. Fig. 32) is the result of all the previous steps. Experimenters may describe the experimental HMS. We will not focus on results here, having already detailed all kinds of results in the section about data records (cf. §4.3.2). This section deals with the impact of results on the descending phase of design. Three main feedbacks aim to update boxes of the design phase.

The first round of feedback (13th step of the method) validates the HMS and updates current models of human operators, assistance systems and processes according to new achievements.

The second round of feedback (14th step of the method) highlights unsuitable aspects in the design of the HMS. Therefore, designers must modify such aspects that may deal with new human tasks, or new assistance tasks, or new Common Work Space, or new organizations. The data processing stage and the proposed tools provide precise information about unsuitable aspects, so designers know precisely where modifications need to be made.

The third round of feedback (15th step of the method) underlines the necessity for participants to have more training with their new tasks, regarding their cooperation with assistance systems, or with the process.

Therefore, the proposed method is an iterative method that experimenters can apply several times until the human-machine system reaches fixed goals. Fig. 55 presents such an iterative procedure based on four main characteristics: the design phase, the evaluation phase, the field of possibilities and the number of iterations. The spiral refinement brings attention to the long design and evaluation phases at the start of HMS design due to the high number of possible systems that could fit the objectives. Nevertheless, the more experimenters iterate the procedure, the more they reduce the number of possible systems and the less phases are long.
<figure><img src="image_51.png" /><figcaption>Fig. 55 Iterative procedure with spiral refinement (Marie-Pierre Pacaux-Lemoine & Flemisch, 2015)</figcaption></figure>

## 4.4 Experimental dimensions

As a conclusion for this part on methodological aspects, we provide a synthesis of all the types of experiments we conducted with a presentation using three dimensions. The first one comprises of the experimental conditions, described according to the realism of the situation, which is the human activity context and type of process. The second dimension deals with the realism of human behavior that evolves from a small samples of humans, who are more or less aware of the process, and the use of new systems, to huge samples equivalent to all end-users. The third dimension is the realism of new system and, above all, its maturity. These three dimensions may help to define the right number of experiments and the right steps from one experiment to the next to be sure of the relevance of these experiments (cf. Fig. 56).

<figure><img src="image_52.png" /><figcaption>Fig. 56 Three dimensions for experimental design and evaluation in the automotive domain (adapted from (Marie-Pierre Pacaux-Lemoine & Flemisch, 2015))</figcaption></figure>
Each dimension shows examples of states, but more states in between might exist. On Fig. 56, the central point at the intersection of the three axes highlights what we could consider as an end to experiments. However, before achieving this point, several other points describing several types of experiments may exist, and experimenters have to explain and justify how they go from one point to another one in order to be sure of improvements on each axis, as well as the consistency of the successive results.

The next step should be the implementation of new systems as well as the training of human operators. Nevertheless, when implementation of the HMS is complete, the humans involved in the management or in the use of such HMS must make sure that it functions appropriately. Many characteristics of HMS may change and destabilize its proper functioning. Indeed, as made clear by the arrows of the three dimensions, they point toward an “unknown world” in which many unexpected events, good or bad, could occur. Parts of this “unknown world” will hopefully be studied in a new future.
## SYNTHESIS ON RESEARCH CONTRIBUTION

My research contribution dealt with the design aspect of HMS, as well as its evaluation (cf. Fig. 57). From the designer's perspective (purple part of the figure), we proposed three main models. The first can be considered as the basis since it defines what is a **cooperative agent**. The definition uses the concepts of **Know-How** and **Know-How-to-Cooperate** which were presented at the beginning of my research work. The **Common Work Space** also dates from this period and aims to support cooperation between cooperative agents. As for these concepts, we constructed tools to identify what could be cooperation between two agents, and we defined **levels of cooperation**. This definition is a response to the several taxonomies of levels of automation. Therefore, we proposed a method to identify the **levels of automation** designers may need regarding the problematics they have to tackle, and we highlighted the way to make such levels **adaptable**. Therefore, we extend the model of cooperation between two agents to a model of cooperation between several agents involved at different decisional levels through the definition of the **layers of cooperation**.

<figure><img src="image_53.png" /><figcaption>Fig. 57 Research contributions</figcaption></figure>

The other part of my research contribution concerns the evaluation of the HMS provided by the designer. I propose the extension of a method proposed in LAMIH. **The method** is based on the previous models for the design and lists several existing or already developed tools for the evaluation phase.

Nevertheless, the HDR is only the state of the research activity at this specific time and even if this work had already been welcomed by the research and industrial community, many other improvements could still be made. The next chapter presents my current intentions regarding such improvements. I am not sure that all the points will be studied, as this will depend on opportunities in terms of projects and collaborations.
**Chapter 5. Objectives for the future**

"It is probably true quite generally that in the history of human thinking the most fruitful developments frequently take place at those points where two different lines of thought meet. These lines may have their roots in quite different parts of human culture, in different times or different cultural environments or different religious traditions: hence if they actually meet, that is, if they are at least so much related to each other that a real interaction can take place, then one may hope that new and interesting developments may follow".
Heisenberg (1962), *Physics and Philosophy*.

"Myth embodies the nearest approach to absolute truth that can be stated in words", A. K. Coomaraswamy, *Hinduism and Buddhism*, 1943.

"(robot) Daneel said, "There is a law that is greater than the First Law. A robot may not injure humanity or, through inaction, allow humanity to come to harm. I think to it now as the Zeroth Law of Robotics." *Robot of the dawn, Isaac Asimov, 1983*

"Robots, perhaps! They lessened the dependence of people upon people. They filled the interstices between. They were the insulation that diminished the natural attraction people had for each other, so that the whole system fell apart into isolates." *Robot and empire, Isaac Asimov, 1985*
The HDR gave me the opportunity to confirm my desire to continue the research activity I had been able to conduct as research engineer, and even increasing it, if possible, with several goals I would like to reach. As it is difficult for me to disconnect my teaching duties from my research work, in the following sections about my objectives, some teaching aspects are also described. Six interwoven objectives will be presented with two sub-parts for each one.

## 5.1 The generic models

Two main models have been proposed. The first one deals with Cooperative agents' interaction described according to the Know-How and Know-How-Cooperate of each agent, the Common Work Space to support their cooperation (cf. Fig. 23, p. 48) and its use for defining adaptability of levels of automation. The second model deals with levels and layers of cooperation (cf. Fig. 31, p. 59). The models can be improved according to two directions, the analysis of the potential symmetry between Human and machine in order to extract possible functions that can be shared, especially the KHC functions that were not studied enough. Nevertheless, such new shareable functions must be analyzed with great caution, taking in to account ethical aspects. The other improvement concerns the way we use the models in the design phase of the method.

### 5.1.1 Improvement of the models

The first direction of the improvement is to continue studying the possible symmetry between Human and machine in the system. This was a strong hypothesis when we proposed the model of cooperative agents' interaction and that was fruitful. Indeed, more comparisons could be made in order to check all the aspects in which symmetry might be possible or not. The second direction concerns the answer to the question of symmetry and deals with ethical aspects. Indeed, even if symmetry might be technically possible it might not be ethically acceptable to authorize machine to make risky decisions on behalf of humans and humanity.

#### 5.1.1.1 More symmetry between Human and machine?

With the increase in technological capabilities, system designers can imagine new possibilities for humans to interact with smart systems and delegate some of human functions or tasks to these systems. The interactions and support tools provided by the systems have the potential to be so useful that some authors already argue that they should be considered as symbiotic (Griffith, 2006; Licklider, 1960; Romero, Bernus, Noran, Stahre, & Fast-Berglund, 2016; Tzafestas, 2006). Symbiosis is well known in biology, but if we look at the definition proposed in the Cambridge, Oxford or Larousse dictionaries, we can understand that symbiosis relates to a tight, close, and harmonious relationship that is mutually beneficial yet can lead to vital dependence. From our perspective, this approach implicitly starts to generate some risks. Some have already been identified, like affective (with friends), cognitive (e.g. memory, planning support), and psychological (fear of not having access to the machine) dependencies that might emerge as a result of excessive symbiosis (Brangier et al., 2010).
<figure><img src="image_54.png" /><figcaption>Fig. 58 Where is the equilibrium point for efficient cooperative activity?</figcaption></figure>

Nevertheless, we could adopt a cautious approach to try to improve symmetry between human and machine without falling into risky symbiosis (M.-P. Pacaux-Lemoine & Trentesaux, 2019). We already addressed the notion of trust in the previous chapters, trust hits the human perception of the machine. Quite recently, regarding automation, Balfe et al. identified several characteristics that can contribute to trust (Balfe et al., 2015). The automation must be reliable, competent, visible, observable, understandable, directable, robust, and accountable. Moreover, the automation must provide proactive control and prevent skill degradation. Another kind of topic to approach such characteristics is the notion of transparency.

Transparency:

This concept is not new in Human-Machine System design and analysis. Sheridan and Verplank already explored transparency in the case of study of undersea teleoperation (T. B. Sheridan & Verplank, 1978), Lauwers and Lantz also cited it in regards to shared window systems (Lauwers & Lantz, 1990), while Rasmussen and Pejtersen dealt with transparency of functionality (Rasmussen, Pejtersen, & Schmidt, 1990). The concept of transparency has previously been used in the design of several types of system and may address different parts of the Human-Machine System, e.g. the machine reasoning (Van Dongen & Van Maanen, 2013), the interaction with machine (Stubbs, Hinds, & Wettergreen, 2007) and the process (Grote, 2009). Transparency is included in guidelines for the design of complex processes, e.g. in Air Traffic Management (Dorneich, McGrath, Dudley, & Morris, 2013), in Air Traffic Control (Göritzlehner et al., 2014; Jans, Borst, Paassen, & Mulder, 2019), in robotics (J. Y. C. Chen & Barnes, 2014; Coppin & Legras, 2012), in manufacturing (Guerin, Rauffet, Chauvin, & Martin, 2019) and in car driving (Debernard, Chauvin, Pokan, & Langlois, 2016). It was recently used to address cooperative activity between several actors within the framework of crisis management (Coletti, Mays, & Widera, 2017; Donovan, Salmon, Horberry, & Lenné, 2018). The importance of transparency has now been demonstrated more and more with the once again increasing capabilities of machines. Correlations have been done with degrees of automation (Chien, Lewis, Sycara, Kumru, & Liu, 2019), with a focus on human-robot interaction (Barnes et al., 2019), and its usefulness regarding the potential impact of Artificial Intelligence and especially machine learning abilities (Fallon & Blaha, 2018).

I would like to address this concept by conducting a deeper analysis of all the references previously mentioned in order to improve the models regarding the transparency of machine. However, particular attention could also be drawn to the transparency of “Human for other Humans”, but for machines too. Machines may improve their models of humans if they are more transparent, though ethical questions might be raised.
5.1.1.2 More criteria for function allocation?

The main criteria used in the studies we conducted were agents’ competence, workload and awareness. More
criteria can be identified and used to manage function or task allocation. Other measurements can also be used
to evaluate criteria, like for example mental workload (W. Chen, Sawaragi, & Horiguchi, 2019). Indeed, some
types of measurements can be better adapted to the type of user/participant involved in the experiments, and
to the type of process and environment studied in a project.

As for other criteria, we already mentioned human transparency, but that must be covered by ethical aspects.
Other criteria could be sought and offered to humans, like user preferences (Alexandre, Navarro, Reynaud, &
Osiurak, 2019). Another way to try to find the most suitable function allocation would be to adapt the
transparency of machine to the awareness needs of the user. As we saw above, not enough transparency may
affect Human-machine cooperation due to a lack of information about the other and then a decrease in
awareness and trust. However, too much transparency may affect cooperation because the other is too present
and disturb the individual activity. In Fig. 27 presented in §3.3 (p.51), we suggested using an extension of the
model of Rasmussen to detail three types of cooperative behavior between human and machine; cooperative
activities can be skill-, rule- or knowledge-based as for individual activities. Moreover, we consider that these
behaviors might be present on each layer of cooperation (M.-P. Pacaux-Lemoine & Flemisch, 2016) and on
each level of activity (Abbink et al., 2018). In a study on car driving, Navarro et al. observed the improvement
in recovery maneuvers with a motor priming device that is associated to the motor cue with proprioceptive
pre-activation of the gesture (Navarro, Mars, Forzy, El-Jaafari, & Hoc, 2010). Motor priming can be
considered as a skill-based cooperative behavior. Human supervision of a robot that is moving in a risky
environment may be considered as a cooperative behavior based on rules. Indeed, the human knows the
cooperative rules and may control the robot’s respecting of such rules to be sure that its actions are appropriate
regarding the objectives. Examples of cooperative behavior based on knowledge are numerous since this
happens each time a human learns how to interact and cooperate with a new machine. Human-machine (UK
Ministry of Defence Crown, 2018), robot (Dias et al., 2008), agent (Barnes et al., 2019; J. Y. C. Chen &
Barnes, 2014; Wohleber, Stowers, & Chen, 2017), autonomy (J. Y. C. Chen, Barnes, Selkowitz, & Stowers,
2016), -AI (Abbass, 2019), swarm (Hussein et al., 2018) teaming deal with such an objective: how can we
build a team between human and technical agent similarly to all the examples we met between humans in Air
Traffic Control, in Military robotics and in two-seater fighter aircraft? The studies referenced above need to
be analyzed further in order to extract perhaps more criteria. Nevertheless, I would like to go deeper into the
analysis of cooperation between layers of cooperation as we began with Makoto Itoh (Marie-Pierre Pacaux-
Lemoine & Itoh, 2015) and with Frank Flemisch (Marie Pierre Pacaux-Lemoine & Flemisch, 2018) in order
to improve models and support tools we will design in the future for crisis management, but also with in the
new projects on manufacturing management (cf. §7.6.3, p. 152) and in railway traffic management (cf. §7.7,
p. 154). Such an objective has recently been proposed in the case of earth-space cooperation with the Multi-
Layer Human-Robot Teaming concept (Drepper, Baltzer, Cowley, & Flemisch, 2019).
Ethics:

While Human-machine teaming is an interesting objective, we must nevertheless, as we did for Human-machine symbiosis, take ethical issues into consideration. With Damien Trentesaux, we have recently started to use the model of cooperative agent to identify ethical risks in Human-machine interaction (M.-P. Pacaux-Lemoine & Trentesaux, 2019). Some ethical risks have been identified, such as programmed common sense and bad conscience, decisional ambiguity, mutual interaction, master-slave dependency, emotional dependency, but more work is needed on these aspects. Fortunately, more and more research groups and studies are tackling ethical risks, such as:

* European Group on Ethics in Science and New Technologies (European Group on Ethics in Science and New Technologies, 2018),

* Center of New America Security and Technology and National Security Program on autonomy in weapon systems (Scharre & Horowitz, 2015),

* IEEE regarding ethics of autonomous and intelligent systems (Chatila & Havens, 2019),

* The Royal Academy of Engineering in the UK on Autonomous Systems and the social, legal and ethical Issues (Engineering, 2009),

* and in France, Allistene, which is the alliance of digital sciences and technologies, supports economic and social changes due to digital technology, and in particular with robotics within the framework of CERNA, "Commission de réflexion sur l'Éthique de la Recherche en sciences et technologies du Numérique d'Allistene" (Cerna Collectif, 2014).

Several ethical principles and democratic prerequisites have been written and explained by the European Group on Ethics in Science and New Technologies: "While there is growing awareness of the need to address such questions, AI and robotics are currently advancing more rapidly than the process of finding answers to these thorny ethical, legal and societal questions". The principles and prerequisites address: — Human dignity, — all 'autonomous' technologies regarding delegation of decisions and actions, transparency and predictability, — responsibility, — justice, equity, and solidarity, — democracy, — rule of law and accountability, — security, safety, bodily and mental integrity, — data protection and privacy, — sustainability. Some of these principles haven been mentioned when the models of cooperation were described, but several ones have to be tackled. We have to be aware and integrate the results from all works and issues from these institutes and commissions. That is an important objective for the future if we want to continue with our research on Human(s)-Machine(s) Cooperation, still with a multi-disciplinary approach, even more extended than what we have already done (e.g. added value provided by biological, legal, economical aspects). This will be taken into consideration, especially with the design and evaluation methodology, and the way we use the models.

5.1.2 Improvement in the use of the models

One of the results of our studies is that further information about the human working environment is required, in particular with a better and wider consideration of the multi-level and hierarchical aspects of this working environment. We started to go deeper and wider in the analysis of the working environment for which we have to design HMS within the framework of the HUMANISM project (cf. §7.6.2, p.150) and the ongoing **PhD of**
**Quentin BERDAL.** The main objective of the project is to put forwards a methodology for designing cooperative assistance systems to support Humans in Industry 4.0, especially regarding risks for them in facing complex and uncontrollable systems. The Human-Machine cooperation approach and the way we use our models could be reinforced by the Cognitive Work Analysis approach (Berdal, Pacaux-Lemoine, Trentesaux, & Chauvin, 2019)(cf. Fig. 59).

Cognitive Work Analysis (CWA) was already suggested a long time ago now (Rasmussen et al., 1990) and has been used in several domains of application (Borst, Mulder, & Van Paassen, 2010; Chauvin, Rauffet, Berruet, & Lassalle, 2015; Dadashi, Wilson, Sharples, & Golightly, 2009; Debernard et al., 2016; Horiguchi, Burns, Nakanishi, & Sawaragi, 2004; Morineau & Flach, 2019; Rauffet, Chauvin, Morel, & Berruet, 2015; Neville A Stanton & Bessell, 2014). We are using the CWA approach to define tasks and functions of humans and machines in the case of the control of a flexible cell composed of self-organized systems (smart products) supplied in raw materials and unloaded by mobile robots (Marie-Pierre Pacaux-Lemoine & Trentesaux, 2019). Nevertheless, in the CWA approach, cooperative functions are missing, such as strategies for task allocation, cooperative tasks, detection and management of interferences, as well as cooperation tools. Several points have already been detailed in the HMC approach. Therefore, we would like to analyze how we can combine both approaches to propose a more complete method, especially in domains of application that are vast and complex. A careful reading of previous references may help extract modifications of CWA that have already be made, and help propose added steps.

<figure><img src="image_55.png" /><figcaption>Fig. 59 Use the complementarity of CWA and CMH to propose a stronger method to design HMS, adapted from (Marie-Pierre Pacaux-Lemoine & Trentesaux, 2019).</figcaption></figure>

For example, such a study will be also useful to conduct projects that we have recently started in the railway industry (cf. §7.7, p.154) and will start soon in a clothing factory (cf. §7.6.3, p.152). In both projects, we have to analyze current human activity and existing machines in order to offer and test new jobs and new systems. In the case of manufacturing systems, a lot of work has been carried out regarding formalization and use of competencies, and all levels of activity of the factory are involved. Such work may be useful for having a better overview of the needs (e.g. (Boucher, Bonjour, & Grabot, 2007)).
Simulation:

Another interesting way to improve the use of the models is the ability to use the KH and KHC functions, as well as criteria for allocation, as inputs for a simulator to validate their completeness and complementarity from a technical point of view. The flowchart presented in Fig. 24 and Fig. 25 (p.53) can be programmed and executed with several scenarios in order to support the HMS designer. Software environments might support such an objective, with different types of algorithms or methods. Studies in manufacturing systems are used to design such environments in order to support designers to plan human and machine activity taking into account criteria, like for example human fatigue and reliability (e.g. (Mouayni, Demesure, Bril-el Haouzi, Charpentier, & Siadat, 2019)), or variability of scenario and uncertainty of data (e.g. (Chaabane, Chaari, & Trentesaux, 2009)). FRAM (Functional Resonance Analysis Method) model will be studied again in order to evaluate its suitability for our problematic (e.g. (Hirose & Sawaragi, 2020)). However, the implementation of our models in a simulator may also support the online control of task/function allocation during the HMS execution. Methods using petri nets (e.g. (Idris, Enea, & Lewis, 2016)) and state machines (e.g. (Benloucif, Popieul, & Sentouh, 2016; Rachedi, Berdjag, & Vanderhaegen, 2013)) have already given interesting results. This objective seems to be similar to the one dealing with digital twin and virtual commissioning in manufacturing systems. Digital twin can simulate a process and the agents involved in this process control (e.g. (Borangiu, Trentesaux, Leitão, Giret Boggino, & Botti, 2020)). Virtual commissioning aims to evaluate system functionality, performance and safety before its physical assembly and commissioning. In addition, consideration must be given to work dealing with serious games (e.g. (Alvarez et al., 2014; Marie-Pierre Pacaux-Lemoine & Mareschi, 2015; Sawaragi, Fujii, Horiguchi, & Nakanishi, 2016)), especially with a comparison with what we have already achieved with micro-worlds and simulated environments. It would be interesting to complement serious game use with the abilities we developed with the Wizard of Oz and theater technics presented in §4.3.3.2. In general, it would be necessary to study the different types of simulation, their advantages/disadvantages regarding the objectives of researches.

## 5.2 From generic models to implementation

The previous section aimed to improve our models and the methods that may lead to more precise and better adapted models. Nevertheless, one of the interests in having model is to instantiate and implement it to design systems. The following sections propose two main ways that may support the transfer from generic models to functional systems. Both ways are based on research and techniques from two different disciplines, control engineering and computer science.

### 5.2.1 Choice of control laws

Since our models have identified the KH and KHC that humans and assistance systems are supposed to have in order to cooperate and reach their common goals, the next steps are the implementation of these abilities. Concerning humans, they have to learn and train themselves to achieve the requested tasks and to cooperate with assistance systems. As for assistance systems, they must be programmed by designers. Control engineering can propose many tools to control systems, and most of the time designers use the tools they know or were handled by colleagues involved in the same project, but not the approach that is best suited to the model that is being implemented.
<figure><img src="image_56.png" /><figcaption>Fig. 60 Know-How (deep blue) and Know-How-to-Cooperate (light bleu) of robot translate to control laws: example with Model Predictive Control</figcaption></figure>

In the project named SUCRé (cf. §7.4.3, p.140), we chose the Model Predictive Control to implement the KH and KHC of robots, but other kinds of implementation might have been better (cf. Fig. 60). We did not have the time to discuss the choice of other techniques with colleagues who were involved in the study, but that is something I would like to explore in the future within the framework of national and international research groups.

The discussions can be conducted to be beneficial for all researchers involved. On one hand, Human-Machine Cooperation approach is oriented to suit control models. On the other hand, problematics from Human-Machine Cooperation approach may question control model possibilities and may lead to their improvement or their extension.

Therefore, control engineering may provide adapted tools to improve the KH and KHC of the machines, but as we already saw in the analysis of the different taxonomies of levels of automation, a mix between control engineering and computer science approaches can be achieved. Both disciplines may provide useful and complementary tools to reach the goals presented previously. These tools are detailed in the next section.

### 5.2.2 Computer sciences approaches

Computer science objectives, studies and tools may help take advantage of the capacity of Artificial Intelligence systems, Multi-Agents Systems (MAS) and Object-Oriented Programming to complement control engineering models. Such an objective is not new but has rarely been fully implemented. However, thanks to the software environments of both disciplines, which are now beginning to become closer, combination could be implemented and evaluated. This is especially the case when researchers tackle definitions of levels of automation, levels of autonomy and Human-AI cooperation (Abbass, 2019). Different levels of intelligence have also been defined that use foundations from both disciplines (Wang et al., 2019).
They are described as the following:

* reflexive intelligence for sensory-driven intelligence,

* imperative intelligence for trigger-, time-, event-driven intelligence,

* adaptive intelligence for analogy-based, feedback-modulated, regressive-optimized, environment-awareness intelligence,

* autonomous intelligence for perceptive, problem-driver, goal-driver, deductive intelligence,

* cognitive intelligence (knowledge-based, learning-based, inference-driven, inductive intelligence)

Another interesting point is that researchers from MAS are accustomed to organizing functions between several agents, definitely artificial agent most of the time, but the connection and communication are mature. In addition, in control engineering studies, we have an increasing need of these abilities to enable cooperation between several humans and artificial agents from several decisional levels. MAS may help implement such a cooperation.

I had intended to pursue this objective some years ago but did not have the ability to go deeper in that direction (Marie-Pierre Pacaux-Lemoine & Grislin-Le Strugeon, 1998). I hope that the new attempt I initiated in LAMIH with the set-up of the AiAIDo project to respond to the France – Germany –Japan trilateral call dealing with Artificial Intelligence will succeed. We are focusing our attention on the objectives to build a bridge between the human-centered perspectives and the technical and artificial intelligence centered perspectives, with a particular focus on trust, ethics, but also controllability, management of interferences between Human and AI system. We would like to develop and use a new way to anticipate and/or to manage conflicts between human and AI system based on the principles and techniques of the martial art AiKiDo (F. O. Flemisch et al., 2020).

<table><caption>Fig. 61 Example of the use of a different approach, e.g. AiKiDo approach, to analyze and manage Human-AI interferences</caption><thead><tr><td>Human-human interference (AiKiDo)</td><td>Generic principles</td><td>Human-machine systems interference (AiAIDo)</td></tr></thead><tbody><tr><td>From competition to cooperation, from opponent to partner, all joined to reach common objective</td><td>From Competition to Cooperation</td><td>Human and machine must have and reach same goals, e.g. by embedding explicit or implicit goals into the machines</td></tr><tr><td>Escalation- and de-escalation path from harmless to more dangerous back to harmless techniques</td><td>Ai..Do</td><td>Identification of dangerous patterns, development of benevolent and effective human-machine patterns</td></tr><tr><td>Tori uses Uke's momentum and strength to get into the center of the attack, tries to control the situation towards a solution</td><td>Controllability from Instability to dynamic stability to harmony (Irimi)</td><td>Arbitration mediator orchestrates a dynamic stabilization of the Human-machine system</td></tr><tr><td>Tori manage attacks from several Uke (Randori)</td><td>From one-one to several-several</td><td>Basic principles from One Human/One machine system to multi-human/multi-AI systems</td></tr><tr><td>No exclusion, black and white belts work together</td><td>No exclusion Adaptiveness</td><td>Design for all with adaptation and adaptability</td></tr><tr><td>Tori/Uke must remain in visual, touch and haptic contact to know other intends</td><td>Keep in contact, transparency</td><td>Human and machine are coupled in a multimodal way and communicate intents</td></tr><tr><td>Tori/Uke must not only focus on the other, but develops a mental model of themselves and environment as a common system</td><td>Meta-System-SA (Situation awareness)</td><td>Humans and machines (e.g. AI) build model of themselves, of others and the environment to predict and harmonize actions and results towards common goals</td></tr><tr><td>Tori/Uke know techniques and patterns to counter any attack</td><td>Cooperative patterns</td><td>Human and machines have cooperative, compatible patterns to interact</td></tr><tr><td>From Uke close to Tori to learn, to Uke farther from Tori to attack faster and stronger</td><td>From simple to complex</td><td>With training human can cooperate with more complex systems.</td></tr></tbody></table>
Multi-agent systems and object-oriented object programming might help consider both Human and machine as similar agents, from the program's point of view, and with humans represented by a kind of “ avatar ”. Nevertheless, these computer science approaches usually consider humans as a disturbance who are removed from the general system most of the time. Discussions, mutual understanding, common objectives and cooperative work in system implementation and evaluation might address reticence and allow very interesting common studies.

Another field of computer science that can also be connected is that which deals with human-machine interface
design. This field is presented in the next section.

**5.3 From Common Work Space to Human-machine interface**

The models and methods previously mentioned lead us to identify several aspects of Human-machine
interaction, and one is the Common Work Space, it supports the external KHC by allowing the collection of
information about process and other agents involved in the process control. It supports the implementation of
actions as well. In our studies, and as I have already written in this document, we did not focus our research
on the ergonomics aspects of the interface available to humans, since we focused on function allocation.
Nevertheless, we identified information that must be observable and handled by humans through an interface.
Even though we always develop interfaces to implement CWS, cooperation with HMI experts would help
improve interaction between human and machine.

5.3.1 Human-machine interface

Several attempts have already been made to use other kinds of interaction instead of the usual visual and sound ones. The project in robotics applied to disabled users, offering them Emulated Haptic Feedback (cf. §7.4.4, p.143). Thanks to a collaboration with the University College of London and the associate professor Tom Carlson, we were able to support cooperation between human and robot by providing an adapted interface, and compensating for the inability of some individuals to control robot except with their brain. Human and robot were able to communicate through the visual interface. It took into account the mental effort of the human to share commands sent to robot (cf. §7.4.4, p.143).

<figure><img src="image_57.png" /><figcaption>Fig. 62 Emulated Haptic Feedback interface</figcaption></figure>
This project highlighted the great importance of the interface, which is sometimes neglect by control engineering designers. I would like to go deeper in the connection between the information we identified with our design method based on our model, and the access to such information and action through interfaces. Various interfaces have been used in each project, sometimes based on existing interfaces like in air traffic control, sometimes completely reimagined like in the human-robots cooperation in the SUCRé project. New displays have been designed, like smart clothes that can support cooperation between robots and firefighters who may not be able to use their hands and watch screens. Haptic feedback is another interface I would like to analyze deeper since it is rich in information, e.g. (Devigne et al., 2019). Cooperative patterns could be studied through haptic feedback using a stick in the trilateral France - Germany - Japan project we mentioned, but other kinds of device could be imagined, such as the one put forwards by the Disney research lab. They propose a force jacket which is a “pneumatically-actuated jacket for embodied haptic experiences” (Delazio, Nakagaki, Klatzky, Hudson, & Lehman, 2018). Similar techniques could be used to control natural gesture of Human when sending a command with hand and wrist movement, for example.

These examples deal with the interface between one human and one machine, but several projects underline the increasing need to design interfaces adapted to several humans who have to cooperate, humans who may be assisted by several kinds of machine. This objective is presented in the next section.

### 5.3.2 Human(s)-machine(s) interface(s)

The need to design interfaces for several humans who have different roles and authorities, but who have to reach a common goal, was especially evident in crisis management projects. In the SUCRé project, the analysis of firefighters' activities highlights such a need. That was due to lack of information shared between the staff who are in the field, in the command post vehicles and in the crisis unit. In this example, there were only three kinds of units among all units that might be deployed in case of crisis. Such an objective has already been mentioned above with the concept of transparency. Nevertheless, in this part I would like to address the technical tools designers can use to implement transparency, namely the kind of interface that can support layers of cooperation.

We started to test the ability to use a tabletop with tangible objects within the framework of a LAMIH project in order to design a cooperative environment for several humans to make strategic or tactical decisions, and to send commands through the modification of position of objects moving on the table (cf. Fig. 63). We proved the technical feasibility. Nevertheless, improvements are necessary. Moreover, experiments must be conducted with well-adapted experiment protocol and several teams of participants. This type of equipment could be very useful in crisis management, especially in the case of firefighters when, in the command post vehicle, several officers may have to make a collective decision. The interest in tangible objects is mainly to make clear, to all those who must be involved in the decision, the intentions of people or machines. Indeed, people handle objects to analyze and decide. The tangible tabletop is one technical technique but other kinds of facilitative tools could be interesting too, like the collaborative transparent displays (Li, Greenberg, & Sharlin, 2017).
<figure><img src="image_58.png" /><figcaption>Fig. 63 Tangible tabletop – collective decision making (Merrad, Habib, Héloir, Kolski, & Krüger, 2019; Merrad, Héloir, & Kolski, 2017)</figcaption></figure>
# Chapter 6. General conclusion

## 6.1 Synthesis on the research perspectives

The research objectives presented in this manuscript are synthesized in Fig. 64. As for the research contributions, they still concern the design and the evaluation aspects. It is indeed difficult, for me, to improve one aspect without thinking on how to improve another. With the increasing of abilities and capacities of systems, defined in terms of *Know-How* and *Know-How-to-Cooperate*, new challenges must be considered when designing systems, such as *ethical risks* (in purple on Fig. 64). Nevertheless, such risks can be studied partly through improvements we could make with focus on the *transparency* of cooperative agents and the *ergonomics* of the Human-Machine Interface. Other improvements might also concern tools for designers, such as the use of *generic models*, the *simulation* of the instantiation of these models to check their appropriateness according to objectives, and the way they can be *implemented* in the Human-Machine System.

<figure><img src="image_59.png" /><figcaption>Fig. 64 Synthesis on research perspectives</figcaption></figure>

In connection with the objectives of the HMS design, the evaluation phase (in green on Fig. 64) may take advantage of the generic models to improve the *on/off line analysis* of the cooperative agents' activities. In addition, a deep analysis of the possibilities provided by the simulators must be explored, like the *simulation of assistance systems*, the *simulation of the technology* (car, train, aircraft, process) that has to be assisted, or the *simulation of the environment* in which the technology is running. Each type of simulation can be improved as well as their interconnections.

The last objective, I would like to present, is the **multi-disciplinary approach**. We already had some work that brought together several disciplines, but they were casual and dependent on opportunities provided by projects. Deeper work could be conducted in order to go forwards with a more integrative methodology, in which disciplines are not only juxtaposed, but parts of their concepts may be shared, perhaps made collaborative and/or may enrich each other.
## 6.2 Perspectives on teaching aspects

The teaching aspects have not been described in detail, but they formed part of most of the studies we conducted, especially when some systems were developed and tested with students within the framework of internships or student projects in the lab. One perspective is that this enables us to reach a better balance and organization between lectures, exercises, practical works and projects (cf. Fig. 65). Indeed, since the presentation of research projects during lectures arouses the curiosity of students, an interesting way to maintain curiosity is to explain how exercises and practical work may provide the tools to be able, one day, to join such research projects. Therefore, the continuity between the different types of courses is important and may justify to students the involvement professors are requesting.

<figure><img src="image_60.png" /><figcaption>Fig. 65 Perspectives on teaching aspects</figcaption></figure>

As I had started to do last year, I would like to set up courses in which students are sometimes designers, sometimes users and sometimes experimenters. Organizing this is not easy since students, who are members of a same group, know each other and usually do not want to penalize other students. Therefore, the scoring for evaluation is difficult to determine and to implement. Nevertheless, the techniques used during the evaluation phase of the method presented in chapter 4 may contribute to reaching such an objective.

As was the case for the research perspective, this teaching perspective also deals with multi-disciplinary projects. We know that students must be good in the disciplines they are learning, in computer and control engineering and signal processing, i.e. the 61st section of the French National Board of Universities. However, in their future jobs they will have to work with people with other competencies. Even though courses about project management are already taught, with a presentation of what other disciplines may bring, it could be beneficial for students to experiment exchange of information, organization of cooperative tasks and achievement of objectives within the framework of lab or industrial project.
6.3 Final conclusion

That is the end of the manuscript, but not the end of my desire to have deeper involvement in the research and teaching activities. On the contrary, this manuscript is not only a synthesis of the contributions gathered for several years (152 documents among which 15 articles in journals and 70 articles in international conferences), in 14 research projects, with 10 PhD students, with numerous and fruitful contacts with French and foreign researchers, French technicians and engineers from labs and industries. It is, in fact, several syntheses correlated, through the manuscript, to provide complementary tools for the design of Humans-Machines Systems (2nd and 3rd chapters) and their evaluation (4th chapter). These three chapters push the synthesis forward a stronger and stable state of my thoughts.

The manuscript also highlights connections between teaching and research, respectively linked to the use and the improvement of computer and control engineering tools. Moreover, these reframed bases underline the research directions I would like to explore, some of which I have already begun to do so. Nevertheless, with the "Habilitation à Diriger des Recherches", I would be able to adapt "the level of autonomy" I am able to accord myself, from no responsibility to full autonomy and responsibility, depending on the topics research projects and calls may offer, and in agreement with the heads of the institutes with whom I will work.

To conclude, I would like to reiterate one of the objectives presented in the introduction. The outcome of the research activity, as well as the concepts, methods and abilities provided to students have to converge towards real advances for humans, for humanity and for its environment.
## References

Abbass, H. A. (2019). Social Integration of Artificial Intelligence: Functions, Automation Allocation Logic and Human-Autonomy Trust. *Cognitive Computation, 11*(2), 159-171.
https://doi.org/https://doi.org/10.1007/s12559-018-9619-0

Abbink, D. A., Carlson, T., Mulder, M., De Winter, J. C. F., Aminravan, F., Gibo, T. L., & Boer, E. R. (2018). A topology of shared control systems-finding common ground in diversity. *IEEE Transactions on Human-Machine Systems, 48*(5), 509-525. https://doi.org/10.1109/THMS.2018.2791570

Abbink, D. A., Mulder, M., & Boer, E. R. (2012). Haptic shared control: smoothly shifting control authority? *Cognition, Technology & Work, 14*, 19-28. https://doi.org/10.1007/s10111-011-0192-5

Alexandre, B., Navarro, J., Reynaud, E., & Osiurak, F. (2019). Which cognitive tools do we prefer to use, and is that preference rational? *Cognition, 186*, 108-114. https://doi.org/10.1016/j.cognition.2019.02.005

Allen, J., & Ferguson, G. (2002). Human-Machine Collaborative Planning. In *Third International NASA Workshop on Planning and Scheduling for Space*. Houston, TX.

Altendorf, E., Baltzer, M., Canpolat, Y., Lopez, D., Schreck, C., Weßel, G., & Flemisch, F. (2016). Automated Driving using Shared and Cooperative Guidance and Control: Revisiting past Research with a new Model. *13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems*. https://doi.org/10.1016/j.ifacol.2016.10.480

Alvarez, J., Haudegond, S., Havrez, C., Kolski, C., Lebrun, Y., Lepreux, S., & Libessart, A. (2014). From screens to devices and tangible objects: A framework applied to serious games characterization. *Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*, 8512 LNCS(PART 3), 559-570. https://doi.org/10.1007/978-3-319-07227-2_53

Anceaux, F., Mouchel, M., Paglia, C., & Richard, P. (2018). *TC-RAIL, Livrable L5.2: État des lieux de l'activité de conduite et de la téléopération et préconisations*.

Auberlet, J.-M., Pacaux, M.-P., Anceaux, F., Plainchault, P., & Rosey, F. (2010). The impact of perceptual treatments on lateral control: a study using fixed-base and motion-base driving simulators. *Accident; Analysis and Prevention, 42*(1), 166-173. https://doi.org/10.1016/j.aap.2009.07.017

Auberlet, J.-M., Rosey, F., Anceaux, F., Aubin, S., Briand, P., Pacaux, M.-P., & Plainchault, P. (2012). The impact of perceptual treatments on driver's behavior: from driving simulator studies to field tests--first results. *Accident; Analysis and Prevention, 45*, 91-98. https://doi.org/10.1016/j.aap.2011.11.020

Bainbridge, L. (1983). Ironies of automation. *Automatica, 19*(6), 775-779.

Baker, M., & Yanco, H. a. (2004a). Autonomy mode suggestions for improving human-robot interaction. *Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics, 3*, 2948-2953.

Baker, M., & Yanco, H. A. (2004b). Autonomy Mode Suggestions for Improving Human-Robot Interaction. In *Proceedings of the IEEE SMC, The Hague, Netherlands*.
Balfe, N., Sharples, S., & Wilson, J. R. (2015). Impact of automation: Measurement of performance, workload and behaviour in a complex control environment. *Applied Ergonomics*, 47, 52–64.
https://doi.org/10.1016/j.apergo.2014.08.002

Baltzer, M. C. A., & Flemisch, F. O. (2019). Interaction Patterns for Cooperative movement with vehicles. In *CHI’19 Workshop on “Looking into the Future: Weaving the Threads of Vehicle Automation.”* Glasgow, UK.

Barber, D., Davis, L., Nicholson, D., Finkelstein, N., & Chen, J. Y. C. (2008). The Mixed Initiative Experimental (Mix) Testbed For Human Robot Interactions With Varied Levels Of Automation. UNIVERSITY OF CENTRAL FLORIDA ORLANDO INST FOR SIMULATION AND TRAINING.

Barnes, M., Elliott, L. R., Barnes, M., Elliott, L. R., Wright, J., Scharine, A., & Chen, J. (2019). Human – Robot Interaction Design Research : From Teleoperations to Human – Agent Teaming. CCDC Army Research Laboratory Aberdeen Proving Ground United States.

Benloucif, M. A., Popieul, J., & Sentouh, C. (2016). Architecture for multi-level cooperation and dynamic authority management in an Automated Driving System – A case study on lane change cooperation. *13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, 49*(19), 615–620. https://doi.org/10.1016/j.ifacol.2016.10.631

Bentley, R., Rodden, T., Sawyer, P., & Sommerville, I. (1992). An architecture for tailoring cooperative multi-user displays. In *Proceedings of Computer Supported Cooperative Work, CSCW’92*.

Berdal, Q., Pacaux-Lemoine, M. P., Trentesaux, D., & Chauvin, C. (2019). Human-machine cooperation in self-organized production systems: A point of view. In T. Borangiu, D. Trentesaux, A. Thomas, & S. Cavalieri (Eds.), *Studies in Computational Intelligence* (Vol. 803, pp. 123–132).
https://doi.org/10.1007/978-3-030-03003-2_9

Biester, L., & Bosch, R. (2005). The Concept of Cooperative Automation in Cars: Result from the experiment "Overtaking on highway." In *Proceedings of the 3rd International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design* (pp. 342–348).

Billings, C. E. (1996). Human-Centered Aviation Automation : Principles and Guidelines. Moffett Field, Calif. : National Aeronautics and Space Administration, Ames Research Center.

Bolstad, C., & Endsley, M. R. (2000). Shared displays and team performance. In *Human Performance, Situation Awareness and Automation Conference* (pp. 1–6).

Borangiu, T., Trentesaux, D., Leitão, P., Giret Boggino, A., & Botti, V. (2020). Service Oriented, Holonic and Multi-agent Manufacturing Systems for Industry of the Future, Proceedings of SOHOMA 2019. *Studies in Computational Intelligence, 853*. https://doi.org/DOI=10.1007/978-3-030-27477-1

Bordel, S., Somat, A., Barbeau, H., Anceaux, F., Greffeuille, C., Menguy, G., ... Gallenne, M.-L. (2014). From technological acceptability to appropriation by users: methodological steps for device assessment in road safety. *Accident; Analysis and Prevention, 67*, 159–165.
https://doi.org/10.1016/j.aap.2014.01.016

Borst, C., Mulder, M., & Van Paassen, M. M. (2010). A review of Cognitive Systems Engineering in aviation. In *IFAC PapersOnLine* (Vol. 11). https://doi.org/10.3182/20100831-4-fr-2021.00040

Boucher, X., Bonjour, E., & Grabot, B. (2007). Formalisation and use of competencies for industrial performance optimisation: A survey. *Computers in Industry, 58*(2), 98–117.
https://doi.org/10.1016/j.compind.2006.09.004
Boy, Guy A. (2006). Conclusion and Perspectives. From Automation to Interaction Design. In *The Handbook of Human-Machine Interaction*. (pp. 431–443). London: CRC Press. https://doi.org/https://doi.org/10.1201/9781315557380

Boy, Guy André. (2018). Human-centered design of upcoming human missions to Mars. *Journal of Space Safety Engineering*, 5(1), 9–13. https://doi.org/10.1016/j.jsse.2018.01.001

Brangier, É., Hammes-Adelé, S., & Bastien, J. M. C. (2010). Analyse critique des approches de l’acceptation des technologies : de l’utilisabilité à la symbiose humain-technologie-organisation. *Revue Europeenne de Psychologie Appliquee*, 60(2), 129–146. https://doi.org/10.1016/j.erap.2009.11.002

Brézillon, P., & Pomerol, J.-C. (1999). Contextual knowledge sharing and cooperation. *Le Travail Humain*, 62(3), 223–246.

Cappaert-Blondelle, S. (2012). Metro Automation Facts, Figures and Trends. *International Association of Public Transport*.

Cerna Collectif. (2014). *Éthique de la recherche en robotique*.

Chaabane, S., Chaari, T., & Trentesaux, D. (2009). Integrated sizing support system using simulation and design of experiments. *2009 International Conference on Computers and Industrial Engineering, CIE 2009*, 735–741. https://doi.org/10.1109/iccie.2009.5223808

Chatila, R., & Havens, J. C. (2019). The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. In E. K. E. Aldinhas Ferreira M., Silva Sequeira J., Singh Virk G., Tokhi M. (Ed.), *Robotics and Well-Being. Intelligent Systems, Control and Automation: Science and Engineering* (Vol. 95). Springer, Cham. https://doi.org/https://doi.org/10.1007/978-3-030-12524-0_2

Chauvin, C., & Hoc, J.-M. (2014). Integration of ergonomics in the design of human-machine systems. In *Designing human-machine cooperation systems* (pp. 43–81). John Wiley & Sons.

Chauvin, C., Rauffet, P., Berruet, P., & Lassalle, J. (2015). Using Cognitive Work Analysis to design smart-grid interface Using. In D. de Waard, K. A. Brookhuis, A. Toffetti, A. Stuiver, C. Weikert, D. Coelho, ... M. N. (Eds.), *Human Factors and Ergonomics Society Europe Chapter 2015 Annual Conference*.

Chen, J. Y. C., & Barnes, M. J. (2014). Human-Agent Teaming for Multirobot Control: A Review of Human Factors Issues. *IEEE Transactions on Human-Machine Systems*, 44(1), 13–29. https://doi.org/10.1109/THMS.2013.2293535

Chen, J. Y. C., Barnes, M. J., Selkowitz, A. R., & Stowers, K. (2016). Effects of Agent Transparency on Human-Autonomy Teaming Effectiveness. *IEEE International Conference on Systems, Man, and Cybernetics*, 1838–1843. https://doi.org/10.1109/SMC.2016.7844505

Chen, W., Sawaragi, T., & Horiguchi, Y. (2019). Measurement of Driver’s Mental Workload in Partial Autonomous Driving. *IFAC PapersOnLine*, 52(19), 347–352. https://doi.org/10.1016/j.ifacol.2019.12.083

Chien, S. Y., Lewis, M., Sycara, K., Kumru, A., & Liu, J. S. (2019). Influence of Culture, Transparency, Trust, and Degree of Automation on Automation Use. *IEEE Transactions on Human-Machine Systems*. https://doi.org/10.1109/THMS.2019.2931755

Coletti, G. S., Mays, R. E., & Widera, A. (2017). Bringing Technology and Humanitarian Values Together : A Framework to Design and Assess Humanitarian Information Systems. In *International Conference on Information and Communication Technologies for Disaster Management*.
Coppin, G., & Legras, F. (2012). Autonomy spectrum and performance perception issues in swarm supervisory control. *Proceedings of the IEEE, 100*(3), 590–603.
https://doi.org/10.1109/JPROC.2011.2174103

Cosenzo, K. A., Chen, J. Y. C., Drexler, J. M., Barnes, M. J., Hudson, I. L., Reinerman-Jones, L., & Nicholson, D. (2010a). Dynamic Task Allocation for Secure Mobility To Enhance Soldier Effectiveness. *Proceedings of the Bi-Annual Army Science Conference*.

Cosenzo, K. A., Chen, J. Y. C., Drexler, J. M., Barnes, M. J., Hudson, I. L., Reinerman-Jones, L., & Nicholson, D. (2010b). DYNAMIC TASK ALLOCATION FOR SECURE MOBILITY TO ENHANCE SOLDIER EFFECTIVENESS. In *Bi Annual Army Science Conference*.

Cummings, M L, & Mitchell, P. J. (2007). Operator scheduling strategies in supervisory control of multiple UAVs, 11, 339–348. https://doi.org/10.1016/j.ast.2006.10.007

Cummings, Mary L. (2003). Automation and Accountability in Decision Support System Interface Design. *The Journal of Technology Studies*, 23–31.

Dadashi, N., Wilson, J. R., Sharples, S., & Golightly, D. (2009). Cognitive Work Analysis practicality and limitations in complex control systems : a case of alarm handling, (1999).

Debernard, S., Chauvin, C., Pokan, R., & Langlois, S. (2016). Designing Human-Machine Interface for Autonomous Vehicles. In *13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems*. Kyoto, Japan. https://doi.org/10.1016/j.ifacol.2016.10.629

Decortis, F., & Pavard, B. (1994). Communication et coopération : de la théorie des actes de langage à l'approche ethnométhodologique. In *modélisation à la conception*. (In Pavard). Systèmes coopératifs de la Toulouse Octarés.

Delazio, A., Nakagaki, K., Klatzky, R., Hudson, S. E., & Lehman, J. F. (2018). Force Jacket : Pneumatically-Actuated Jacket for Embodied Haptic Experiences.

Devigne, L., Pasteau, F., Babel, M., Narayanan, V. K., Guegan, S., & Gallien, P. (2019). Design of a Haptic Guidance Solution for Assisted Power Wheelchair Navigation. *Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018*, 3231–3236.
https://doi.org/10.1109/SMC.2018.00547

Dias, M. B., Kannan, B., Browning, B., Jones, E. G., Argall, B., Dias, M. F., ... Stentz, A. J. (2008). Sliding autonomy for peer-to-peer human-robot teams. *Intelligent Autonomous Systems* 10, IAS 2008, 332–341.
https://doi.org/10.3233/978-1-58603-887-8-332

Donovan, S. L., Salmon, P. M., Horberry, T., & Lenné, M. G. (2018). Ending on a positive: Examining the role of safety leadership decisions, behaviours and actions in a safety critical situation. *Applied Ergonomics*, 66, 139–150. https://doi.org/10.1016/j.apergo.2017.08.006

Dorneich, M. C., McGrath, K. a., Dudley, R. F., & Morris, M. D. (2013). Analysis of the Characteristics of Adaptive Systems. *2013 IEEE International Conference on Systems, Man, and Cybernetics*, 888–893.
https://doi.org/10.1109/SMC.2013.156

Drepper, A., Baltzer, M., Cowley, A., & Flemisch, F. O. (2019). Multi-Layer Human-Robot Teaming : From Earth to Space. In *14th IFAC Symposium on Analysis Design and Evaluation of Human Machine Systems*. Tallinn, Estonia.

Duquesne, L. (2009). UNE APPROCHE D ' AIDE MULTICRITÈRE À LA DÉCISION POUR L ' ACCEPTATION D ' UNE SIGNALISATION ROUTIERE. Valenciennes, France.
Duquesne, L., Millot, P., Pacaux-Lemoine, M.-P., & Crevits, I. (2009). APPROACH OF MULTICRITERIA DECISION AID FOR THE ACCEPTABILITY OF DYNAMIC ROAD SIGNS. *28th European Annual Conference on Human Decision-Making and Manual Control*.

Durso, F. T., Hackworth, C. A., Truitt, T. R., Crutchfield, J., Nikolic, D., & Manning, C. A. (1998). Situation Awareness As a Predictor of Performance in En Route Air Traffic Controllers. *Air Traffic Control Quarterly*, 6(1), 1-20. https://doi.org/10.2514/atcq.6.1.1

Endsley, M. R. (1995a). Measurement of Situation Awareness in Dynamic Systems. *Human Factors*, 37(1), 65-84.

Endsley, M. R. (1995b). Toward a theory of situation awareness in dynamic systems: Situation awareness. *Human Factors*, 37(1), 32-64.

Endsley, M. R., & Kaber, D. B. (1999). Level of automation effects on performance, situation awareness and workload in a dynamic control task. *Ergonomics*, 42(3), 462-492.

Endsley, M. R., & M. Robertson, M. (2000). Situation awareness in aircraft maintenance teams. *International Journal of Industrial Ergonomics*, 26(2), 301-325.

Engineering, T. R. A. of. (2009). *Autonomous Systems: Social, Legal and Ethical Issues*.

Fallon, C. K., & Blaha, L. M. (2018). Improving Automation Transparency: Addressing Some of Machine Learning 's Unique Challenges. *International Conference on Augmented Cognition*. Springer, Cham.

Finzi, A., & Orlandini, A. (2005). Human-Robot Interaction Through Mixed-Initiative Planning for Rescue and Search Rovers, 483-494.

Fitts, P. (1951). Human engineering for an effective air-navigation and traffic-control system. *Washington, DC: National Research Council*. Washington: DC: National Research Council.

Flemisch, F., Abbink, D. A., Itoh, M., Pacaux-Lemoine, M. P., & Weßel, G. (2019). Joining the blunt and the pointy end of the spear: towards a common framework of joint action, human-machine cooperation, cooperative guidance and control, shared, traded and supervisory control. *Cognition, Technology and Work*, 21(4), 555-568. https://doi.org/10.1007/s10111-019-00576-1

Flemisch, F., Abbink, D., Itoh, M., Pacaux-Lemoine, M.-P., & Weßel, G. (2016). Shared control is the sharp end of cooperation: Towards a common framework of joint action, shared control and human machine cooperation. *IFAC-PapersOnLine*, 49(19). https://doi.org/10.1016/j.ifacol.2016.10.464

Flemisch, F. O., Adams, C. A., Conway, S. R., Goodrich, K. H., Palmer, M. T., & Schutte, P. C. (2003). The H-Metaphor as a Guideline for Vehicle Automation and Interaction. *Control*, (December), 1-30. https://doi.org/10.1176/956

Flemisch, F. O., Pacaux-Lemoine, M.-P., Herzberger, N., Wasser, J., Baltzer, M., Vanderhaegen, F., ... Itoh, M. (2020). Conflicts in Human-machine systems as an Intersection of Bio- and Technosphere: Cooperation and interaction patterns for Human and Machine Interference and Conflict Resolution. In *IEEE International Conference on Human-Machine Systems*. Rome, Italy.

Flemisch, Frank, Heesen, M., Hesse, T., Kelsch, J., Schieben, A., & Beller, J. (2012). Towards a dynamic balance between humans and automation: Authority, ability, responsibility and control in shared and cooperative control situations. *Cognition, Technology and Work*, 14(1), 3-18. https://doi.org/10.1007/s10111-011-0191-6
Flemisch, Frank, Kelsch, J., Löper, C., Schieben, A., & Schindler, J. (2008). Automation spectrum, inner/outer compatibility and other potentially useful human factors concepts for assistance and automation. *Human Factors for Assistance and Automation*, (2008), 1–16.

Galster, C. L. S., Barnes, C. M., Cosenzo, K., Galster, S., Hollnagel, E., Miller, C., ... Breda, L. Van. (2007). HUMAN AUTOMATION INTEGRATION. In R. & T. O. North Atlantic Treaty Organisation (Ed.), *Uninhabited Military Vehicles (UMVs): Human Factors Issues in Augmenting the Force, RTO Technical report* (pp. 1–108).

Goodrich, M. A., Olsen, D. R. J., Crandall, J. W., & Palmer, T. J. (2001). Experiments in Adjustable Autonomy Robot Effectiveness Neglect. *Time*, 3, 1624–1629.

Göritzlehner, R., Borst, C., Ellerbroek, J., Westin, C., Van Paassen, M. M., & Mulder, M. (2014). Effects of transparency on the acceptance of automated resolution advisories. *Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics, 2014-Janua*(January), 2965–2970. https://doi.org/10.1109/smc.2014.6974381

Gould, J. D., & Lewis, C. (1985). Designing for usability: key principles and what designers think. *Communications of the ACM*, 28(3), 300–311.

Griffith, D. (2006). Neo-symbiosis: a system design philosophy for diversity and enrichment. *International Journal of Industrial Ergonomics*, 36, 1075–1079. https://doi.org/10.1016/j.ergon.2006.09.008

Grislin-Le Strugeon, E., & Millot, P. (1999). Specifying artificial cooperative agents through a synthesis of several models of cooperation. *Proceedings of the 7th European Conference on Cognitive Science Approaches to Process Control CSAPC'99*, (September).

Grote, G. (2009). Limits to Control in Automated Systems. In *Management of Uncertainty*. https://doi.org/10.1007/978-1-84882-373-0_6

Gruyer, D., Royère, C., Du Lac, N., Michel, G., & Blosseville, J.-M. (2006). SiVIC© and RTMaps©, Interconnected platforms for the conception and the evaluation of driving assistance systems. In *ITS World congress*. London, UK.

Guerin, C., Rauffet, P., Chauvin, C., & Martin, E. (2019). Toward production operator 4.0: modelling Human-Machine Cooperation in Industry 4.0 with Cognitive Work Analysis. In *14th IFAC Symposium on Analysis Design and Evaluation of Human Machine Systems* (pp. 73–78). Tallinn, Estonia.

Gutwin, C., & Greenberg, S. (1999). A Framework of Awareness for Small Groups in Shared-Workspace Groupware. Technical Report 99-1, Department of Computer Science, University of Saskatchewan, Canada.

Habib, L., Pacaux-Lemoine, M.-P., & Millot, P. (2017a). Adaptation of the level of automation according to the type of cooperative partner. In *2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017* (Vol. 2017-Janua). https://doi.org/10.1109/SMC.2017.8122718

Habib, L., Pacaux-Lemoine, M. P., & Millot, P. (2017b). A method for designing levels of automation based on a human-machine cooperation model. *IFAC-PapersOnLine*, 50(1). https://doi.org/10.1016/j.ifacol.2017.08.235

Habib, Lydia. (2019). Niveaux d'automatisation adaptables pour une coopération homme-robots. *Doctorat Spécialité Automatique, Université Polytechnique Hauts-de-France*.

Haider, A., & Catarrasi, B. (2016). *Future Unmanned System Technologies: Legal and Ethical Implications of Increasing Automation*. Kalkar, Germany: The Joint Air Power Competence Center.
Hart S. G., & Staveland L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P. A. Hancock and N. Meshkati (Eds.) *Human Mental Workload*. Amsterdam: North Holland Press.

Hault-Dubrulle, A., Robache, F., Drazétic, P., & Morvan, H. (2009). Pre-Crash Phase Analysis Using a Driving Simulator. Influence of Atypical Position on Injuries and Airbag Adaptation. *The 21st International Technical Conference on the Enhanced Safety of Vehicles Conference (ESV)*, 1–13.

Hault-Dubrulle, A., Robache, F., Pacaux, M.-P., & Morvan, H. (2011). Determination of pre-impact occupant postures and analysis of consequences on injury outcome. Part I: a driving simulator study. *Accident; Analysis and Prevention*, 43(1), 66–74. https://doi.org/10.1016/j.aap.2010.07.012

Hirose, T., & Sawaragi, T. (2020). Extended FRAM model based on cellular automaton to clarify complexity of socio-technical systems and improve their safety. *Safety Science*, 123 (November 2019), 104556. https://doi.org/10.1016/j.ssci.2019.104556

Hoc, J. M., & Amalberti, R. (2007). Cognitive Control Dynamics for Reaching a Satisficing Performance in Complex Dynamic Situations. *Journal of Cognitive Engineering and Decision Making*, 1(1), 22–55.

Hoc, Jean-Michel. (1996). *Supervision et contrôle de processus: la cognition en situation dynamique*. Grenoble, France: Presses Universitaires de Grenoble.

Hoc, Jean-Michel. (2000). From Human-Machine Interaction to Human-Machine Cooperation. *Ergonomics*, 43(7), 833–843.

Hoc, Jean-Michel, Cacciabue, P. C., & Hollnagel, E. (1995). *Expertise and Technology: Cognition and Human-Computer Cooperation*. Hillsdale, NJ: Erlbaum.

Hoc, Jean-Michel, & Lemoine, M.-P. (1998a). Cognitive Evaluation of Human-Human and Human-Machine Cooperation Modes in Air Traffic Control. *The International Journal of Aviation Psychology*, 8(1), 1–32. https://doi.org/10.1207/s15327108ijap0801_1

Hoc, Jean-Michel, & Lemoine, M.-P. (1998b). Cognitive Evaluation of Human-Human and Human-Machine Cooperation Modes in Air Traffic Control. *International Journal of Aviation Psychology*, 8(1), 1–32. https://doi.org/10.1207/s15327108ijap0801_1

Hoc, Jean Michel. (2001). Towards a cognitive approach to human-machine cooperation in dynamic situations. *International Journal of Human Computer Studies*, 54(4), 509–540. https://doi.org/10.1006/ijhc.2000.0454

Hollnagel, E., & Woods, D. D. (2005). *Joint cognitive systems: Foundations of cognitive systems engineering*. Boca Raton, FL: CRC Press / Taylor & Francis.

Hollnagel, Erik, & Woods, D. D. (1983). Cognitive systems engineering: New wine in new bottles. *International Journal of Human Computer Studies*, 18, 583–600. https://doi.org/10.1006/ijhc.1982.0313

Horiguchi, Y., Burns, C. M., Nakanishi, H., & Sawaragi, T. (2004). Visualization of Control Structure in Human-Automation System Based on Cognitive Work Analysis. In *12th IFAC Symposium on Analysis, Design, and Evaluation of Human-Machine Systems* (pp. 423–430). Las Vegas, USA.

Hussein, A., Ghignone, L., Nguyen, T., Salimi, N., Nguyen, H., Wang, M., & Hussein, A. (2018). Towards Bi-Directional Communication in Human-Swarm Teaming: A Survey, (March).

Idris, H., Enea, G., & Lewis, T. A. (2016). Function Allocation between Automation and Human Pilot for Airborne Separation Assurance, (Landry 2011). https://doi.org/10.1016/j.ifacol.2016.10.456
Inagaki, T. (2003). *Adaptive Automation: Sharing and Trading of Control*. In E. Hollnagel (Ed.), *Handbook of Cognitive Task Design* (pp. 147–169). Lawrence Erlbaum Associates. https://doi.org/10.1201/9781410607775

Inagaki, T. (2006). *Design of human–machine interactions in light of domain-dependence of human-centered automation*. *Cognition, Technology & Work*, 8(3), 161–167. https://doi.org/10.1007/s10111-006-0034-z

Inagaki, T., Moray, N., & Itoh, M. (1998). *Trust, Self-Confidence and Authority in Human-Machine Systems*. *IFAC Proceedings Volumes*, 31(26), 431–436. https://doi.org/10.1016/s1474-6670(17)40131-5

Inagaki, Toshiyuki, & Itoh, M. (2010). *Theoretical framework for analysis and evaluation human’s overtrust in and overreliance on advanced driver assistance systems*. In *European Conference on Human Centred Design for Intelligent Transport Systems*. Berlin, Germany: HUMANIST publications.

Inagaki, Toshiyuki, & Itoh, M. (2013). *Human’s Overtrust in and Overreliance on Advanced Driver Assistance Systems: A Theoretical Framework*. *International Journal of Vehicular Technology*, 2013. https://doi.org/10.1155/2013/951762

Inagaki, Toshiyuki, & Sheridan, T. B. (2008). *Authority and Responsibility in Human-Machine Systems: Is Machine-Initiated Trading of Authority Permissible in the Human-Centered Automation Framework?* In *Applied Human Factors and Ergonomics*, AEI 2008, Las Vegas 14-17 July, 2008.

Inagaki, Toshiyuki, & Sheridan, T. B. (2018). *A critique of the SAE conditional driving automation definition, and analyses of options for improvement*. *Cognition, Technology & Work*, 21(4), 569–578. https://doi.org/10.1007/s10111-018-0471-5

Itoh, M., Pacaux-Lemoine, M.-P., Robache, F., & Morvan, H. (2013). *Analysis of human’s emergency avoiding maneuver while driving a car*. In *Proceedings of the SICE Annual Conference*.

Itoh, Makoto, & Pacaux-Lemoine, M.-P. (2018). *Trust View from the Human-Machine Cooperation Framework*. *Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018*, 3213–3218. https://doi.org/10.1109/SMC.2018.00544

Jans, M., Borst, C., Paassen, M. M. Van, & Mulder, M. (2019). *Effect of ATC Automation Transparency on Acceptance of Resolution Advisories*. In *14th IFAC Symposium on Analysis Design and Evaluation of Human Machine Systems* (pp. 353–358). Tallinn, Estonia.

Jones, D. G., & Endsley, M. R. (2004). *Use of Real-Time Probes for Measuring Situation Awareness*. *International Journal of Aviation Psychology*, 14(4), 343–367. https://doi.org/10.1207/s15327108ijap1404_2

Jones, P., & Jasek, C. (1997). *Intelligent Support for Activity Management (ISAM): An Architecture to Support Distributed Supervisory Control*. *IEEE Systems, Man and Cybernetics, Part A: Systems and Humans*, 27(3). https://doi.org/10.1109/3468.568737

Jou, Y. T., Yenn, T. C., Lin, C. J., Yang, C. W., & Chiang, C. C. (2009). *Evaluation of operators’ mental workload of human-system interface automation in the advanced nuclear power plants*. *Nuclear Engineering and Design*, 239(11), 2537–2542. https://doi.org/10.1016/j.nucengdes.2009.06.023

Kaber, D. B., & Endsley, M. R. (2004). *The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task*. *Theoretical Issues in Ergonomics Science* (Vol. 5). https://doi.org/10.1080/1463922021000054335
Kaber, D. B., Perry, C. M., Segall, N., McClernon, C. K., & Prinzel, L. J. (2006). Situation awareness implications of adaptive automation for information processing in an air traffic control-related task. *International Journal of Industrial Ergonomics*, 36(5), 447-462.
https://doi.org/10.1016/j.ergon.2006.01.008

Karsenty, L. (2000). Cooperative work : the role of explanation in creating a shared problem representation. *Le Travail Humain*, 63(4), 289-309.

Kiener, J., & Stryk, O. Von. (2010). Towards cooperation of heterogeneous, autonomous robots: A case study of humanoid and wheeled robots. *Robotics and Autonomous Systems*, 58(7), 921-929.
https://doi.org/10.1016/j.robot.2010.03.013

Krobusek, R. D., Boys, R. M., & Palko, K. D. (1988). Levels of Autonomy in a Tactical Electronic Crewmember. In *Proceedings of the Human-Electronic Crew Workshop, Ingolstadt, FRG*.

Lauwers, J. C., & Lantz, K. A. (1990). Collaboration awareness in support of collaboration transparency: requirements for the next generation of shared window systems. In *CHI '90: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (pp. 303-311).
https://doi.org/https://doi.org/10.1145/97243.97301

Lee, J. D., & See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. *Human Factors*, 46(1), 50-80.

Lee, J., & Moray, N. (1992). Trust, control strategies and allocation of function in human-machine systems. *Ergonomics*, 35(10), 1243-1270.

Lemoine, M.-P., Debernard, S., Crevits, I., & Millot, P. (1996). Cooperation Between Humans and Machines: First Results of an Experiment with a Multi-Level Cooperative Organisation in Air Traffic Control. *Computer-Supported Cooperative Work*, 5(2), 299-321.

Lemoine, M. P. (1998). Coopération Hommes-Machines dans les procédés complexes : modèles techniques et cognitifs pour le Contrôle de Trafic Aérien. *Doctorat Spécialité Automatique Des Systèmes Industriels et Humains, Université de Valenciennes et Du Hainaut-Cambrésis*.

Li, J., Greenberg, S., & Sharlin, E. (2017). A two-sided collaborative transparent display supporting workspace awareness. *International Journal of Human Computer Studies*, 101 (November 2016), 23-44.
https://doi.org/10.1016/j.ijhcs.2017.01.003

Licklider, J. C. R. (1960). Man-computer symbiosis. *IRE Transactions on Human Factors in Electronics*, (March), 4-11.

Lindström, V., & Winroth, M. (2010). Aligning manufacturing strategy and levels of automation: A case study. *Journal of Engineering and Technology Management - JET-M*, 27(3-4), 148-159.
https://doi.org/10.1016/j.jengtecman.2010.06.002

Loiselet, A., & Hoc, J. (2001). La gestion des interférences et du référentiel commun dans la coopération: implications pour la conception. *Psychologie Française*, 46, 167-179.

Lyons, J. B., & Guznov, S. Y. (2018). Individual differences in human-machine trust: A multi-study look at the perfect automation schema. *Theoretical Issues in Ergonomics Science*, 0(0), 1-19.
https://doi.org/10.1080/1463922X.2018.1491071

Merrad, W., Habib, L., Héloir, A., Kolski, C., & Krüger, A. (2019). Tangible tabletops and dual reality for crisis management: case study with mobile robots and dynamic tangible objects. In *ANT 2019, The 10th International Conference on Ambient Systems, Networks and Technologies*. Leuven, Belgium.
Merrad, W., Heloir, A., & Kolski, C. (2017). Tangible interfaces and dual reality for collaborative problems solving around distributed tabletops. In *29ème conférence francophone sur l'Interaction Homme-Machine, AFIHM*. Poitiers, France.

Michon, J. (1985). Human Behavior and Traffic Safety. *Human Behavior and Traffic Safety*, 485–520.
https://doi.org/10.1007/978-1-4613-2173-6

Miller, C. a, & Parasuraman, R. (2007). Designing for flexible interaction between humans and automation: delegation interfaces for supervisory control. *Human Factors*, 49(1), 57–75.

Millot, P. (1995). La Coopération Homme-Machine dans la Supervision : les enjeux, les méthodologies, les problèmes. *Séminaire Supervision et Coopération Homme-Machine*.

Millot, P., & Pacaux-Lemoine, M.-P. (2014). *Human-Machine Cooperation and Situation Awareness. Designing Human-Machine Cooperation Systems*. https://doi.org/10.1002/9781118984390.ch9

Millot, P., & Willaeys, D. (1985). An Approach of Dynamical Allocation of Supervision Tasks Between Man and Computer in Control Rooms of Automatized Production Systems. *IFAC Proceedings Volumes*, 18(10), 159–165. https://doi.org/10.1016/S1474-6670(17)60214-3

Millot, Patrick. (1988). *Supervision des procédés automatisés et ergonomie* (Collection). Paris: Hermes.

Millot, Patrick. (2015). Situation Awareness: is the glass half empty or half full? *Cognition Technology & Work*, 17(2), 169–177. https://doi.org/10.1007/s10111-015-0322-6

Millot, Patrick, & Debernard, S. (1993). Men machines cooperative organizations: Methodological and Practical Attempts in air traffic control. In *IEEE Int. Conf. Systems, Man, and Cybernetics*. Le Touquet. France.

Millot, Patrick, & Hoc, J.-M. (1997). Human-Machine Cooperation: Metaphor or possible reality? In *2nd European Conference on Cognitive Science ECCS'97* (pp. 165–174). Manchester, U.K.

Millot, Patrick, & Kamoun, A. (1988). An implicit method for dynamic task allocation between man and computer in supervision posts of automated processes. In *IFAC CONCRESS "Analysis, Design and Evaluation of Human-Machine Systems"*. Oulu, Finland.

Millot, Patrick, & Lemoine, M.-P. (1998). An Attempt for generic concepts toward Human-Machine Cooperation. In *IEEE International Conference on Systems, Man and Cybernetics*. San Diego, USA.

Millot, Patrick, & Mandiau, R. (1995). Men-Machine Cooperative Organizations: Formal and Pragmatic implementation methods. In J-M Hoc, P.-C. Cacciabue, & E. Hollnagel (Eds.), *Expertise and Technology: Cognition Computer Cooperation* (pp. 213–228, 199). New Jersey: Lawrence Erlbaum Associates.

Millot, Patrick, & Pacaux-Lemoine, M. P. (2013). A common work space for a mutual enrichment of human-machine cooperation and team-situation awareness. *IFAC Proceedings Volumes (IFAC-PapersOnline)*, 12(PART 1), 387–394. https://doi.org/10.3182/20130811-5-US-2037.00061

Millot, Patrick, & Roussillon, E. (1991). Man-Machine Cooperation in Telerobotics: Problematic and Methodologies. In *in Proc. of the second Symposium on Robotics*. Gif-sur-Yvette, France.

Morgan, B. B., Glickman, A. S., Woodard, E. A., Blaiwes, A. S., & Salas, E. (1986). *Measurement of team behaviors in a Navy environment (NTSC Tech. Rep. No. 86-014)*.
Morineau, T., & Flach, J. M. (2019). The heuristic version of Cognitive Work Analysis: A first application to medical emergency situations. *Applied Ergonomics*, 79, 98–106.
https://doi.org/10.1016/j.apergo.2018.07.008

Mouayni, I. El, Demesure, G., Bril-el Haouzi, H., Charpentier, P., & Siadat, A. (2019). Jobs scheduling within Industry 4.0 with consideration of worker's fatigue and reliability using Greedy Randomized Adaptive Search Procedure. *IFAC-PapersOnLine*, 52(19), 85–90. https://doi.org/10.1016/j.ifacol.2019.08.242

Muslim, H., Itoh, M., & Pacaux-Lemoine, M.-P. (2017). Driving with shared control: How support system performance impacts safety. In *2016 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2016 - Conference Proceedings*. https://doi.org/10.1109/SMC.2016.7844303

Navarro, J., Mars, F., Forzy, J. F., El-Jaafari, M., & Hoc, J. M. (2010). Objective and subjective evaluation of motor priming and warning systems applied to lateral control assistance. *Accident Analysis and Prevention*, 42(3), 904–912. https://doi.org/10.1016/j.aap.2009.07.008

Nevo, D., Benbasat, I., & Wand, Y. (2012). The knowledge demands of expertise seekers in two different contexts: Knowledge allocation versus knowledge retrieval. *Decision Support Systems*, 53(3), 482–489.
https://doi.org/10.1016/j.dss.2012.03.005

Nielsen, J. (1993). *Usability Engineering*. London: Academic Press.

Pacaux-Lemoine, M.-P., & Debernard, S. (2002). Common work space for human-machine cooperation in air traffic control. *Control Engineering Practice*, 10, 571–576. https://doi.org/10.1016/S0967-0661(01)00060-0

Pacaux-Lemoine, M.-P., & Flemisch, F. (2016). Layers of Shared and Cooperative Control, assistance and automation. *IFAC-PapersOnLine*, 49(19). https://doi.org/10.1016/j.ifacol.2016.10.479

Pacaux-Lemoine, M.-P., Itoh, M., Morvan, H., & Vanderhaegen, F. (2012). Car driver behavior during pre-crash situation: Analysis with the BCD model. *Advances in Transportation Studies*, 159–170.
https://doi.org/10.4399/978885484657915

Pacaux-Lemoine, M.-P., & Trentesaux, D. (2019). Ethical risks of Human-Machine Symbiosis in Industry 4.0: insights from the Human-Machine Cooperation approach. *IFAC PapersOnLine*, 52(19), 19–24.
https://doi.org/10.1016/j.ifacol.2019.12.077

Pacaux-Lemoine, M P, & Loiselet, A. (2002). A common work space to support the cooperation into the cockpit of a two-seater fighter aircraft. *Cooperative Systems Design: A Challenge of Mobility Age*, 157–172.

Pacaux-Lemoine, Marie-Pierre, Berdal, Q., Enjalbert, S., & Trentesaux, D. (2018). Towards Human-based Industrial Cyber-Physical Systems. In *IEEE Industrial Cyber-Physical Systems* (pp. 615–620). Saint Petersburg, Russia. https://doi.org/10.1109/ICPHYS.2018.8390776

Pacaux-Lemoine, Marie-Pierre, & Crévits, I. (2010). Methodological approach and road safety system evaluation. In *IFAC Conference on "Analysis, Design and Evaluation of Human-Machine Systems"*. Valenciennes, France.

Pacaux-Lemoine, Marie-Pierre, & Flemisch, F. (2015). Risk management and methodological approaches: Human-Machine Cooperation in car and aircraft. *Summer School on Risk Management, a Human Centered Approach*.
Pacaux-Lemoine, Marie-Pierre, & Grislin-Le Strugeon, E. (1998). Multi Agents System and Human-Machine Cooperation. In *17th European Annual Conference on Human Decision Making and Manual Control (EAM)*. Valenciennes, France.

Pacaux-Lemoine, Marie-Pierre, & Itoh, M. (2015). Towards vertical and horizontal extension of shared control concept. In *IEEE International Conference on Systems, Man, and Cybernetics* (pp. 3086–3091). Hong Kong, China. https://doi.org/10.1109/SMC.2015.536

Pacaux-Lemoine, Marie-Pierre, & Mareschi, E. (2015). Individual and collective adaptation to situation emergency and complexity: a Human-Machine Cooperation approach. *Summer School: Risk Management, a Human Centered Approach*.

Pacaux-Lemoine, Marie-Pierre, Simon, P., & Popieul, J. (2003). A Driving Simulator for Man Machine Studies in the Field of Driving Safety. In *International symposium on Virtual Concept*. Biarritz, France.

Pacaux-Lemoine, Marie-Pierre, Simon, P., & Popieul, J. (2015). Human-Machine Cooperation principles to support driving automation systems design. In *3rd International Symposium on Future Active Safety Technology Toward zero traffic accidents (FAST-zero'15)*. Gothenburg, Sweden.

Pacaux-Lemoine, Marie-Pierre, & Trentesaux, D. (2019). Cognitive Work Analysis vs. Human-Machine Cooperation in HUMANISM project. In «*Human & Industry 4.0*». Valenciennes, France.

Pacaux-Lemoine, Marie-Pierre, & Vanderhaegen, F. (2013a). Towards levels of cooperation. *Proceedings - 2013 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2013*, 291–296. https://doi.org/10.1109/SMC.2013.56

Pacaux-Lemoine, Marie-Pierre, & Vanderhaegen, F. (2013b). Towards Levels of Cooperation. *2013 IEEE International Conference on Systems, Man, and Cybernetics*, 291–296. https://doi.org/10.1109/SMC.2013.56

Pacaux-Lemoine, Marie Pierre. (2014). Human-Machine Cooperation Principles to Support Life-Critical Systems Management. *Risk Management in Life-Critical Systems*, 253–277. https://doi.org/10.1002/9781118639351.ch12

Pacaux-Lemoine, Marie Pierre, & Flemisch, F. (2018). Layers of Shared and Cooperative Control, assistance and automation. *Cognition, Technology & Work*, 49(19), 159–164. https://doi.org/10.1016/j.ifacol.2016.10.479

Pacaux, M.-P., Debernard, S., Godin, A., Rajaonah, B., Anceaux, F., & Vanderhaegen, F. (2011). Levels of automation and human-machine cooperation: Application to human-robot interaction. In *Proceedings of the 18th IFAC World Congress, Milano (Italy) August 28 - September 2, 2011* (pp. 6484–6492).

Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A Model for Types and Levels of Human Interaction with Automation, 30(3), 286–297.

Piaget, J. (1977). *Études sociologiques [Sociological studies] (3ème éd.)*. Genève: Droz.

Piaget, Jean. (1947). *La psychologie de l'intelligence*. Paris: Armand Colin.

Prewett, M. S., Johnson, R. C., Saboe, K. N., Elliott, L. R., & Coover, M. D. (2010). Computers in Human Behavior Managing workload in human-robot interaction: A review of empirical studies. *Computers in Human Behavior*, 26(5), 840–856. https://doi.org/10.1016/j.chb.2010.03.010

Rachedi, N. D., Berdjag, D., & Vanderhaegen, F. (2013). Operator behavior modeling: Unexpected situations management. *IFAC Proceedings Volumes (IFAC-PapersOnline)*, 12(PART 1), 381–386.
Rajaonah, B., Tricot, N., Anceaux, F., & Millot, P. (2008). The role of intervening variables in driver-ACC cooperation. *International Journal of Human Computer Studies*, **66**(3), 185–197.
https://doi.org/10.1016/j.ijhcs.2007.09.002

Rasmussen, J. (1981). Human Systems Design Criteria: State of the Art [1979] and Future Perspectives. *Computers in Industry*, **2**(4), 297–309.

Rasmussen, J. (1983). Skills, Rules, and Knowledge; signals, Signs, and Symbols, and Other Distinctions in Human Performance Models. *IEEE Transactions on Systems, Man, and Cybernetics*, **13**(3), 257–266.

Rasmussen, J., & Goodstein, L. P. (1987). Decision Support in Supervisory Control of High-risk Industrial Systems. *Automatica*, **23**(5), 663–671. https://doi.org/10.1016/0005-1098(87)90064-1

Rasmussen, J., Pejtersen, A. M., & Schmidt, K. (1990). Taxonomy for Cognitive work analysis. *Technical Report Risø-M-2871*.

Rauffet, P., Chauvin, C., Morel, G., & Berruet, P. (2015). Designing sociotechnical systems: a CWA-based method for dynamic function allocation. In *ECCE 2015 – European Conference on Cognitive Ergonomics* (p. 21). https://doi.org/10.1145/2788412.2788433

Reason, J. (1988). *Human error*. Cambridge University Press.

Rieger, C. A., & Greenstein, J. S. (1982). The allocation of tasks between the human and computer in automated systems. *Proceedings - International Conference on Cybernetics and Society*, 204–208.

Robache, F. (2017). Évaluation sur simulateur de conduite du comportement humain en situation de pré-crash. Application à l'amélioration des airbags. *Doctorat Spécialité Mécanique, Université de Valenciennes et Du Hainaut-Cambrésis*.

Rognin, L., Salembier, P., & Zouinar, M. (2000). Cooperation, reliability of socio-technical systems and allocation of function. *Int. J. Human-Computer Studies*, **52**, 357–379.

Romero, D., Bernus, P., Noran, O., Stahre, J., & Fast-Berglund, Å. (2016). The Operator 4.0: Human Cyber-Physical Systems & Adaptive Automation Towards Human-Automation Symbiosis Work Systems. In I. Nääs, O. Vendrametto, J. Mendes Reis, R. F. Gonçalves, M. T. Silva, G. von Cieminski, & D. Kiritsis (Eds.), *Advances in Production Management Systems. Initiatives for a Sustainable World: IFIP WG 5.7 International Conference, APMS 2016, Iguassu Falls, Brazil, September 3-7, 2016, Revised Selected Papers* (pp. 677–686). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-51133-7_80

Rouse, W. B., Cannon-Bowers, J. A., & Salas, E. (1992). The role of Mental Models in team Performance in Complex Systems. *IEEE Transactions on System, Man, and Cybernetics*, **22**(6), 1296–1308.

Royce, W. (1970). Managing the development of large software system. In *in Proc. IEEE Western Computer Conference*. Los Almitos, CA: IEEE computer Society Press.

Royer, V. (1994). Partage de croyances : condition nécessaire pour un système coopératif ? In I. P. B. (Ed) (*Systèmes coopératifs de la modélisation à la conception* (pp. 253–270). Toulouse Octarés.

SAE. (2013). Summary of SAE International 's Draft Levels of Automation for On-Road Vehicles. *SAE On-Road Automated Vehicle Standards Committee Draft Information Report J3016: Taxonomy and Definitions*, (July 2013).

Salas, E., Prince, C., Baker, D., & Shrestha, L. (1995). Situation awareness in Team Performance: Implications for Measurement and Training. *Human Factors*, **37**(1), 123–136.
Salmon, P. M., Stanton, N. A., Jenkins, D. P., Walker, G. H., Young, M. S., & Aujla, A. (2007). What Really Is Going on? *Review, Critique and Extension of Situation Awareness Theory*. *Engineering, Psychology and Cognitive Ergonomics*, 407–416.

Sanderson, P., Scott, J., Johnson, T., Mainzer, J., Watanabe, L., & James, J. (1994). MacSHAPA and the enterprise of exploratory sequential data analysis (ESDA). *International Journal Of Human-Computer Studies*, 41, 633–681.

Sarter, N. B., & Woods, D. D. (1995). How in the world did we get into that mode? Mode error and awareness in supervisory control. *Human Factors*, 37, 5–19.

Sarter, N. B., Woods, D. D., & Billings, C. E. (1997). AUTOMATION SURPRISES. In G. Salvendy (Ed.), *Handbook of Human Factors & Ergonomics, second edition* (Wiley, pp. 1–25).

Sasangohar, F., Scott, S. D., & Cummings, M. L. (2014). Supervisory-level interruption recovery in time-critical control tasks. *Applied Ergonomics*, 45(4), 1148–1156.
https://doi.org/10.1016/j.apergo.2014.02.005

Sauer, J., Wastell, D. G., Hockey, G. R. J., Crawshaw, C. M., & Downing, J. (2003). Designing micro-worlds of transportation systems: the computer-aided bridge operation task. *Computers in Human Behavior*, 19(2), 169–183.

Sawaragi, T., Fujii, K., Horiguchi, Y., & Nakanishi, H. (2016). Analysis of Team Situation Awareness Using Serious Game and Constructive Model-Based Simulation. *IFAC-PapersOnLine*, 49(19), 537–542.
https://doi.org/10.1016/j.ifacol.2016.10.617

Scerbo, M. W. (1996). Theoretical perspectives on adaptive automation. In *Automation and human performance: Theory and applications* (In R. Para, pp. 37–63). Mahwah, NJ: Erlbaum.

Scharre, P., & Horowitz, M. C. (2015). An introduction to autonomy in weapon systems. *Working Paper, Center for a New American Security*, 1–23. Retrieved from
https://s3.amazonaws.com/files.cnas.org/documents/Ethical-Autonomy-Working-Paper_021015_v02.pdf?mtime=20160906082257

Schieben, A., Heesen, M., Schindler, J., Kelsch, J., & Flemisch, F. (2009). The theater-system technique: Agile designing and testing of system behavior and interaction, applied to highly automated vehicles. *Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2009*, (September), 43–46. https://doi.org/10.1145/1620509.1620517

Schirner, G., Erdogmus, D., Chowdhury, K., & Padir, T. (2013). The future of human-in-the-loop cyber-physical systems. *IEEE Computer*, 46, 36–45. https://doi.org/http://dx.doi.org/10.1109/MC.2012.428.

Schmidt, K. (1991). Cooperative Work: A Conceptual Framework. In J. Rasmussen, B. Brehmer, & J. Leplat (Eds.), *Distributed Decision Making: Cognitive Models for Cooperative Work* (pp. 75–109). Chichester: John Wiley & Sons.

Schmidt, K. (2002). The Problem with ‘Awareness.’ In *Computer Supported Cooperative Work* (Vol. 11, pp. 285–298). Kluwer Academic.

Sheridan, T. B. (1984). Supervisory control of remote manipulators, vehicles and dynamic processes: experiments in command and display aiding. In W. B. Rouse (Ed.), *Advances in Man-Machine Systems Research* (JAI Press, pp. 49–137). NY.

Sheridan, T. B. (2002). Humans and Automation: System Design and Research Issues. *System Engineering and Management: HFES Issues in Human Factors and Ergonomics*, 3.
Sheridan, T. B., & Verplank, W. L. (1978). *Human and Computer Control of Undersea Teleoperators*. Massachusetts Inst of Tech Cambridge Man-Machine Systems Lab.

Sheridan, T., & Ferrell, W. (1974). *Man-machine systems: information, Control and Decision Models of Human Performance*. Cambridge, Massachusetts, and London, England: The MIT Press.

Sheridan, Thomas B. (1992). *Telerobotics, automation, and human supervisory control*. MIT Press.

Shu, Y., & Furuta, K. (2005). An inference method of team situation awareness based on mutual awareness. *Cognition, Technology and Work, 7*(4), 272–287. https://doi.org/10.1007/s10111-005-0012-x

Simmons, R., Singh, S., Heger, F. W., M., H. L. S., Koterba, S., Melchior, N., & Sellner, B. (2007). Human-Robot Teams for Large-Scale Assembly. In *Proceedings of NASA Science Technology Conference*. Adelphi, MD.

Skjerve, A. B. M., & Skraaning, G. (2004). The quality of human-automation cooperation in human-system interface for nuclear power plants. *International Journal of Human Computer Studies, 61*(5), 649–677. https://doi.org/10.1016/j.ijhcs.2004.06.001

Sonnenwald, D. H., & Pierce, L. G. (2000). Information behavior in dynamic group work contexts: Interwoven situational awareness, dense social networks and contested collaboration in command and control. *Information Processing and Management, 36*(3), 461–479. https://doi.org/10.1016/S0306-4573(99)00039-4

Sperber, D., & Wilson, D. (1987). Precis of Relevance : Communication and Cognition. *Behavioral and Brain Sciences, 10*, 697–754.

Stanton, N A, Stewart, R., Harris, D., Houghton, R. J., Baber, C., Master, R. M. C., ... Green, D. (2006). Distributed situation awareness in dynamic systems : theoretical development and application of an ergonomics methodology. *Ergonomics, 49*(12–13), 1288–1311. https://doi.org/10.1080/00140130600612762

Stanton, Neville A, & Bessell, K. (2014). How a submarine returns to periscope depth : Analysing complex socio-technical systems using Cognitive Work Analysis. *Applied Ergonomics, 45*(1), 110–125. https://doi.org/10.1016/j.apergo.2013.04.022

Stubbs, K., Hinds, P. J., & Wettergreen, D. (2007). Autonomy and Common Ground in Human-Robot Interaction. *IEEE Intelligent Systems, 22*(2).

Taborin, V., & Millot, P. (1989). Cooperation between man and decision aid system in supervisory loop of continuous processes. Copenhagen, Denmark: 8th European Annual Conference on Decision making and Manual Control.

Trentesaux, D., & Millot, P. (2016). A Human-Centred Design to Break the Myth of the ``Magic Human'' in Intelligent Manufacturing Systems. In T. Borangiu, D. Trentesaux, A. Thomas, & D. McFarlane (Eds.), *Service Orientation in Holonic and Multi-Agent Manufacturing* (pp. 103–113). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-30337-6_10

Tricot, N., Rajaonah, B., Popieul, J.-C., & Millot, P. (2006). Design and Evaluation of an Advanced Driver Assistance System: The case of an Auto Adaptive Cruise Control. *Le Travail Humain, 69*, 129–152.

Tzafestas, S. (2006). Concerning human-automation symbiosis in the society and the nature. *International Journal of Factory Automation, Robotics and Soft Computing, 1*(3), 6–24.

UK Ministry of Defence Crown. (2018). *Human-Machine Teaming, Joint Concept Note 1/18, Development, Concepts and Doctrine Centre*.
Vakil, S. S., Hansman, R. J., Midkiff, A. H., & Vaneck, T. (1995). Mode Awareness in Advanced Autoflight Systems. *IFAC Proceedings Volumes*, 28(15), 371–376. https://doi.org/10.1016/s1474-6670(17)45260-8

Van Dongen, K., & Van Maanen, P.-P. (2013). A framework for explaining reliance on decision aids. *Journal of Human Computer Studies*, 71(4), 410–424. https://doi.org/10.1016/j.ijhcs.2012.10.018

Vanderhaegen, F., Crevits, I., Debernard, S., & Millot, P. (1994). Man-Machine Cooperation: Toward an activity regulation assistance for different air Traffic Control Levels. *International Journal on Human Computer Interaction*, 6(1), 65–104. https://doi.org/10.1080/10447319409526084

Vanderhaegen, Frédéric. (2004). The Benefit-Cost-Deficit (BCD) model for human analysis and control. In *Proceedings of the 9th IFAC/IFORS/IEA symposium on Analysis, Design, and Evaluation of Human-Machine Systems*. Atlanta, GA, USA.

Wang, Y., Plataniotis, K., Kwong, S., Leung, H., Yanushkevich, S., Karray, F., ... Patel, S. (2019). On Autonomous Systems: From Reflexive, Imperative and Adaptive Intelligence to Autonomous and Cognitive Intelligence. *2019 IEEE 18th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC'19)*, 1–6.

Wohleber, R. W., Stowers, K., & Chen, J. Y. C. (2017). Effects of Agent Transparency and Communication Framing on Human-Agent Teaming. *International Conference on Systems, Man, and Cybernetics (SMC)*, 3427–3432. https://doi.org/10.1109/SMC.2017.8123160

Woods, D. (1997). Human-Centered Software Agents: Lessons from Clumsy Automation. In J. Flanagan, T. Huang, P. Jones, & S. Kasif (Eds.), *Human centered systems: information, interactivity and intelligence* (pp. 288–293). DC: National Science Foundation.

Woods, D. D. (1986). Paradigms for Intelligent Decision Support. In *Proceedings of the NATO Advanced Study Institute on Intelligent Decision Support on Intelligent decision support in process environments* (pp. 153–173). https://doi.org/10.1007/978-3-642-50329-0_11

Woods, D. D., Tittle, J., Feil, M., & Roesler, A. (2004). Envisioning Human – Robot Coordination in Future Operations. *IEEE Transactions on Systems, Man and Cybernetics*, 34(2), 210–218. https://doi.org/10.1109/TSMCC.2004.826272

Zieba, S., Polet, P., & Vanderhaegen, F. (2011). Using adjustable autonomy and human-machine cooperation to make a human-machine system resilient - Application to a ground robotic system. *Information Sciences*, 181(3), 379–397. https://doi.org/10.1016/j.ins.2010.09.035
# Chapter 7. Appendix 1: Projects, implementation to various application areas

## 7.1 Air Traffic Control

**Academic and socio-economic partners:** CENA (the French acronym for Center for Air Navigation Studies), DGAC (the French acronym for General Direction of Civil Aviation), En-route air traffic control centers of Reims and Athis-Mons (Paris), Eurocontrol.

**Projects:** SPECTRA V2, SWIFT (1992-1998)

**Role:** I was a PhD student

**Scientific outcomes:**

* 3 articles in international journals: [10][11][12]

* 11 articles in proceedings of international conference with selection committee: [82][85][86][87][88][89][90][91][92][93][94]

* Several reports among which [151][152]

**Scientific Context:** The objective of the project conducted in air traffic control was to control the workload of the “radar” air traffic controller, the controller who communicates with pilots to provide acknowledges and orders to avoid air-misses between planes. Literature and past projects in this field highlighted the possibility of assisting controllers with a system able to detect potential air-misses and to provide solutions for the simplest air-misses between two planes. Therefore, the assistance system, called SAINTEX, controlled a portion of the air-misses according to two types of authority management, i.e. human or machine decides allocation. The results reflect the outcome of the usefulness of SAINTEX, but suggested another way to control task allocation that was an additional task for the radar controller. Air control organization was the base to propose new assistance system and new studies. Radar controllers works with planning controllers who coordinate the traffic in their own sector with the traffic in other sectors in order to avoid insoluble conflicts at the sector’s borders. They were well aware of their teammate workload. Nevertheless, an assistance system able to detect radar controller overload, called PLAFF, offered possible task allocations for SAINTEX to planning controllers. Therefore, the multilevel aspects of cooperation with planning and tactical levels were identified as a solution for improved safety and performance of the general humans-machines system, but as well as for improved human confidence and consequently acceptance of the automatic control.
<figure><img src="image_61.png" /><figcaption>Fig. 66 Air traffic control platform</figcaption></figure>

**Research activities within this context:** I was involved in the design and implementation of both assistance systems on the experimental platform that was also designed in the lab. I participated in the setting up and management of experiments, as well as contributing to data analysis, a conference and journal papers and writing reports.

We conducted several studies in the field and spent several days with air traffic controllers, in Athis-Mons and Reims, in order to be sufficiently immersed to feel and understand controllers’ needs and requests regarding the increase of air traffic. We were able to perceive individual and cooperative activities in the control room. Therefore, we identified, proposed and evaluated the new notions of Know-How and Know-How-to-Cooperate, as well as Multi-Level Cooperation, especially through the analysis of experimental data supported by MacShapa software. This software supports the analysis of several types of data, and especially the coding of verbal reports recorded during experiments and during debriefings.

We conducted experiments in the control centers on the platform designed by LAMIH, and also on the platform designed within the framework of the European project SWIFT (Specification for controller working positions in future air traffic control). European experiments evaluated new interfaces, like the electronic strip board (a strip describes the flight plan of each aircraft entering the controlled sector), as well as new assistance systems, like the conflicts detection, now implemented on the air traffic controllers’ workstation.
## 7.2 Car driving

More and more studies deal with car driving and the objective of finding solutions to tackle the increase of traffic as well as the increase of drivers' needs. The increase in traffic entails an increase in the complexity of the driving environment. The increase in drivers' needs includes the drivers' comfort, but also the need to be autonomous, e.g. to move despite some disabilities due to accident or age. Automation can be a possible answer to such needs. Systems designed for the car-driving domain today are able to fully control the car in specific driving situations. They also gather information from the environment and diagnose problems regarding driving conditions and other road users. They have even started to diagnose driver behavior regarding driving situations and the driver state. Advanced Driving Assistance Systems (ADAS) can partially or even fully take over from humans. However, what is the risk of implementing such a system? Systems can improve the way automobiles are used today, but what will the impact be on drivers' behavior, on their interaction with the ADAS and the other road users involved in the traffic that are progressively equipped?

The objective of this section is to present several studies conducted in order to provide some preliminary responses to the previous highlighted needs and questions. This section presents some ADAS and on-road systems, as well as results stemming from experiments conducted in the field and on driving simulators. Before such descriptions, an overview of risk taking in automobiles by drivers, but also by system designers, presents a model gathering different types of interactions between driver and several other entities involved in the driving environment. This model is useful to highlight the objectives of the projects presented in the following sections.

<figure><img src="image_62.png" /><figcaption>Fig. 67 Driving environment and driving assistance systems</figcaption></figure>

The driving environment is a dynamic process based on the combination of infrastructure, weather conditions and road users. Drivers are part of this dynamic process by controlling their cars in order to reach their destination safely (cf. Fig. 67). The model highlights the distinction between the command part and the physical part of the driver, as well as between the vehicle and systems. Both parts are of course strongly interconnected, but such distinction underlines the distinctions also made for systems design. In fact, systems
are most of the time designed independently of each other. Nevertheless, with the increase in systems' abilities and complexity, more and more projects focus on systems' compatibility and coherence. The cognitive part is the problem-solving ability that allows the driver to analyze the driving situation and make the right decision at the right time. Embedded or on-road driving assistance systems may support this part by warning against risky driving situation or by controlling the car for driver comfort and safety (command part of the vehicle). The physical part includes the ability of the driver to gather information from the environment, from the vehicle and the systems, and the ability to act in the vehicle in order to implement decisions. However, the physical part is also the part that safety systems and compartment must protect when an accident occurs by taking into account the driver's position in order to adjust protections dynamically.

### 7.2.1 Car driving - Embedded Driving Assistance Systems

**Academic and socio-economic partners:** PSA, Renault, OKTAL, University of Tsukuba (Japan)

**Projects:** PREDIT ARCOS (2002-2005), ST2 (2004-2008), ANR COCOVEA (2013-2016), International collaboration

**Scientific outcomes:**

* 14 articles in proceedings of international conference with selection committee: [38][46][52][64][70][71][72][74][75][76][77][78][79][80]

* 8 reports: [123][134][135][136][137][138][140][142]

* 3 seminars: [108][109][117]

**Scientific Context:** The objective of these three projects was to design and evaluate advanced driving assistance systems (ADAS) on the Université Hauts-de-France's driving simulator. Warning systems about traffic situations and cruise control systems (cf. Fig. 68) were simulated with the Wizard of Oz method (ST2) and/or implemented (ARCOS, COCOVEA) to be evaluated with participants. Main results dealt with the awareness of drivers regarding ADAS mode, mode changes as well as ADAS abilities according to driving situations. We improved and adapted the analysis method designed in the air traffic control domain, supporting the coding of human and system activities during experiments based on log files and verbal reports.

**Research activities within this context:** I was involved in the design of ADAS, in the design and implementation of the Wizard of Oz method to control ADAS, but also traffic, from the experimental room that allows the control of the driving simulator and experiments (scenarios, assistance systems). I participated in the settings up and management of experiments, as well as contributing to the data analysis, conference papers, a book chapter and writing reports.
<figure><img src="image_63.png" /><figcaption>Fig. 68 Control room – Risky driving situations – Driving simulator of the Université Hauts-de-France</figcaption></figure>
## 7.2.2 Car driving - On-road driving assistance system

**Academic and socio-economic partners:** LCPC & INRETS (now IFSTTAR), LRPC Saint-Brieuc, CETE-NC, Erdyn consultants, GMConseil, LAUREPS (Université Rennes), LAA (Université René Descartes), ESEO (Ecole d'ingénieur Angers).

**Project:** PREDIT SARI (2005-2010)

**Role:** Leader of AJISE sub-project for LAMIH.

**Scientific outcomes:**

* 3 articles in international journals: [4][5][8]

* 5 articles in proceedings of international conference with selection committee: [47][58][59][60][61]

* 2 articles in proceedings of national conference with selection committee: [95][96]

* 7 reports: [124][125][126][127][129][132][133]

* 1 seminar: [113]

**Scientific Context:** The objective of the SARI (Automatic Road Condition Monitoring to provide Information to Drivers and Road Managers) project was to study new technological solutions to road safety, especially on rural roads with the limited budget of local communities and the inability to improve road security by rebuilding roads. In 2004, the severity (deaths/100 injuries) was also 3.9 times higher on rural roads than in urban areas. The project proposed and evaluated new solutions according to four main sub-projects (cf. Fig. 69):

* RADARR (Research into Attributes for Early Diagnosis of Road Discontinuity) dealt with risks due to dangerous bends combined to the position of the vehicle in the lane;

* IRCAD (Informing road users about the risk of discontinuities in the route due to poor weather) dealt with risks due to slippery roads combined to vehicle speed;

* VIZIR (Intelligent vision of risky roads and zones) dealt with risks due to loss of visibility because of road layout or masking (vegetation, buildings).

* AJISE (Acceptability in legal terms, to individuals and society and from the economic standpoint) dealt with the ability to analyze previous risks according to different points of view.

**Research activities within this context:** I was involved in the AJISE sub-project to coordinate technical solutions proposed by the other sub-projects and to conduct experiments on driver simulators and in the field on several risky roads identified in Côte d'Armor and in Maine-et-Loire French departments.
<figure><img src="image_64.png" /><figcaption>Fig. 69 Technical solutions implemented and evaluated in PREDIT-SARI project</figcaption></figure>

I was especially involved in the study focused on the risk related to crest vertical curves (CVC) on rural roads, using a field site managed by a local operator of the French road network (Conseil Général de Maine-et-Loire, 49). The final goal was to test one road treatment on the real road. The first step, using driving simulators, determined the two most effective perceptual treatments (i.e., rumble strips on both sides of the centerline and sealed shoulders) in order to help drivers to maintain lateral control when driving on CVCs. In the second step, where we were evaluating the impact of perceptual treatment on the field site, a diagnostic device was developed and installed, in order to track the individual paths of vehicles around the target CVC in the field site. By observing these paths, the third step involved collecting the data during two periods. The first before setting up the centerline rumble strips (CRS), and the second after the setting them up. This way, we evaluated the impact of the CRS on the driver's behavior, and we compared the results obtained in the field study with those of the driving simulators' studies. This comparison showed that the trends observed, both in driving simulators and on the real site, were similar.

<figure><img src="image_65.png" /><figcaption>Fig. 70 Evaluation of the impact of perceptual treatment in the field and on two driving simulators: fixed-based one on the left and motion-based one on the right</figcaption></figure>

## Acknowledgments

The research was performed as part of the French national project PREDIT-SARI (sub-themes VIZIR and AJISE) sponsored by the Department of the Environment, Energy, Sustainable Development and Sea (MEEDDM). We would like to thank Isabelle Aillerie (INRETS-MSIS) for constructing the 3D databases and for her help with replicating the experiment with the fixed- and the motion-based driving simulators. And I would like to thank Jean-Michel Auberlet from Institut Français des Sciences et Technologies des Transports, de l'Aménagement et des Réseaux (IFSTTAR) for our rich and fruitful discussions and works.
### 7.2.3 Car driving - Driver behavior during pre-crash situation

**Academic and socio-economic partners:** LAMIH: "Human and Life Sciences", "Automation and Control" and "Mechanical Engineering" departments, LAB (Laboratoire d'Accidentologie de Biomécanique et d'étude du comportement Humain), and University of Tsukuba (Japan) within the framework of the visiting professor application with Makoto Itoh (2011-2012)

**Project:** ST2 (French acronym for Science and Technology for Transport Safety) (2004-2008)

**Scientific outcomes:**

* 3 articles in international journals: [6][7][9]

* 2 articles in national journal:[13][15]

* 8 articles in proceedings of international conference with selection committee: [53][55][56][65][66][67][68][69]

* 2 articles in proceedings of national conference with selection committee: [97][98]

* 1 seminar: [114]

**Scientific Context:** The objective of the project dealt with drivers' emergency-avoiding behavior while they are driving a car in a rural area. A significant improvement in active and passive safety technologies for vehicles has helped to reduce the number of accidents and the damage due to an accident significantly. Active safety systems work before a crash occurs and nowadays have preventive functions to avoid a crash. Passive safety concerns the period after the crash in order to protect occupants or to reduce the occupant injuries. In order to quantify the efficiency of the passive safety systems, European normalized crash tests are performed with crash test dummies. Therefore, the tests of the passive systems efficiency do not take into account a large panel of driver anthropometries, the real comfort driving position and the real driver reactions facing a crash. Our objective was to test the ability to evaluate drivers' behavior during a virtual crash situation on a driving simulator, to analyze risks of drivers Out Of Position (OOP) regarding the way and the time airbags are triggered and to propose different ways to protect the occupants. The potential solution studied was the possibility of detecting crashes earlier and to have better control over airbag inflation. Based on behavioral data obtained from experiments conducted with the driving simulator of Université Polytechnique Hauts-de-France, we analyzed the driver's behavior just before the collision. The results showed that it would be possible to characterize OOP of drivers. Madymo® digitally reproduced such positions with the ellipsoid Hybrid III dummy model and performed a numerical simulation of a frontal collision at 56 km/h. The results of the computational runs put forward injurious situations when the driver's arm was in front of the steering wheel. Indeed, in this case, the arm could hit the head under airbag deployment and cause the neck to be bent seriously and the head to be thrown violently back.

Another result highlighted the tendency of drivers to choose a “rational” behavior to avoid the collision by using the available visual cues even though the situation is extremely time critical. Indeed, the result was interesting in a sense that people can choose an “appropriate” alternative in time even under severely critical conditions. However, the choice of the alternative is based only on available information and not on hidden hazards. If the hidden hazards are taken into account, the alternative chosen by the human may be inappropriate. This finding may be useful to design protective ADAS that work in crash situation.
<figure><img src="image_66.png" /><figcaption>Fig. 71 Control of experiment and data recording; Scenario; digital car and OOP</figcaption></figure>

**Research activities within this context:** I was involved in the ST2 project to manage two experiments conducted, first on the fixed-based driving simulator that became the motion-based one used by Université Polytechnique Hauts-de-France today. About one hundred drivers experienced a virtual crash situation during the first or the second experiment (cf. Fig. 72). We were able to analyze the behavior of 76 of them. The main difficulty was the synchronization between all types of data. Log files and video records supported the extraction of the force applied on the brake pedal, the moment applied on the steering wheel, the speed of the steering wheel, the car speed, the distance from line crossing and the time for drivers to move from the accelerator pedal to the brake pedal. Thanks to the use of the so-called BCD (Benefit/Cost/Deficit) model, we found that emergency (deficit) in the crash situation could be detected with an average time of 0.98 s.

<figure><img src="image_67.png" /><figcaption>Fig. 72 Scenario of the unavoidable crash situation; Sequence of the falling block on the windscreen</figcaption></figure>

## Acknowledgments

This research work was supported by International Campus on Safety and Intermodality in Transportation the Nord-Pas-de-Calais Region, the European Community, the Regional Delegation for Research and Technology, the Ministry of Higher Education and Research, and the National Center for Scientific Research. We gratefully acknowledge the support of these institutions.
### 7.2.4 Car driving - Infrastructure impact

**Academic and socio-economic partners:** Cofiroute (the French acronym for Industrial and Financial Company of Motorways), Oktal, Renault (French car manufacturer)

**Project:** Project commissioned by the French Transports minister to Cofiroute (2002-2003)

**Role:** Leader of the experimentations conducted on the driving simulator.

**Scientific outcomes:**

* 1 article in proceedings of international conference with selection committee: [73]

* 2 reports: [139][141]

**Scientific Context:** The French government selected Cofiroute as the main contractor for building, operating and maintaining the A86 west tunnels. This future tunnel will house two superposed, independent carriageways, each carrying one-way traffic. Each level is divided into a two-lane carriageway. The tunnel is 10 kilometers long and between 20 and 90 meters deep. The ceiling height is 2.55 meters and the speed is limited to 70 km/h. Literature dealt with the most stressful aspect of this type of environment, its brightness. Moreover, the tunnel only provided a hard shoulder for emergency use and refuge areas on the left side, when the usual positioning in right-hand drive countries is the right side. Therefore, the study had to analyze drivers' perception and behavior during normal and risky driving situations, as well as the impact of the positioning of the hard shoulder on the left side of the road on traffic patterns.

<figure><img src="image_68.png" /><figcaption>Fig. 73 Tunnel virtual mock-up; Tunnel characteristics</figcaption></figure>

**Research activities within this context:** I was involved in the definition of the experimental protocol built with nine traffic situations. We conducted experiments with 38 participants. We recorded data about the interactions between drivers, the vehicle and the environment, as well as driver posture, spontaneous verbalizations and the drivers' comments during the replay. The experiments took place on a motion-based driving simulator installed at Renault's Technocentre. It is a full-scale instrumented Renault Clio cockpit (cf. Fig. 74).
<figure><img src="image_69.png" /><figcaption>Fig. 74 Driving simulator in the Technocentre of Renault</figcaption></figure>

The results showed that in normal driving conditions, with no risky traffic situations and without offences, drivers have in mind the position of the hard shoulder and, even if it is on the left side of the road, they never use it. This behavior is perhaps due to the particular conditions of a simulation, where drivers felt someone was watching their behavior and so respected the hard shoulder rules. However, during two risky situations, when the driver or another vehicle had to go on the hard shoulder because the vehicle is breaking down, problems appeared. Drivers exhibited behavior based on reflex and stopped the vehicle on the right-hand lane instead of going on the hard shoulder on the left. Some drivers made the mistake of overtaking vehicles using the hard shoulder. Some drivers also thought that it was too dangerous to cross the left lane to go to the hard shoulder, because in France the left-hand lane is the faster one. Drivers' comments made it clear that they need better signs and markings for the hard shoulder, perhaps with zebra markings and/or sound rough markings. They would have confidence in the tunnel's surveillance cameras and hoped that traffic infraction would be punished. Several drivers specified that they preferred to drive in the central lane of the tunnel, i.e. in the left-hand lane if the hard shoulder is on the left and on the right-hand lane if the hard shoulder is on the right. They said that the slowest lane should be close to the hard shoulder, because if they have a problem, they should be able to join it quickly.

## Acknowledgments

We wish to thank Oktal and Renault for their contribution to this study.
7.3 Fighter aircraft

**Academic and socio-economic partners:** CNRS (the French acronym for Scientific Research National Center), Dassault-Aviation, DGA (the French acronym for French defense)

**Project:** Human-Human cooperation and its assistance in the field of fighter aircraft piloting (1998-2001)

**Role:** Leader on the engineering control component and responsible for the design of the experimental platform in LAMIH.

**Scientific outcomes:**

* 3 articles in proceedings of international conference with selection committee: [63][81][83]

* 8 reports: [143][144][145][146][147][148][149][150]

**Scientific Context:** A scientific collaboration between Dassault-Aviation, CNRS and DGA dealt with human-human cooperation and its assistance in the field of fighter aircraft piloting. The research program had many goals. The first one was to study cognitive cooperation in the cockpit of a two-seater fighter aircraft, between the pilot and the weapon system officer (WSO). A preliminary study leads to studying their real tasks and to observing their activity in a simulated situation. The second goal offered some possible directions for designing assistance systems to support pilot and WSO cooperation in degraded and unexpected conditions (deteriorated situations or new missions). Moreover, their means of cooperation might be helpful to design assistance systems to support cooperation between the pilot and other actors of the battlefield (e.g. controllers of AWACS (Airborne Warning And Control System)), as well as cooperation between remote human operators and UCAV (Unmanned Combat Air Vehicle). The main goal of this study was to determine the impact of different levels of development of the Common Work Space (CWS) on the cooperative activities in the cockpit. The CWS consisted in enriching the representation of team activity according to temporal and spatial formats, which was mainly based on the usual tactical situation representation (SITAC). With the CWS, humans could access to information provided by other humans, and supply their understanding of the situation by providing their information gathering, diagnosis, decision-making and actions in a more or less automatic manner. Results showed that the integration of such a rich representation of the battlefield improved cooperative activity. More recent fighter aircrafts are now equipped with such an interface.

<figure><img src="image_70.png" /><figcaption>Fig. 75 Fighter aircraft experimental platform and Common Work Space (more details on Fig. 48)</figcaption></figure>
**Research activities within this context:** I was involved in the design of the experimental platform, a generic two-seater fighter aircraft simulator, which was an adaptation of Dassault-aviation aircraft simulators. With this platform, we were able to implement and control new IHMs, and to control scenarios during experiment, thanks to the Wizard of Oz method. We conducted experiments with pilots and WSO of Air Force Bases of Luxeuil (12 crews) and Cambrai (4 crews) in France. Both pilots and Weapon System Officers were able to perform every task (dynamic task allocation is possible, as it is in actual work, but to a larger extent). Missions are divided, most of the time, into three or four main stages: (i) going near the target (entering the enemy territory and flying to the target, steering clear of enemy threats), (ii) handling one or two targets (reconnaissance or destruction mission), and then (iii) leaving the enemy territory.

The crew mainly made up the content of the CWS during mission planning. They added all the information they wanted along a spatial representation of the flight plan, e.g. mission orders, targets, the flight plan, flight rules, and individual and cooperative tasks of each one like ATC or AB3C radio contacts, activation or deactivation of functions and function options, altitude and speed control and the preparation of weapons.

The analysis of the answers to the questionnaires, cooperative activity during mission completion and use of assistance systems, were possible thanks to the automatic and manual coding of all crews' activities. Results showed that the crews used the CWS as short-term and long-term assistance tools. They handled this interface during low workload periods, whereas during high workload periods they did not have sufficient resources available to do so. Nevertheless, the automated update of common information may help. Our results also made lack of training clear. The crews did not have enough time to learn how to make use of the assistance tools and to embed their usual work procedures in them. However, we showed that this type of cooperative support prevented the crews from interrupting each other's tasks, especially when they were fully involved in their own task. Another important result was that CWS facilitated crews' cooperation even if the mission forbad radio communication for security and safety reasons.

## Acknowledgments

This study was done in collaboration with Dassault-Aviation. I would like to thank the Dassault-Aviation Istres test base staff Dassault-Aviation Saint-Cloud staff for their collaboration. The experimental field was the Air Force, especially Air Base 116 and 107. I must thank 2/4, 1/4 and 1/12 Air Force squadrons for their warm welcome and their cooperation. I would also like to thank the IMASSA-CERMA staff for their help, J.-M. Hoc and R. Amalberti for their precious advice during this study.
## 7.4 Robotics

The origin of the word “robot” is a Slavic stem (robota) dealing with chore or serfdom. The theatre play by Karel Čapek, “Rossum's Universal Robots”, used “robot” for the first time in 1920. With the technological improvements of sensors, processors and other mechanical and electronic components, robots are considered more and more as a machine that can be very close to human activity for several aspects, like a human companion. Indeed, robot can perform all functions of the cooperative agent model (cf. chapter 0), with tactical as well as with operational objectives. Therefore, this domain of application is very useful to evaluate our models and methods, and four projects support such studies. We conducted three studies with safety objectives (§7.4.1 to §7.4.3) and one study to support disable people (§7.4.4).

### 7.4.1 Ground robotics in a simulated military context – MIX

**Academic and socio-economic partners:** Master’s students

**Project:** Master’s students’ project (2013-2014)

**Role:** Supervisor of Sarah Hammiche and Maxime Sédilot (Master’s students)

**Scientific outcomes:**

* 1 article in proceedings of international conference with selection committee: [39]

**Scientific Context:** The objective of this study was the evaluation the impact of Unmanned Ground Vehicle (UGV) autonomy, combined with task complexity, on human performance and situation awareness. The virtual environment software MIX (Mixed Initiative eXperimental), dedicated to studying warfighters’ needs in the domain of robotic military equipment and unmanned systems, supported such evaluations (Barber et al., 2008; Cosenzo et al., 2010a). Humans and machines performed three types of task in order to reach a common goal. The operational tasks were to move into a city and to detect threats. The strategic task consisted in designing the UGV trajectory given a geographic goal and detecting threats. Operators had pictures of targets they had to find in the environment. Two LoAs defined the way operators could interact with the UGV. On the first level, operators remotely controlled UGVs using a joystick (teleoperation). On the second level, operators preplanned UGVs’ routes defined by series of waypoints. MIX includes an Operator Control Unit (OCU) (cf. Fig. 76).

**Research activities within this context:** I was involved in the implementation of the experimental environment, in order then to explain it to students and support them in the design of experiments. They conducted four experiments to compare different conditions regarding the
performance of participants and their acceptability. The conditions relied on two factors: task complexity and UGV autonomy. The task complexity depended on the number of UGVs to control.

<figure><img src="image_71.png" /><figcaption>Fig. 76 MIX (The Mixed Initiative eXperimental testbed)</figcaption></figure>

During the first experimental condition, participants defined trajectories to achieve a goal predefined by the experimenters, and they remotely controlled one UGV following this trajectory. During the second experimental condition, participants defined trajectories to reach another goal and the UGV followed this trajectory autonomously. During the third experimental condition, participants defined two different trajectories to reach two different predefined goals. One UGV followed the preplanned trajectory autonomously, and participants remotely controlled a second UGV following the second trajectory. During the fourth experimental condition, participants defined two trajectories, autonomously followed by one UGV. Each experimental condition took place in different areas in order to avoid a learning effect due to familiarity with the environment. Experimental protocol randomly organized conditions in order to avoid a possible learning effect due to the virtual environment and the autonomous UGV use.

Five participants took part to the experiments and results showed that participants' situation awareness was better when they remotely controlled UGVs. This was inconsistent with literature. The number of detected threats was lower with two UGV and there was no significant influence of LoA. Controlling and supervising more than one UGV decreased participants' capabilities and higher LoAs did not compensate task complexity in the way we defined them. Indeed, the relatively short time of each task did not provide enough time to participants to better observe the environment. This empirical study highlighted the relevance of adapting LoAs according to human and autonomous vehicle competences and performance, therefore of designing more useful adaptive LoAs based on Know-How, Know-How-to-Cooperate and task allocation criteria.

## Acknowledgments

I would like to thank Sarah Hammiche and Maxime Sédilot for their good work.
7.4.2 Ground robotics in a military context - TAROT

**Academic and socio-economic partners:** DGA (French acronym for Defense Procurement Agency), THALES OPTRONICS (Land & Joint Systems - Robotics & Mini-UAV Department).

**Project:** PEA TAROT (French acronym for Advanced Study Programme: Autonomy of decision-making technologies for ground robots) (2006-2009)

**Scientific outcomes:**

* 1 article in proceedings of international conference with selection committee: [57]

* 2 reports: [128][130]

* 3 seminars: [99][115][116]

<figure><img src="image_72.png" /><figcaption>Fig. 77 Experimental platform and R-Trooper robot</figcaption></figure>

**Scientific Context:** The study aimed to define and implement optimal cooperation between a human, for whom we have to define a new job, and a robot partly able to control its trajectory, for which we have to define new cooperative functions. KH and KHC functions were defined (cf. Table 5). During the mission, a human is distant from the operational theater in a command post vehicle, and robot motion is performed on a targeted area. Robot could be partly autonomous according to situations and objectives. Staff officers from a strategic, decisional level plan missions and provide an initial task allocation between the human and the robot at the tactical and operational levels. At the operational level, the objective is to allocate, if possible, the motion control to the robot, deciding which algorithm to use (edge, object and itinerary tracking, safety function), and to allocate information tracked by robot to the human. Nevertheless, when the robot might not be able to be autonomous because of complex environment, the human must take back the control, temporarily or until the end of the mission. The tactical level is more mission-completion oriented in that it is about respecting the plan of the mission. Both the human and the robot have the ability to identify goals and to change from one to another, but only a human can add new goals. A CWS has been designed in order to
inform the human about the robot's goals (geographic point), information analysis carried out by the robot, the choice of the robot's motion algorithm used, and the degree of trust it has in the selected algorithm and in the autonomous mode. Tow displays make up the CWS, and task allocation is the most important information, while other information mainly justifies it (cf. Table 5). We evaluated four LoAs, but whichever the level, human had always the authority to get the control back.

<table><caption>Table 5 Human and robot KH and KHC for LoA identification</caption><thead><tr><th></th><th colspan="2">Information acquisition</th><th colspan="2">Information analysis</th><th colspan="2">Decision and action selection</th><th colspan="2">Action implementation</th></tr><tr><th></th><th>Human</th><th>Robot</th><th>Human</th><th>Robot</th><th>Human</th><th>Robot</th><th>Human</th><th>Robot</th></tr></thead><tbody><tr><th colspan="9">Know-how</th></tr><tr><th>Strategic level</th><td>Commands / orders</td><td></td><td>Mission completion</td><td></td><td>Plan modification</td><td></td><td>To modify plan</td><td></td></tr><tr><th>Tactical level</th><td>Current objectives (mobility and reconnaissance)</td><td>Current objectives (only mobility)</td><td>Goal and object identification for reconnaissance</td><td>Next goal</td><td>Goals modification and reconnaissance of objects</td><td>Goal use or lost</td><td>To modify goal</td><td></td></tr><tr><th>Operational level</th><td>All objects</td><td>Objects (edge, target, obstacle)</td><td>Trajectory</td><td>Choice of its sensorimotor behaviors</td><td>Stopping or not the mobility</td><td>About stopping or not the mobility</td><td>To teleoperate or to stop mobility</td><td>To continue or to stop mobility</td></tr><tr><th colspan="9">Know-how-to-cooperate</th></tr><tr><th>Strategic level</th><td>Task allocation for all the mission</td><td>Task allocation for all the mission</td><td>Task allocation analysis according to criteria</td><td>Allocation problem</td><td>Allocation modification</td><td>Allocation modification</td><td>To decrease or increase level of autonomy</td><td>To decrease level of autonomy</td></tr><tr><th>Tactical level</th><td>Robot behaviour</td><td>Object to use as goal</td><td>Robot behaviour analysis according to current or new goal</td><td>Problem with current goal</td><td>New goal to add in the plan and robot behaviour to change or to learn</td><td>Selection of next goal</td><td>To provide information concerning new goal</td><td>To choose next goal or to ask information</td></tr><tr><th>Operational level</th><td>Using robot sensors</td><td>Feedback about sensorimotor behavior</td><td>Robot behaviour according to trajectory</td><td>Information human operator can't have</td><td></td><td>Use of new sensorimotor behavior and obstacle</td><td></td><td>To provide information concerning trajectory</td></tr></tbody></table>

Interesting results demonstrate the usefulness of the methodological technique, the Wizard of Oz, as well as the way we defined of LoAs. Participants provided also important points about task allocation and sharing control, especially regarding their cooperation with human teammates today, highlighting their mutual understanding and team awareness.

**Research activities within this context:** I was involved in the design of robot and CWS using the model of cooperative agents' interactions to identify current KH and KHC of each agent, with the assistance of Thalès engineers who knew the capabilities of robots they designed well. Two experiments were conducted in two different areas (simple vs. complex) and with two different experimental controls (with vs. without Wizard of Oz). The first experiments aimed to evaluate the benefit of using autonomous robots, and the human had no other objective other than respecting the trajectory. During the second experiment, the human had the objective of doing reconnaissance by searching for objects in the environment in addition to the previous
task. Only the human was able to perform the reconnaissance task. The missions took place within Thales group infrastructures for the first experiment and on an industrial wasteland for the second. The first one was limited to be easier for the robot to perform task; the second one had several sparse areas of vegetation and debris from a destroyed factory, to provide several potential obstacles.

As for the main results, the human always felt the need to validate the allocation of control to robot even if it was not necessary, in order to check everything was in order before agreeing robot autonomy. Whatever the selected LoA, participants seemed to have more confidence in the robot during the first experiment than in the second. Indeed, the second experiment imposed reconnaissance tasks, which forced participants to have less time to supervise robot even though this task was dependent on robot motion. A lack of coordination between decision making and action of each agent at the operational level can impact trust. The increase of the robot's KHC might solve such a problem. Another explanation for the greater level of trust in the first experiment is the use of Wizard of Oz method. This technique allows technical problems, such as losing a target because of fog or rain, to be avoided. Moreover, robot motion and decision seemed to be more "human-like" than when we used programmed algorithms. As perspectives, during debriefing, participants underlined the importance of preparing the mission plan themselves and thus deciding task allocation according to their own task allocation criteria.

## Acknowledgments

The present research work was supported by the French Defense Procurement Agency and coordinated by THALES OPTRONICS (Land & Joint Systems - Robotics & Mini-UAV Department). I gratefully acknowledge the support of these institutions.
### 7.4.3 Ground robotics in a crisis management context – SUCRé

**Academic and socio-economic partners:** Ecole Nationale Supérieur des Arts et Industries Textiles (GEMTEX), Université d'Artois (LGI2A), Université de Lille 1 (previously LAGIS, now CRYSTAL), Ecole des Mines de Douai (URIA), Université de Reims Champagne Ardenne (CRESTIC), Université de Technologies de Troyes (Tech-CICO and LM2S).

**Project:** Regional project SUCRé (the French acronym for Identification, design and evaluation of technological tools to support cooperative activities of several actors involved in crisis management: Human-Human and Human-Robots Cooperation in hostile environment) (2014-2017).

**Role:** Co-supervisor of Lydia Habib (PhD) with P. Millot, and supervisor of S. Boulnois (Engineer)

**Scientific outcomes:**

* 4 articles in proceedings of international conference with selection committee: [29][33][35][39]

* 2 book chapters: [19][20]

* 4 reports: [119][120][121][122]

* 2 seminars: [103][105]

**Scientific Context:** The project dealt with the design of methodological and technological tools for the management and control of crisis for French Department Fire and Rescue Service, with an original and multidisciplinary approach. The project dealt with sociotechnical systems such as military organization or civil security. They manage crises, taking advantage of cooperation between command structures, human operators in the field and all available resources. The use of automation offers opportunities to better exploit information to improve understanding, decision-making and tempo, as well as supporting headquarters for more agile command and control. Among these resources, in hostile environments, mobile robots may prevent human operators from being exposed to danger. These robots can have cooperative abilities in order to increase the efficiency of such a sociotechnical system. Therefore, the focus of the project is the study and design of assistance systems able to support Human(s)-Robot(s) cooperation inside sociotechnical systems, and complex and dynamic environments. Furthermore, robot-robot cooperation based on a self-organized swarm has to reach objectives ordered by the crisis unit. A remote human operator manages tactical tasks from a command post vehicle according to the mission plan provided at the strategic level, which here is conducted by a crisis unit.

The remote human operator supervises the swarm supported by assistance systems to manage humans' and robots' task allocation in case of unexpected events from the operational level or
new objectives from the strategic level. Other aspects of cooperation were studied but not evaluated, namely the cooperation between human and robot through smart clothes. Technical points were defined and implemented, and robot was able to use data about a human (temperature, fall...) in order to alert other humans and to stay close to human to facilitate visual/audio communication if necessary.

<figure><img src="image_73.png" /><figcaption>Fig. 78 Experiments to evaluate Humans-Robots cooperation in crisis management</figcaption></figure>

**Research activities within this context:** I was involved in the conceptual aspects about methodology and cooperation definition, as well as in the development of the experimental platform for crisis management. We conducted experiments to evaluate three LoAs defined according to KH and KHC functions allocated to humans and/or robots. At the operational level, the mission (task) of the human-robots team dealt with obstacle detection and avoidance, path planning and tracking, and reconnaissance mapping. They were sharable tasks, except for the reconnaissance mapping subtask because robots were not equipped for image processing. At the tactical level, the interface focused on the mission plan, the robot positions in real time, and the human operator's reconnaissance mapping regarding the localization of victims and dangers (risk of fire, risk of explosion, toxic material and gas valve). Eighteen participants took part in the experiment, with varying degrees of training in video games and the control of robots. We conducted two experimental conditions different according to the complexity of the environment (visibility, density of obstacles, hypothesis for number of victims and dangers) displayed on the map or not. Subjective measures dealt with human operator workload (assessed through Nasa Task Load) and human operator Situation Awareness. Objective measures dealt with human operator performance based on the number of located victims and dangers, and the time required to fulfil the mission.
**Acknowledgment:** The project SUCRé was partly funded by the Regional Council of the French Region "Hauts de France" and supported by GRAISYHM programs. We gratefully acknowledge these institutions and the Nord-Pas-de-Calais department Fire and Rescue Service for the time they spent explaining their activities and for the opportunity to observe training sessions of firefighters, especially Commander Laurent Foucrier and Commander Eric Mareschi. I would like to thank GEMTEX lab of ENSAIT for our very pleasant and fruitful collaboration.
7.4.4 Ground Robotics – Handicap

**Academic and socio-economic partners:** University College of London

**Project:** Project conducted within the framework of the visiting professor application with Tom Carlson (2016-2018)

**Role:** Co-supervisor of Lydia Habib (PhD) with P. Millot.

**Scientific outcomes:**

* 2 articles in proceedings of international conference with selection committee: [32][24]

**Scientific Context:** The project had the objective of providing people who have lost motor or sensory capabilities with assistance systems to enable them to communicate with others and interact with their environment. The only way to communicate for such people might be to use brain signals directly. The goal is therefore to provide people with tetraplegia, confined to their room, with a telepresence tool that facilitates the daily interactions so many of us take for granted. In our case, the telepresence tool is a robot that is remotely controlled. The system is therefore composed of a BCI and a visual interface to implement an “emulated haptic shared control” of the robot (cf. Fig. 79). The aim is to share motion control between the human and the robot according to the difficulty of the situation. We implemented two layers of cooperation. At the operational layer, the user and the robot have to focus on obstacle avoidance. The human can detect obstacles thanks to a live video stream of the environment from the robot’s perspective; and the robot can detect obstacles thanks to its actuated ultrasonic sensor (shared information gathering and analysis). Both can make decisions (shared decision), but only the robot can apply the decision by modifying its trajectory. The tactical layer manages modifications to the mission plan, i.e. how to modify the trajectory to reach the target, after the avoidance of an unexpected obstacle. However, in this case of telepresence, we consider that the target does not impose the design of a precise plan, because it could be changed as and when different opportunities arise for the human (new information to search, other people to meet...). Moreover, it could be difficult for the human to provide precise details of the target to the robot, due to the relatively low resolution and throughput of the BCI as an interface. In this case, the robot is not able to know the target explicitly and to modify the trajectory itself to reach it.

Both human and robot detect and manage interference in obstacle detection and command through the Common Work Space that implements the “emulated shared control”. The final decision concerning the command to be executed (right or left) is made by the robot, due to the human’s limited capacity to react quickly to a situation. The idea is to emulate haptic behavior based on muscles effort with a visual display. Depending on where obstacles are detected, the
robot/BCI system makes it easier or more difficult—in terms of (mental) effort—for the human
to deliver a command and this is conveyed to the user through our standard visual feedback.

<figure><img src="image_74.png" /><figcaption>Fig. 79 Human-BCI-robot cooperation - Shared control with “emulated haptic feedback”</figcaption></figure>

The user moves the grey bar to the left and right by performing motor imagery (cf. Fig. 79: Grey bar between the red and blue arrows). When the bar hits one of the arrows, it delivers the corresponding turn left or turn right command. Dynamically changing the thresholds to deliver BCI commands is illustrated by changing the size of the arrow and essentially makes it (mentally) easier or more difficult to turn in one or both directions.

**Research activities within this context:** I was involved in the conceptual aspects of the “emulated haptic feedback”, as well as in the development of the experimental platform composed of the robot, visual interface and BCI. We conducted two experiments, the first to test the technical feasibility and the second to test the concept. During the experiments, the participants had to control the robot in order to follow the trajectory indicated by the white tape on the floor, whilst avoiding obstacles. The robot is not able to follow the set trajectory because it does not know it, just as it would not know where a user would like it to go with a remote control. The users, who were not disable people, had to keep their hands relaxed loosely on their lap and only have to imagine the kinesthetic movement of their left or the right hand. Two experimental conditions have been defined, combining the possibility to use emulated haptic feedback or not. Results showed that with the “emulated Haptic feedback”, the mean number of collisions is smaller and the robot makes fewer stops, perhaps because it is more difficult for the participant to send the robot in a direction where there is clearly an obstacle. On average, users complete more “correct” sections of the trajectory. In future studies, we would like to give more assistance to the human operator by adapting dynamically the behavior of the robot according to the human’s state. In fact, we assume that such a task required a high level of concentration and mental workload.

**Acknowledgment:** We would to thank Tom Carlson and Nadia Sciacca for their venue in Valenciennes and their involvement in the experiments.
## 7.5 Manufacturing systems

An objective of the Industry 4.0 and the factory of the future is to reach a high level of mass customization, low level of energy consumption through more agility in the global manufacturing system. Such a demand implies designing a more flexible and more resilient manufacturing system. One way is to involve new technologies from digital and manufacturing technology fields. Nevertheless, at the same time, humans must be kept at the center of the global process for decision making and control. Indeed, Intelligent Manufacturing Systems (IMS) tend to implement new autonomous systems, nevertheless so autonomous that humans could be less and less aware of plant and controller states. Such lack of human situation awareness can lead to poor IMS performance. Indeed, Humans are the ultimate agent to manage unforeseen problems or situations not managed by the automated part. Hence, it is necessary to design human-centered supervisory control tools for IMS, taking advantages of human capabilities.

The following projects deal with these objectives and aim to design IMS with the Human-Machine Cooperation approach.
## 7.6 Manufacturing Systems - Diagnosis Assistance System

**Academic and socio-economic partners:** University of Reims (CReSTIC), Nancy (CRAN UMR CNRS 7039).

**Project:** GIS 3SGS ADEXEC (Approach to Detect and Explain Errors of Command) (2007-2012)

**Scientific outcomes:**

* 1 article in a national journal: [14]

* 3 articles in proceedings of international conference with selection committee: [48][54][48]

**Scientific Context:** This project took place within the framework of the GIS 3SGS project (the French acronym for Scientific Interest Group – Systems Monitoring, Dependability and Security). The main objective of the project is to support the diagnosis of a command implemented to control a process, which might not be suitable anymore because maintenance workers or production workers have modified it due to technical problems or new production goals. We placed inside a PLC (Programmable Logical Controller), so independent to the command, a robust filter based on safety constraints and forbidden states identified by experts and validated by model checking. Therefore, the filter was able to detect dangerous forbidden states of the process and stop it. This approach maintains process safety by avoiding damage to products and manufacturing system. Nevertheless, the filter's decision to stop a process may suppress important information that human operators might use to diagnose and correct the command. Therefore, we complemented the implementation of the filter with assistance systems that provide four levels of explanation to human operators to diagnose command errors.

<figure><img src="image_75.png" /><figcaption>Fig. 80 Experimental platform: ITS-PLC to simulate the operative part of the process (on the left) and the assistance system (on the right) providing explanations about process stop and command errors</figcaption></figure>
We conducted experiments and the general trend was that the more the assistance system provides explanations about its analysis of the process, i.e. if participants have more explanations than just current and projection views, the more they detect and correct the error in the command. Nevertheless, more precise the levels of explanation would require more difficult and time-consuming design. Designers must therefore find a compromise.

**Research activities within this context:** Conceptual aspects about the identification of levels of explanation were the main focus, as well as the design and management of experiments. We defined four levels of explanation, based on the concept of human-machine cooperation, to assess the adaptation of the level of explanation according to the type of human operator who would have to diagnose, i.e. according to their expertise and experience with the process. We created new interfaces with a “top view” of the process showing sensors and actuators, and information according to the levels of explanation, as well as for the “current view”, displaying sensors and actuator states at the stopped time. The first level of explanation dealt with the “projection view”, that is, what would have been the state of the process if the process had not been stopped by the assistance system, or in other words, negative consequences due to command error. The second level of explanation, “status before stop”, is the presentation of the state of the process just before it was stopped. This level of explanation focuses on the process elements that had triggered the filter and presents the result of the erroneous command. The third level of explanation, “advanced representation”, is the presentation of the events (rising and falling edge) which had reached a forbidden process state. Here, the display presents two process states, before the forbidden state and after, and explains the undesirable behavior of the command. The fourth level of explanation is the presentation of hypotheses concerning the command errors. This level explains the behavior that the command should have, and presents information concerning the non-use of information to reach an output, or the explanation of unwanted output. Nine participants took part in the experiments. Objective data stemming from command error detection and solving, and subjective stemming from answers to questionnaires provided the results previously presented.

**Acknowledgment:** Special thanks to Pascale Marangé, Alexandre Philippot and Bernard Riera for our close cooperation.
### 7.6.1 Manufacturing Systems – Supervision and control of Self-organized system

**Academic and socio-economic partners:** Master’s students

**Project:** Master’s students project (2013-2015)

**Role:** Co-supervisor of Thomas Laurain and Sébastien Boulnois (Master’s students).

**Scientific outcomes:**

* 1 article in an international journal: [3]

* 2 articles in proceedings of international conference with selection committee: [37][45]

* 1 report: [118]

**Scientific Context:** The objective of this study was the design of a manufacturing system able to take advantage of available intelligence systems technologies while keeping human skills and competences globally in control. The idea was to evaluate the impact of human-machine cooperation on manufacturing system performance compared to a techno-centered design. The system studied within this framework was based on the specific artificial self-organizing intelligent control for a flexible manufacturing system (FMS). Artificial self-organizing (ASO) systems are a possible way of implementing IMS principles. They have been selected for their potential reactivity in the face of disturbances, including human intervention.

<figure><img src="image_76.png" /><figcaption>Fig. 81 Flexible manufacturing system at Université Polytechnique Hauts-de-France (AIP-PRIMECA FMS) – Emulated system – CWS with the tactical assistance</figcaption></figure>

The suggested ASO manufacturing control system, based on holonic and bio-inspired paradigms is a pure techno-centered approach. The problem is that since the ASO system is self-organized, then it may behave in an unexpected way, leading to, at the very least, poor performance, but also potentially disruptive or hazardous situations. This prompted us to re-consider the potential benefits of a human operator because of their ability to react to the unpredictable and to spontaneously conceive solutions. However, the behavior of the ASO is highly reactive and hard to predict for humans, thus a human-centered approach was considered
within the framework of the HMC principles. Instead of focusing on the ASO at the operational level, we also analyzed tactical and strategic levels to provide human with assistance systems (cf. Fig. 82). At the tactical level, the human has to supervise and, if necessary, update the manufacturing plan progress controlled by the ASO. Nevertheless, ASO systems are complex and highly reactive systems that humans may have difficulty understanding and manage without stopping the process. In order to avoid such an extreme situation, a tactical assistance system, linked to ASO system, enables the testing of possible decisions.

**Research activities within this context:** I was involved in the conceptual aspects regarding the increase in KHC of the IMS to support humans in its understanding and control. I was also involved in the supervision of the Master's students who designed and conducted experiments. Subjective data recording dealt with workload evaluation through online and offline (NASA-TLX) questionnaires. We calculated objective data according to the expected performance of the manufacturing system. The goal was not to maximize the throughput but to maintain it between a lower limit (set to 4 products/300s) and an upper limit (set to 6 products/300s). Keeping the throughput constrained helps to maintain control over energy consumption, for which a maximum level was set to 20 kWh. We compared — performance without human in the loop, i.e. techno-centered approach, — performance with human in the loop but without the tactical assistance system, — and performance with human in the loop assisted by the proposed tactical decision assistance system i.e. human-centered approach. When in the loop, the human was aware of process states and acted to reach requested goals. Nevertheless, without the suggested tactical decision assistance, the human was unable to maintain the indicator within the limits, as the number of products and thus the energy consumption threshold exceeded. Therefore, the best results were obtained when the human was involved and assisted. Results stemming from the online and offline questionnaires showed a decrease in the mental demand thanks to the help provided by the assistance system to anticipate the future ASO behavior, an increase in self-confidence and trust in the assistance system, and a decrease in effort because the assistance system performs part of the task.

**Acknowledgment:** I would like to thank Thomas Laurain and Sébastien Boulnois for their excellent work.
7.6.2 Manufacturing Systems – Humans-Intelligent systems Cooperation

**Academic and socio-economic partners:** Université de Reims Champagne Ardenne (CRESTIC), Université de Bretagne-Sud, Lorient (Lab-Sticc)

**Project:** ANR HUMANISM (HUman-MAchines cooperationN for flexible production SysteMs) (2017-2021)

**Role:** Co-supervisor of Quentin Berdal (PhD) with D. Trentesaux and C. Chauvin.

**Scientific outcomes:**

* 4 articles in proceedings of international conference with selection committee: [17][27][28][31]

* 3 seminars: [100][101][102]

**Scientific Context:** The objective of this project is to propose a methodology for designing cooperative systems to support human operators of the Factory of the Future by maintaining their situation awareness and providing adapted assistance systems to make decisions and to manage uncertainties. The Human-Machine cooperation approach, reinforced by the Cognitive Work Analysis approach and the Diagnosis approach, aim to design levels of automation adaptable to self-organized systems, suitable to human needs, and despite the increase in systems complexity and the on-going quest for production systems performance. We are studying three use cases: Human-Robots-ASO cooperation, Humans-Cobot cooperation and Human-Diagnosis support system cooperation (cf. Fig. 82). The use cases are complementary examples of human-machine cooperation that exists or may exist soon in Industry 4.0. They focus on the operational and tactical layers of cooperation. Each agent involved in each layer is in cooperation with all agents of the layers involved in the control of the process. The combination of all individual and cooperative abilities of agents may handle the ethical risks and lead to the best organization between them.

<figure><img src="image_77.png" /><figcaption>Fig. 82 Human-Robots-ASO cooperation, Université Polytechnique Hauts-de-France – Humans-Cobot cooperation, Université Bretagne-Sud, Lorient – Human-Diagnosis support system cooperation, Université de Reims</figcaption></figure>
**Research activities within this context:** My activities focused on the link between Human-Machine cooperation and Cognitive Work Analysis approaches, especially the study of their complementarity and how they can be combined to increase the capacity of the design methodology. We identified possibilities that were implemented for a use case dealing with the control of a production cell, loaded and unloaded by ground robots with two levels of automation. We evaluated assistance systems to supervise and control the production cell, the robots and their interactions. Experiments took place with participants with different levels of expertise and experience regarding manufacturing systems and robotics.

**Acknowledgment:** The research presented in this paper was carried out in the context of the HUMANISM N° ANR-17-CE10-0009 research program, funded by the ANR "Agence Nationale de la Recherche". I gratefully acknowledge the contribution of this institution.
### 7.6.3 Manufacturing systems – Industry 4.0 in Garment factory

**Academic and socio-economic partners:** École Nationale Supérieure des Arts et Industries Textiles (GEMTEX lab), Camaïeu (French retail-clothing company)

**Project:** Smart Fashion 4.0 (2019-2022)

**Role:** Co-supervisor of Mehran Majidan (PhD) with F. Vanderhaegen and S. Thomassey

**Scientific Context:** The project aims to tackle limits of retail-clothing company still based on international supply chains and with long delays to market standard products, while consumption patterns are changing and online sales are booming. The project's objective is to decrease of design and production time for the fashion and luxury goods sectors. New technologies are being studied and will be proposed to respond to this objective. Assistance systems may support managers in the selection of the products regarding the short-term and medium-term trends of consumers. Big databases, virtual shops, virtual prototypes may be used to identify new products. Assistance systems may support manufacturers to make decisions regarding humans-machines organizations that would be adapted according to types of products, human and machine competences and capacities. Adaptable organizations would be evaluated according to human acceptability and performance.

<figure><img src="image_78.png" /><figcaption>Fig. 83 Industry 4.0 in textile-clothing chain</figcaption></figure>
**Research activities within this context:** The research will focus on the improvement of methodology supporting the design and evaluation of adaptable human(s)-machine(s) organization. The methodology will take into account the current ability of machines to manufacture clothes and the current ability of human operators at several decisional levels in order to extend such abilities to perform new functions and to cooperate with each other. New jobs, and so new professional training opportunities, will be created. Such a methodology may help identify and evaluating new humans-machines organizations and task allocation according to objectives and constraints provided by managers. These organizations will be then implemented on one production line of Camaïeu, partly to test the feasibility, but mainly to test the benefit of using the method, the new organizations and their impact on global performance and human acceptability.

**Acknowledgment:** I would like to thank Xianyi Zeng, Ludovic Koehl, Sébastien Thomassey, and Guillaume Tartare from ENSAIT – GEMTEX for their friendly cooperation which continues to grow.
## 7.7 Railway

**Academic and socio-economic partners:** Railenium, SNCF, Thales C&S, Actia Telecom, CNES

**Project:** TC-RAIL project: Train remote control (2017-2021)

**Role:** Co-supervisor of Quentin Gadmer (Research engineer in Railenium) with Philippe Richard

**Scientific outcomes:**

* 1 article in proceedings of international conference with selection committee: [25]

**Scientific Context:** Levels of automation implemented in the railway domain are already high, especially with the use of automated metro systems. Four Grades of Automations (GoA) have been defined. Automatic Train Protection (ATP) systems define GoA1 and are under the control of the train driver. GoA2 is distinguished by the implementation of ATP complemented by the Automatic Train Operation (ATO) system, while still remaining under the control of the train driver. GoA3 is the driverless control grade, and finally GoA4 is the unattended train (UTO). Nevertheless, high levels of automation in the urban sector cannot be directly transposed to train control in mainline or freight sectors, since the train driving environment is more likely to face various unexpected events. The autonomous train is still the final objective, nevertheless before reaching such a high level of automation, the first step will be to design intermediate levels where a human operator is kept "in the loop" of the control of the train. As for the scope of the TC-Rail project, the development of remote driving is a necessary first stage that serves two objectives: the optimization of "last kilometers" for freight railway traffic and a way to recover control from an autonomous train, as a degraded driving mode.

<figure><img src="image_79.png" /><figcaption>Fig. 84 Remote control of train and human-ADASs cooperation</figcaption></figure>
**Research activities within this context:** Railway literature rarely addresses these topics as remote driving may lead to highly risky and complex situations. Therefore, the first goal is to identify how such risks can be addressed through a Human-machine cooperation approach. Literature and several studies dealing with remote control point out that, due to the loss of most of the proprioceptive information, cognitive workload may increase, and perception, attention and sense of presence may decrease. For this reason, a state of the art analysis on cognitive mechanisms in railway driving was conducted, completed by an activity analysis of railway driving activities (Anceaux, Mouchel, Paglia, & Richard, 2018). Since new systems must be adapted to current trains and infrastructure, we propose to design three types of system to support remote train drivers in the control of the train. The first one deals with train driver work position mainly based on a human-machine interface allowing the remote control of the train with the digital representation of devices currently used in the train cabin. Therefore, this work position provides a video feedback to the driver and the usual screen control and command devices. The second system augments the work position by providing tools for information gathering, information analysis, decision making and action, i.e. to support the KH of the driver. Moreover, the second system may support the driver KHC by managing cooperation with the train, but also with other human agents involved in the traffic management. Indeed, the train driver will be cut off from the usual human agents such as ground agents (from the train station or technicians operating on the train after an incident), rail traffic controllers and other drivers.

**Acknowledgment:** I would like to thank Quentin Gadmer for his excellent work.
# Chapter 8. Appendix 2

Publications in which I was involved as first author or co-author.

## 8.1 International journal

[1] Flemisch F., Abbink D., Itoh M., Pacaux-Lemoine M.-P., Wessel G. (2019). Joining the blunt and the pointy end of the spear: towards a common framework of joint action, human-machine cooperation, cooperative guidance and control, shared, traded and supervisory control. Cognition Technology and Work, 21 (4), pp. 555-568 [DOI=10.1007/s10111-019-00576-1].

[2] Pacaux-Lemoine M.-P., Flemisch F. (2018). Layers of shared and cooperative control, assistance, and automation. Cognition, Technology & Work [DOI=10.1007/s10111-018-0537-4].

[3] Pacaux-Lemoine M.-P., Trentesaux D., Zambrano rey G., Millot P. (2017). Designing Intelligent Manufacturing Systems through Human-Machine Cooperation Principles: A Human-Centered Approach. Computers & Industrial Engineering, 111, pp. 581-595. [IF=2.623] [DOI=10.1016/j.cie.2017.05.014].

[4] Bordel S., Somat A., Barbeau H., Anceaux F., Grefeuille C., Menguy G., Pacaux-Lemoine M.-P., Subirats P., Terrade F., Gallenne M.-L. (2014). From technological acceptability to appropriation by users: Methodological steps for device assessment in road safety. Accident Analysis and Prevention, 67, pp. 159-165, ISSN ISSN 0001-4575. [IF=1.964] [DOI=http://dx.doi.org/doi:10.1016/j.aap.2014.01.016].

[5] Auberlet J.-M., Rosey F., Anceaux F., Pacaux-Lemoine M.-P., Aubin S., Briand P., Plainchault P. (2012). The impact of perceptual treatments on driver's behavior: >From driving simulator studies to field tests—First results. Accident Analysis and Prevention, 45, pp. 91-98

[6] Pacaux-Lemoine M.-P., Itoh M., Morvan H., Vanderhaegen F. (2012). Car driver behavior during pre-crash situation: analysis with the BCD model. Advances in Transportation Studies, an International Journal, 26, pp. 159-170

[7] Hault-Dubrulle A., Robache F., Pacaux-Lemoine M.-P., Morvan H. (2011). Determination of pre-impact occupant postures and analysis of consequences on injury outcome. Part I: A driving simulator study. Accident Analysis and Prevention, 43(1), pp. 66-74, ISSN 0001-4575. [IF=1.867]

[8] Auberlet J.-M., Pacaux-Lemoine M.-P., Anceaux F., Plainchault P., Rosey F. (2010). The impact of perceptual treatments on lateral control: A study using fixed-base and motion-base driving simulators. Accident Analysis and Prevention, 42, pp. 166-173

[9] Morvan H., Tan K., Robache F., Pacaux-Lemoine M.-P., Drazetic P. (2007). Pre-crash investigation using a driving simulator and numerical analyses to determine the influence
of the arms positions. International Journal of Crashworthiness, 12(5), pp. 531-539, ISSN 1358-8265

[10] Pacaux-Lemoine M.-P., Debernard S. (2002). A Common Work Space to support the Air Traffic Control. Control Engineering Practice, A Journal of IFAC, 10 (5), pp. 571-576

[11] Hoc J.-M., Pacaux-Lemoine M.-P. (1998). Cognitive evaluation of human-human and human-machine cooperation modes in air traffic control. International Journal of Aviation Psychology, 8, pp. 1-32

[12] Pacaux-Lemoine M.-P., Debernard S., Crévits I., Millot P. (1996). Cooperation between humans and machines: first results of an experimentation of a multi-level cooperative organisation in air traffic control. Computer Supported Cooperative Work, 5 (2), pp. 299-321

## 8.2 National journal

[13] Itoh M., Pacaux-Lemoine M.-P., Robache F., Morvan H. (2015). An Analysis of Driver's Avoiding Maneuver in a Highly Emergency Situation. SICE Journal of Control, Measurement, and System Integration, Vol. 8, N° 1, pp. 027-033

[14] Marangé P., Debernard S., Gellot F., Pacaux-Lemoine M.-P., Philippot A., Poulain T., Riera B., Pétin J.-F. (2014). Approach to detection and explanation of control by a robust filter. Journal Européen des Systèmes Automatisés JESA, Vol. 48, N° 4-5-6, pp. 339-372

[15] Itoh M., Pacaux-Lemoine M.-P., Robache F., Morvan H. (2014). Analyse de manœuvres d'évitement en situation d'urgence dans le cadre de la conduite automobile. Journal Européen des Systèmes Automatisés JESA, Vol. 48, N° 4-5-6, pp. 493-509

## 8.3 Book direction

[16] Flemisch F., Abbink D., Itoh M., Pacaux-Lemoine M.-P. (2019). Special issue on shared and cooperative control, Cognition, Technology & Work, , Volume 21, Number 4. Springer

[17] Pacaux-Lemoine M.-P., Sacile R., Flemisch F., Leitão P., Sallak M. (ongoing) Special Issue on Human and Industry 4.0, Cognition, Technology & Work, , Volume 21, Number 4. Springer

## 8.4 Proceedings of Conference organized by LAMIH

[18] Vanderhaegen F., Caulier P., Pacaux-Lemoine M.-P., Polet P. (2006). Proceedings of the 25th European Annual Conference on Human Decision Making and Manual Control, EAM'06. , Valenciennes, France, ValenSciences, PUV, Valenciennes, ISBN 2-905725-87-7
## 8.5 Book chapter

[19] Tartare G., Pacaux-Lemoine M.-P., Koehl L., Zeng X. (2019). Développement d'un vêtement intelligent pour l'aide à la gestion de crises - application au contrôle d'incendies. In F. Vanderhaegen, C. Maaoui, M. Sallak, D. Berdjag, Défis de l'automatisation des systèmes socio-techniques, ISTE éditions, pp. 300-317, ISBN 9781784055394

[20] Tartare G., Pacaux-Lemoine M.-P., Koehl L., Zeng X. (2019). Development of an Intelligent Garment for Crisis Management: Fire Control Application. In Frédéric Vanderhaegen, Choubeila Maaoui, Mohamed Sallak, Denis Berdjag, Automation Challenges of Socio-technical Systems (Chapter 9), ISTE Ltd and John Wiley & Sons, pp. [DOI=10.1002/9781119644576.ch9].

[21] Millot P., Pacaux-Lemoine M.-P. (2014). Human machine Cooperation and Situation Awareness. In P. Millot, Designing Human Machine Cooperation Systems, ISTE-Wiley, London, pp. 349-378, ISBN 978-1-84821-480-4

[22] Pacaux-Lemoine M.-P. (2014). Human-Machine Cooperation Principles to support Life Critical Systems management. In P. Millot, Risk Management in Life Critical Systems, ISTE-Wiley, London, pp. 253-277, ISBN 978-1-84821-480-4

[23] Millot P., Pacaux-Lemoine M.-P. (2013). Coopération homme-machine et Situation Awareness. In P. Millot, Ergonomie des Systèmes Homme-machine : conception et coopération, Hermés-Lavoisier, Paris, pp. 349-378

## 8.6 Proceedings of International Conference with selection committee

[24] Marie-Pierre Pacaux-Lemoine, Lydia Habib, Nadia Sciacca and Tom Carlson (2020) Emulated haptic shared control for brain-computer interfaces improves human-robot cooperation, IEEE Int. Conference on Human-Machine System, April 6-8, 2020, Rome, Italy.

[25] Marie-Pierre Pacaux-Lemoine, Quentin Gadmer and Philippe Richard (2020) Train remote driving: A Human-Machine Cooperation point of view, IEEE Int. Conference on Human-Machine System, April 6-8, 2020, Rome, Italy.

[26] Frank Flemisch, Marie-Pierre Pacaux-Lemoine, Frederic Vanderhaegen, Emmanuelle Grislin, Makoto Itoh, Yuichi Saito, Joscha Wasser, Nicolas Herzberger and Marcel Baltzer (2020). Conflicts in Human-machine systems as an Intersection of Bio-and Technosphere, IEEE Int. Conference on Human-Machine System, April 6-8, 2020, Rome, Italy.

[27] Pacaux-Lemoine M.-P., Trentesaux D. (2019). ethical risks of human-machine symbiosis in industry 4.0: insights from the human-machine cooperation approach. IFAC-PapersOnLine, 14th IFAC Symposium on Analysis Design and Evaluation of Human Machine Systems (IFAC HMS'19), 52(19), Elsevier, Tallinn, Estonia, pp. 19-24, September.
[28] Berdal Q., Pacaux-Lemoine M.-P., Trentesaux D., Chauvin C. (2019). Human-Machine cooperation in self-organized production system: a point of view. Borangiu, Trentesaux, Thomas, Cavalieri, Service Orientation in Holonic and Multi-Agent Manufacturing, 803, Studies in computational intelligence, Springer, pp. 123-132, January. [DOI=10.1007/978-3-030-03003-2_9].

[29] Habib L., Pacaux-Lemoine M.-P., Millot P. (2018). Human-robots team cooperation in crisis management mission. IEEE International Conference on Systems, Man, and Cybernetics, Miyazaki, Japan, October.

[30] Itoh M., Pacaux-Lemoine M.-P. (2018). Trust View from the Human-Machine Cooperation Framework. IEEE International Conference on Systems, Man, and Cybernetics, Miyazaki, Japan, October.

[31] Pacaux-Lemoine M.-P., Berdal Q., Enjalbert S., Trentesaux D. (2018). Towards Human-based Industrial Cyber-Physical Systems. IEEE Industrial Cyber-Physical Systems (ICPS), Saint Petersburg, Russia, pp. 615-620, may. [DOI=10.1109/ICPHYS.2018.8390776].

[32] Pacaux-Lemoine M.-P., Carlson T., Habib L. (2018). Human-Robot Cooperation through Brain-Computer Interaction and Emulated Haptic Supports. IEEE International Conference on Industrial Technology, Lyon, France, February.

[33] Habib L., Pacaux-Lemoine M.-P., Millot P. (2017). Adaptation of the level of automation according to the type of cooperative partner. IEEE International Conference on Systems, Man, and Cybernetics, Banff, Canada, October.

[34] Flemisch F., Canpolat Y., Altendorf E., Wessel G., Itoh M., Baltzer M., Pacaux-Lemoine M.-P., Abbink D., Schutte P. (2017). Shared and cooperative control of ground and air vehicles: Introduction and General overview. IEEE International Conference on Systems, Man, and Cybernetics, Banff, Canada, October.

[35] Habib L., Pacaux-Lemoine M.-P., Millot P. (2017). A method for designing levels of automation based on a human-machine cooperation model. IFAC World Congress, Toulouse, France, July.

[36] Pacaux-Lemoine M.-P., Tartare G., Habib L., Koehl L., Zeng X. (2017). Human-Robots Cooperation through Intelligent Garment. IEEE International Symposium on Industrial Electronics, Edinburgh, Scotland, June.

[37] Pacaux-Lemoine M.-P., Trentesaux D., Zambrano rey G. (2016). Human-Machine Cooperation to design Intelligent Manufacturing Systems. 42nd Annual Conference of IEEE Industrial Electronics Society (IECON16), Florence, Italy, pp. 5904-5909, October. [DOI=10.1109/IECON.2016.7793180].

[38] Muslim H., Itoh M., Pacaux-Lemoine M.-P. (2016). Analysis of Human-Machine Cooperation When Driving with Different Reliabilities of Shared Control Systems. IEEE Systems, Man, and Cybernetics, Budapest, Hungary, October.