# Attention Analysis for Assistance in Assembly Processes

Lucas Paletta

► To cite this version:

Lucas Paletta. Attention Analysis for Assistance in Assembly Processes. 9th International Precision Assembly Seminar (IPAS), Dec 2020, Held virtually, Unknown Region. pp.310-323, 10.1007/978-3-030-72632-4_23. hal-03520413

HAL Id: hal-03520413

https://inria.hal.science/hal-03520413

Submitted on 11 Jan 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Attention Analysis for Assistance in Assembly Processes

Lucas Paletta

DIGITAL - Institute for Information and Communication Technologies
JOANNEUM RESEARCH Forschungsgesellschaft mbH, 8010 Graz, Austria
lucas.paletta@joanneum.at

**Abstract.** Human attention processes play a major role in the optimization of human-machine interaction (HMI) systems. This work describes a suite of innovative components within a novel framework in order to assess the human factors state of the human operator primarily by gaze and in real-time. The objective is to derive parameters that determine information about situation awareness of the human collaborator that represents a central concept in the evaluation of interaction strategies in collaboration. The human control of attention provides measures of executive functions that enable to characterize key features in the domain of human-machine collaboration. This work presents a suite of human factors analysis components (the Human Factors Toolbox) and its application in the assembly processes of a future production line. Comprehensive experiments on HMI are described which were conducted with typical tasks including collaborative pick-and-place in a lab based prototypical manufacturing environment.

**Keywords:** Human factors analysis, intuitive assistance, dynamic attention.

## 1 Introduction

Human-machine interaction in manufacturing has recently experienced an emergence of innovative technologies for intelligent assistance [1]. Human factors are crucial in Industry 4.0 enabling human and machine to work side by side as collaborators. Human-related variables are essential for the evaluation of human-interaction metrics [2]. To work seamlessly and efficiently with their human counterparts, complex manufacturing work cells, such as for assembly, must similarly rely on measurements to predict the human worker's behavior, cognitive and affective state, task specific actions and intent to plan their actions. A typical, currently available application is anticipatory control with human-in-the-loop architecture [3] to enable robots to perform task actions that are specifically based on recently observed gaze patterns to anticipate actions of their human partners according to its predictions.

In order to characterize the human state by means of quantitative measurement technologies we expect that these should refer to performance measures that represent psychologically relevant parameters. Executive functions [18] were scientifically investigated to represent activities of cognitive control, such as, (i) inhibition in selective attention, (ii) task switching capabilities, or (iii) task planning. These functions are dependent on dynamic attention and are relevant too many capability requirements in assembly. (i) appropriate focus at the right time to manual interaction, (ii) switching attention between
multiple tasks at the same time, and (iii) action planning in short time periods for opti-
mized task performance. Consequently, measuring and modeling of the state of cogni-
tion relevant human factors as well as the current human situation awareness are man-
datory for the understanding of immediate and delayed action planning. However, the
state-of-the-art does not yet sufficiently address these central human cognitive functions
to an appropriate degree. This work intends to contribute to fill this gap by presenting a
novel framework for estimating important parameters of executive functions in assembly
processes by means of attention analysis. On the basis of mobile eye tracking that has
been made available for real-time interpretation of gaze we developed software for fast
estimation of executive function parameters solely on the basis of eye tracking data. In
summary, this work describes a review of novel methodologies developed at the Human
Factors Lab of JOANNEUM RESEARCH to measure the human factors states of the
operator in real-time with the purpose to derive fundamental parameters that determine
situation awareness, concentration, task switching and cognitive arousal as central for
the interaction strategies of collaborative teams. The suite of gaze based analysis soft-
ware is referred as Human Factors Toolbox.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Human-robot collaboration and intuitive interface (HoloLens, eye tracking, markers for OptiTrack localization) for the assessment of human factors state to characterize key features in the collaboration domain. HRC within a tangram puzzle assembly task. (a) The operator collaborates with the robot in the assembly (only robot can treat ‘dangerous’ pieces). (b) Egocentric operator view with augmented reality based navigation (arrow), piece localization, gaze (blue sphere), current state of mental load (L) and concentration (C) in the HoloLens display. (c) Recommended piece (arrow) and gaze on currently grabbed puzzle piece.</figcaption></figure>

Human situation awareness is determined on the basis of concrete measures of eye
movements towards production relevant processes that need to be observed and evalu-
ated by the human. Motivated by the theoretical work of [4] on situation awareness the
presented work specifically aims at dynamically estimating (i) distribution of attentional
resources with respect to task relevant ‘areas of interaction’ over time, determined by
features of 3D gaze analysis and a precise optical tracking system, and (ii) derive from
this human factors in real-time, such as, (a) human concentration on a given task, (b)
human mental workload, (c) situation awareness and (d) executive functions related
measure, i.e., task switching rate. Gaze in the context of collaboration is analyzed in
terms of - primarily, visual - affordances for collaboration. In this work we demonstrate
the relevance of considering eye movement features for a profound characterization of
the state of human factors in manufacturing assembly cells by means of gaze behavior,
with the purpose to optimize the overall human-machine collaboration performance.

In the following, Section 2 provides an overview on emerging attention analyses
technologies in the context of manufacturing human-machine interaction. Section 3 pro-
vides insight into the application of a learning scenario in an assembly working cell.
Section 4 provides conclusions and proposals for future work,

2 Human Factors Measurement Technologies for Human-Machine Interaction

2.1 Intuitive Multimodal Assistive Interface and Gaze

In the conceptual work on intuitive multimodal assistive interfaces, the interaction
design is fully aligned with the user requirements on intuitive understanding of technol-
ogy [5]. For intuitive interaction, one opts for a human-centered approach and starts from
inter-human interactions and the collaborative process itself. The intuitive interaction
system is based on the following principles:

*   **Natural interaction:** Mimicking human interaction mechanisms a fast and intuitive interaction processes has to be guaranteed.

*   **Multimodal interaction:** Implementation of gaze, speech, gestural, and Mixed-Reality interaction offers as much interaction freedom as possible to the user.

*   **Tied modalities:** Linking the different interaction modalities to emphasize the intuitive interaction mechanisms.

*   **Context-aware feedback:** Feedback channels deliver information regarding task, environment to the user. One has to pay attention at what is delivered when and where.

A central component entitled ‘Interaction Model’ (IM) acts as interaction control and undertakes the communication with the periphery system. The IM also links the four interaction modalities and ensures information exchange between the components. It triggers any form of interaction process, both direct and indirect, and controls the context-sensitivity of the feedback. It is further responsible for dialog management and information dispatching.

In human factors and ergonomics research, the analysis of eye movements enables to develop methods for investigating human operators' cognitive strategies and for reasoning about individual cognitive states [6]. Situation awareness (SA) is a measure of an individual's knowledge and understanding of the current and expected future states of a situation. Eye tracking provides an unobtrusive method to measure SA in environments
where multiple tasks need to be controlled. [7] provided first evidence that fixation duration on relevant objects and balanced allocation of attention increases SA. However, for the assessment of executive functions, the extension of situation analysis towards concrete measures of distribution of attention is necessary and described as follows.

## 2.2 Recovery of 3D Gaze in Human-Robot Interaction

Localization of human gaze is essential for the localization of situation awareness with reference to relevant processes in the working cell. [8] Firstly proposed 3D information recovery of human gaze with monocular eye tracking and triangulation of 2D gaze positions of subsequent key frames within the scene video of the eye tracking system. Santner et al. [9] proposed gaze estimation in 3D space and achieved accuracies ≈ 1 cm with RGB-D based position tracking within a predefined 3D model of the environment. In order to achieve the highest level of gaze estimation accuracy in a research study, it is crucial to track user's frustum / gaze behavior with respect to the worker's relevant environment. Solutions that realize this include vision-based motion capturing systems: OptiTrack<fnref n="1" /> can achieve high gaze estimation accuracy (≈ 0.06 mm).

## 2.3 Situation Awareness

Based on the cognitive ability, flexibility and knowledge of human beings on the one hand and the power, efficiency and persistence of industrial robots on the other hand, collaboration between both elements is absolutely essential for flexible and dynamic systems like manufacturing [10]. Efficient human-robot collaboration requires a comprehensive perception of essential parts of the working environment of both sides. Human decision making is a substantial component of collaborative robotics under dynamic environment conditions, such as, within a working cell. Situation awareness and human factors are crucial, in particular, to identify decisive parts of task execution.

In human factors, situation awareness is principally evaluated through questionnaires, such as, the Situational Awareness Rating Technique (SART [11]). Psychological studies on situation awareness are drawn in several application areas, such as, in air traffic control, driver attention analysis, or military operations. Due to the disadvantages of the questionnaire technologies of SART and SAGAT, more reliable and less invasive technologies were required, however, eye tracking as a psycho-physiologically based, quantifiable and objective measurement technology has been proven to be effective [7,12]. In several studies in the frame of situation awareness, eye movement features, such as dwell and fixation time, were found to be correlated with various measures of performance. [13,14] have developed measurement / prediction of Situation Awareness in Human-Robot Interaction based on a Framework of Probabilistic Attention, and real-time eye tracking parameters.

## 2.4 Cognitive Arousal and Concentration Estimation

For quantifying stress we used cognitive arousal estimation based on biosensor data. In the context of eye movement analysis, arousal is defined by a specific parametrization of fixations and saccadic events within a time window of five seconds so that there is good correlation ($r=.493$) between the mean level of electro-dermal activity (EDA) and the outcome of the stress level estimator [15].

<footnote n="1">¹ http://www.naturalpoint.com/optitrack</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 2:** Measure of attentional concentration on tasks. (a) Concentration level during a session without assistance (red line is the mean; red is standard deviation), and (b) concentration level during the session with assistance. On average, the concentration increased when using the intuitive assistance.</figcaption></figure>

For the estimation of concentration or sustained attention, we refer to the areas of interaction (AOI) in the environment as representing the spatial reference for the task under investigation. Maintaining the attention on task related AOI is interpreted as the concentration on a specific task [16], or on session related tasks in general. Various densities of the fixation rate enable the definition of a classification of levels of actual concentration within a specific period of time, i.e., within a time window of five seconds.

## 2.5 Estimation of Task Switching Rate

Task switching, or set-shifting, is an executive function that involves the ability to unconsciously shift attention between one task and another. In contrast, cognitive shifting is a very similar executive function, but it involves conscious (not unconscious) change in attention. Together, these two functions are subcategories of the broader cognitive flexibility concept. Task switching allows a person to rapidly and efficiently adapt to different situations [17].

In a multi-tasking environment, cognitive resources must be shared or shifted between the multiple tasks. Task switching, or set-shifting, is an executive function that involves the ability to unconsciously shift attention between one task and another. Task switching allows a person to rapidly and efficiently adapt to different situations. The task-switching rate is defined by the frequency by which different tasks are actually operated. The difference between tasks is defined by the differences in the mental model which is necessary to represent an object or a process in the mind of the human operator. Mental models are subjective functional models; a task switch requires the change of the current mental model in consciousness and from this requires specific cognitive resources and a load.

In the presented work, processing of a task is determined by the concentration of the operator on a task related area of interaction (AOI). Interaction is defined by areas in the operating environment where the operator is manipulating the location of puzzle objects, i.e., grabbing puzzle pieces from a heap of pieces, or putting pieces onto a final position
in order to form a tangram shape. Whenever the gaze of the operator intersects with an AOI that belongs to a specific task, then it is associated with an on-going task. The task switch rate is then the number of switches between tasks per period of time, typically the time of a whole session (see **Fig. 3** for a visualization). Task switching has been proposed as a candidate executive function along with inhibition, the maintenance and updating of information in working memory, and the ability to perform two tasks at the same time. There is some evidence not only that the efficiency of executive functions improves with practice and guidance, but also that this improvement can transfer to novel contexts. There are demonstrable practice-related improvements in switching performance [18,19,20].

<figure><img src="image_4.png" /><figcaption>**Fig. 3:** Task switching between collaborative and the single task. (a) Placing a puzzle piece to goal area in the collaborative task. (b) The operator places a puzzle piece to the goal area of single task. (c) Switch between collaborative task (S1, above, green) and single task (S2, below, red). Task duration is determined by the human gaze being focused within an AOI related to the specific task.</figcaption></figure>

## 2.6 Cognitive Arousal and Attention in Multitasking

Stress is defined in terms of psychological and physical reactions of animate beings in response to external stimuli, i.e., by means of stressors. These reactions empower the
animate to cope with specific challenges, and at the same time there is the impact of
physical and mental workload. [21,22] defined stress in terms of a physical state of load
which is characterized as tension and resistance against external stimuli, i.e., stressors
which refers to the general adaptation syndrome. Studies confirm that the selectivity of
human attention is performing better when impacted by stress [35] which represents the
'narrowing attention' effect. According to this principle, attention processes would per-
form in general better with increasing stress impact.

Executive functions are related to the dynamic control of attention which refers to a
fundamental component in human-robot collaboration [36]. Executive functions refer
to mechanisms of cognitive control which enable a goal oriented, flexible and efficient
behavior and cognition [37,38], and from this the relevance for human everyday func-
tioning is deduced. Following the definition of [23], executive functions above all in-
clude attention and inhibition, task management or control of the distribution of atten-
tion, planning or sequencing of attention, monitoring or permanent attribution of atten-
tion, and codification or functions of attention. Attention and inhibition require direc-
tional aspects of attention in the context of relevant information whereas irrelevant in-
formation and likewise actions are ignored. Executive functions are known to be im-
pacted by stress once the requirements for regulatory capacity of the organism are sur-
passed [24,40]. An early stress reaction is known to trigger a saliency network that
mobilizes exceptional resources for the immediate recognition and reactions to given
and in particular surprising threats. At the same time, this network is down-weighted
for the purpose of executive control which should enable the use of higher level cogni-
tive processes [41]. This executive control network is particularly important for long-
term strategies and it is the first that is impacted by stress [25].

In summary, impact of stress can support to focus the selective attention processes under single task conditions, however, in case of multiple task conditions it is known that stress impacts in a negative way the systematic distribution of attention onto several processes and from this negatively affects the performance of executive functions. Evaluation and prevention of long-term stress in a production work cell is a specific application objective. Stress free work environments are more productive, provide more motivated co-workers and error rates are known to be lower than for stressed workers. The minimisation of interruption by insufficient coordination of subtasks is an important objective function which has executive functions analysis as well as impact by stress as important input variables.

**Multi-Tasking, Executive Functions and Attention.** Multi-tasking activities are in indirect relation with executive functions. Following Baddeley's model of a central executive in the working memory [38] there is an inhibitor control and cognitive flexibility directly impacting the (i) multi-tasking, (ii) the change between task and memory functions, (iii) selective attention and inhibition functions. The performance in relation to an activity becomes interrupted once there is a shift between one to the other ("task switch"). The difference between a task shift which particularly requires more cognitive resources, and a task repetition is referred to as 'switch cost' [26] and in particular of relevance if the task switch is more frequently, and also referring to the frequency of interruptions after which the operator has to continue the activity from a memorized point where the switch started. Task switches in any case involve numerous executive
function processes, including shift of focus of attention, goal setting and tracking, resources for task set reconfiguration actions, and inhibition of previous task groups. Multi-Tasking and the related switch costs define and reflect the requirements for executive functions and from this the cognitive control of attention processes, which provides highly relevant human factors parameters for the evaluation of human-robot collaboration systems.

The impact of executive functions on decision processes was investigated in detail by [27]. Specifically they researched on aspects of applying decision rules based on inhibition and the consistency in the perception of risk in the context of ‘task shifting’. The results verify that the capacity to focus and therefore to inhibit irrelevant stimuli represents a fundamental prerequisite for successful decision processes. At the same time, the change between different contexts of evaluation is essential for consistent estimation of risk.

<figure><img src="image_5.png" /><figcaption>**Fig. 4:** Bio-sensor and eye tracking glasses setup in the Human Factors Lab at JOANNEUM RESEARCH. (a) Bio-sensor setup with EDA sensor, HRV and breathing rate sensors. (b,c) Eye-hand coordination task interrupted by a robot arm movement. (d) Target areas with holes and gaze intersections centered around the center as landing point of eye-coordination task.</figcaption></figure>

**Measurements of Emotional and Cognitive Stress.** Stress or activity leads to psychophysiological changes, as described by [28]. The body provides more energy to handle the situation; therefore the hypothalamic-pituitary-adrenal (HPA) axis and the sympathetic nervous system are involved. Sympathetic activity can be measured from recordings of, for example, electro-dermal activity (EDA) and electrocardiogram
(ECG). Acute stress leads to a higher skin conductance level (SCL) and also to more spontaneous skin conductance reactions (NS.SCR). [29] showed in his summary of EDA research that these two parameters are sensitive to stress reactions. In a further step, [30] examined the reaction of stress by measuring EDA with a wearable device, during cognitive load, cognitive and emotional stress, and eventually classified the data with a Support Vector Machine (SVM) whether the person received stress. The authors conclude that the peak high and the instantaneous peak rate are sensitive predictors for the stress level of a person. The SVM was capable to classify this level correctly with an accuracy larger than 80%. The electrocardiogram shows a higher heart rate (HR) during stress conditions meanwhile the heart rate variability (HRV) is low [31], and the heart rate was found to be significantly higher during cognitive stress than in rest. [32] found in their survey that the HRV and the EDA are the most predictive parameters to classify stress. [33] used a cognitive task in combination with a mathematical task to induce stress. Eye activity can be a good indicator for stressful situations as well. Some studies report more fixations and saccades guided by shorter dwell time in stress, other studies reported fewer fixations and saccades but longer dwell time. Many studies use in addition the pupil diameter for classification of stress, such as reported. In the survey from [32], the pupil diameter achieved good results for detecting stress. A larger pupil diameter indicates more stress. In the study of [34], blink duration, fixations frequency and pupil diameter were the best predictors for work load classification.

<figure><img src="image_6.png" /><figcaption>**Fig. 5:** Measurement of stress using eye movement features in relation to EDA measures. (a) The relative increase and decrease of EDA and 'stress level' method based output are cor- related with Pearson r=.493 and from this the gaze based method provides an indication of arousal as measured by EDA. (b) The stress level measured with higher and lower cognitive stress (NC and SC condition) very well reflects the required cardinal ranking.</figcaption></figure>

**Stress level estimation.** Eye movement based classification method, i.e., the 'stress level' estimation method, very well estimates the EDA arousal level, as well as the cardinal ranking for different levels of stress impact. The stress level estimation is computed by thresholds on the number of saccades and the mean dwell time during an observation window of 5000 ms. For level 2 the number of saccades should be between 15 and 25, and mean dwell time below 500 ms, For level 3, more than 25 saccades and mean dwell time below 250 ms is requested.

**Sample study on stress and attention in multitasking human-robot interaction.**
The presented work aimed to investigate how the impact of cognitive stress would affect the attentional processes, in particular, the performance of the executive functions
that are involved in the coordination of multi-tasking processes. The study setup for the proof-of-concept involved a cognitive task as well as a visuomotor task, concretely, an eye-hand coordination task, in combination with an obstacle avoidance task that is characteristic in human-robot collaboration. The results provide the proof that increased stress conditions can actually be measured in a significantly correlated increase of an error distribution as consequence of the precision of the eye-hand coordination. The decrease of performance is a proof that the attentional processes are a product of executive function processes. The results confirm the dependency of executive functions and decision processes on stress conditions and will enable quantitative measurements of attention effects in multi-tasking configurations.

## 3 Use Case for Real-time Human Factors Measurements in Assembly Work Cells

The use case for an assembly work cell is depicted in Fig. 6. The worker learns about to assemble a transformer with its individual components, interacting with various areas in its environment, e.g., to pick up a component from one area of a table and to place it to another part of a table.

The learning scenario demonstrates the feasibility of the concentration measurement technology. (1a) View on the worker with focus on work (1a,3a) and being interrupted by a chat with a colleague (2a). Accordingly, the level of concentration measured drops down during the chat (2c) but is high during focused work (1c, 3c). (1b, 2b, 3b) demonstrates the egocentric view of the worker by means of the HoloLens headset with gaze cursor (blue) and concentration (orange) and stress (yellow) measurement bars.

The development of the demonstrator about the learning scenario is in progress. It is overall based on the human factors measurement technologies described above. Upon completion, a supervisor worker will be capable to select specific tasks from a novel assembly work and investigate how well a rooky worker is performing in comparison to expert data which are matched towards the novel data from the rooky. A distance function will finally provide insight into parts of the interaction since with substantial distance to the expert profile and from this enable to put the focus on parts of the sequence that most need training. Furthermore, the pattern of interaction, i.e., gaze on objects and following interactions, is further investigated for additional optimisations if applicable.

## 4 Conclusions and Future Work

This work presented innovative methodologies for the assessment of executive functions to represent psychologically motivated quantitative estimations of human operator performance in assembly processes. We estimated dynamic human attention from mobile gaze analysis with the potential to measure, such as, attention selection and inhibition, as well as task switching ability, in real-time. We presented a suite of different
components with gaze based human factors analysis that is relevant for the measure-
ment of cognitive and psychophysiological parameters. In this manner, this work in-
tends to provide a novel view on human operator state estimation as a means for the
global optimization of human-machine collaboration.

<figure><img src="image_7.png" /><figcaption>**Fig. 6:** Learning scenario demonstrating the feasibility of the concentration measurement technology. (1a) View on the worker with focus on work (1a,3a) and being interrupted by a chat with a colleague (2a). Accordingly, the level of concentration measured drops down during the chat (2c) but is high during focused work (1c, 3c). (1b, 2b, 3b) demonstrates the ego-centric view of the worker by means of the HoloLens headset with gaze cursor (blue) and concentration (orange) and stress (yellow) measurement bars.</figcaption></figure>

Within a typical learning scenario of a human operator within an assembly work cell
and the study setup including application of state-of-the-art intuitive assistance tech-
nology for the performance of collaborative tasks, we illustrated the potential of inter-
pretation from gaze based human factors data in order to evaluate a collaborative sys-
tem.

In future work we will study the potential of more complex eye movement features,
such as, for planning processes, to enable a more detailed analysis of the dynamic dis-
tribution of attentional resources during the tasks.

**Acknowledgments.** This work has been supported by the Austrian Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK) within projects MMASSIST (no. 858623) and FLEXIFF (no. 861264).

**References**

1. Saucedo-Martínez, J. A., et al., Industry 4.0 framework for management and operations: a review, Journal of Ambient Intelligence and Humanized Computing, 2017, pp. 1-13.

2. A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz, and M. Goodrich, "Common metrics for human-robot interaction", *Proc. ACM SIGCHI/SIGART Human-robot interaction*, 2006.

3. C.-M. Huang, and B. Mutlu. "Anticipatory robot control for efficient human-robot collaboration", *Proc. ACM/IEEE HRI 2016*, 2016.

4. Endsley, M. R. "Toward a theory of situation awareness in dynamic systems," *Human Factors*, vol. 37, no. 1, pp. 32-64, 1995.

5. Paletta, L., Pszeida, M., Ganster, H., Fuhrmann, F., Weiss, W., Ladstätter, S., Dini, A., Murg, S., Mayer, H., Brijacak, I., and Reiterer, B. (2019). Gaze based Human Factors Measurements for the Evaluation of Intuitive Human-Robot Collaboration in Real-time. Proc. 24th IEEE Conference on Emerging Technologies and Factory Automation, ETFA 2019, Zaragoza, Spain, September 10-13, 2019.

6. K. Holmqvist, M. Nyström, R. Andersson, R. Dewhusrt, H. Jarodzka, and J. van de Weijler. Eye Tracking – A Comprehensive Guide to Methods and Measures, Oxford University Press, 2011, pp. 187.

7. K. Moore, and L. Gugerty, “Development of a novel measure of situation awareness: The case for eye movement analysis,” *Human Factors & Ergon. Soc. Ann. Meeting*, vol. 54:19, 2010

8. S. M. Munn, and J.B. Pelz, "3D point-of-regard, position and head orientation from a portable monocular video-based eye tracker", *Proc. ETRA 2008*, pp. 181-188.

9. K. Santner, G. Fritz, L. Paletta, and H. Mayer, “Visual recovery of saliency maps from human attention in 3D environments”, *Proc. ICRA 2013*, pp. 4297-4303.

10. C. Heyer, "Human-robot interaction and future industrial robotics applications," *Proc. IEEE/RSJ IROS*, pp. 4749-4754, 2010.

11. R. M. Taylor, "Situational awareness rating technique (SART): The development of a tool for aircrew systems design", in *Situational Awareness in Aerospace Operations*, 1990, pp. 3/1-3/17.

12. Stanton, N.A., P.M. Salmon, G.H. Walker, and D.P. Jenkins. Genotype and phenotype schemata and their role in distributed situation awareness in collaborative systems. *Theoret. Issues in Ergon.Sci*, 2009.
13. Dini, A., Murko, C., Paletta, L., Yahyanejad, S., Augsdörfer, U., and Hofbaur, M. (2017) Measurement and Prediction of Situation Awareness in Human-Robot Interaction based on a Framework of Probabilistic Attention, *Proc. IEEE/RSJ IROS 2017*, 2017.

14. Paletta, L., Pittino, N., Schwarz, M., Wagner, V., and Kallus, W., Human factors analysis using wearable sensors in the context of cognitive and emotional arousal, *Proc. 4th International Conference on Applied Digital Human Modeling*, AHFE 2015, July 2015.

15. L. Paletta, M. Pszeida, R. Marton, T. Haspel, and Nauschnegg, B. (2019). Stress Measurement in Multi-Tasking Decision Processes Using Executive Functions Analysis, *Proc. AHFE 2019 International Conference on Human Factors and Ergonomics*, 2019

16. Bailey, Brian P.; Konstan, Joseph A. (July 2006). On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state. *Computers in Human Behavior*. 22 (4): 685-708.

17. Stephen Monsell, "Task Switching", *Trends in Cognitive Sciences*, Vol.7 No.3 March 2003.

18. A. Miyake, N.P. Friedman, M.J. Emerson, A. Witzki, A. Howerter, T.D. Wager. "The unity and diversity of executive functions and their contributions to complex 'frontal lobe' tasks: A latent variable analysis". *Cognitive Psychology*, 41, 49-100, 2000.

19. A.T. Jersild. "Mental set and shift". *Archives Psychol.*, 89.1927.

20. Kramer, A. F., Hahn, S., & Gopher, D. (1999). Task coordination and aging: Explorations of executive control processes in the task switching paradigm. *Acta Psychol.*, 101, 339-378.

21. Selye H. 1936. A syndrome produced by diverse nocuous agents. *Nature* 138, July 4):32.

22. Lazarus, R. S., & Folkman, S. (1984). *Stress, appraisal, and coping*. New York: Springer.

23. Smith, E., and Jonides, J., Storage and Executive Processes in the Frontal Lobes, *Science*, Vol. 283, March 1999.

24. Dickerson, Sally & Kemeny, Margaret. (2004). Acute Stressors and Cortisol Responses: A Theoretical Integration and Synthesis of Laboratory Research. *Psychological Bulletin*. 130. 355-91. 10.1037/0033-2909.130.3.355

25. Diamond A. (2013). Executive functions. *Annu. Rev. Psychol.* 64 135–168. 10.1146/anurev-psych-113011-143750.

26. Meyer, D. E., Evans, J. E., Lauber, E. J., Gmeindl, L., Rubinstein, J., Junck, L., & Koepp, R. A. (1998). The role of dorsolateral prefrontal cortex for executive cognitive processes in task switching. *Journal of Cognitive Neuroscience*, 1998, Vol. 10.

27. F.Del Missier, T. Mäntyla, W. Bruine de Bruin (2010), Executive functions in decision making: An individual differences approach. *Thinking & Reasoning*, 2010, 16 (2), 69–97.

28. W.B. Cannon, The wisdom of the body, New York, Norton, 1932

29. W. Boucsein, *Electrodermal Acivity*, New York, Plenum, 1992.

30. C. Setz, B. Arnrich, J. Schumm, R. La Marca, G. Tröster, U. Ehlert, Discriminating Stress from Cognitive Load Using a Wearable EDA Device, *IEEE Transactions of Information Technology in Biomedicine*, Vol. 14, No. 2, March 2010.

31. J. Taelman, S. Vandeput, A. Spaepen, S. Van Huffel, Influence of Mental Stress on Heart Rate and Heart Rate Variability, *Proc. Conf. for International Federation for Medical and Biological Engineering*, pp. 1366 – 1369, 2008.

32. N. Sharma, T. Gedeon, Objective measures, sensors and computational techniques for stress recognition and classification: A survey. *Computer Methods and Programs in Biomedicine* 108, pp. 1287 – 1301, 2012.

33. F. Sun, C. Kuo, H.-T. Cheng, S. Buthpitiya, P. Collins, M. Griss, Activity-aware Mental Stress Detection Using Physiological Sensors, *Mobile Computing, Applications, and Services*, pp. 211 – 230, Springer Berlin Heidelberg, 2011.
34. K.F. Van Orden, W. Limbert, S. Makeig, Eye Activity Correlates of Workload during a Visuospatial Memory Task, *Human Factors: The Journal of the Human Factors and Ergonomics Society*, 43, pp.111 – 121, 2001.

35. E. Chajut and D. Algom (2003). Selective Attention Improves Under Stress: Implications for Theories of Social Cognition, *Journal of Personality and Social Psych.*, 2003, Vol. 85

36. J. Bütepage, and D. Kragic (2017). Human-Robot Collaboration: From Psychology to Social Robotics. arXiv:1705.10146.

37. Baddeley A. D., Hitch G. (1974). Working memory, in *The Psychology of Learning and Motivation: Advances in Research and Theory*, Vol. 8 ed. Bower G. A., editor. pp. 47–89.

38. Lezak M. D. (1995). *Neuropsychological Assessment*. New York, NY: Oxford Univ. Press.

39. Koolhaas J. M., Bartolomucci A., Buwalda B., De Boer S. F., Flügge G., Korte S. M., et al. (2011). Stress revisited: a critical evaluation of the stress concept. *Neurosci. Biobehav. Rev.* 35 1291–1301.

40. Hermans E. J., Henckens M. J. A. G., Joels M., Fernandez G. (2014). Dynamic adaptation of large-scale brain networks in response to acute stressors. *Trends Neurosci.* 37 304–314.