From Human Perception and Action Recognition to Causal Understandingof Human-Robot Interaction in Industrial Environments ∗Stefano Ghidoni1 , Matteo Terreran1 , Daniele Evangelista1 , Emanuele Menegatti1 ,Christian Eitzinger2 , Enrico Villagrossi3 , Nicola Pedrocchi3 , Nicola Castaman4 , Marcin Malecha5Sariah Mghames6 , Luca Castri6 , Marc Hanheide6 , Nicola Bellotto1,61
Dept. of Information Engineering, University of Padova, Italy, 2 Profactor GmbH, Austria,
3STIIMA-CNR, Italy, 4 IT+Robotics srl, Italy, 5 Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR),
Germany, 6 Lincoln Centre for Autonomous Systems, United Kingdom{stefano.ghidoni,matteo.terreran,daniele.evangelista,emanuele.menegatti,nicola.bellotto}@unipd.it,
christian.eitzinger@profactor.at, {enrico.villagrossi,nicola.pedrocchi}@stiima.cnr.it,
nicola.castaman@it-robotics.it, marcin.malecha@dlr.de,{smghames,lcastri,mhanheide,nbellotto}@lincoln.ac.uk
Abstract
Human-robot collaboration is migrating fromlightweight robots in laboratory environments to industrial applications, where heavy tasks and powerful robots are more common. In this scenario, a reliable perception of the humans involved in the process and related intentions and behaviors is fundamental. This paper presents two projects investigating the use of robots in relevant industrial scenarios,
providing an overview of how industrial humanrobot collaborative tasks can be successfully addressed.

1

Introduction

Human-robot interaction plays a fundamental role in industrial applications, where it is a key element in the Industry 4.0 paradigm. This is an enabling technology for masscustomization and special production processes, because thehuman experience and dexterity can be combined with thepower of a robot to perform heavy duty tasks.
This paper presents two projects that are strongly based onhuman-robot collaboration. The DrapeBot project, detailedin Section 2, focuses on the co-manipulation of large carbon
fiber parts, lead by the human operator and supported by therobot. The cooperation is based on the AI-driven detection
and understanding of the human activity observed by a camera network equipping the workcell. The DARKO project,presented in Section 3, proposes a new generation of mobile
robot manipulators for intralogistics applications. The robots
are equipped with advanced human-robot interaction capabilities including predictive planning, mutual intention communication, context-based modelling, and causal reasoning. The
goal is for the robot to understand and anticipate the dynamics of potentially unknown environments to enhance safety
and operational efficiency.
∗This research has received funding from the European Union’sHorizon 2020 research and innovation program under grant agreements No. 101006732 and No. 101017274.

Figure 1: A human operator cooperates with a robot for carbon fibertransportation.

2The DrapeBot Project

Draping is one of the most complex operations in carbon fibermanufacturing. It is carried out by transporting the carbon
fiber fabric onto the preform and adapting its shape. The
success of this process depends on the manual ability of thehuman operator to correctly place the fabric and model theshape: this means that a high degree of manual ability isneeded, that can hardly be achieved using an automatic process. This process is usually performed by human operators.
When it comes to large components, usually two or more
highly skilled operators are needed—one usually controlling
the process, and the other(s) acting as a support. This is, atthe same time, expensive and unneeded, because one single
skilled operator could manage the draping process, while thetask of the additional operators could be carried out by an automatic system capable of following the indications provided,
as exemplified in Fig. 1. This means to better exploit the skillsof the human operator who acts as the leader of the draping
process, guiding the whole process and assessing its quality,
while, at the same time, employing robots for executing theFigure 2: Example of body pose estimation system, a key elementof the smart workcell.

more trivial but heavy duties.

2.1

Human-Robot Interaction for Carbon Fiber
Parts Manufacturing

The European project DrapeBot (www.drapebot.eu) has thegoal of developing a novel human-robot interaction technology capable of supporting one single human operator handling large carbon fiber parts. This is based on cutting edge
AI tools, both at the perception and the action stage. In suchcontext, a thorough understanding of the scene is needed toproperly feed the AI systems that are in charge of determining
the robot action and motion in real-time.

2.2

The Smart Workcell

DrapeBot aims at developing a smart workcell, that is, aworkcell that is aware of the processes that take place inside
the workspace, thanks to the sensors and the artificial intelligence modules that equip the cell. Understanding human behavior starts from perceiving the human body posture, involving Action Recognition [1], people re-identification [2] andhuman-computer interaction—a typical output of a body pose
estimation system is shown in Fig. 2. In recent years, severalefforts have been done to obtain fast and reliable human motion estimates in real-time. Several works, like [3], focused
specifically on employing image processing techniques togain knowledge about the human pose in real-time. In somecases, filtering has been also applied to provide smoother results [4].
The current status of the workcell is another key element.
For example, similar gestures may have different meanings
depending on the current status of the workcell. To sense theenvironment in the most flexible way, semantic segmentation
is employed–this is a per-pixel classification task based on aset of classes representing important elements in the workcell.

2.3Workcell Environment Understanding

Focusing on the perception stage, DrapeBot aims at enhancing the state of the art and the industrial practice by combining visual sensors with inertial sensors to obtain a fast andaccurate perception of how human workers move in the workcell. This will be accompanied by a semantic interpretationof the environment. In this way, the system will ensure asmooth and reliable interaction with the human(s), that will
not be blocked nor slowed down due to temporary perception
failures caused, for example, by an occlusion preventing thevision system to perceive the human operator. Such high-end
perception systems will be employed to provide a thoroughunderstanding of the working environment and the body poseof the person(s) present in the working area of the robot. A
probabilistic model for human body pose estimation will bederived, capable of taking advantage of both visual and inertial sensors, enhancing the detection when both are available but capable of working when only one sensory system
is working. An important element of the project will be thedefinition of the sensing accuracy needed to meet the safety
requirements of the robotic workcell.
The semantic environment perception equipping the workcell will be a key element for detecting in detail the current
status of the workcell, because it can provide a high-level description of the components of the environment, enabling automatic evaluation of the current status of the production process, possible misalignments, and possible risks when workers are in proximity of dangerous moving elements.
The DrapeBot project will enhance the state of the art
and the industrial practice by deploying an integrated layered
framework where the Task and Motion Planning (T&MP) andthe low-level control will be jointly designed to work in fullsynergy. The T&MP will manage human and context information to do both: reason on the user model and interpret
several sensorial information. Sensor data will come from therobotic system and the human perception system, that will include an AI-based human action and intention understanding
module. This will enable optimization of task sharing between humans and robots and selecting the processing areas
to minimize human interference. The low-level control will
exploit the estimated human intention control dealing withthe compensation of the model inaccuracies, and allowing acontinuous adaptation on the best task and motion plan computed by the T&MP module.
Considering the T&MP layer, the core advancement willconsist in the deployment of modules designed for efficient
on-line computation of a sub-plan after performing each action. Thus, exploiting at each iteration updated sensory readings, it implicitly takes into account unexpected changes inthe estimated state due to previous sensors and actuators failures, or environment re-configurations. This framework will
integrate a human-aware motion planner that will generate acollision-free trajectory provided in real-time leveraging ona data-driven approach (e.g, a Deep Reinforcement Learningframework) to improve the planning effectiveness.

3The DARKO Project

Mobile robot manipulators can play a crucial role in future
agile production processes. In order to be effective though,they must demonstrate high levels of technical capability,
adaptability to dynamic environments, and advanced humanrobot interaction skills. Despite significant progress in recentyears, the current state-of-the-art does not meet yet the requirements of tomorrow’s industry.

Figure 3: Concept model of the DARKO project. Mobile robot manipulators operate in intralogistic environments exploiting predictiveplanning and safe human-robot spatial interaction.

The goal of the H2020 project DARKO1 – Dynamic Agile production Robots that learn and optimise Knowledge andOperations – is to develop a new generation of production
robots for highly dynamic manipulation tasks, which are ableto navigate safely and efficiently thanks to new predictiveplanning and human-robot interaction capabilities (Fig. 3).
In particular, the project aims to answer scientific and technical questions about dynamic manipulation (e.g. throwingan object) enriched by context- and human-aware navigation
in intralogistic settings.

3.1

Human-Robot Spatial Interaction

One of the key objectives of DARKO is efficient and safe
human–robot co-production, which is addressed by advancing the state-of-the-art in human-robot spatial interaction(HRSI), i.e. the way robot and human move with respect toeach other. This includes new methods to learn, predict, andexploit activity patterns in shared environments; communicate mutual navigation intents; and create a framework forrisk-aware robot planning and coordination.
The project will extend previous work of the partners on
human detection, tracking, and motion prediction. It willfurther investigate the use of spatial augmented reality andanthropomorphic gestures for robot’s intents communication. Finally, new models for context-aware representation of
HRSI will be created, including novel solutions for causal understanding of the environment’s dynamics, so that the robot
can operate safely and efficiently even in case of unexpected
situations (i.e. no training data).

3.2

Context-aware Representations of HRSIs

Human spatial interactions, defined as the mutual influenceof motion behaviours between two or more people, depend on
both human activities (e.g. speed and destination) and objects
or constraints (e.g. nearby door, narrow corridor, etc). Similarly, HRSIs are influenced by the robot’s goal and nearby
humans, but also other factors outside the direct control of theinteracting agents (e.g. narrow corridor in Fig. 4). In many1

https://darko-project.eu/

Figure 4: Example of HRSI where the robot stops to let the operator
pass in a narrow corridor.

real-world scenarios (e.g warehouse, hospitals, etc), the environment is highly dynamic, with sudden variations of agentsand potential interactions. The success of a production line isdirectly linked to each worker’s tasks, whether these are done
collaboratively or individually. The efficient completion ofsuch tasks on the ability of the agents to communicate theirmutual intents, so that pre-defined plans can be adapted andoptimised. A requirements for intent communication the ability of the robot to predict and implement the most opportune
spatial interactions with the surrounding agents.
In order to better capture the nature of HRSIs in realistic
scenarios, as the logistic ones, and therefore enable safer andmore socially acceptable robot motion behaviours, DARKO
will create new context-aware representations of HRSI embedding the relative motion with respect to other relevant
features of typical production environments (e.g. workcells,storage areas, forklifts, etc.). However, given the complexityof real-world dynamic environments, a continuous or purely
quantitative description of such relations is computationallyunfeasible for the operational requirements of a production
robot. Therefore, a neuro-symbolic representation of HRSIs
is adopted, which is based on a theoretically-sound Qualitative Trajectory Calculus (QTC) for moving objects [5]. This
representations will extend previous work on QTC-based
HRSI to explicitly model and predict the relative motion ofthe robot with respect to nearby humans and other static (e.g.
doors, pallets) or dynamic (e.g. forklifts) objects. The increased complexity of these models, driven by the number ofpeople co-existing in a workplace, requires new algorithmic
solutions for dealing with space dimensionality, uncertainties, and possible lack of information (e.g partially observablespace).
The proposed qualitative models will provide an interface
between the classic sensor-level perception of objects andhumans in the robot’s environment, and the higher-level in-terpretation required to understand the causal relations between its agents, focusing in particular on human-human andhuman-robot spatial interactions. These models can also beused to infer the safest robot motion behaviour in proximity topeople and to provide a human-readable description of HRSIs
for post-hoc analyses and risk assessment.
Current work in that direction includes the development ofpredictive models for qualitative interactions between agentsin highly dynamic scenarios, considering robot, humans, andtheir neighboring objects (e.g doors). To this end, someof the available open-source datasets in public spaces (e.g.
JackRabbot2 ) are being considered. To assess the computational complexity and efficiency of algorithms that canadapt to domain shifts for different environments and scenarios, a performance comparison is being performed between
different hybrid symbolic/deep learning-based architectures
for predicting qualitative interactions between human-humanand human-robot spatial interactions.

3.3

Causal Reasoning for Safe HRSI

Safety is a key requirement of any human-robot interaction.
New methods for the analysis of different safety aspects areneeded to increase this requirement. In DARKO, safety will
be intensively put at test since the robot could face many different complex situations, including working and sharing theenvironment with humans, the behaviour of whom can be affected by other agents and objects in the environment. Dealing with such situations implies a deeper understanding of
human motion intentions and possible factors that influencethem, besides the robot’s presence. Causal Inference, i.e. the“science” of understanding cause and effect [6], could represent a turning point in how robots cooperate with humans
flexibly and, most importantly, interact with them safely.
One of DARKO’s goals is to make the robot able to learn
the effect of its behaviours by observing how humans react
to its actions. This ability will require the robot to discoverand reason on cause-effect relations and, given a desired motion plan, choose the most effective HRSI, taking into account both safety and task performance. The project therefore
will develop new algorithms to learn and reason on causal
models of HRSI, enabling the robot to compute intervention(e.g. “what happens if I move and get close to that human?”)and counterfactual probabilities (e.g. “what would have happened if I remained still instead of moving?”), as illustrated
in Fig. 5. By using these probabilities, the robot can choose
the most appropriate behaviour in proximity of humans, increasing safety and efficiency of HRSIs.
To this end, causal discovery algorithms are being investigated with application to real-world scenarios similar toDARKO (e.g. THÖR3 dataset), in order to study the behaviours of human agents in warehouse environments andto reconstruct the underlying causal model which describes
them.
23

http://svl.stanford.edu/projects/jackrabbot/
http://thor.oru.se/

Figure 5: Example of intervention (blue trajectory) and counterfactual (red trajectory) in HRSI.

4Conclusions

This paper presented two projects investigating the humanrobot interaction scenarios and applications in industrial contexts, DrapeBot and DARKO. The former proposes the designof a smart workcell that can actively support the human operators in a production process. The sensory system installedin the workcell can understand human behavior and intention thanks to a wide use of AI systems, that are integrated
with the robot task and motion controller. The latter focuseson the development of a new generation of mobile robot manipulators that use advanced AI methods for modelling andpredicting human-robot spatial interactions, including algorithms for causal reasoning to improve safety and operational
efficiency in dynamic environments.

References[1] F. Han, X. Yang, C. Reardon, Y. Zhang, and H. Zhang,“Simultaneous Feature and Body-Part Learning for realtime robot awareness of human behaviors”, 2017 IEEE
Int. Conf. on Robotics and Automation (ICRA), pp.
2621-2628.
[2] S. Ghidoni, and M. Munaro, “A multi-viewpoint
feature-based re-identification system driven by skeleton keypoints”, Robotics and Autonomous Systems, 90,
45-54, 2017.
[3] Z. Cao, T. Simon, S.E. Wei, and T. Sheikh, “Realtime multi-person 2D pose estimation using part affinityfields”, in Proc. of IEEE Conf. on Computer Vision andPattern Recognition, pp. 7291-7299, 2017.
[4] A. Malaguti, M. Carraro, M. Guidolin, L. Tagliapietra, E. Menegatti, and S. Ghidoni, “Real-time trackingby-detection of human motion in RGB-D camera networks”, in 2019 IEEE Int. Conf. on Systems, Man andCybernetics (SMC), pp. 3198-3204.
[5] Delafontaine, M., Chavoshi, S. H., Cohn, A. G., Vande Weghe, N. (2012). A qualitative trajectory calculus
to reason about moving point objects. In Qualitative
spatio-temporal representation and reasoning: Trends
and future directions (pp. 147-167). IGI Global.
[6] Pearl, J., Mackenzie, D. (2019). The book of why. Penguin Books.