Preprint version of the IEEE 2023 European Conference on Mobile Robots (ECMR), Coimbra, Portugal 4 – 7 September 2023

Dynamic Human-Aware Task Planner for Human-Robot Collaborationin Industrial Scenario
Alberto Gottardi1,2,† , Matteo Terreran1 , Christoph Frommel3 ,Manfred Schoenheits3 , Nicola Castaman2 , Stefano Ghidoni1 , Emanuele Menegatti1
Abstract— The collaboration between humans and robots inindustrial scenarios is one of the key challenges for Industry 4.0.
In particular, industrial robots offer accuracy and efficiency,
while humans have experience and the capability to manage
complex situations. Combining these features can enhance theindustrial process by avoiding the user manipulates heavy
weights and allowing him to dedicate his efforts to tasks
where flexibility, quality and experience make the difference inthe final product. However, the collaboration between humans
and robots raises several new problems to be addressed likesafety, tasks scheduling and operator ergonomics. For example,
human presence in the robot workspace introduces various
elements of complexity into robot planning due to its dynamism
and unpredictability. Planning must take into account how tocoordinate the tasks between the robot and the human andbe quick in re-planning to respond reactively to the operator’s
trigger. For this purpose, this work proposes a hierarchical
Human-Aware Task Planner framework capable of generatea suitable plan to complete the process and manage userinterrupts in order to have a constantly updated plan. The
method is evaluated in a real industrial scenario and in a specificcomplex assembly task like the draping of carbon fiber plies.
Index Terms— Human-Aware Task Planner, Human Action
Recognition, Human-Robot Collaboration, Dynamic industrial
scenario

Fig. 1: Dynamic Human-Aware Task Planning Frameworkfor Human-Robot Collaboration.
Since a shared industrial environment between humans
and robots is a highly dynamic scenario, the classical taskplanner approaches are infeasible: they assume that theworkspace is deterministic, the state is fully-observable, therobot is the only agent that can change the workspace, andactions are instantaneous. Therefore, to be usable in dynamic
environments, task planners must deal with unpredictable andpartially uncontrollable situations, especially due to humanbehaviour [2]. Several approaches have been investigatedover the last few years to handle the dynamic scenario:
from Artificial Intelligent Planning like PDDL [3] or Markov
Decision Process [4] through Finite State Machine [5] totimeline-based approaches [6]. Multi-level programming [7]and Task Allocation [8] solutions are also very common approaches. Finally, game-theory and Reinforcement Learning(RL) models and methodologies are widely applied to multiagent task scheduling problems [9], [10].
This paper proposes the Dynamic Human-Aware Task
Planner framework for HRC in an industrial scenario summarized in Fig. 1. In particular, it focuses on the dynamicscheduling of shared human-robot activities within a manufacturing environment where humans and robots have tocollaborate to complete complex tasks like object sorting,production line assembly [11] or draping [12].
Draping is one of the most complex operations in carbon
fibre manufacturing. It is carried out by transporting thecarbon ply onto the mould and adapting its shape to themould. Nowadays, this process is completely manual andperformed by expert human operators. In addition, the humanoperator is in charge of transporting the plies from a table

I. INTRODUCTION
Human-Robot Collaboration (HRC) in the industrial scenario is one of the most important technological challengesof recent years. The synergy between the robot’s abilities,
like precision, accuracy, efficiency and repeatability, alongwith human intelligence, flexibility and experience provides
several advantages because it reduces the operator’s effort
and improves ergonomics during the operations, ensures
the production quality and accuracy [1]. To be able tofully benefit from these advantages, while at the same timeensuring user safety when working with the robot, intelligent
task coordination between humans and robots is required.
A task planner takes over this intelligent coordination ofactivities. Moreover, this implies that it is necessary to use
a planner that takes into account the user.
† Corresponding Author
1 Intelligent Autonomous System Lab, Department of Information Engineering, University of Padova, 35131 Padua, Italy. gottardial,matteo.terreran, ghidoni, emg@dei.unipd.it
2

IT+Robotics

srl,

36100

Vicenza,

Italy.

nicola.castaman@it-robotics.it
3(DLR),

Deutsches
ZentrumfürLuftund
Raumfahrt
e.V.
Augsburg,Germany.
christoph.frommel,

manfred.schoenheits@dlr.de979-8-3503-0704-7/23/$31.00 ©2023 IEEE1

to the mould and then draping it. The EU project DrapeBot1
aims at developing an HRC system capable of assisting an
operator working on the draping of carbon fibre parts. In
order to manage that process, a collaborative task plannermust be used.
To address the requirements outlined above, we proposea hierarchical task planner that exploits the symbolic description of the process, the definition of Primitives andComposites actions and human commands to generate asuitable and continuously updated plan for the assembly
process. In detail, our contribution follows:• Dynamic Human-Aware Task Planner framework,
which is able to compute human and robot activities
and handle the user commands to update the plan.
• Using primitive actions to create more complex, socalled composite actions, which contribute to the creation of the final plan.
• Intelligent Action Recognition to trigger activities orrobot behaviour not foreseen in the plan but which theexpert user wants to perform.
II. RELATED WORKS
In recent years, task planning problems for HRC havebeen investigated. Existing works like [13], [14] tried to
explore the knowledge encoded in the CAD model to extract
the product’s assembly sequence. Other works focused on
sub-problems such as scheduling human and robot actions
through Petri Nets [5], [15], or cooperative planning at a
symbolic level [16], [17]. These approaches work better
only in a classical static environment. Indeed, they cannot
handle dynamism, uncertainty and the possibility of the user
triggering unforeseen actions as a Human-Aware dynamic
scenario requires and as proposed in our approach.
Nikolakis et al. [18] proposed a hierarchical method
based on multi-criteria decision-making for an offline task
allocation and a dynamic replanning due to unexpected
events. Related works which used multi-criteria decisionmaking framework are [19], [20]. In these works, the authors
considered robots and humans as resources. They developed
task allocation approaches capable of handling unexpected
events but not capable of handling specific commands/actions
desired by the user. An unexpected event can be considered
as a generic trigger where different events could correspondto a generic reaction. A user’s command, instead, is a specifictrigger, i.e., each command corresponds to a specific reaction.
However, this part is crucial because the operator is an important subject inside the process, and his ability is fundamental
to improve the process. Our method proposes solving thisgap using the action recognition module connected to thetask planner.
Graph-based approaches are described in [21], [22]. The
modelling of the process takes place via AND/OR graphthat can handle the parallelism of two actions assigned totwo different resources. However, they cannot handle theorder of precedence constraints typical of assembly tasks
1 https://www.drapebot.eu/

and how our approach aims to address and resolve. In otherwork, instead of using graph-based approaches, the authors
exploit the advantages of the Behaviour Tree (BT) [23],[24]. In particular, Lamon et al. [24] have combined the BT
approach with a Mixed-Integer Linear Programming (MILP)
based role allocation method that allows individual andcollaborative roles within the same formulation. However,human uncertainty is not modelled and considered. But, an
intelligent system has to consider human intentions in its
decision-making rather than force the operator to follow astrict, predefined assembly plan. In our proposed method,the operator can directly interact with the system and force
the robot to execute some tasks by the action recognition
module. Human intentions are typically modelled through thePartially Observed Markov Decision Process (POMDP) [25].
Approaches which shared similarities with Cramer et al. [25]modelled the collaborative task like hierarchical task network(HTN) [26], [27], [28]. The latter directly employs first-orderlogic to enable the robot to estimate its partner’s goals andanticipate correctly in the presence of human variability andnon-deterministic sensing. Another work related to the HTN
is [29], where the task planner is able to divide the plan into
multiple streams for multiple agents, humans included.
Most of the approaches described above have an implicit
representation of the time. Actions are supposed to beinstantaneous so that the action effects become true when
the action itself is applied and changes the environment or
the situation. Therefore, states and goals are not supposed
to have a temporal extension such that they hold only fora limited temporal interval, or that they must be achieved
within known temporal bounds. The planners that follow thisapproach are temporal planning, and the main feature is thatthey synthesize plans by combining causal reasoning withtime and resource reasoning [30], [6].
In [6], Umbrico et al. proposed a timeline-based planner
called PLATINUm with the ability to deal with temporal
uncertainty at the planning and plan execution levels. The
same authors then improved that tool by proposing an evolution of it, called TENANT [31], capable of setting objectives, defining tasks and establishing operational constraints,
despite the inherent complexity required in planning androbotics. Although these works are evaluated in industrial
applications, these approaches are not able to be adapted or
rescheduled based on real-time observations by the operator.
Indeed, adapting plans on the fly can be difficult, especiallywhen the original plan heavily relies on strict time constraints
like in these approaches.
Finally, significant advances in Deep Reinforcement
Learning (DRL) have been witnessed in many outstandinglarge-scale sequential decision-making problems [10], [32].
Hu et al. in [33] exploited the combination of timed-place
Petri nets with the deep Q-network with GCN to manage thedynamic scheduling problem of an industrial manufacturing
scenario. In the same application, Kim et al. in [34] proposed the RL approach where intelligent agents evaluate thepriorities of jobs and distribute them through negotiation.

Fig. 2: Example of a plan with composite and primitive
actions.
III. SYSTEM ARCHITECTURE
This section presents the Dynamic Human-Aware Task
Planner framework that handles the entire process andhuman-robot planning. Fig. 1 depicts an overview of thesystem proposed.
The Task Planner module coordinates the human androbot activities. It is responsible for creating a continuously
updated plan that serves as a guideline for the workflow andwill be composed of the sequence of actions to achieve theassigned task. Finally, the Task Planner must manage thehuman intentions to adapt the computed task plan to meetthe collaboration needs dynamically or to handle unexpected
situations and use recovery actions to return to a safe state.
A simple example of the plan for a draping process isshown in Fig. 2 where it is composed of composite action
like Transport that represents the activity to transfer the
carbon fibre ply from a picking table to the mould, andprimitives like Draping and Inspection that represent theactions performed by the human operator that drapes theply into the mould and checks that no defects have formed
during the previous activity.
The second important module is Action Recognition,
which recognises the gestures associated with triggering specific actions. The associated command is sent to the Central
Node, which is the module that monitors the operation of thesystem and sends the commands to the other modules that are
in charge of performing the action in the plan. The Central
Node manages the information the sensors provide in theworkcell. Finally, a state-of-art Motion Planner is involvedin order to generate a collision-free trajectory for the robot.
A. Task Planner
The task planner structure is outlined in Fig. 3 and isdeveloped following a hierarchical approach consisting of 3
different layers:• A low layer consisting of Primitive Actions.
• A middle layer consisting of Composite Actions.
• A high layer consisting of the Plan of the entire process.
Each layer has its own distinctive characteristics and adifferent level of abstraction with respect to the final task.
In the lower layer, we have the Primitive actions which
represents the activities to be performed (e.g. Move, Draping,
Inspection, etc.). The robot and the human alone could
execute this activity, or both agents are required. A seriesof preconditions and effects characterise a primitive action.
The preconditions are verified directly by the primitive itself,while the effects describe the state changes. When one of thepreconditions is not satisfied, the current state is invalid andthe primitive cannot be sent to execution, i.e. the related
activity cannot be performed. Therefore, the primitive itself
notifies the Central Node that the state is invalid. The central

Fig. 3: Hierarchical structure of the Task Planner: the primitive actions are in the lower layer, the composite actions arein the middle layer and the final plan is in the highest layer.
node triggers the related recovery behaviour in order to return
to a valid state. A recovery behaviour is a specific and simple
action, strictly related to the primitive itself, that is responsible for returning the system to a valid state, thus allowing
the process to continue while limiting external intervention
to a minimum. For example, if a robot’s gripper fails to
grasp an object, recovery could be deactivating the gripper
and retrying the grasping operation, i.e., reverting the state,
performing the object detection again and re-evaluating theprecondition primitives for the grasp action. Another more
complex example could be the Piece-Detection primitive,
which involves localizing the object on the pick-up tableand providing a suitable grasping point. If the algorithm does
not find the desired object, two recovery behaviours can be
activated: the first starts a second scan of the table by moving
the camera in a slightly different position; the second, if theprocess allows (i.e. without violating precedence constraints)
searches for the next object to pick-up. The action associatedto recovery behaviour is defined as a primitive, with its
precondition (if needed) and effects. After the interventionof a recovery behaviour, it is verified whether the current
plan is still valid and whether the preconditions of that
action are now valid. If this happens, the primitive is reexecuted, otherwise a re-planning is required. A set of
specific configuration files defines the primitives and theirstructure.
In the middle layer, we have the Composite actions defined
as a logical sequence of primitive actions. The composite
has both preconditions and effects, corresponding to the firstand last primitives, respectively. Similar to the primitives, thecomposite has a set of recovery behaviours triggered whena transition between one primitive and the next fails. The
sequence of the primitives is defined in an external configuration by an expert operator who has to collaborate withthe robot in the workcell. The definition of primitives andcomposite actions are provided in input to the Task Planneras shown in Fig. 1. The symbolic language used to model theprimitives is the PDDL [35] because its action is precisely
defined by a set of parameters, preconditions, and effects
required by the Task Planner. Fig. 4 depicts an example of
the Transport composite action which includes the primitives

Fig. 4: Example of the Transport Composite Action (CA) as
the sequence of Primitive Actions (PA).
Move, Piece-Detection, Activation and Deactivation of thegripper.
Finally, in the highest layer, we have the Plan for theentire process. By construction, this is the most abstract layer
and where the definitions of composites and primitives areused together with information from the assembly process,environment and human operator to create the process plan.
Assembly applications contain precedence constraints tomanage and define the order in which the main components
should be mounted. For these reasons, it was decided to use
a Direct Acyclic Graph (DAG) approach with weighted arcs
to represent all the possible alternative plans. In particular,
each node of the graph represents an action (primitive orcomposite), while each arc represents the dependencies thatmust be fulfilled. For example, some objects must be placed
before others in assembly tasks. Therefore, during the building of the graph, the task planner has to take into account that
aspect. For example, as shown in Fig. 5, the Action 5 must be
performed only after Action 3 and Action 4. In addition, eacharc has an associated cost representing the effort of the robot
and the user respectively in performing the action associatedwith the transition between the two nodes. The goal is tocreate a plan that minimises the user’s effort and maximises
the robot’s effort by exploiting the possibility of having
the robot perform some actions while the user performs
others in the same collaborative workcell. Therefore, using
the DAG (Alg1 - line 1) it is possible to find a topologicalordering which describes the sequence of actions to complete
the process (Alg1 - line 2). However, a DAG may containmore than one valid topological ordering. For this reason,
the Depth-first search (DFS) algorithm was used for the
topological search and optimised the cost function. The costfunction used to calculate effort is the same for user androbot and it is the sum of the weights on the arcs in the
DAG in Fig. 5. The mathematical formulation of the costfunction follows:
X
X
X
C(w) =
wi,u +
wi,ru −
wi,r
(1)
u

ru

u

where wi,j represents the weight of ith arc and j represents to which agent the weight is associated, whether tothe user (u), the robot (r) or both (ru) when that action
is to be performed by the two agents together. In order to
optimize the cost function and obtain the plan that minimizes
the user’s effort, the arg minw C(w) is taken (Alg1 - line 3).
In this way, a plan can be found to complete the process anddescribed by the sequence of actions to be performed by the
robot and user.
B. Human Action Recognition
The Human Action Recognition module monitors humanactivities during the collaboration, such as phases of theFig. 5: Plan described by the Direct Acyclic Graph (DAG)
where the action is represented by the node and the effort
associated by the arc’s weight.
Algorithm 1 Task Planner
Input: P A Primitive Action set, CA Composite Action set, Pd
Process description, state current state, a action failed, hhuman command
Output: P Plan
1: G ← buildDAG(P A, CA, Pd , state, a, h)
2: < T P, C(w) >← f indAllT opologicalOrdering(G)3: P ← arg minw C(w) ∈< T P, C(w) >4: return P

Algorithm 2 Central Node
Input: P A Primitive Action set, CA Composite Action set, Pd
Process description, H human command set
1: P ← T askP lanner(P A, CA, Pd )2: for all a ∈ P do
3:valid ← evaluateP recondition(a)4:
if valid then5:state ← Execute(a)
6:
else
7:
(state, valid) ← RecoveryBehaviour(a)8:if valid then9:
go to 3
10:
else
11:
P ← T askP lanner(P A, CA, Pd , state, a, null)12:
end if13:
end if14:h ← HumanActionRecognition(), h ∈ H
15:
if h then16:
P ← T askP lanner(P A, CA, Pd , state, null, h)17:
end if
18: end forprocess (e.g., draping) or particular gestures to provide
commands to the robot (e.g., request a new ply). Suchinformation allows the task planner to be constantly updated
on the current activities of the human operator: the taskplanner can periodically check whether the human is stillengaged in particular tasks (e.g., draping) or whether throughthe use of gestures it is requesting specific actions from therobot that require the generation of a new task plan.
The human action recognition module is based on aprevious work [36], where a graph convolutional neuralnetwork was proposed to recognize common human actions
and gestures which arise in a collaborative manufacturing
scenario. Such network takes as input a sequence of human
3D poses (i.e., skeletons) and tries to classify human movements according to a set of actions of interest by analyzing
both spatial and temporal information.
In this work, human poses are provided by the state-of-the performance of the Task Planner analyzing the computational time spent to create a suitable plan, considering also
the replanning phase, and the performance of the Human
Action Recognition system. Finally, a qualitative analysis ina real scenario is provided.
Fig. 6: Example of replanning due to user command.
the-art monocular 3D pose estimator MeTRAbs [37]. Suchestimator outputs human poses composed of 19 keypoints
describing the main body joints (e.g., torso, arms, legs).
Differently from [36], where actions were recognizedusing an ensemble of various models specialized for different
body parts (e.g., body and hands), in this work we focused
only on the body, as during a real manufacturing application
the hands are thickly occluded and difficult to estimate
accurately.
C. Central Node
The Central Node executes the plan by activating the correct primitives and supervising their execution. In addition,
this module is always aware of the state of the workcell
through the sensors present in the scene, e.g. a camera
network positioned around the robot workcell, laser scanners,
etc. A second purpose of the central node is to handle invalid
state situations one may find oneself in during the executionof the process plan. When a primitive precondition is notsatisfied (Alg2 - line 3), it is notified that the state is invalid
and the central node will trigger the corresponding recovery
behaviour in order to return to a valid state (Alg2 - line
7). When this happens, the primitive is asked to re-verify
whether the preconditions are satisfied (Alg2 - line 9) inorder to verify whether the current plan is still valid or if are-planning is necessary (Alg2 - line 11). In the last case, thecentral node notified the task planner module that the current
plan was unfeasible and a new plan was required. A similar
situation occurs when the user wants to make the robot
perform a task not foreseen in the plan (Alg2 - line 14). In
this case, through the Human Action Recognition system, theuser executes a specific command associated with a specificaction to be done, be it a primitive or a composite. For
example, as depicted in Fig. 6, when the Inspection action
was finished, the user noticed some defects and decided totake a picture of the area where the defects were present. In
order to perform this action, which was not included in theoriginal plan, he performed the specific gesture associatedwith the composite TakePicture. The central node receives
this information and sends it to the Task Planner module
which is in charge of creating a new plan where the requested
action is the first action to be performed. The requested
action is treated as a precedence constraint to add to the
DAG.
IV. EXPERIMENTS
The Dynamic Human-Aware Task Planner developed inthis work was tested in a specific assembly scenario whichis the draping of fibre carbon plies. First, we described thedraping process and the actions involved, then we evaluated

A. Case Study
Draping is a complex industrial operation that requires
an advanced skilled user who is not only responsible fordraping onto the mould but also for a series of activities
such as inspecting the part, noting by text and/or photos ofcertain areas if they have slight defects, and checking thatthe orientation of the fibres is correct. The transport of aply could be executed by the operator or robot alone, or it
could be a collaborative transport where human and robot
are involved. Thus, the coordination of the activities, likethe robot’s motion, activation/deactivation of the gripper toperform the pick and place and the detection of the piece inthe picking table is crucial.
In addition, the operator could use a gesture to trigger
an action which it was not in the original plan (Fig. 6) or
to notify the central node that the current action has been
completed and to move on to the next one. An exampleof this situation is when the operator has finished draping
the ply and wants to notify the central node so that it canperform the next action. This is a simple and intuitive way
for the user to interact with the robots and provide his/herexperience into the system. Therefore, analyzing the process
and the activities to be done, a set of gestures of interest
has been defined based on the possible human interactions
which can arise during the process. In particular, a set of 6
human actions has been considered: Detection, Inspection,
Transport, Draping, Drape Next and Take Picture.
As described in the previous paragraph, a configuration fileis used to represent the precedence constraints and define
the order in which the plies are draped. Also, the sets ofprimitives, composite and gesture are defined in order toperform the entire draping process. Table I provides an
overview of the actions used to evaluate the Task Planner,
while a detailed list of the human actions of interest and theirdescription is provided in Table II.
Action name

Description

Move
Gripper Activation
Gripper Deactivation
Piece Detection
Draping
Transport
TakePicture
Annotation
Inspection

Primitive that represents the motion of the robot and/or the operator
Primitive that represents the activation of the gripper to pick up the ply
Primitive that represents the deactivation of the gripper to place the ply
Primitive that represents the detection of the plies on the table
Primitive that represents the draping of the ply onto the mould by the operator
Composite that represents the transport of the ply from the picking table to the mould
Composite that represents the saving of an image of a certain mould area
Composite that represents the saving of information by the user operator
Composite that represents the inspection of the draping plies onto the mould

TABLE I: Overview of actions used to evaluate the Task
Planner.
B. Task Planner Evaluation
The Task Planner was evaluated through the entire draping
process using the actions shown in Table I and 20 independent process trials were performed. For each trial, theplan considered the draping of 5 plies. As mentioned above,

Action name

Meaning

Description

Detection
Inspection
Transport
Draping
Drape Next
Take Picture

Trigger the Object Detection
Stop current robot operation
Collaborative transport
Manual draping
Require next ply
Take a photo of the ply status

Clap with stretched arms above head
Raise one hand with a stretched arm
Move while holding one side of the ply
Drape the ply on the mould
Move the right arm bend 90°
Point at the desired location to be framed

TABLE II: Set of actions of interest considered for evaluating
the human action recognition module in the proposed casestudy.
Trial

First Plan Time [ms]Replanning Time [ms]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

3.36
3.58
3.79
3.81
2.95
2.14
2.5
2.4
3.52
3.04
2.42
3.86
3.15
3.62
3.35
2.32
2.42
2.61
3.76
2.70

3.16
4.08
5.9
4.1
6.79
5.72
5.95
3.43
3.13
3.5
3.49
5.09
3.81
4.18
5.86
3.67
4.37
6.12
6.52
5.56

Average

3.065

4.7215

Validity

19/20

19/20

TABLE III: Time for the first plan (left) and average time
for replanning (right) in ms.
the computational time is used to evaluate the performanceand the interaction with the user was done by the Human
Action Recognition system. Therefore, when the Central
Node received the gesture it would either perform the next
action in the plan or send a message to the Task Plannerthat a re-planning was necessary. Task Planner and Central
Node were running in a Lenovo ThinkPad with 11th Intel
Core i7 processor and 16GB of RAM. The results obtained
are summarized in Table III where the time is expressed inmilliseconds (ms).
As shown in Table III, the planner demonstrated highefficiency in generating the first plan, with an averagecomputational time of 3.065 ms. However, when the planner
had to replan in response to a user’s command, it was slightly
slower, with an average computational time of 4.72 ms. The
minimum and maximum computational times observed were
2.14 ms and 3.86 ms for the first plan, and 3.13 ms and 6.79
ms for the replanning phase, respectively. All the values in
the replanning column are the average of all replanning that
happened during the trial. In fact, in this way, it is possible
to consider when the rescheduling has taken place. If you
replan at the beginning of the process, there are a lot of tasks

Fig. 7: Evaluation of the action and gesture recognition
performance on the test set as confusion matrix.
afterwards, so it takes a lot of time; vice versa if you replan
close to the end of the process, it will be faster because there
are fewer tasks left to complete the process. Out of all thetrials conducted, there was only one instance (Trial 13) where
the Task Planner provided an invalid plan both as the firstplan and during the replanning phase. The computationaltime was recorded in this case, and the trial was aborted.
These findings highlight the efficiency of the planner ingenerating initial plans, with most plans being valid andfeasible. The slight increase in computational time during
the replanning phase suggests that the process of revisingand generating a new plan takes slightly more time than
the initial planning stage. Additionally, the occurrence of aninvalid plan during replanning emphasizes the importance ofthorough testing and verification to ensure the reliability andsafety of the system.
C. Human Action Recognition Evaluation
For evaluating the Human Action Recognition module, adataset has been collected for 5 different subjects, each one
performing five times all the human actions considered inTable II. In order to improve the reliability of the action
classifier, also a general “unknown” class has been considered in the dataset acquisition to learn better to distinguishthe movements associated with the gesture from movements
related to general movements of the worker within theworkcell not related to the overall process as walking andstanding. We considered 10 sequences for each subject forevaluating performance on the “unknown” gesture since it
includes a larger variability of possible movements.
The action recognition module is trained on the collected
dataset, using the sequence relative to 4 subjects. The sequences related to the fifth subject are reserved as a test
set, on which the action recognition classifier is evaluated interms of accuracy. This allows to evaluate the action classifier
on a set of data not used to train the model, assessing
the classifier’s ability to generalize on novel data. The total
number of test sequences is 40: five sequences for each of

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 8: Evaluation during a draping task: the robot detects the required ply (a) and moves to the “pick” position (b); theoperator waits until the robot reaches the final position on the mould (c) and then starts the manual draping phase (d); when
draping is finished, the worker requests a new ply using a Drape Next gesture (e); while the robot starts moving towards a
new ply, the human raise the right arm to trigger a manual “inspection” causing a replanning (f).
the six human actions defined in Table II, and ten sequences
involving common movements annotated as “unknown”.
The action recognition module has been evaluated interms of Top1 and Top3 accuracy. The former represents
the percentage of correctly predicted gestures in the test set.
At the same time, the Top3 accuracy is the percentage ofactions whose correct prediction falls in the three highest
softmax scores estimated by the network. The performanceachieved are Top1=90.00% and Top3=97.50%. As shown inthe confusion matrix, Fig. 7, the classifier performs well interms of accuracy on the considered test set since most of
the actions of the fifth subject are correctly recognized.
D. Experimental Validation
The proposed framework has been validated in a realindustrial scenario, targeting a collaborative draping task
shown in Figure 8. In particular, the human operator and therobot work together to drape a series of plies on a mould: therobot provides the material transport and accurate placementon the mould (Figure 8c), while the human operator performs
the actions that require high manual dexterity, such as
manually draping the material over the mould (Figure 8d).
At any time, the user can request particular actions from therobot by means of gestures, such as a request for a newply (Figure 8e) or a request to stop the current operation toallow the operator to manually inspect the draping quality(Figure 8f). The proposed framework is able to monitor
the worker’s activity and handle sudden requests from theoperator. For example, in the experimental validation shownin Figure 8, at the end of the manual draping, the operator

makes a Drape Next gesture, thus triggering the task plannerto generate a plan to move the robot towards a new ply;
immediately afterwards, when the robot starts to move, theuser makes a new Inspection gesture forcing the task planner
to delete the previous plan and generate a new one.
V. CONCLUSIONSIn this paper, we proposed a Human-Aware Task Plannerfor Human-Robot Collaborative industrial applications. The
advantages of this approach are the ability to create a plan
starting from the description of the process and the actions
in order to share that activities both from humans androbots. In addition, it is able to handle user interaction
through the dynamic rescheduling of the plan followinguser interruptions or to handle unexpected events. The usercommands are handled by an intelligent Human Action
Recognition module based on Deep Learning technique.
Another main contribution is the ability of the planner tocreate actions starting from primitives. The framework has
been validated in an industrial collaborative scenario derived
from the DrapeBot European research project. The results
obtained demonstrate the applicability and effectiveness ofthe proposed approach. In future works, we plan to integrate
the Task Planner with an ergonomic Motion Planner in orderto evaluate the complete Task and Motion Planner (TAMP)
application in a dynamic industrial scenario where one of
the collaborative activities between robot and human is thecollaborative transport of plies.

ACKNOWLEDGMENT
This project has received funding from the European
Union’s Horizon 2020 research and innovation program
under grant agreement No 101006732, ”DrapeBot – COLLABORATIVE DRAPING OF CARBON FIBER PARTS”.
R EFERENCES[1] G. Chryssolouris, N. Papakostas, and D. Mavrikios, “A perspective
on manufacturing strategy: Produce more with less,” CIRP Journalof Manufacturing Science and Technology, vol. 1, no. 1, pp. 45–52,
2008.
[2] S. Pellegrinelli, A. Orlandini, N. Pedrocchi, A. Umbrico, and T. Tolio,“Motion planning and scheduling for human and industrial-robot
collaboration,” CIRP Annals, vol. 66, no. 1, pp. 1–4, 2017.
[3] M. Cashmore, M. Fox, D. Long, D. Magazzeni, B. Ridder, A. Carrera,
N. Palomeras, N. Hurtos, and M. Carreras, “Rosplan: Planning in
the robot operating system,” in Proceedings of the international
conference on automated planning and scheduling, vol. 25, 2015, pp.
333–341.
[4] B. Lacerda, D. Parker, and N. Hawes, “Optimal and dynamic planningfor markov decision processes with co-safe ltl specifications,” in 2014
IEEE/RSJ International Conference on Intelligent Robots and Systems.
IEEE, 2014, pp. 1511–1516.
[5] V. A. Ziparo, L. Iocchi, P. U. Lima, D. Nardi, and P. F. Palamara,“Petri net plans: A framework for collaboration and coordination in
multi-robot systems,” Autonomous Agents and Multi-Agent Systems,
vol. 23, pp. 344–383, 2011.
[6] A. Umbrico, A. Cesta, M. Cialdea Mayer, and A. Orlandini, “Platinu
m: A new framework for planning and acting,” in AI* IA 2017
Advances in Artificial Intelligence: XVIth International Conference of
the Italian Association for Artificial Intelligence, Bari, Italy, November
14-17, 2017, Proceedings 16. Springer, 2017, pp. 498–512.
[7] M. S. Malvankar-Mehta and S. S. Mehta, “Optimal task allocation in
multi-human multi-robot interaction,” Optimization Letters, vol. 9, pp.
1787–1803, 2015.
[8] F. Chen, K. Sekiyama, F. Cannella, and T. Fukuda, “Optimal subtask
allocation for human and robot collaboration within hybrid assembly
system,” IEEE Transactions on Automation Science and Engineering,
vol. 11, no. 4, pp. 1065–1075, 2013.
[9] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitiveenvironments,” Advances in neural information processing systems,
vol. 30, 2017.
[10] T. Yu, J. Huang, and Q. Chang, “Optimizing task scheduling in humanrobot collaboration with deep multi-agent reinforcement learning,”
Journal of Manufacturing Systems, vol. 60, pp. 487–499, 2021.
[11] L. Tagliapietra and E. Tosello, “Curami: human-robot collaboration
for intelligent assembly tasks.”[12] M. Terreran, S. Ghidoni, E. Menegatti, V. Enrico, P. Nicola, N. Castaman, A. Gottardi, E. Christian, V. Luca, S. Giuseppe, et al., “Asmart workcell for cooperative assembly of carbon fiber parts guided
by human actions,” in Atti della 4ª Conferenza Italiana di Robotica e
Macchine Intelligenti, 2022.
[13] M. H. Arbo, Y. Pane, E. Aertbeliën, and W. Decré, “A system architecture for constraint-based robotic assembly with cad information,”in 2018 IEEE 14th International Conference on Automation Scienceand Engineering (CASE). IEEE, 2018, pp. 690–696.
[14] G. Michalos, J. Spiliotopoulos, S. Makris, and G. Chryssolouris, “A
method for planning human robot shared tasks,” CIRP journal of
manufacturing science and technology, vol. 22, pp. 76–90, 2018.
[15] A. Casalino, A. M. Zanchettin, L. Piroddi, and P. Rocco, “Optimalscheduling of human–robot collaborative assembly operations withtime petri nets,” IEEE Transactions on Automation Science andEngineering, vol. 18, no. 1, pp. 70–84, 2019.
[16] Y. Cheng, L. Sun, and M. Tomizuka, “Human-aware robot taskplanning based on a hierarchical task model,” IEEE Robotics andAutomation Letters, vol. 6, no. 2, pp. 1136–1143, 2021.
[17] A. Ham and M.-J. Park, “Human–robot task allocation and scheduling:
Boeing 777 case study,” IEEE Robotics and Automation Letters, vol. 6,
no. 2, pp. 1256–1263, 2021.
[18] N. Nikolakis, N. Kousi, G. Michalos, and S. Makris, “Dynamicscheduling of shared human-robot manufacturing operations,” Procedia CIRP, vol. 72, pp. 9–14, 2018.

[19] P. Tsarouchi, G. Michalos, S. Makris, T. Athanasatos, K. Dimoulas,and G. Chryssolouris, “On a human–robot workplace design andtask allocation system,” International Journal of Computer Integrated
Manufacturing, vol. 30, no. 12, pp. 1272–1279, 2017.
[20] P. Tsarouchi, S. Makris, and G. Chryssolouris, “On a human and dualarm robot task planning method,” Procedia CIRP, vol. 57, pp. 551–
555, 2016.
[21] L. Johannsmeier and S. Haddadin, “A hierarchical human-robot
interaction-planning framework for task allocation in collaborative
industrial assembly processes,” IEEE Robotics and Automation Letters,
vol. 2, no. 1, pp. 41–48, 2016.
[22] K. Darvish, E. Simetti, F. Mastrogiovanni, and G. Casalino, “A hierarchical architecture for human–robot cooperation processes,” IEEE
Transactions on Robotics, vol. 37, no. 2, pp. 567–586, 2020.
[23] F. Rovida, B. Grossmann, and V. Krüger, “Extended behavior trees
for quick definition of flexible robotic tasks,” in 2017 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS).
IEEE, 2017, pp. 6793–6800.
[24] E. Lamon, F. Fusaro, E. De Momi, and A. Ajoudani, “A comprehensivearchitecture for dynamic role allocation and collaborative task planningin mixed human-robot teams,” arXiv preprint arXiv:2301.08038, 2023.
[25] M. Cramer, K. Kellens, and E. Demeester, “Probabilistic decisionmodel for adaptive task planning in human-robot collaborative assembly based on designer and operator intents,” IEEE Robotics andAutomation Letters, vol. 6, no. 4, pp. 7325–7332, 2021.
[26] R. Caccavale and A. Finzi, “Flexible task execution and attentional
regulations in human-robot interaction,” IEEE Transactions on Cognitive and Developmental Systems, vol. 9, no. 1, pp. 68–79, 2016.
[27] J. Cacace, R. Caccavale, A. Finzi, and R. Grieco, “Combining humanguidance and structured task execution during physical human–robot
collaboration,” Journal of Intelligent Manufacturing, pp. 1–15, 2022.
[28] M. Rizwan, V. Patoglu, and E. Erdem, “Human robot collaborative
assembly planning: An answer set programming approach,” Theory
and Practice of Logic Programming, vol. 20, no. 6, pp. 1006–1020,
2020.
[29] R. Lallement, L. de Silva, and R. Alami, “Hatp: hierarchical agentbased task planner,” in International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2018), 2018.
[30] S. Stock, M. Mansouri, F. Pecora, and J. Hertzberg, “Online task
merging with a hierarchical hybrid task planner for mobile service
robots,” in 2015 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS). IEEE, 2015, pp. 6459–6464.
[31] E. Foderaro, A. Cesta, A. Umbrico, and A. Orlandini, “Simplifying
the ai planning modeling for human-robot collaboration,” in 2021
30th IEEE International Conference on Robot & Human Interactive
Communication (RO-MAN). IEEE, 2021, pp. 1011–1016.
[32] H. Oliff, Y. Liu, M. Kumar, M. Williams, and M. Ryan,“Reinforcement learning for facilitating human-robot-interaction in
manufacturing,” Journal of Manufacturing Systems, vol. 56, pp.
326–340, 2020. [Online]. Available: https://www.sciencedirect.com/
science/article/pii/S0278612520301084[33] L. Hu, Z. Liu, W. Hu, Y. Wang, J. Tan, and F. Wu, “Petrinet-based dynamic scheduling of flexible manufacturing system via
deep reinforcement learning with graph convolutional network,”
Journal of Manufacturing Systems, vol. 55, pp. 1–14, 2020.
[Online]. Available: https://www.sciencedirect.com/science/article/pii/
S0278612520300145[34] Y. G. Kim, S. Lee, J. Son, H. Bae, and B. D. Chung,“Multi-agent system and reinforcement learning approach for
distributed intelligence in a flexible smart manufacturing system,”
Journal of Manufacturing Systems, vol. 57, pp. 440–450, 2020.
[Online]. Available: https://www.sciencedirect.com/science/article/pii/
S0278612520301916[35] C. Aeronautiques, A. Howe, C. Knoblock, I. D. McDermott, A. Ram,
M. Veloso, D. Weld, D. W. SRI, A. Barrett, D. Christianson, et al.,“Pddl— the planning domain definition language,” Technical Report,
Tech. Rep., 1998.
[36] M. Terreran, M. Lazzaretto, and S. Ghidoni, “Skeleton-based action
and gesture recognition for human-robot collaboration,” in Intelligent
Autonomous Systems 17: Proceedings of the 17th International Conference IAS-17. Springer, 2023, pp. 29–45.
[37] I. Sárándi, T. Linder, K. O. Arras, and B. Leibe, “Metrabs: metric-scale
truncation-robust heatmaps for absolute 3d human pose estimation,”
IEEE Transactions on Biometrics, Behavior, and Identity Science,
vol. 3, no. 1, pp. 16–30, 2020.