Submitted version of the contribution presented at the IEEE ETFA 2022 conference in Stuttgart.
Published manuscript available at DOI: 10.1109/ETFA52439.2022.9921553

Inverse Optimal Control for the identification ofhuman objective: a preparatory study for physicalHuman-Robot Interaction
1st Paolo Franceschi

2nd Nicola Pedrocchi

3rd Manuel Beschi

CNR-STIIMA; UniBs
Milano, Italy; Brescia, Italy
paolo.franceschi@stiima.cnr.it

CNR-STIIMA
via Alfonso Corti 12, 20133, Milano, Italy
nicola.pedrocchi@stiima.cnr.it

UniBs
via Branze 38, 25123, Brescia, Italy
manuel.beschi@unibs.it

Abstract—Nowadays, many applications involving humans androbots working together require physical interaction. It is known
that, during an interaction, the mutual understanding andknowledge of the partner’s goal improves and allows naturalinteraction. For this purpose, this work proposes Inverse OptimalControl (IOC) to recover the cost function of a human performinga reaching task with a robot in passive impedance control. This
work presents the potentialities and limitations of the presented
IOC method to describe human objectives. This work representsa preparatory study toward smooth and natural physical HumanRobot Interaction (pHRI), intending to understand the basic
information on humans’ behavior.
Index Terms—Inverse Optimal Control, physical HumanRobot Interaction

I. I NTRODUCTION
With the large spread of collaborative robots and applications, the need to make Human-Robot Interaction as natural as
possible arises, especially when they must cooperate throughphysical interaction, in the so-called physical Human-Robot
Interaction (pHRI) [1], [2]. Indeed, in pHRI, the humanoperator and the robot must execute a task together, and theaction of the one has immediate consequences on the other[3].
It was shown that humans understand each other’s intentions
while physically interacting to perform a task [4]. While thisnaturally comes for humans, the same does not apply to therobotic partner. Then, it is also interesting to allow the robot
to understand its partner’s objective.
Many techniques and models are studied and presented
in the literature to describe human and human-machine behavior. Such models span from elementary models (humans
as a spring) to complex neuromuscular models and OptimalControl models, depending on the need. For a complete andup-to-date review, please see [5].
Provided that Game Theory (GT) represents a robust framework to describe an individual’s behavior during interaction[6], this work focuses on the identification of optimal behaviors of humans, and the recovery of the human cost function,
This project has received funding from the European Union’s Horizon
2020 research and innovation program under grant agreement No 101006732(Drapebot).

as this modeling allows direct integration into GT frameworks.
Optimal Control (OC) aims to find a control action for a
dynamic system over a time window such that a cost function
is minimized. Conversely, Inverse Optimal Control (IOC) techniques are adopted to recover the cost function that produced
observed control actions and state histories.
The most common strategy used in IOC is to parametrize
an unknown objective function as a weighted sum of relevant features (or basis functions) with unknown weights [7].
This model is also adopted to describe human behaviors in
performing different tasks. In [8], human jumping is studied,highlighting the possibility that the cost function varies during
the task. In [9] and [10] study human movement by detectingchanges to the optimization criterion during a squat task.
In [11], human arm reaching is studied, considering freespace reaching motions. The results show a trade-off between
kinematics and dynamics-based controllers depending on thereaching task. Interestingly, this trade-off depends on the initialand final arm configurations. [12] also studies human arm
motions, with applications to human-robot collaboration in ashared workspace. These works involve cost functions withmultiple features but consider only motions in the free space,
without accounting for interactions.
If the only cost function’s features considered are the stateand control action, the problem is reduced to a typical Linear
Quadratic Regulator (LQR). In this case, more specific IOC
techniques exist. Such techniques compute the weight matrices
starting from the feedback matrix and not the complete control
and state histories[13] [14] [15]. Interestingly, such methods
do not require the entire trajectory to identify the cost function,but only the knowledge of the gain matrix is sufficient. Finally,such approaches appear suitable to study humans in interaction
with autonomous agents, described by Game Theory, as in[16], [17], [18].
Motivated by the previous studies, this work aims at implementing IOC to understand the fundamental behavior ofhumans interacting with a variable, passive system. It is
possible to obtain basic yet valuable information to develop
natural pHRI controllers. The simple LQR model is studied.
Indeed, despite its simplicity, as shown, LQR cost functions

can capture essential human behaviors during the interaction.
Moreover, modeling the human’s cost function as the LQR costfunction allows easy integration in Linear Quadratic GameTheoretic (LQGT) framework, which provides useful tools for
pHRI controllers design [19], [20], [21].
Even though it was shown that the recovered cost functionsof the humans have different values if recovered from manualcontrol or shared control [22], this work, as a preparatory
study, aims at understanding the differences and things incommon between various subjects cost functions. This provides useful insights into humans’ behavior, which should beconsidered for future studies.
II. METHOD

A. System modeling
Working in the Cartesian space is more intuitive and naturalfor the human operator; hence the desired robot motion at theend-effector is implemented as an impedance model in the
Cartesian space:(1)

where Mi , Di and Ki ∈ R6×6 are the desired inertia, damping
and stiffness impedance matrices, respectively, x0 (t) ∈ R6 is thevector containing the equilibrium positions at the end-effector,uh (t) ∈ R6 represents the human effort applied to the system.
The Cartesian coordinates in x and xre f are defined according
to [23], with the vector x = [pT φ T ]T where pT are the position
coordinates and φ T the set of Euler angles that defines therotation matrix describing the end-effector orientation. Withẋ = [vT ω T ]T is indicated the vector containing the linear andangular velocities.
After some reformulation, (1) can be rewritten in a linearized state-space formulation around the working point asż = Az + Bh uh

Many previous works describe human’s intention as theminimization of a quadratic cost function. In particular, it
is possible to assume the cost function as quadratic on thestate and control input, leading to the LQR formulation of
the problem, (as examples see [19], [21], [24]). Therefore, thehuman objective can be described as the minimization of aquadratic cost function, defined as
Z ∞

Jh =

0

(z Qh z + uh Rh uh ) dt

 6×6 0
12 × 6Bh=
Mi−1

are the state and input matrices, respectively, with 06×6 denoting a 6 × 6 zero matrix and Ja the analytical Jacobian matrix,
with the dimensions of the considered Cartesian components.
Robot controllers typically accept as control input reference
positions or velocities in the joint space; hence, it is worth
converting the reference velocities from the Cartesian space
to the joint space.

(4)

where Jh is the cost that the human incurs, Qh ∈ R2n×2n is
a matrix containing weights on the state, and Rh ∈ R6×6 is amatrix of weights on the control input.
Remark 1. The human’s model weights Qh and Rh areunknown and must be identified. A reasonable estimate of thehuman parameters, Q̂h and R̂h , can be obtained with the IOC
method as presented in the following section.
Given the control objective in (4), and the system dynamics
in (2), the LQR Optimal Control Problem can be summarizedasZ ∞min Jh =u0(z Qh z + uh Rh uh ) dt
s.t.ż = A z + Bh uh(5)z(t0 ) = z0(2)where z = [x − x0 ẋ]T ∈ R12 is the state space vector, while thematrices
 6×6

Ja
0
A=
−Mi−1 Ki −Mi−1 Diandwith q̇re f (t) ∈ Rn , where n represents the number of joints, arethe reference velocities in the joint space, J(q)+ is the pseudoinverse of the geometric Jacobian matrix. Simple integration
allows commanding joint positions instead of velocities to therobot. Considering that today’s robots have excellent tracking
performance in the frequency range excitable by the operator,
in this work, hypothesis q̇ ≃ q̇re f is assumed to hold.
B. Optimal Control Problem

In this section, the robot behavior is modeled as a Cartesian
impedance subject to the external force applied by the human,
and the system is rewritten in a state-space formulation.
Optimal control to model human behavior is introduced, andthe Inverse Optimal Control technique is adopted to recover
the unknown human control objective.

Mi ẍ(t) + Di ẋ(t) + Ki (x(t) − x0 (t)) = uh (t)Given the reference velocity in the Cartesian space, thefollowing relation allows obtaining reference velocities in the
joint spaceq̇re f (t) = J(q)+ ẋ(t),(3)

The LQR optimal control has the feedback formuh = −K z(t)(6)

in which the human control action is described as linear
feedback proportional to the system’s state. In (6), the matrix
K represents the feedback gain matrix, copmuted asT
K = R−1h B P(7)in which, the matrix P is the unique solution for the feedback
linear-quadratic optimal control problem, solution of the infinite horizon Continuous Algebraic Riccati Equation (CARE),
given byT
AT P + PA − PBh R−1
h Bh P + Qh = 0.

(8)

C. Human objective identification
Given the definition of the direct Optimal Control Problem
as in (5), this section addresses the Inverse Optimal Control
problem. That is, given observed state and control histories,
denoted as z̄ and ūh , given the system dynamics in (2), recover
the cost function (4) that produced such control histories.
As discussed in the Introduction, different IOC techniques
exist. Among the various, this work implements the one
presented in [15], considering the human as the only active
player, as briefly described in this section.
1) Gain identification: The method relies on the knowledgeof the feedback gain matrix. Because the gain matrix of thehuman is not known, it has to be recovered. If the complete
trajectory (or a sufficient part of it) is known, it is possible toapply the Least Square Method (LSM), and the matrix Kh is
obtained fromZ tf

K̂h = arg min
Khti∥Kh z̄(t) + ūh (t)∥2 dt(9)where ti and t f indicate the initial and final time of theˆtrajectory (or a sufficient portion of it), and the symbol (·)
indicates an estimate of the real value.
2) Inverse Optimal Control: This section briefly presents
the main steps to recover the cost function of a player. Referto [15] for the full treatment. As preliminary, denote ⊕ as the
Kronecker sum, defined as
X ⊕Y = (X ⊗ Iq ) + (Ir ⊗Y )(10)

with X ∈ Rr×r and Y ∈ Rq×q .
Given A, B, and K̂h , the Inverse Optimal Control (IOC)problem can be solved as follows. First, denote F, F⊕ , K̂⊕and Z asF
= A − BK̂h ,F⊕ = F T ⊕ F T ,K̂⊕ = K̂hT ⊕ K̂hT ,
−1
T
Z
= (I
 n ⊗ B )F⊕ T

M = Z Z K̂⊕ + K̂ ⊗ I
Then, denote θ as the vector of vectorized weights as
θ = [vec(Q̂) vec(R̂)]T .

(11)

As demonstrated in [15], if θ satisfies
Mθ = 0(12)

the Q̂ and R̂ are the correct unkown parameter for (8).
Given that (12) is a reformulation of (8), the parameter setof the inverse LQ differential system is given byθ = ker(M)(13)

with convex boundaries such that Rh > 0. As typical in IOC
problems, a residual is defined asr = Mθ

(14)

to consider non-optimal behaviors and imperfect modeling ofK̂. The following quadratic problem is formulated to minimize
the residual
1
min ||r||22 = θ T H θθ2(15)
s.t. I θ ≥ 0
R>0with H = 2M T M.
Remark 2. All the λ θ , λ > 0 solutions are acceptable, since
the solution of problem (5) is the same for any λ J, λ > 0.
Remark 3. The constraints in (15) are imposed by OptimalControl necessary conditions, i.e. matrix Q semi-positive definite and matrix R strictly positive definite.
III. E XPERIMENTS
In this section, experimental results are presented. First, thedesign of experiments is presented, with comments on theevaluation procedure, then results are presented and analyzed.
A. Design of experiments
Three subjects are involved in this study. Each subject is
asked to move the robot end-effector to a fixed, unknown setpoint along the z-axis. The impedance parameters of the robot
are set to M = 10, and three values of damping D = [25, 50, 75]
are tested to verify how different systems modify the control
objective. The stiffness is set K = 0. Each subject is asked toperform 12 set-point reaching for each damping value for a
total of 36 reaching tasks. Each reaching is along the x–axis
only, with the set-point at various, random distances. Before
each recording session, the subject is allowed to practice fora while to gain confidence with the system.
The measured exchanged forces, the reference position, andthe actual positions compute the IOC problem. The forces aremeasured with a Robotiq FT300 sensor mounted on the robot
tip. The robotic system gives the positions directly, and thevelocities are computed by differentiating the positions.
The human cost function matrices, as typically happens[16], [18], are assumed to be nonzero only on the diagonalterms, resulting in Q = diag([q1 , q2 ]), and R = r. Moreover,since the solution to the optimal control problem is the samefor any λ Jh with λ scalar, as additional constraint to (15) is
set q1 = 1. In this way, all the recovered cost functions arecomparable.
A first analysis is done on the recovered features (q1 , q2
and r) from (15) to check their variation with respect to thevariations of the system. In the ideal scenario, the weights
can be different between different subjects. Despite this, eachsubject should have constant weight values, even when varying
the system’s parameters.
A measure of the control input is defined to evaluate thegoodness of the recovered costs as
R tf

Eu =

t0 u(t) − û(t) dt

umax

(16)

1.2

10 -3

5

d=25
d=50
d=75

1

d=25
d=50
d=75

4

0.8
3
0.6
2
0.4
1

0.2
0

subj 1subj 20subj 3Fig. 1: Mean and standard deviations of the recovered control
weightssubj 1subj 2

subj 3Fig. 2: risetime over setpoint computation

B. Results
where umax is the maximum measured control effort, and
it is used to normalize the errors to make the comparison
fair. The recovered control input û(t) is computed by runninga system simulation. In the ideal case, this value should be
zero. Defining the set-point as xsp = xre f − x0 , and the Raise
Time RT as the time required to move the system from x0 to0.95 xsp , it is possible to define a ratio
RT
,
xsp

RTsp =(17)

to check how the different systems influence the raise time byremoving dependency from the set-point distance. In the ideal
case, the ratio should be the same for a given system with any
set-point and should increase by increasing the damping.
Finally, a possible relation between the distance from setpoint and weight on control is investigated. In the ideal case,the values of R should always be the same, without regard forthe set-point.

As previously discussed, the weight q1 is constrained tobe q1 = 1, and the other weights q2 and r are computed
consequently. From the experiments, all the subjects have anegligible q2 , with values in the order of 1e − 4. This meansthat the human feedback is based mainly on position, as also
happens in [22]. Since the values of q2 are very close to 0,
the rest of this paper will approximate them with 0 without
further analysis.
Given q2 = 0, the ratio between q1 and r drives the system’s
response. The analysis of the values of R, visible in figure 1,
shows that humans have a low weight also in how much effort
is put into the task. The low value of r, compared with q1 ,means that a fast set-point reaching is more favorable thansaving effort.
Moreover, three subjects tend to decrease the value of r as
the damping increase, which means that reaching the desired
position at a specific time should remain almost constantagainst changes in the system.

0.9

0.9

0.9
fitted curve

fitted curve

fitted curve

0.85
0.8

0.8

0.8

0.7

0.75

0.7

0.6

sp

sp

sp

0.7

0.6

0.65

0.6

0.5

0.5

0.4

0.4

0.55

0.5

0.45
1

2

3

4

R

5

6

7

0.3
0.5

(a) control weight identified with
D=25

1

1.5

2

R

10 -4

2.5

3

3.5

0.3
0.2

0.4

0.6

0.8

1

10 -4

1.2

R

1.4

1.6

1.8

2
10 -4

(b) control weight identified with(c) control weight identified with

D=50

D=75

Fig. 3: Correlation between control weight identified and different setpoints, for subject 3

C. Discussion
80

d=25
d=50
d=75

70

This work’s analysis suggests human modeling as an optimal controller and hints about real human intention. The
most relevant result is that humans tend to keep the required
time as constant as possible when performing a task. This isprobably a consequence of the fact that a human control works
as feedback on a visual stimulus and keeps this control -which
is not considered in the presented study- constant.
Despite this, optimal control modeling of a human interacting with a robot can still be adopted, keeping in mind
some baseline rules shown in the results section. The human’s
weight on control input decreases if the system becomes morerigid against motions and decreases as the desired position
becomes closer to the current one.

60
50
40
30
20
10
0

subj 1subj 2

subj 3Fig. 4: Mean and standard deviations of the control inputs

IV. C ONCLUSIONSThis work presents a study of the human behavior modeled
as optimal control during physical Human-Robot Interaction.
Theoretical modeling of the system and control model ispresented, and the formulation of the Inverse Optimal Control
is presented. IOC is used to recover the cost functions of threesubjects performing a reaching task interacting with different
robot behavior. The analysis shows that OC modeling of ahuman may be used and gives good results with some caution.
Such a model presents applications in designing gametheoretical-based controllers for pHRI, in which the knowledgeof the other’s cost function is necessary. Moreover, it can also
be used as a reference in modeling human behavior for digitaltwin simulations.
Future works will study more complex models with different
cost functions that consider other features, such as acceleration, jerk, time that results relevant, and energy. Moreover,different interacting strategies, such as cooperative and noncooperative game theory, will be analyzed to understand how
two humans behave when performing interacting tasks andwhich one model interaction better.

Figure 2 also displays such a behavior. shows the ratio RTsp .
The Figure shows a minimal increment in the ratio compared with the increment of damping, particularly in subjects
2 and 3, confirming that humans appear more interested intime than in the effort required.
This study also shows a correlation between the set-point
distance and the weight a human gives to the control effort.
The farther the set-point is, the higher the control weight, asshown in figure 3.
The error between the measured and reconstructed control
input is analyzed. Figure 4 shows the normalized error between
the measured and the simulated control effort of the threesubjects in the three scenarios.

50

50

40

40

40

30

30

30

20

10

0

control effort [N]50control effort [N]control effort [N]Finally, figure 5 presents one control history measured andrecovered for each subject in the study, with the corresponding
normalized error.

20

10

0

-10
0.5

1

1.5

time[s]

2

2.5

3

3.5

10

0

-10
0

20

-10
0

0.5

1

1.5

2

2.5

3

3.5

0

time[s]

0.5

1

1.5

2

2.5

3

3.5

time[s](a) Computed and measured effort of (b) Computed and measured effort of (c) Computed and measured effort ofsubject 1. Eu = 17.81

subject 2. Eu = 46.32

subject 3. Eu = 40.02

Fig. 5: Example of one trial of each subject.

ACKNOWLEDGMENT
This project has received funding from the European
Union’s Horizon 2020 research and innovation program under
grant agreement No 101006732 (Drapebot).
R EFERENCES[1] S. Haddadin and E. Croft, Physical Human–Robot Interaction. Cham:
Springer International Publishing, 2016, pp. 1835–1874.
[2] A. De Santis, B. Siciliano, A. De Luca, and A. Bicchi, “An atlas ofphysical human–robot interaction,” Mechanism and Machine Theory,
vol. 43, no. 3, pp. 253–270, 2008.
[3] E. Matheson, R. Minto, E. G. G. Zampieri, M. Faccio, and
G. Rosati, “Human–robot collaboration in manufacturing applications:
A review,” Robotics, vol. 8, no. 4, 2019. [Online]. Available:
https://www.mdpi.com/2218-6581/8/4/100[4] A. Takagi, G. Ganesh, T. Yoshioka, M. Kawato, and E. Burdet, “Physically interacting individuals estimate the partner’s goal to enhance their
movements,” Nature Human Behaviour, vol. 1, no. 3, pp. 1–6, 2017.
[5] A. Scibilia, N. Pedrocchi, and L. Fortuna, “Human control model
estimation in physical human;machine interaction: A survey,” Sensors,
vol. 22, no. 5, 2022. [Online]. Available: https://www.mdpi.com/14248220/22/5/1732[6] N. Jarrassé, T. Charalambous, and E. Burdet, “A framework to describe,
analyze and generate interactive motor behaviors,” PLOS ONE, vol. 7,
no. 11, pp. 1–13, 11 2012.
[7] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in Proceedings of the twenty-first international
conference on Machine learning, 2004, p. 1.
[8] K. Westermann, J. F.-S. Lin, and D. Kulić, “Inverse optimal control
with time-varying objectives: application to human jumping movement
analysis,” Scientific reports, vol. 10, no. 1, pp. 1–15, 2020.
[9] W. Jin, D. Kulić, J. F.-S. Lin, S. Mou, and S. Hirche, “Inverse optimal
control for multiphase cost functions,” IEEE Transactions on Robotics,
vol. 35, no. 6, pp. 1387–1398, 2019.
[10] J. F.-S. Lin, V. Bonnet, A. M. Panchea, N. Ramdani, G. Venture, andD. Kulić, “Human motion segmentation using cost weights recoveredfrom inverse optimal control,” in 2016 IEEE-RAS 16th International
Conference on Humanoid Robots (Humanoids), 2016, pp. 1107–1113.
[11] O. S. Oguz, Z. Zhou, S. Glasauer, and D. Wollherr, “An inverse optimal
control approach to explain human arm reaching control based on
multiple internal models,” Scientific reports, vol. 8, no. 1, pp. 1–17,
2018.

[12] J. Mainprice, R. Hayne, and D. Berenson, “Goal set inverse optimal
control and iterative replanning for predicting human reaching motionsin shared workspaces,” IEEE Transactions on Robotics, vol. 32, no. 4,
pp. 897–908, 2016.
[13] M. Menner and M. N. Zeilinger, “Convex formulations and algebraic
solutions for linear quadratic inverse optimal control problems,” in 2018
European Control Conference (ECC), 2018, pp. 2107–2112.
[14] F. Jean and S. Maslovskaya, “Inverse optimal control problem: the linearquadratic case,” in 2018 IEEE Conference on Decision and Control(CDC), 2018, pp. 888–893.
[15] J. Inga, E. Bischoff, T. L. Molloy, M. Flad, and S. Hohmann, “Solution
sets for inverse non-cooperative linear-quadratic differential games,”
IEEE Control Systems Letters, vol. 3, no. 4, pp. 871–876, 2019.
[16] F. Köpf, J. Inga, S. Rothfuß, M. Flad, and S. Hohmann, “Inverse
reinforcement learning for identification in linear-quadratic dynamic
games,” IFAC-PapersOnLine, vol. 50, no. 1, pp. 14 902–14 908, 2017.
[17] S. Rothfuß, J. Inga, F. Köpf, M. Flad, and S. Hohmann,“Inverse optimal control for identification in non-cooperative
differential games,” IFAC-PapersOnLine, vol. 50, no. 1, pp. 14 909–
14 915, 2017, 20th IFAC World Congress. [Online]. Available:
https://www.sciencedirect.com/science/article/pii/S2405896317334602[18] J. Inga, M. Flad, and S. Hohmann, “Validation of a human cooperative
steering behavior model based on differential games,” in 2019 IEEE
International Conference on Systems, Man and Cybernetics (SMC),
2019, pp. 3124–3129.
[19] Y. Li, G. Carboni, F. Gonzalez, D. Campolo, and E. Burdet, “Differential
game theory for versatile physical human–robot interaction,” Nature
Machine Intelligence, vol. 1, no. 1, pp. 36–43, Jan 2019.
[20] Y. Li, K. P. Tee, R. Yan, W. L. Chan, and Y. Wu, “A framework ofhuman–robot coordination based on game theory and policy iteration,”
IEEE Transactions on Robotics, vol. 32, no. 6, pp. 1408–1418, 2016.
[21] Y. Li, K. P. Tee, W. L. Chan, R. Yan, Y. Chua, and D. K. Limbu,“Continuous role adaptation for human–robot shared control,” IEEE
Transactions on Robotics, vol. 31, no. 3, pp. 672–681, 2015.
[22] J. Inga, M. Eitel, M. Flad, and S. Hohmann, “Evaluating human behavior
in manual and shared control via inverse optimization,” in 2018 IEEE
International Conference on Systems, Man, and Cybernetics (SMC),
2018, pp. 2699–2704.
[23] B. Siciliano and L. Villani, “Robot force control,” 2000.
[24] X. Na and D. J. Cole, “Linear quadratic game and non-cooperative
predictive methods for potential application to modelling driver–afs
interactive steering control,” Vehicle system dynamics, vol. 51, no. 2,
pp. 165–198, 2013.