<figure><img src="image_1.png" /></figure>

HAL
open science

# Holistic view of Inverse Optimal Control by introducing projections on singularity curves

Jessica Colombel, David Daney, François Charpillet

► To cite this version:

Jessica Colombel, David Daney, François Charpillet. Holistic view of Inverse Optimal Control by introducing projections on singularity curves. 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, May 2023, London, United Kingdom. pp.12240-12246, 10.1109/ICRA48891.2023.10161163. hal-03926294

HAL Id: hal-03926294

https://hal.science/hal-03926294

Submitted on 6 Jan 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Holistic vision of Inverse Optimal Control

Jessica Colombel, David Daney, François Charpillet

**Abstract**—Inverse Optimal Control (IOC) is a problem that exists in many fields. In the context of human motion analysis, many methods for solving this problem have been proposed. This paper presents the Projected Inverse Optimal Control (PIOC) approach which puts forward a simple and complete vision of the problem. PIOC is composed of several independent elements allowing to cover problems of dimensioning, measurement noise and reliability of solution. The first element is the parameterization of the data, allowing to reduce the dimensions of the problem. The second element corresponds to the decoupling of the conditions including the conditions of identifiability of the parameters, singularity of the problem and feasibility of the solution. The last element refers to the choice of solution. We show with this approach that the classical solution methods are in fact projections in the space of trajectories. PIOC also allows us to propose a simple algorithm for the choice of basis, a recurrent problem in IOC.

Inverse Optimal Control, Human Motion Analysis, Identification.

## I. INTRODUCTION

The Inverse Optimal Control (IOC) problem is a complex problem studied in many disciplines. In particular, the IOC is related to the Inverse Reinforcement Learning problem whose expression and proposed resolution methods differ [1]. For human motion analysis, the IOC is a well known problem. Thanks to this method, the literature analyzes the movement in order to imitate it with a robot [2], [3], to predict human movements to better interact with humans [4], [5] or even to split the movement into subtasks [6]. Two methods have been particularly used in the literature: the Bilevel method [2], [7] and the AIOC method [8], [9]. In the context of human motion analysis, we are particularly interested in IOC under uncertainties: noisy data, uncertain model, unknown cost function and parameter selection.

This paper proposes a new approach for Inverse Optimal Control problem, when observed data are noisy, which raise up the problem of optimality of the observed trajectory. We called it Projected Inverse Optimal Control (PIOC). This approach is divided into three elements, the parametrization of the measurement, the decoupling of conditions of viability of the problem and the choice of solution. Our approach brings a simple and complete vision of the IOC resolution problem. In particular, we show that the resolutions methods proposed in the literature are equivalent to particular projections in the trajectory space. The choice of the resolution method is equivalent to a choice of solution according to different criteria.

Thus, the originality of this paper lies in a new simple vision of the IOC problem through:

* parameterization of the measurement states and its impact on the dimensionality and identifiability of the problem;

* decoupling of the conditions: identifiability, singularity and feasibility conditions;

* illustration of the choice of the solution by a projection;

* solution for the choice of the cost function.

First, this paper introduces the problem statement of IOC by defining Direct Optimal Control (DOC) and Bilevel and AIOC resolution methods. Then, the PIOC approach is presented by detailing its different steps: parametrization, decoupling conditions and choice of solution. We also provide a simple example to illustrate our approach. Finally, we will discuss the perspectives and limitations of the proposed approach before concluding this paper.

## II. PROBLEM STATEMENT

### A. Context

The analysis of human movement by the inverse optimal control is based on the assumption that the movement is optimal according to a criterion. Bipedal walking is considered as an optimal movement [10], as well as simpler movements such as picking where the hand is following a trajectory minimizing the criterion of torque variation [11]. Moreover, the biological sensorimotor control would itself be governed by optimal laws [12]. Given the multiplicity of movements and their representation, many criteria have been proposed to generate or analyze through IOC human movement [6], [8], [13]–[18]. These criteria, or cost functions, are the basis for the expression of DOC. While the DOC allows to generate trajectories, the IOC allows to analyze trajectories and to understand how they were generated.

The following subsections will present the DOC as well as two methods for solving the IOC problem. We will simplify their presentation as much as possible in order to stay pedagogical. Finally, we will discuss the limitations of these methods.

### B. Direct Optimal Control

In the context of human (or robot) motion, the aim of the DOC is to generate a trajectory $s^*$ optimal for a convex criterion $C(s) = \sum_{k=1}^{n_c} \omega_k C_k$, composed by a sum of weighed $(\omega = [\omega_1, \dots, \omega_{n_c}])$ cost functions $(C_k, k = 1, \dots, n_c)$, under $n_f$ equality constraints $f_i(s) (i = 1, \dots, n_f)$ and $n_h$ inequality constraints $h_j(s) (j = 1, \dots, n_h)$.

$$s^* = \arg \min_{s} C(s) \qquad (1)$$

$$\text{s.t.} \quad f_{1,...,n_f}(s) = 0, \quad h_{1,...,n_h}(s) \le 0$$

where $s \in \mathbb{R}^{n_s \times n_t}$ is a trajectory composed, by $n_t$ of frames; each are defined by a state $s_t$ of dimension $n_s$, such that $s = [s_1, \dots, s_t, \dots, s_{n_t}]^T$.

The existence of a solution to this problem is assured by the convexity of the base [19]. For that, the literature suggest
convexed cost functions (e.g. quadratic) and positive weights $\omega$ [8].

### C. Inverse Optimal Control

The IOC is the problem of finding the unknown weights $\omega$ associated to known cost functions. The constraints are also known. A lot of resolution methods were proposed to find the solution of the problem of IOC in the context of human movement analysis ( [2], [3], [15], [16], [20]) but we will focus on two of them: the Bilevel resolution method and the Approximate IOC method.

1) **Bilevel IOC:** In the context of human motion analysis, the Bilevel IOC resolution method has been used for locomotion [2], [14] and for the arm [21]. It is a resolution method that uses two levels of calculation (lower-level: DOC that generates optimal trajectory; upper-level: distance of generated trajectory to observed trajectory) that loop until they converge to the result: the optimal trajectory closest to the observed one [22]. The IOC Bilevel is expressed as follows:

$$ 
\begin{array}{ll}
\min_{\omega} & \|\mathbf{s}^* - \mathbf{s}_M\|^2 \\
\textbf{with } \mathbf{s}^* = \underset{\mathbf{s}}{\operatorname{argmin}} & \sum_{k=1}^{n_c} \omega_k \sum_{t=1}^{n_s} C_k(\mathbf{s}_t) \\
\textbf{s.t.} & f_{1,...,n_f}(\mathbf{s}) = 0, \quad h_{1,...,n_h}(\mathbf{s}) \le 0
\end{array}
\quad (2) $$

with $\mathbf{s}_M$ the observed trajectory. We can already note in this expression that if the data are noisy, we converge to the optimal trajectory closest to the noisy one.

2) **Approximated IOC:** The so-called "Approximate" resolution methods are methods based on properties of an optimal problem and resulting conditions. The best known conditions are the Karush-Kuhn-Tucker conditions. They have been used a lot in the framework of IOC [6], [9], [13], [23]–[25]. Other conditions have also been used for IOC, such as the Jacobi-Bellman equations [26]–[28].

The KKT conditions are expressed as follows:

$$ 
\begin{array}{ll}
\displaystyle\sum_{k=1}^{n_c} \omega_k \frac{\partial C_k}{\partial \mathbf{s}}(\mathbf{s}^*) + \sum_{i=1}^{n_f} \lambda_i \frac{\partial f_i}{\partial \mathbf{s}}(\mathbf{s}^*) & \\
\displaystyle\qquad + \sum_{j=1}^{n_h} \nu_j \frac{\partial h_j}{\partial \mathbf{s}}(\mathbf{s}^*) = \mathbf{0} & \text{stationnary} \\
\\
\left. \begin{array}{l} f_i(\mathbf{s}^*) = 0, \quad i = 1, \dots, n_f \\ h_j(\mathbf{s}^*) \le 0, \quad j = 1, \dots, n_h \end{array} \right\} & \text{primal feasibility} \\
\\
\nu_j \ge 0, \quad j = 1, \dots, n_h & \text{dual feasibility} \\
\\
\nu_j h_j(\mathbf{s}^*) = 0, \quad j = 1, \dots, n_h & \text{complementary slackness}
\end{array}
\quad (3) $$

The stationnary condition can also be written with matrices:

$$ \underbrace{[\mathbf{J}_{\omega}, \mathbf{J}_{\lambda}, \mathbf{J}_{\nu}]}_{\mathbf{J}} \underbrace{\begin{pmatrix} \omega \\ \lambda \\ \nu \end{pmatrix}}_{\mathbf{z}} = \mathbf{0} \quad (4) $$

with $\mathbf{J}$ the Identification Matrix and $\mathbf{J}_{\omega}, \mathbf{J}_{\lambda}$ and $\mathbf{J}_{\nu}$ the sub-matrices associated with the parameters $\omega$ (what the IOC wants to retrieve) and $\lambda, \nu$ the Lagrangian coefficients of the constraints of equality and inequality, respectively.

The basis $\mathbf{C}$ is convex, so that these conditions are considered necessary but also sufficient for the optimality of the problem [29].

In the literature, these conditions are used to solve the IOC problem by using them as "residuals". The solution is then done by minimizing residuals [6], [9], [13], [23].

### D. Limitations of the methods

Each of these resolution methods has limitations that do not allow for optimal use in human motion analysis. The main drawback of Bilevel is the computation time for the double optimization loop [28]. The approximate methods pose problems of robustness [30]. In particular, it is necessary to know to what extent one can trust the results obtained [31].

However classical IOC methods work well if the problem is well posed and the data are perfect. In particular, if the basis chosen for the IOC correspond well to the one used to generate the reference trajectory, which must be provided without uncertainties, the resolution methods previously mentioned will find the solution without difficulty. In practice, this case cannot be considered: it is then necessary to take into account the natural uncertainty of the problem, in particular for the analysis of human motion which interests us particularly.

Moreover according to Ab Azar [1], the IOC problem suffers more broadly from several problems:

* Ill-posedness problem (existence and uniqueness of the solution);

* Non-convexity of functions and/or constraints;

* Data availability (incomplete or noisy data);

* Non-linearity, complexity and dimensionality;

* Feature selection (relevance of parameters for dimensionality reduction);

* Generalization (choice of cost functions/results obtained).

In our approach, we want to address these difficulties and propose elements to overcome them. We therefore propose to unify the different methods by highlighting the central role of the projection of uncertain trajectory on an ideal provided by the choice of bases.

## III. PSEUDO INVERSE OPTIMAL CONTROL APPROACH

Our approach, called PIOC, is particularly interesting in the presence of uncertainties, whether they are on the data or the choice of cost functions or constraints.

### A. Illustration

This article corresponds to the presentation of our general vision of the IOC. Thus we illustrate the different notions of our approach from a simple example already used in litterature [20]. We propose the same example in our previous article were the detailed equations are given [31]: a 2 bars with two degrees of freedom $\theta_1$ et $\theta_2$. Both the DOC and the IOC will be modeled from $n_c = 2$ cost functions of torques: $\mathbf{C} = \{\tau_1^2, \tau_2^2\}$. For each degree of freedom, the state is represented by the angle, its velocity and its acceleration. There is no inequality constraint in this example.
## B. Parametrization: reduction of the dimension

Parameterization consists in expressing the observed state by parameters such as polynomials or Fourier series. Among the IOC problems cited earlier II-D, trajectory parameterization helps with some of them.

The parameterization allows to reduce the observed parameters, leading to a reduction of the dimensions of the problem while keeping all the information describing the state of the system. This reduction is necessary because the dimension of the state variables can be very large for human motion. For instance, if we observed a human arm we can have 8 angles and their velocity and acceleration, which lead to $n_s = 24$ state variables. Now suppose there are 30s of measurement at 25Hz, so $n_t = 750$, the number of state variables will be $n_s * n_t = 18000$. By reducing the parameters of the problem, the time to solve the problem is reduced accordingly.

The data availability problem is also addressed by the parametrization. The data needs to be interpolated in order to be parameterized. This not only reduces the effects of missing data, but also reduces the impact of noise for the derived data thanks to the analytical expression of the parametrization.

For our approach, we propose to use a polynomial parametrization. This parametrization is already used in the literature to represent human motion trajectories [32], [33], and even for IOC [6], [23]. Note that in our approach, we use the parametrization of the system states and not the cost functions, which is then called polynomial optimization [34].

For the observation of human motion, expressing the state by polynomials not only reduces the number of observed parameters, but also the number of constraints. The constraints concerning the start and end points and the kinematics are removed because they are directly included in the coefficients of the polynomial.

Let $Q(t)$ be a polynomial describing the trajectory of a state parameter $q$, such that $Q(t) = \sum_{k=0}^{n_d} \alpha_k t^k$, with $n_d$ the degree of the polynomial. To simplify the calculations, we choose to define the starting and ending point at $t = 0$ and $t = 1$, respectively. It is then possible to associated start and end points with the polynomial coefficients and its derivation. The representation of the trajectories and their derivatives (first and second) by polynomials can be represented by a linear system in polynomial coefficients. The number of rows and columns of this linear system correspond to the initial and final constraints and to the number of parameters parameterizing the trajectory, respectively. The difference between the two gives the dimension of the space defining the set of trajectory allowing to go from an initial point to a final point. Theoretically, the larger this space is, the finer it can represent the observed trajectories. If $n_d = 6$, we obtain:

$$ \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 2 & 0 & 0 & 0 & 0 \\ 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ 0 & 1 & 2 & 3 & 4 & 5 & 6 \\ 0 & 0 & 2 & 6 & 12 & 20 & 30 \end{pmatrix} \begin{pmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2 \\ \alpha_3 \\ \alpha_4 \\ \alpha_5 \\ \alpha_6 \end{pmatrix} - \begin{pmatrix} q_0 \\ \dot{q}_0 \\ \ddot{q}_0 \\ q_1 \\ \dot{q}_1 \\ \ddot{q}_1 \end{pmatrix} = \mathbf{0} \quad (5) $$

This system is made of 6 equations for 7 unknowns, so it is possible to define 6 of the parameters in function of the 7th one. We can then express the polynomial $Q$ only in terms of $\alpha_6$, as follows:

$$ Q(t) = a + b t + c t^2 + (d - \alpha_6) t^3 + (e + 3 \alpha_6) t^4 + (f - 3 \alpha_6) t^5 + \alpha_6 t^6 \quad (6) $$

with $a, b, c, d, e$ and $f$ scalars only depending on the starting and ending points.

In this example, $\alpha_6$ is the free coefficient which can express the whole polynomial. Note that if $n_d = 7$, there will be 2 free parameters for each state. We propose to use the free coefficients as parameters for the IOC problem. It is important to understand that if $n_d \le 5$, there is no free parameter, and therefore there is only one trajectory that corresponds to the start and end points. If there is only one trajectory described by this constraints, this trajectory is not the result of an optimization process but simply of constraints. Therefore, it is not possible to use a polynomial degree strictly lower than 6 to describe a trajectory supposedly resulting from an optimization. We insist on this lower bound because parameterizations of human motion tend to 5th-order polynomial [33] for the study of the Jerk, but in the context of this optimization problem, this degree is not sufficient.

## C. Decoupling of conditions

In our approach, the main principle is the decoupling of the conditions. As already introduced in our previous paper [31], the solution of the problem is based on the verification of different conditions derived from KKT conditions. We consider an acceptable result when each of these conditions is met:

* Identifiability conditions;

* Singularity condition;

* Feasibility conditions.

1) **Identifiability conditions:** For a problem such as IOC, to admit a solution, all parameters must be identifiable. As detailed in our previous paper [31], sub matrices ($J_\omega, J_\lambda, J_\nu$) associated with parameters need to be full ranks. If not a QR decomposition is suggested to eliminate the superfluous parameters.

**Dimensionality:** The identifiability is also dependent on the dimensions of the problem. In particular, the problem must have more (or at least equal) parameters than equations (cost functions and constraints). Note that in the context of parametric approach, as we proposed, for the matrix, this translates into having as many or more rows (of size $(n_d - 5) * n_q$) than columns (of size $n_c + (n_f - n_{fp}) + n_h$ with $n_{fp}$ constraints eliminated by the parametric). In other words $(n_d - 5) * n_q \ge n_c + (n_f - n_{fp}) + n_h$. This means that for a given problem, model and constraints impact directly the degree of the polynomial allowing to solve the problem.

For example with our illustration III-A: polynomial trajectories eliminate all the equality constraints ($n_f = n_{fp}$) from our problem and there is no constraints of equality ($n_h = 0$). If the polynomials are of degree $n_d = 6$, there are only two free parameters (one for each angle) so the matrix obtained is of size $2 \times 2$. If we want to study $n_c = 4$ cost functions, we must increase the degree of the polynomial to $n_d = 7$. This
link between the degree of the polynomial and the problem studied is very important to take into account. All the more so as one must be careful since the higher the degree of the polynomial, the more likely the trajectory will have unwanted curve oscillations.

<figure><img src="image_3.png" /><figcaption>Fig. 1: Illustration of a singularity curve from the example in section III-A. Two axis represents the free parameters $\alpha_6$ and $\beta_6$ from the angle $\theta_1$ and $\theta_2$, respectively. The color map is the determinant of the matrix $det(J^T J)$.</figcaption></figure>

2) **Singularity condition:** Based on the KKT stationary condition, this condition is a necessary but not sufficient condition to express the optimality of the trajectory. We call it singularity condition because the Identification Matrix $J$ needs to be singular to admit a non-trivial solution.

The singularity of a matrix is equivalent to at least one of the singular values is null or the determinant of the matrix is null. For our problem, the matrix $J$ is often non-square, so the determinant should be expressed as: $det(J^T J)$. The determinant

has the advantage of having an analytical expression, unlike singular values.

With our example, it is possible to display this condition in the trajectory space. We call the curve obtained the "singularity curve" (Fig. 1).

Note that this determinant is a necessary but not sufficient condition (KKT conditions on constraints are necessary to be sufficient). Figure 1 proposes a zoom on a portion (red square) on which all the trajectories from the DOC are.

3) **Feasibility conditions:** In our approach, we distinguish two types of feasibility conditions: the conditions associated to the constraints and the condition associated to the convexity of the problem.

As the KKT conditions express it, the conditions associated with the constraints are feasibility conditions. In a context of uncertainty, the constraints may not be respected. We consider that the verification of these constraints should be done according to the desired accuracy. This verification can even be optional.

The condition associated to the convexity of the problem is defined by the DOC (sec. II-B): the positivity of the weights obtained are necessary. Thus if there are not positives, the results are not considered reliable. It is important to understand that in our approach it is a *condition* and not a *constraint*. For example, in the literature, AIOC methods use the positivity of $\omega$ as a constraints [6], [23], [24]. We consider that the constraint forces the $\omega$ to be positive and thus modify the result to fit the constraint, even if it means a huge deviation from the observed trajectory. Using it as a condition allows us to invalidate the result if the weights are negative.

### D. Choice of solution: projection

As previously mentioned, our approach is reserved for the problem of IOC under uncertainty. The parameterization and the decoupling of the conditions allow to tackle the difficulties of selection of the parameters, dimensionality, convexity and expression of the inverse problem. Thus, if the data are perfect, these steps are sufficient to obtain a reliable result whatever the chosen solution method. If the data are perfect and each of the previous conditions are met, the observed trajectory will be on a singularity curve. However, if the data are noisy the observed trajectory will not be on a singularity curve. Therefore, the resolution methods correspond to a projection of the observed trajectory to a singular trajectory on the singularity curve. Different methods do not project in the same way.

1) **Type of projection:** In this paper, we have mentioned two methods used in the literature for the application of IOC to human motion analysis: Bilevel and AIOC (Sec. II-C). These methods of solving the IOC problem correspond to different projections, one orthogonal and the other one that we will call "iso-weighted". Figure 2 illustrates these two projections for two uncertain trajectories.

a) **Iso-weighted projection:** The iso-weighted projection is obtained with the resolution of an unconstrained AIOC on $\omega$. It corresponds to this equation:

$$ 
\begin{aligned}
\mathbf{z} = & \arg \min_{\mathbf{z}} \|\mathbf{J}\mathbf{z}\|^2 \\
\text{s.t.} \quad & \det(\mathbf{J}(\mathbf{s})^T \mathbf{J}(\mathbf{s})) = 0
\end{aligned} 
\quad (7) 
$$
<figure><img src="image_4.png" /><figcaption>Fig. 2: Comparison of projections for two trajectories in green and yellow on a singularity curve graph (Fig. 1). For each trajectory, the projection in plain line is iso-weighting and the dotted lines are "orthogonal". Note that for the green trajectory has two dotted projections, one has weight constraints ($\omega \ge 0$) the other has no constraints.</figcaption></figure>

In reality, we only use a Singular Value Decomposition and the solution to this minimization should be the singular vector associated to the smallest singular value (more details in [31]). Figure 3 shows for each trajectory, which weight $\omega$ is associated. In our example, $\omega$ is a vector of size $n_c = 2$, so it is possible to express this vector as an angle. The yellow/red areas are the areas in which the $\omega$ is positive. They are therefore the areas in which it is possible to obtain a solution to the IOC problem with this resolution method.

It is to be noted that the resolution of AIOC is not a real "projection" on the trajectory space. Indeed, as the goal of the IOC is only to recover the omega, there is no real need to find the optimal trajectory corresponding to the weights found. We present it as a projection to show which singular trajectory the uncertain observed trajectory corresponds to. It is interesting to understand that for all iso-weighted trajectories, the minimum cost corresponding to this weighting is on the singularity curve. If the objective of the resolution is to minimize the cost of the observed trajectory, this solution is to be preferred.

b) **Orthogonal Projection:** The resolution of the IOC problem with a Bilevel corresponds an orthogonal projection constrained by positive omega. The high level is a minimization of the distance between two trajectories and in the polynomial framework the distance between two polynomials is linearly dependent on the free parameters. Let $\xi_p$ be the path error between two polynomials $Q(t)$ et $Q'(t)$, such that:

$$ \xi_p = \int_0^1 (Q(t) - Q'(t)) dt \qquad (8) $$

<figure><img src="image_5.png" /><figcaption>Fig. 3: Illustration of the iso-weighted area. Two axis represents the free parameters $\alpha_6$ and $\beta_6$ from the angle $\theta_1$ and $\theta_2$, respectively. The black curve is the singularity curve. The colormap is associated to the angle (in degree) that represents the vector $\omega$ of size 2. In yellow/red, $\omega$ is positiv.</figcaption></figure>

If the constraints of the model and if the start and finish are identical, according to the equation (Eq. 6), we obtain:

$$ \begin{aligned} \xi_p &= \int_0^1 \left( -(\alpha_6 - \alpha'_6) t^3 + 3(\alpha_6 - \alpha'_6) t^4 - 3(\alpha_6 - \alpha'_6) t^5 + (\alpha_6 - \alpha'_6) t^6 \right) dt \\ &= \Delta_{\alpha} \int_0^1 \left( -t^3 + 3t^4 - 3t^5 + t^6 \right) dt \\ &= -\frac{\Delta_{\alpha}}{140} \end{aligned} \qquad (9) $$

with $\Delta_{\alpha} = \alpha_6 - \alpha'_6$. The same reasoning can be applied to each observed state as well as with a higher polynomial degree. We therefore have an error that is linear between the error on the trajectory and the error on the parameters. Now, in the space of parametric trajectories, the minimum squared norm is the shortest distance, which corresponds to an orthogonal projection between the measured trajectory parameters and those of the optimal trajectories.

Moreover, since the low level is a DOC, $\omega$ are necessarily positive. Choosing the solution corresponding to a Bilevel then consists in taking the optimal trajectory closest to the observed trajectory. In other words, an orthogonal projection constrained by positive omega.

**Orthogonal projection without constraints:** As we said before, it does not seem relevant to us to constrain the positivity of $\omega$ but rather to use it as a condition (Sec III-C). This choice of projection which would be purely orthogonal would correspond to a method also used in the literature: the Onelevel IOC [35]. This method is composed by the high level of the bilevel but under KKT condition constraints. Onelevel IOC has the advantage of being faster than the Bilevel which has 2 levels of optimization. Moreover, according to our approach, we even propose to rewrite the Onelevel method as follows:

$$ \begin{aligned} \mathbf{s}^* &= \arg \min_{\mathbf{s}} \|\mathbf{s} - \mathbf{s}_M\|^2 \\ \text{s.t.} \quad &\det(\mathbf{J}(\mathbf{s})^T \mathbf{J}(\mathbf{s})) = 0 \end{aligned} \qquad (10) $$
As already mentioned in the paper, the use of the determinant rather than all the KKT conditions brings the advantage of being simpler and especially of decoupling the feasibility problem from the resolution, which is at the heart of our approach.

2) *Choice of basis:* The approach proposed in this paper aims to address many problems of IOC resolution in the context of human movement. The choice of the cost functions used to generate the observed trajectory is one of them. Figure 4 shows three singularity curves corresponding respectively to torque, angular power and acceleration multiplied by torque. If we observe a trajectory in the middle of its curves, how can we choose the best basis?

We propose a simple algorithm to choose a cost functions basis. This algorithm relies on properties of singularity curves and conditions proposed by our approach.

In a first step, the algorithm proposes to compute the determinants of each of the identification matrices associated to their basis. Once done, basis are sorted in ascending order for the determinants, so that the smallest determinant is first tested with a IOC resolution method. Doing the resolution of IOC on the smallest determinant allows the algorithm to be faster, especially if many basis are tested. However, this criterion cannot be unique because the $\omega$ also give clues on the reliability of the result. Thus, if $\omega$ is not positive, the next basis is tested. The first basis tested with a positive $\omega$ vector corresponds to the chosen basis. If no basis tested gives positive $\omega$, then the algorithm returns that there are no basis associated with this trajectory.

Note that this algorithm is generic and can be used with any solving method (for example, AIOC or Bilevel). If Bilevel is chosen and the computation time of the algorithm is not a problem, it is possible to choose as selection criterion the basis corresponding to the optimal trajectory closest to the observed trajectory.

Be careful, the use of the determinant as a comparison criterion between different basis raises the question of the normalization of matrices.

## IV. DISCUSSION AND PERSPECTIVES

The PIOC approach provides a holistic and generic vision of the IOC problem. In particular, it provides answers to issues of the IOC problem in the context of human motion: model, dimensionality, measurement noise, etc. Each of the proposed solutions is independent of the others and our approach is meant to be generalizable. We show in this paper that two methods of resolution of the IOC problem, Bilevel and AIOC, were just two different types of projection in the trajectory reference frame. This approach opens up many elements. Each of the steps proposed in our approach can be transposed with other methods. Concerning the parametrization, it would be possible to test with other parameters than the polynomial. Similarly, the optimality conditions on which we rely are the KKT conditions. The same approach can be taken with the Jacobi-Bellman conditions. As we work in an uncertain framework, the main interest of this approach is the decoupling between the resolution (or the choice of solution) and the different conditions allowing to choose if the result obtained is valid or not. This approach opens the door to many perspectives: better

<figure><img src="image_6.png" /><figcaption>Fig. 4: Illustration of 3 singular curves: $C = [\tau_1, \tau_2]$ (yellow), $C' = [\dot{\theta}_1 \tau_1, \dot{\theta}_2 \tau_2]$ (blue) and $C'' = [\ddot{\theta}_1 \tau_1, \ddot{\theta}_2 \tau_2]$ (purple). The green point correspond to an uncertain trajectory. It is not obvious which cost function basis to choose in this configuration.</figcaption></figure>

choice of solution according to the application, better choice of cost functions basis, improvement of calculation times (thanks to the Onelevel for example), better vision of the IOC problem, better understanding of the reliability of the results obtained and generalization of the application to human movement.

## V. CONCLUSION

This paper presents our holistic approach called Projected Inverse Optimal Control. This approach is complete and transversal in the framework of uncertainty related to the observation of human movement. It is to be understood that it is not a simple method of resolution. It proposes different elements, each independent of the others, to improve expression and better understand the resolution of the IOC’s problem. We have shown in this paper that our approach overcomes some difficulties. The parameterization of the trajectories allows to reduce the dimensions of the problem. In the polynomial framework, we even show that there is a minimum degree depending on the state, the constraints and the number of cost functions. We also propose to decouple the conditions of the problem from its solution. The identifiability condition verifies that the problem is well posed, the singularity condition verifies an optimality condition necessary to solve the problem and the feasibility condition verifies the reliability of the results. Finally, we show that the solution of the IOC problem could be illustrated simply by projections in the space of trajectories. This vision of the resolution allowed us to explain the differences between the classical resolution methods of the literature as well as to discuss an algorithm allowing to choose a base of cost functions.
REFERENCES

[1] N. Ab Azar, A. Shahmansoorian, and M. Davoudi, "From inverse optimal control to inverse reinforcement learning: A historical review," *Annual Reviews in Control*, vol. 50, pp. 119-138, Jan. 2020.

[2] K. Mombaur, A. Truong, and J.-P. Laumond, "From human to humanoid locomotion—an inverse optimal control approach," *Autonomous Robots*, vol. 28, no. 3, pp. 369-383, Apr. 2010.

[3] T. Park and S. Levine, "Inverse Optimal Control for Humanoid Locomotion," p. 5, 2013.

[4] J. Mainprice, R. Hayne, and D. Berenson, "Predicting human reaching motion in collaborative tasks using Inverse Optimal Control and iterative re-planning," in *2015 IEEE International Conference on Robotics and Automation (ICRA)*, May 2015, pp. 885-892.

[5] S. Gaurav and B. Ziebart, "Discriminatively Learning Inverse Optimal Control Models for Predicting Human Intentions," in *Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems*, ser. AAMAS '19. Montreal QC, Canada: International Foundation for Autonomous Agents and Multiagent Systems, May 2019, pp. 1368-1376.

[6] J. F.-S. Lin, V. Bonnet, A. M. Panchea, N. Ramdani, G. Venture, and D. Kulić, "Human motion segmentation using cost weights recovered from inverse optimal control," in *2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)*, Nov. 2016, pp. 1107-1113.

[7] S. Albrecht, P. Basili, S. Glasauer, M. Leibold, and M. Ulbrich, "Modeling and Analysis of Human Navigation with Crossing Interferer Using Inverse Optimal Control," *IFAC Proceedings Volumes*, vol. 45, no. 2, pp. 475-480, Jan. 2012.

[8] A. Panchea, *Inverse Optimal Control for Redundant Systems of Biological Motion*. Orléans, Dec. 2015.

[9] A.-S. Puydupin-Jamin, M. Johnson, and T. Bretl, "A convex approach to inverse optimal control and its application to modeling human locomotion," in *2012 IEEE International Conference on Robotics and Automation*, May 2012, pp. 531-536.

[10] R. M. Alexander, "The Gaits of Bipedal and Quadrupedal Animals," *The International Journal of Robotics Research*, vol. 3, no. 2, pp. 49-59, Jun. 1984.

[11] Y. Uno, M. Kawato, and R. Suzuki, "Formation and control of optimal trajectory in human multijoint arm movement," *Biological Cybernetics*, vol. 61, no. 2, pp. 89-101, Jun. 1989.

[12] E. Todorov, "Optimality principles in sensorimotor control," *Nature Neuroscience*, vol. 7, no. 9, pp. 907-915, Sep. 2004.

[13] P. Carreno-Medrano, T. Harada, J. F.-S. Lin, D. Kulić, and G. Venture, "Analysis of Affective Human Motion During Functional Task Performance: An Inverse Optimal Control Approach," in *2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids)*, Oct. 2019, pp. 461-468.

[14] D. Clever, R. Malin Schemschat, M. L. Felis, and K. Mombaur, "Inverse optimal control based identification of optimality criteria in whole-body human walking on level ground," in *2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob)*, Jun. 2016, pp. 1192-1199.

[15] D. Clever and K. Mombaur, "An Inverse Optimal Control Approach for the Transfer of Human Walking Motions in Constrained Environment to Humanoid Robots," in *Robotics: Science and Systems XII. Robotics: Science and Systems Foundation*, 2016.

[16] B. Berret, E. Chiovetto, F. Nori, and T. Pozzo, "Evidence for Composite Cost Functions in Arm Movement Planning: An Inverse Optimal Control Approach," *PLOS Computational Biology*, vol. 7, no. 10, p. e1002183, Oct. 2011.

[17] K. Mombaur, A.-H. Olivier, and A. Crétual, "Forward and Inverse Optimal Control of Bipedal Running," in *Modeling, Simulation and Optimization of Bipedal Walking*, ser. Cognitive Systems Monographs, K. Mombaur and K. Berns, Eds. Berlin, Heidelberg: Springer, 2013, pp. 165-179.

[18] J. R. Rebula, S. Schaal, J. Finley, and L. Righetti, "A Robustness Analysis of Inverse Optimal Control of Bipedal Walking," *IEEE Robotics and Automation Letters*, vol. 4, no. 4, pp. 4531-4538, Oct. 2019.

[19] S. P. Boyd and L. Vandenberghe, *Convex Optimization*. Cambridge, UK : New York: Cambridge University Press, 2004.

[20] W. Jin, D. Kulić, S. Mou, and S. Hirche, "Inverse Optimal Control with Incomplete Observations," *arXiv:1803.07696 [cs]*, May 2019.

[21] S. Albrecht, M. Ulbrich, and M. Leibold, "A bilevel optimization approach to obtain optimal cost functions for human arm movements," *Numerical Algebra, Control and Optimization*, vol. 2, no. 1, pp. 105-127, Mar. 2012.

[22] A. Sinha, P. Malo, and K. Deb, "A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications," *IEEE Transactions on Evolutionary Computation*, vol. 22, no. 2, pp. 276-295, Apr. 2018.

[23] A. M. Panchea and N. Ramdani, "Towards solving inverse optimal control in a bounded-error framework," in *2015 American Control Conference (ACC)*, Jul. 2015, pp. 4910-4915.

[24] P. Englert, N. A. Vien, and M. Toussaint, "Inverse KKT: Learning cost functions of manipulation tasks from demonstrations," *The International Journal of Robotics Research*, vol. 36, no. 13-14, pp. 1474-1488, Dec. 2017.

[25] N. Aghasadeghi and T. Bretl, "Inverse optimal control for differentially flat systems with application to locomotion modeling," in *2014 IEEE International Conference on Robotics and Automation (ICRA)*, May 2014, pp. 6018-6025.

[26] E. Pauwels, D. Henrion, and J.-B. Lasserre, "Inverse optimal control with polynomial optimization," in *53rd IEEE Conference on Decision and Control*, Dec. 2014, pp. 5581-5586.

[27] T. L. Molloy, J. J. Ford, and T. Perez, "Finite-horizon inverse optimal control for discrete-time nonlinear systems," *Automatica*, vol. 87, pp. 442-446, Jan. 2018.

[28] M. Johnson, N. Aghasadeghi, and T. Bretl, "Inverse optimal control for deterministic continuous-time nonlinear systems," in *52nd IEEE Conference on Decision and Control*, Dec. 2013, pp. 2906-2913.

[29] M. A. Hanson, "On sufficiency of the Kuhn-Tucker conditions," *Journal of Mathematical Analysis and Applications*, vol. 80, no. 2, pp. 545-550, Apr. 1981.

[30] A. Aswani, Z.-J. M. Shen, and A. Siddiq, "Inverse Optimization with Noisy Data," *Operations Research*, vol. 66, no. 3, pp. 870-892, Jun. 2018.

[31] J. Colombel, D. Daney, and F. Charpillet, "On the Reliability of Inverse Optimal Control," in *2022 International Conference on Robotics and Automation (ICRA)*, May 2022, pp. 8504-8510.

[32] C. Taylor, B. konskyvon Konsky, and C. Kirtley, "Polynomial approximations of gait for human motion analysis and visualization," in *Proceedings of the 20th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Vol.20 Biomedical Engineering Towards the Year 2000 and Beyond (Cat. No.98CH36286)*, vol. 5, Nov. 1998, pp. 2490-2493 vol.5.

[33] N. Hogan and D. Sternad, "Sensitivity of Smoothness Measures to Movement Duration, Amplitude, and Arrests," *Journal of Motor Behavior*, vol. 41, no. 6, pp. 529-534, Nov. 2009.

[34] J.-B. Lasserre, "Inverse polynomial optimization," *arXiv:1103.3284 [math]*, Oct. 2012.

[35] K. Hatz, J. P. Schlöder, and H. G. Bock, "Estimating Parameters in Optimal Control Problems," *SIAM Journal on Scientific Computing*, vol. 34, no. 3, pp. A1707-A1728, Jan. 2012.