Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costs
Authors: Dominik Straub
Matthias Schultheis
Heinz Koeppl
Constantin A. Rothkopf
Venue: NeurIPS 2023 poster (2023)
Keywords: inverse optimal control
probabilistic modeling
motor control
cognitive science

Abstract: Inverse optimal control can be used to characterize behavior in sequential decision-making tasks. Most existing work, however, is limited to fully observable or linear systems, or requires the action signals to be known. Here, we introduce a probabilistic approach to inverse optimal control for partially observable stochastic non-linear systems with unobserved action signals, which unifies previous approaches to inverse optimal control with maximum causal entropy formulations. Using an explicit model of the noise characteristics of the sensory and motor systems of the agent in conjunction with local linearization techniques, we derive an approximate likelihood function for the model parameters, which can be computed within a single forward pass. We present quantitative evaluations on stochastic and partially observable versions of two classic control tasks and two human behavioral tasks. Importantly, we show that our method can disentangle perceptual factors and behavioral costs despite the fact that epistemic and pragmatic actions are intertwined in sequential decision-making under uncertainty, such as in active sensing and active learning. The proposed method has broad applicability, ranging from imitation learning to sensorimotor neuroscience.

Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costs
Dominik Straub
Matthias Schultheis Centre for Cognitive Science
Centre for Cognitive Science
Technische Universität Darmstadt
Technische Universität Darmstadt
dominik. Satraabbtuuddramsttaa
matthias, schul www.:sssssschhhihtoeutoocoooee thwsttttaaaattttttattt.........
Heinz Koeppl
Constantin Rothkopf
Centre for Cognitive Science Technische Universität Darmstadt heinz. koe happlltttddarmtttttt. de
Centre for Cognitive Science
Technische Universität Darmstadt
constantin wrt.hiitnnotoootoeeooeoooo Stu-danssaatt de
Abstract
Inverse optimal control can be used to characterize behavior in sequential decision- making tasks. Most existing work, however, is limited to fully observable linear systems, requires the action signals to be known. Here, we introduce prob- abilistic approach to inverse optimal control for partially observable stochastic non-linear systems with unobserved action signals, which unifies previous ap- proaches to inverse optimal control with maximum causal entropy formulations. Using an explicit model of the noise characteristics of the sensory and motor sys- tems of the agent in conjunction with local linearization techniques, we derive an approximate likelihood function for the model parameters, which can be computed within a single forward pass. We present quantitative evaluations on stochastic and partially observable versions of two classic control tasks and two human be- havioral tasks. Importantly, we show that our method can disentangle perceptual factors and behavioral costs despite the fact that epistemic and pragmatic actions are intertwined sequential decision-making under uncertainty, such as in active sensing and active learning. The proposed method has broad applicability, ranging from imitation learning to sensorimotor neuroscience.
Introduction
Inverse optimal control (IOC) is the problem of inferring an agent's cost function and other properties of their internal model from behavior. While IOC has been fundamental task in artificial intelligence and machine learning, particularly reinforcement learning (RL) and robotics, it has widespread applicability in several scientific fields including behavioral economies, psychology, and neuro- science. For example, in cognitive science and sensorimotor neuroscience, optimal control models have explained key properties of behavior, such speed-accuracy trade-offs [I] the minimum intervention principle (2]. But, while researchers usually build an optimal control model and compare its predictions behavior, certain parameters of the agent's internal processes are typically unknown. For example, an agent might have uncertainty about their perception or experience intrinsic costs of behavior. These parameters are different between individuals and inferring them from observed behavior can help to understand internal tradcoffs between behavioral goals, perceptual and cognitive
"qqual contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

processes, and predict behavior under novel conditions. Applying IOC in these domains poses several challenges that make most previous methods not viable.
First, most IOC methods assume the agent's 's control signals to be known. This assumption, while convenient in simulations or robotics, where the control signals may be easily quantified, does not hold in many other real-world applications. In transfer learning or behavioral experiments, the control signals are internal quantities of an animal or human, e.g., neural activity or muscle activations, and are therefore not straightforwardly observable. Thus, we consider the scenario where a researcher has observations of the system's 's state only, i.e., measurements of behavior.
Second, most IOC methods model the variability of the agent using a stochastic policy in a maximum causal entropy formulation [M Behavioral variability in biologicalsystemss however, is known to arise from multiple distinct sources [4]. There is noise in the sensory system, which makes the
state of the world partially observable, and in the motor system. In sensorimotor neuroscience, the uncertainty in the sensory and motor systems can be characterized quantitatively by formulating accurate models, which are helpful understand behavioral variability [5).
Third, many IOC methods are based on matching feature expectations of the cost function between the model and data 33, and are thus not easily adapted to infer other model parameters. In behavioral experiment, however, researchers are often interested in inferring the noise characteristics of the sensorimotor system or other properties of the agent's 's internal model besides the cost function [6].
Fourth, while the theory of Seeear-uadratie-Gaaan Gaussian (L-G)) control 7 is suited to deal with the issues above, many real-world systems are not captured by the LQG assumptions. First, the dynamics may be non-linear, e.g., in roboties and motor control when controlling joint angles in a kinematic chain. Second, the variability of the system may not be captured by normal distributions, e.g., in sensorimotor control, where the standard deviation of sensory and control signals scales with their means. While iterative methods for solving the optimal control problem such iterative variants of LQG g exist, here we consider the corresponding inverse problem.
To address these issues, we adopt a probabilistic perspective. We distinguish between the control problem faced by the agent and the IOC prob- lem faced by the researcher. From the agent's perspective, the problem consists of acting in a partially observable Markov decision process (POMDP; Fig. A). We consider the setting of continuous states and controls, stochastic non- linear dynamics, partial observations, and finite horizon. For this setting, there are efficient approximately optimal solutions to the estima- tion and control problem (see Section ( The researcher, the other hand, is interested in inferring properties of the agent's model and cost function. The IOC problem from their per- spective can be formulated using a probabilistic graphical model (Fig. B), in which the state of the system is observed, while variables internal to the agent are latent.
Mc+2
Figure I: A Decision network from the agent's perspective [notation from 9. At each time step, the agent receives a noisy observation y, of the state 22 performs control uy, and incurs cost Ct. B Probabilistic graphical model from the re- searcher's perspective, who observes trajectory æi.r of an agent. Quantities internal to the agent, i.e. observations ye, beliefs b, and control signals ue, are not directly observed.
Here, we unify MCE models, which are agnostic regarding the probabilistic structure causing the observed stochasticity of the agent's 's policy, with IOC methods that involve an explicit observation model. First, we define the IOC problem where we allow for both: We employ an explicit observation model, but also allow the agent to have additional stochasticity through MCE policy. Second, we provide a solution to the IOC problem in this setting by approximate filtering of the agent's s state estimate via local linearization, which allows marginalizing over these latent variables and deriving an approximate likelihood function for observed trajectories given parameters (Section 3 By maximizing the approximate likelihood function, an estimate of the optimal parameters can be determined. Third, we evaluate our proposed method on two classic control tasks, pendulum and cart pole, and on two human behavioral tasks, navigation and a manual reaching S3cction[4.2 Section (43)
Fourth, we show that our approach allows disentangling the influences of perceptual uncertainty and behavioral costs on coormationnsceking behavior in the light-dark domain (Section 4.4).
2

---

Related work
Inferring costs or utilities from behavior has been of interest for long time in several scientific fields, such as behavioral economics, psychology, and neuroscience [10 More specific to the problem formulation adopted here, estimating objective functions the field of control was first investigated by Kalman [33 in the context of deterministic linear systems with quadratic costs, More recent formulations were developed first for discrete state and control spaces under the term inverse reinforcement learning [IRL; [4 [5), including formulations allowing for stochasticty in action selection [T6]. this line, the maximum entropy [ME; 17] and MCE formulation [ gave rise to many new methods, e.g., for non-linear continuous systems via linearization sampling [9] for fully observable deterministic systems.
18 or importance
IOC methods for stochastic systems have been developed in the setting of affine control dynamics Arbitrary non-linea stochastic dynamics in the infinite horizon setting have been approached using model-free deep MCE IRL 22 23 The latter approaches, however, do not yield interpretable representations, as the cost function is represented by neural network. Further, past methods based on MCE are limited to estimating cost functions and cannot be used to infer other latent quantities, such as noises or subjective beliefs. The partially observable setting for IOC has previously been addressed for discrete state-action spaces [24) and continuous states with discrete actions [25]. Schmitt et al. (26) addressed systems with linear dynamics and continuous controls for a linear switching observation model. Other work has considered partial observability from the researcher's perspective, e.g., through occlusions [27 28). There are some IOC methods which are applicable to partially observable stochastic systems: In previous work 2 regarded LQG systems, while the work of Chen and Ziebart [30] can be used to estimate cost functions that depend on the state only. Non-linear dynamics in the infinite-horizon setting and the joint estimation of model parameters have been approached by Kwon et al. [31] by training a policy network as a function of the whole parameter space. This work, however, also assumes the control signals to be given and stationary, deterministic policy.
20 21
Applications of IOC methods range from human locomotion 32 over spatial navigation [33] table
tennis (34, to attention switching [35), and target tracking (36]. Other work has been aimed at inferring other properties of control tasks, c.g., the dynamics model (6), learning rules (37), or
discount functions [38]. Several subfields of robotics including imitation and apprenticeship learning
39 as well as transfer learning [40] have also employed IOC.
Background
Before we introduce our probabilistic approach to inverse optimal control, we give an overview of the control and filtering problems faced by the agent and algorithms that can be used to solve it. For summary of our notation in this paper, see Appendix A
2.1 Partially observable Markov decision processes (POMDPs)
a discrete-time stochastic non-linear dynamical system (Fig. A) with states a,, IR" following the dynamics equation 24++1 [ ut vx), where is the dynamics function, u, E R2 are the controls and v, ~ A((0, 7) is o-dimensional Gaussian noise. We assume that the agent has only partial observations y/ E Th. following y: h(a;, uy)) with / the stochastic observation function and w; ~ (O, 1) w-dimensional Gaussian noise. While v, and w, are standard normal random variables, the system can incorporate general control- and state- dependent noises through non-linear transformations within the dynamics function f and observation function h. The agent's goal minimize the expected cost over time horizon T E N, defined by
We consider a special case of POMDPs
T- Σ (a,
J = E (T((T) +
un)
t=1
consisting of final state cost er (æT) and cost at each time step Ce (x uz).
3

---

2.2 Iterative linear-quadratic Gaussian (iLQG)
The control problem from Section 2.1 can be solved approximately using iLQG 8 43 This method iteratively linearizes the dynamics and quadratizes the costs around nominal trajectory. {a,, ū. J.J......... with E; E R", u, E R". and computes the optimal linear control law, u, = (T++) (2) T4) + m, + ü1rr for the approximated system. The quantities Ia and m, the control gain and offset, respectively, and determined through a backward pass for the current reference trajectory. In the following iteration, the determined optimal control law is used to generate a new reference trajectory and the process is repeated until the controller converges.
2.3 Maximum causal entropy (MCE) reinforcement learning
The goal of MCE RL is to minimize the expected cost as in Section 2.2 while maximizing the conditional entropy of the stochastic policy [r(u; (tx), i.e., to minimize [[]( u1:7)
E H(IL, [ This formulation has been used to treat RL as probabilistic inference 444 and model the stochasticity of the agent in IRL 17 3 The objective of IRL maximize the likeli- hood of states and controls { Ut It=1.....N induced by the maximum entropy policy. The resulting optimal policy given by the distribution II, (u, (u, xx) = exp(Q: (z,, Ut) (z+)),, where Qr is the soft Q- Q-function and V, the normalization [7]. For general dynamics and reward functions, it is not feasible compute the soft Q-function exactly. Approximate solutions have been derived using lin- earization [88, importance sampling [1], or deep function approximation 23 For linear dynamics
and quadratic costs, the optimal policy is a Gaussian distribution (u, xx)) = =N(u;: Lext, - -H;²), where L; and H4, result from the optimal LQG controller 48]. More detailed formulas are provided
in Appendix B
2.4 Extended Kalman filter (EKF)
Given the system defined in Section 2.1 the optimal filtering problem is to compute belief distribu- tion of the current state given past observations, i.c., p(x, Y1:-11). For Linear-Gaussie systems, the solution is given in closed form and known as the Kalman filter [49]. In case of non- non-linear systems as in Section 2. a Gaussian approximation to the optimal belief can be computed using the extended Kalman filter (EKF) via b = f(b, u, 0) Ke(ye h(b,, Oj), where b, E R denotes the mean of the Gaussian belief p(æ; 9/...... y/- 1). The matrix K, denotes the Kalman gain for time and is computed by applying the Kalman filter the system leneaiy-ieeeriized around the nominal trajectory
obtained by the approximate optimal control law of iLQG (Section 2.2
3 Probabilistic inverse optimal control
We consider an agent acting in partially observable Markov decision process (POMDP) as introduced in Section 2.1 We assume that the agent acts at time based on their belief b, about the state of the system st, which evolves according to 1 = (b,,, ut, yt). While the helief of the agent defined commonly distribution over the true state, here we model b, as a statistics of the distribution, i.e., b, E R The function 3r R2 R" x R + R² is called belief dynamics. We further assume that the agent follows time-dependent policy R x R R" i.e., u4 = n. (b, E,), which can be stochastic with & ~ N/ 7).
summary
In the inverse optimal control problem, the goal is to estimate parameters O E R" of the agent's optimal control problem given the model and trajectory data. These parameters can include properties of the agent's ' cost function, the sensory and control systems of the agent, or the system's 's dynamics. More precisely, we assume that we are given a parametric form of the system dynamics. belief dynamics, cost function, and initial belief of the agent, which all might depend on the unknown parameters that we aim to infer. Further we assume a set of state trajectories. Importantly, we do not assume knowledge of the agent's 's belief. We follow probabilistic approach to inverse optimal control, i.e., we consider the likelihood function
T II
p ()) =p(æ) 0
p( 2 0),
(1)

---

describing the probability of the observed trajectory data æ1.T := {a1...., 2r given the parameters. For set of trajectories, we assume them to be independent given the parameters, so that the likelihood factorizes into single trajectory likelihoods of the form in Eq. ①T. In this equation, generally, each state 1++1 depends on all previous states 21, 2t, because the agent's internal noisy observations and control signals are not accessible to the researcher (Fig. [B) Therefore, the Markov property does not hold from the researcher's perspective, rendering computation of the likelihood function intractable. To deal with this problem, we employ two key insights: First, the joint dynamical system of the states and the agent's 's belief is Markovian [50]. Second, by keeping track of the distribution over the agent's 's belief. i.., by performing belief tracking (29]. we can iteratively compute the individual factors of the likelihood function in Eq. ( In our IOC method, the goal is to maximize the likelihood w.r.t. the parameters O. To do so, use gradient-based optimization with automatic differentiation to differentiate through the likelihood for computing the optimal parameters (Algorithm D. An implementation of our algorithm is publicly available
We first introduce general formulation of the IOC likelihood involving marginalization over the agent's 's internal beliefs in Section 3.1 Then, we show how to make the computations tractable by
local linearization in Section 3.2 In Section (33 we provide details for suitable linearization points,
which enables us to evaluate the approximate likelihood within a single forward pass.
3.1 Likelihood formulation
We start by defining a joint dynamical system of states and beliefs [50], in which each depends only on the state and belief at the previous time step and the noises. For that, we insert the policy into the dynamics and the policy and observation function into the belief dynamics, yielding the equation
xt+1 b
f(z Te (bb. Ei), v1) (b,,,,, n (b, Et), h( wx)) ga,, b,, Ue, we. Ee)-
=
For given values of æ; and b,, this equation defines the distribution p(1++1, b æt. by). as ve: we & are independent of 1 and b,+1- Importantly, with this formulation, the control signals are only implicitly regarded through the policy, as we assume them to be latent for the researcher. In Section 3.2 we will introduce an approximation via linearization, which leads to a closed-form expression for 1( b Øt, bz).
One can use this Markovian joint dynamical system to compute the likelihood factors for each time step (29). To this end, we first rewrite the individual likelihood terms p(æ:+1 x1:t) of Eq. (I by marginalizing over the agent's belief at each time step, i.c.,
p(a++1 x1.)) / p(x b x1s) db
(3)
As the belief is an internal quantity of the agent and thus not observable to the researcher, we keep track of its distribution, p(b, æ1a)). For this, we rewrite
p(z,11. b x1:)) p( b xt, b.) p(b, x1.e) db;
(4)
where we have exploited the fact that the joint dynamical system of states and beliefs is Markovian. The distribution p(b:l x1:)) acts a summary of the past states and can be computed by conditioning on the current state, i.c.,
p( by x1:-11
p(b x1.e)
(5)
p(æ, xi:t-1)
After determining p(b; (1::), we can propagate it through the joint dynamical system to arrive at the distribution p( bi41 21:)). To obtain the belief distribution of the following time step. p(b:+1 @ 11+++),, we condition on the observed state æ++11 To obtain the likelihood contribution, on
the other hand, we marginalize out b To summarize, starting with an initialization p(bo). we can
compute the individual terms p( (1:))) of the likelihood by executing Algorithm [ (Appendix C
5

---

3.2 Tractable likelihood via local linearization
While the marginalization and propagating operations in the previous section can be done in closed form for linear- Gauss-Ga systems, this is no longer feasible for non-linear systems. Therefore, follow the approach of local linearization used in iLQG (Section (.2) and the EKF (Section (24). For the belief statistics, we consider the mean of the agent's belief, 1.e., b,- E Ela 91..... Ut-1 and initialize the distribution for the first time step as a Gaussian, p(b)) ) = V (N ). We then approximate the distribution p(z b 3t by) Gaussian by applying first-order Taylor expansion of g.
To obtain closed-form expression for 9. which can lincarize, model the agent's policy using iLQG (Section (.2) and the belief dynamics using the EKF (Section 2.4). This choice leads to an affine control and belief given b,, making linearization of p(2 b 22; b,) straightforward. To allow for additional stochasticity in the agent's 's policy, we use the MCE formulation (Section[2.3 For linearized dynamics, the MCE policy is given by Gaussian distribution, so that Iy (b,,, Ee) (b, #1.r) m, + 1:T H. with H, the Cholesky decomposition of H,, and can be marginalized out in closed form.
The approximations we have introduced allow to solve the integral in Eq. (4) in closed form by applying standard equations for linear transformations of Gaussians, resulting in
p(1++1, bi+1 x1a) N (14, Ex),
(6)
with pr = g( "i 0,00, and E, = J6 Jo + JOJ + JwJ + JeJ where J. denotes the Jacobian of g w.r.t evaluated at (xx, (1 0. 0. 0). Under this Gaussian approximation, both remaining operations of Algorithm [ (Appendix C can also be performed in closed form. more detailed derivation and representation of these formulas can be found in Appendix [ If the agent has full observations of the system's 's state, the inverse optimal control problem simplified significantly (see Appendix Eb. Details about the implementation are provided in Appendix F
3.3 Data-based linearization
The forward optimal control problem commonly solved by starting with a randomly initialized nominal trajectory and iterating between computing the locally optimal control law and linearization until convergence. To compute the likelihood in the inverse problem, we can take more efficient approach by linearizing directly around the given trajectory T1.7. We then need to perform only one backward pass to compute an approximately optimal control law given the current parameters, approximately optimal filter. This, in particular, allows efficient
and forward pass to compute computation of the gradient of the likelihood function for the optimization procedure. As we assume compute estimates of the
the controls to be unobservable, but they are needed for the linearization, controls by minimizing the squared difference of the noiseless predicted states and the actual states (see Appendix G). Note that these estimated controls used as observed controls in the IOC likelihood itself. In the case where the full state is not observable, cannot linearize around the trajectory. For these cases, we propose two approaches to compute gradients based on implicit differentiation and differentiating only through the last iteration. As this setting is not the main focus of this paper, details of these approaches are provided in Appendix
only used for the linearization, but not
4 Experiments
We evaluated our method two classic control tasks, i.e., Pendulum and Cart Pole, and two human behavioral tasks, manual reaching and navigation. To evaluate the accuracy of the parameter estimates obtained by our method and to compare it against baseline, we computed absolute relative errors per parameter, i.e., | Ø) /B) This metric makes averages across parameters on different scales more interpretable compared to other metrics such as root mean squared errors. For each task, we simulated 1 100 sets of parameters from a uniform distribution in logarithmic space. For each set of parameters, we simulated 50 trajectories. We then maximized the log likelihood using gradient- optimization with automatic differentiation [L-BFGS algorithm: 51] See Appendix | for a summary of the hyperparameters of our experiments.

---

a) Data simulated with ground truth parameters
b) Log likelihood of parameters c) Data simulated with maximum
likelihood parameter estimates
Low cost, low noise
0.55
0.55
[m]
10-1
MLE
0.45
X
X
0.35
X
10
0.55
0.55
High cost, high noise
[m)
0.45
0.45
X
X
0.35
0.35
X
10
-0.12
-0.02
0.08
10
10
10
-0.32
-0.02
0.08
[m]
X [m]
d) Maximum likellhood parameter estimates
10
estimate
ours
10-4
10t1
10P
10
10
10:2
10"1
10
10-2
10a
10
10t2
10a1
10-2
10-1
100
10a
102
10-1
10-1
10a
107
estimate
10
baselino
10-2
10a2
10:1
107
M/
10
10:2
10
10
10
10
10:
109
10a
102
true
true
true
true
Figure 2: IOC for non-linear reaching. A Simulated trajectories for eight targets. Increasing the control cost and the motor noise affects the trajectories, since reaching the target becomes less important and variability increases. B IOC log likelihood for control cost C, and motor noise dmn The maximum likelihood estimate (pink cross) is close the ground truth parameters (black dot). C Simulated trajectories using the MLEs from B. The simulations are visually indistinguishable from the ground truth data. D True parameters plotted against maximum likelihood estimates. Top row: our method, bottom row: MCE baseline. The columns contain the four different model parameters (control cost Ca. velocity cost c,, motor noise om. observation noise oo).
All tasks we consider have four free parameters: control cost Ca. cost of final velocity Co. motor noise Tm. and observation noise a, the agent. In the fully observable case, leave out the observation noise parameter and only infer the three remaining parameters. For concrete definitions of the parameters in each specific task, see Appendix
4.1 Baseline method
For a comparison to previously proposed methods, we applied baseline method based on the maximum causal entropy (MCE) approach (3] As for this approach, control signals of the observed trajectories are required, we use the estimates of the controls that we determine in our proposed method for the data-based linearization (Section B.3). Note that the baseline, representative for applicable past IOC methods, does not have an explicit model of partial observability. Further note that past methods based on MCE are limited to estimating cost functions, so that parameters such as the agent's noise cannot be inferred. For the specific MCE linearization-bace is actually straightforward to maximize the likelihood with respect the noise parameters, which enables to evaluate noise estimates. To show that this approach constitutes suitable baseline, in
baseline we consider,
7

---

Appendix K. we provide results for the case where the true control signals are known and there is no partial observability. More details of the baseline are provided in Appendix B
4.2 Evaluation on manual reaching task
We evaluate the method on a reaching task with a non-linear two-joint biomechanical arm model, which has been applied to reaching movements in the sensorimotor neuroscience literature [e.g.. 52 [3). The agent's goal is to move its arm towards a target at position e by controlling the torque to the two joints. This objective is expressed as non-quadratic cost function of the joint angles,
J == llr - e²11² + cu l|e
1
+ Co
||ull
since the final position and velocity of the hand er, er are non-linear functions of the joint angles. See Appendix J.1 for details.
We use fully observable [8] and a partially observable version of the task 43 Fig. 2 A shows simulations from the model with two different parameter settings. Evaluating the likelihood function for grid of two of the parameters (Fig. B) confirms that has maxima close to the true parameter values. Simulated data using the maximum likelihood estimates look indistinguishable from the ground truth data (Fig. C).
In Fig. D. we show maximum likelihood estimates and true values for repeated runs with different random parameter settings. The parameter estimates of our method closely align with the true parameter values, showing that we can successfully recover the parameters from data. The baseline method, in contrast, shows considerably worse performance, in particular for estimating noises due to the lacking explicit representation of partial observability. Importantly, even when the true control signals are provided, the noise parameter estimates of the baseline are not well estimated (Appendix K. Estimates for the fully observable case are provided in Appendix K.2) The median absolute relative errors of our method were 0.11, while they were 0.93 for the baseline. The influence of missing control signals and of the lack of an explicit observation model in the baseline can be observed by comparing the results to the fully observable case and the case of given control signals in Appendix K.2 and Appendix
K.3
4.3 Quantitative evaluation on other tasks
To show that our method works for range of different tasks, we evaluated it on the three other tasks (navigation, pendulum and cart pole). In the navigation task, we consider an agent navigating to target under non-linear dynamics while receiving noisy observations from a non-linear observation model. To reach the target, the agent can control the angular velocity of their heading direction and the acceleration with which they move forward. The agent observes noisy versions of the distance to the target and the target's bearing angle. We provide more details about the experiment in Appendix p.2 Maximum likelihood parameter estimates for the navigation task are shown for the partially observable case in Fig- [56] and for the fully observable case in Fig. S10 As for the reaching task, our method provides parameter estimates close to the true ones, while the estimates of the baseline deviate for many trials. Median absolute relative errors of our method were 0.31, while they were 1.99 for the bascline (Fig.
The two classic control tasks (Pendulum and Cart Pole) are based on the implementations in the gym library [54] Because these tasks are neither stochastic nor partially observable in their standard formulations, we introduce noise on the dynamics and turn them into partially observable problems by defining stochastic observation function (see Appendix I.3). In Appendix KK] we show the parameter estimates for the Pendulum (Fig. (54) and for the Cart Pole (Fig. (55) for the partially observable case, while Fig. 588 and Fig. SS9 show the fully observable case, respectively. One can observe that the results are qualitatively similar to the ones in the reaching and navigation tasks, showing that our method provides accurate estimates of the parameters. Median absolute relative errors of our method were 0.12 and 0.41, while for the baseline they were 2.21 and 3.82 (Fig.
8

---

Reaching
Navigation
C Pendulum
D Cart Pole
10a
absolute relative erro
10a
10'
10a
10-1
10-2
ours
10-3
baseline
Fully obs. partially obs.
fully obs. partially abs.
fully obs. partially obs.
fully abs. partially abs.
Figure 3: Evaluation across tasks. Absolute relative errors (log scale) for different tasks. Our method consistently outperforms the MCE baseline.
4.4 Information-seeking behavior in the light-dark domain
Finally, we investigate the ability of our method to disentangle sources of information seeking behav- ior. In the light-dark domain [55], an agent moves in 2D space and receives noisy measurements of its position, whose standard deviation depends on the horizontal distance from a light source:
y 3t + o F
where governs the amount of perceptual uncertainty. This task is common test for information- seeking behavior, because it requires the agent to move towards the light source and at the same time away from the target to reduce the uncertainty about its position relative to the target. When this uncertainty has been reduced, the agent can approach the target (see Appendix [J.4] [.4]for for details). The agent's is goal is to reach the target's position p at the final time step, while minimizing control effort uf:
J (ET P)²
Ei 1= z + c (2t,1 5)2
running cost
Different from the original problem for- mulation, consider the case in which the agent may have an additional inherent desire be close to the light, parameter- ized by c. We recover the original cost repre-
Final state cost
Running costs
Perceptual uncertainty
Ground truth
2.5
function [55] for c 0, but
sent agents that seek light more than neces- sary reduce uncertainty for reaching the goal state with > O. Accordingly, both the reduction of perceptual uncertainty and state-dependent running cost could encour- age the agent to move towards the light
Our estimates
Baseline estimates Vinl
source before approaching the target. For
an external observer, e.g., an experimenter, observing agent moving towards the light might not reveal the different potential sources for this behavior. We now ask if it is possible to disentangle these two factors using proposed IOC algorithm.
x(m)
x [m]
[m]
Figure 4: Light-dark domain. Cost maps and percep- tual uncertainty map plotted with true parameters (A) and inferred parameters using our method (B) and base- line (C). Mean trajectories with start point (circle) and target (cross) are shown in orange.
We simulated 100 trajectories of an iLQG agent [partially observable version, 43
with no inherent desire be near the light source depending on the distance to the light source. The agent first moves towards the light source and then reaches the target (Fig. A). We inferred the parameters a, p, and & using our method and the baseline. Both methods recover the target position. Our method in addition infers values close to the true c, and a. therefore correctly attributes the information-seeking behavior of the agent to the perceptual uncertainty (Fig. 4 B). The baseline method, however, does infer the correct
0) and some perceptual uncertainty (a 0.2)

---

perceptual uncertainty. It instead attributes the agent's 's information-seecle seeking behavior to an inherent desire to be in the right part of the room in the running cost ( This highlights the importance for IOC in partially observable domains to probabilistically take the agent's belief into account, if one is interested in inferring the correct cognitive mechanisms. For a quantitative evaluation of the maximum likelihood estimates in the light-dark domain, see Appendix K.4.
5 Conclusion
In this paper, we introduced a new IOC method for partially observable systems with stochastic non-linear dynamics and missing control signals. Using probabilistic approach, we formulate the IOC problem as maximizing the likelihood of the observed states given the parameters. As the exact evaluation of the likelihood for a general non-linear model is intractable, we developed efficient approximate likelihood by linearizing the system locally around the given trajectories, as in popular approaches such the EKF iLQG. By maintaining a distribution that tracks the agent's belief, an approximate likelihood can be evaluated in closed form within a single forward pass and efficiently optimized. Our proposed formulation is able incorporate multiple sources of the stochasticity of the agent, reconciling the theory of past MCE IOC algorithms [e.g.. 3 and approaches where the
agent's 's stochasticity stems from an explicit stochastic observation and control model [29].
We evaluated our method on two stochastic variants of classic control tasks, pendulum and cart pole, and on two human behavioral tasks, a reaching and a navigation task. In comparison to an MCE baseline, we have found our method to achieve lower estimation errors across all tasks. Further, it successfully inferred noise parameters of the system, which was not possible with the baseline. In the light-dark domain, we were able to infer costs and perceptual uncertainty parameters, two different causes of apparent informntion-ssekknng behavior that could lead to qualitatively similar trajectories. This means that our method is a first step towards distinguishing pragmatic from epistemic controls. To further investigate this, it would be fruitful to examine belief-space planning methods that explicitly include the agent's 's belief covariance in the policy (56].
The limitations of our method mainly due to the lincarization of the dynamical system and the Gaussian approximations involved in the belief tracking formulation of the likelihood function. In more complex scenarios with non- -an-Gassiinn belief distributions, e.g., multimodal beliefs, the method will likely produce inaccurate results. This problem could be addressed by replacing the closed- form Gaussian belief by particle-based methods
(57). Further, focused on tasks which could be solved well by control methods based on linearization and Gaussian approximation (iLQG and EKF), motivated by their popularity in applications in cognitive science and neuroscience. Forward problems that cannot be solved using iLQG are probably not directly solvable using our inverse method. While, in principle, our method is also applicable to other forward control methods that compute differentiable policies, it is an empirical question whether linearizing these policies leads to accurate approximate likelihoods and parameter estimates. A further limitation of our method is that it requires parametric models of the dynamics and noise structure. While missing parameters can be determined using our method, in the case of completely unknown dynamics a model-free approach to IOC would be more suitable. Lastly, while we have shown that inference is feasible, the results probably do not scale to high- nigh-dimensional parameter spaces. One reason for this is that optimization in high-dimensional non-linear space can potentially get stuck in local minima. This problem could be relieved by using more advanced optimization methods. A further, more fundamental, concern with higher-dimensional parameter spaces is that identifiability issues and ambiguous solutions arise. However, our probabilistic approach with a closed-form likelihood opens up the possibility of using Bayesian methods to investigate the identifiability of model parameters (58). For future work exploring the relationship to methods for learning world models in POMDPs might also be a fruitful direction 59 [60]
Our method provides a tool for researchers, e.g., in sensorimotor domains, to model sequential behavior by inferring an agent's subjective costs and internal uncertainties. This will enable answering novel scientific questions about how these quantities are affected by different experimental conditions, how they deviate from intended task goals and provided task instructions, or how they vary between individuals. This particularly relevant to a computational understanding of naturalistie behavior [61-63), for which subjective utilitics are mostly unknown.
10

---

Acknowledgments and Disclosure of Funding
We gratefully acknowledge the computing time on the high-performance computer Lichtenberg at the NHR Centers NHR4CES at TU Darmstadt, and financial support by the project "Whitebox" funded by the Priority Program LOEWE of the Hessian Ministry of Higher Education, Science, Research and Art. We thank Fabian Kessler for helpful discussions and comments and an anonymous reviewer of previous version of the manuscript for surmising that the present method could not infer information-seeking behavior.
References
[1] Christopher M Harris and Daniel M Wolpert. Signal-dependent noise determines motor planning. Nature, 394(6695):700-78) 1998.
[2] Emanuel Todorov and Michael Jordan. Optimal feedback control as a theory of motor coordination. Nature neuroscience, 5(11):1226 1235, 2002.
[3] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning, 2010.
[4] A Aldo Faisal, Luc PJ Selen, and Daniel M Wolpert. Noise in the nervous system. Nature reviews neuroscience, (4):292-333, 2008.
[5] Daniel M Wolpert and Zoubin Ghahramani. Computational principles of movement neuroscience. Nature neuroscience, 3(1 11:1112212-12 2000.
[6] Matthew Golub, Steven Chase, and Byron Yu. Learning an internal dynamics model from control demonstration. In International Conference on Machine Learning, pages 606-614, 2013.
[7] Brian DO Anderson and John Moore. Optimal control: linear quadvatic methods. Prentice-Hall, 1990.
Emanuel Todorov and Weiwei Li. generalized iterative LQG method for locally-optimal feedback control constrained nonlinear stochastic systems. In American Control Conference, pages 300-306. IEEE, 2005.
[9] Mykel Kochenderfer, Tim Wheeler, and Kyle H Wray. Algorithms for decision making, MIT press. 2022.
[10] Frederick Mosteller and Philip Nogee. An experimental measurement of utility. Journal of Political Economy, 59(5):371-404, 1951.
[11] Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263-292, 1979.
[12] Konrad Paul Körding and Daniel M Wolpert. The loss function of sensorimotor learning. Proceedings of the National Academy of Sciences, 101(26)998399966) 9842, 2004.
[13] R E Kalman. When Linear Control System Optimal? Journal of Basic Engincering, 86(1):51-60. 1964.
[14] Andrew Y Ng, Stuart J Russell, al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume 1, 2000.
[15] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Interna- tional Conference Machine Learning, volume 5, 2004.
[16] Constantin A Rothkopf and Christos Dimitrakakis. Preference elicitation and inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference (ECML PKDD), pages 34-48. Springer, 2011.
[17] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In AAAI Conference Artificial Intelligence, volume 23, pages 1433-1438, 2008.
[18] Sergey Levine and Vladlen Koltun. Continuous inverse optimal control with locally optimal examples. In International Conference Machine Learning, pages 475-482, 2012.
[19] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In International Conference Artificial Intelligence and Statistics, pages 182-189, 2011.
11

---

[20] Navid Aghasadeghi and Timothy Bred. Maximum entropy inverse reinforcement learning in continuous state spaces with path integrals. In International Conference on Intelligent Robots and Systems, pages 1561-1566, 2011.
[21] Weiwei Li, Emanuel Todorov, and Dan Liu, Inverse optimality design for biological movement systems. World Congress of the International Federation of Automatic Control (IFAC), 44(1) (1:966229667, 2011.
[22] Chelsca Finn, Sergey Levine, and Picter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pages 19-58. PMLR, 2016.
[23] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song. and Stefano Ermon. IQ-leam: Inverse soft-Q learning for imitation. In Advances in Neural Information Processing Systems, volume 34, pages 4028-4039, 2021.
[24] JD Choi and Kee-Eung Kim. Inverse reinforcement learning in partially observable environments. Journal of Machine Learning Research, 12:691-730, 2011.
[25] Júnior AR Silva, Valdir Grassi, and Denis Fernando Wolf. Continuous deep maximum entropy inverse reinforcement learning using online POMDP. In IEEE International Conference Advanced Robotics, volume 19, pages 382 387, 2019.
[26] Felix Schmitt, Hans-Joachim Bieg. Dietrich Manstetten, Michael Herman, and Rainer Stiefelhagen. Exact maximum entropy inverse optimal control for modeling human attention switching and control. In IEEE International Conference on Systems, Man, and Cybermetics, pages 2807-2813, 2016.
[27] Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting, In European Conference on Computer Vision, pages 201-214. Springer, 2012.
[28] Kenneth Bogert, Jonathan Feng-Shum Lin, Prashant Doshi, and Dana Kulic. Expectation maxiimiaiinniiiiiiiiiii for inverse reinforcement learning with hidden data. In International Conference on Autonomous Agents Multiagent Systems, pages 1034-1042, 2016.
[29] Matthias Schultheis, Dominik Straub, and Constantin A Rothkopf. Inverse optimal control adapted to the noise characteristics of the human sensorimotor system. In Advances in Neural Information Processing Systems, volume 34, pages 9429-9442, 2021.
[30] Xiangli Chen and Brian Ziebart. Predictive inverse optimal control for linear-quadratiicGGausss systems. In Artificial Intelligence and Statistics, pages 165-173, 2015.
[31] Minhae Kwon, Saurabh Daptardar, Paul R Schrater, and Xaq Pitkow. Inverse rational control with partially observable continuous nonlinear dynamics. In Advances in Neural Information Processing Systems, volume 33, pages 7898-79087 2020.
[32] Katja Mombaur, Anh Truong, and Jean-Paul Laumond. From human humanoid locomotion inverse optimal control approach. Autonomous Robots, 28(3):369-33) 2010.
[33] Constantin Rothkopf and Dana Н Ballard. Modular inverse reinforcement learning for visuomotor behavior. Biological cybernetics, 107(4):477-990, 2013.
[34] Katharina Muelling, Abdeslam Boularias, Betty Mohler, Bernhard Schölkopf, and Jan Peters. Learning strategies in table tennis using inverse reinforcement learning. Biological Cybernetics, 108(5):603-619, 2014.
[35] Felix Schmitt, Hans-Joachim Bieg, Michael Herman, and Constantin A Rothkopf. what you see: Inferring sensor and policy models of human real-world motor behavior. In AAAI Conference on Artificial Intelligence, 2017.
[36] Dominik Straub and Constantin A Rothkopf. Putting perception into action with inverse optimal control for continuous psychophysics. Elife, 1 11:e76635, 2022.
[37] Zoe Ashwood, Nicholas A. Roy, Ji Hyun Bak, and Jonathan W Pillow. Inferring learning rules from animal decision-making. In Advances in Neural Information Processing Systems, volume 33, pages 3442-3453, 2020.
[38] Matthias Schultheis, Constantin A Rothkopf, and Heinz Koeppl. Reinforcement learning with non- exponential discounting. In Advances in Neural Information Processing Systems, volume 35, pages 3662. Curran Associates, 2022.
12

---

[39] Takayuki Osa, Joni Pajarinen, Gerhard Neumann. J Andrew Bagnell, Pieter Abbeel. Jan Peters, et al. An algorithnic perspective imitation learning. Foundations and Trends in Roboticx, 7(1-2)11-77, 2018.
2018.
[40] Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, 10(7), 2009.
[41] Karl Johan Åström. Optimal control of Markov processes with incomplete state information I. Journal of Mathematical Analysis and Applications, 10:174-205 1965.
[42] Leslie Pack Kaelbling, Michael L Littman, and Authony Cassandra. Planning and acting partially observable stochastic domains, Artificial intelligence, 101(1-2):99-134, 1998.
[43] Weiwei Li and Emanuel Todorov. Iterative linearization methods for approximately optimal control and estimation of non-linear stochastic system. International Journal of Control, 80(9):1439-1453 2007.
[44] Hilbert Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as a graphical model inference problem. Machine learning, 87(2):159-18) 18299-112222222 2012.
[45] Mare Toussaint. Robot trajectory optimization using approximate inference. In International Conference Machine Learning, pages 1049-1056, 2009.
[46] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, arXiv preprint 3tti::8805.00000 2018.
[47] Adam Gleave and Sam Toyer. A primer maximum causal entropy inverse reinforcement learning. arXiv preprint arXiv: 1140, 2022.
[48] Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Muchine Learning, pages 1-9, 2013.
[49] R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1):35-45, 1960.
[50] Van Den Berg, Pieter Abbeel, and Ken Goldberg. LQG-MP: Optimized path planning for robots with motion uncertainty and imperfect state information. The International Journal of Robotics Research, 30 (7):895 91399-9933900 2011.
[51] Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale count-oonttrineed optimization. ACM Transactions on Mathematical Software (TOMS), 23445550-600 1997.
[52] Amee J Nagengast, Daniel A Braun, and Daniel M Wolpert. Optimal control predicts human performance objects with internal degrees of freedom. PLoS Computational Biology.
2009.
[53] David Knill, Amulya Bondada, and Manu Chhabra. Flexible, dependenn use of sensory feedback to control hand movements. Journal of Neurosciecce, 31(4):2219-122 12309, 2011.
[54] Greg Brockman, Vicki Cheung, Ludwig Pettersson. Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01544 2016.
[55] Robert Plaut, Russell Louis Tedrake, Leslie P. Kaelbling, and Tomas Lozano-Perez. Belief space planning assuming maximum likelihood observations. In Proceedings of Robotics: Science and Systems, 2010. doi: 10. 10.15607/RSS.00 2010. V1.037.
[56] Jur Van Den Berg, Sachin Patil, and Ron Alterovitz. Motion planning under uncertainty using iterative local optimization in belief space. The International Journal of Robotics Research, 31(11):1263 1278, 2012.
[57] Arnaud Doucet, Nando De Freitas, and Neil James Gordon. Sequential Monte Carlo methods in practice. Springer, 2001.
[58] Luigi Acerbi, Wei Ma, and Sethu Vijayakumar. A framework for testing identifiability of bayesian models of perception. In Advances in Neural Information Processing Systems, volume 27, 2014.
[59] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1rX 2019.
13

---

[60] Tadahiro Taniguchi, Shingo Murata, Masahiro Suzuki, Dimitri Ognibene, Pablo Lanillos, Emre Ugur. Lorenzo Jamone, Tomoaki Nakamura, Alejandra Ciria, Bruno Lara, al. World models and predictive coding for cognitive developmental robotics: frontiers challenges. Advanced Robotics, pages 1-27, 2023.
[61] John Krakauer, Asif Ghazanfar, Alex Gomez-Marin, Malcolm Maclver, and David Poeppel. Neuroscience needs behavior: correcting reductionist bias. Newron, 73(3):480-4900 2017.
[62] Paul Cisek and Alexandre Pastor- Bermir. On challenges and mechanisms embodied decisions. Philosophical Transactions of the Royal Society B: Biological Sciences, 369(16553:20134775 2014.
[63] Cory Miller, David Gire, Kim Hoke. Alexander Huk, Darcy Kelley, David A Leopold, Matthew Smear, Frederic Theunissen, Michael Yartsev, and Cristopher M Niell. Natural behavior is the language of the brain. Current Biology, 32(10) (2110))-482-4993 2022.
[64] Christopher M Bishop. Puttern recognition and machine learning. Springer, 2006.
[65] Roy Frostig, Matthew James Johnson, Chris Leary. Compiling machine learning programs high-level tracing. Systems for Machine Learning, 4(9), 2018.
[66] Pauli Virtanen. Ralf Gommers. Travis Oliphant, Matt Haberland, Tyler Reddy. David Cournapeau. Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-72,, 2020.
[67] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López, Fabian Pedregosa, and Jean-Philippe Vert. Efficient and modular implicit differentiation. arXiv preprint arXiv:2105.15183 2021.
[68] Steven George Krantz and Harold R Parks. The implicit function theorem: history, theory, and applications. Springer Science & Business Media, 2002.
[69] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López. Fabian Pedregosa, and Jean Phillpii Vert. Efficient and modular implicit differentiation. In Advances Neural Information Processing Systems, volume 35, pages 5230-5242, 2022.
[70] Weiwei Li. Optimal control for biological movement systems. PhD thesis, UC San Diego, 2006.
14

---

Appendix
A Notation
Table 1: Notation
27 E R
state at time t control at time observation at time / number of time steps system dynamics function observation function
U E R
y: E Rm
T E N
f: IR X R
x _ + Rd I
h: R" X RR
Vc E R
standard multivariate normal dynamics noise standard multivariate normal observation noise cost function at intermediate time steps cost function at final time step summary statisties of agent's belief, p(æ,, y1:t-1) belief dynamics policy of the agent model parameters
W; R
Q. R" x IR + R
CT IR" R
b IR
Be TRC I R. RR
Te IR X IR' + Ra
E IRP
IR" x R x R X
R
joint dynamics of states and belief's
B MCE IRL Baseline
Maximum causal entropy inverse reinforcement learning (MCE IRL) 47 can formulated as maximizing the likelihood
p 0) p( ()) II
p(a++1 2t u. 0)L1, (ux xx),
(10)
t=0
with
11 (ur xx) exp(Q) uz) V/O (x:)).
Here, O is the soft Q-function at time t, given by
(x,, (x,, u:) - (x ut) - E(V; (x
and V the normalization,
Ve (ax)) = log / exp(Q2 (mt, u.)) duy-
u.
For arbitrary systems, computing the soft Q-function exactly is infeasible, therefore common meth- ods apply approximations such as linearization [1], importance sampling 19), or deep function approximation 23 For the case of linear dynamics and quadratic reward, the optimal policy is given by a Gaussian distribution (1; xx) (u;;; L,w,, -H,1), where Le is the controller gain and Hl, matrix resulting from the computation of the LQG controller [88]. As the tasks we consider can be well solved by linearizing the dynamics locally, we choose an approximation by linearization and use the optimal MCE controller for the linearized dynamies to compute the likelihood function for the parameter set 0.
To apply this baseline the setting where control signals missing, the controls as we determine in our proposed method for the data-based linearization (Section 3.3 and Appendix G
the estimates of
We can compute the approximate likelihood function by performing the following steps:
1. Estimate the missing control signals using Eq.
15

---

2. Linearize the system as described in Section 3.3
3. Compute the MCE policy for the linearized system, i.., compute Qf and V
4. Compute the likelihood (in log space) using Eq. (TO)
To maximize the (log) likelihood efficiently, one needs to compute the gradient of the likelihood function, which is straightforwardly achieved by backpropagating the gradient using automatic differentiation in step 4.
Note that there is explicit model of partial observability and we use point estimates for the control signals, therefore the likelihood in Eq. (TO) decomposes independent factors given states and controls. In contrast, in our approach, when incorporating partial observability and missing controls, one has to compute approximate likelihood contributions within forward pass.
C Algorithm to compute an approximate likelihood for IOC
Algorithm Approximate likelihood computation
Output: Approximate likelihood of parameters p(æ1.r 0)
Input: Parameters 0, Data æi.r. Model f,
1: Determine the policy using iLQG
2: Determine the belief dynamics /3 using the EKF
3: for in f1,..., 1} do
4: Compute p(z by+1 11:)) using Eq.
5: Update pbbt11 01.(+1) using Eq. (5)
6: Obtain p(1++1 (1:e) using Eq. 33)
7: end for
D Inverse optimal control derivations
We start with defining the joint dynamics of states and estimates, which returns the next state and estimate as a function of the previous state and estimate and the noises,
E++1 b
= g(x,, br, U., Wr, Et).
To do so, we insert the policy and observation function into the system and belief dynamics, giving
Ft++1 m. Tb, Ex), vt). b 3, me (b. Et). h( w.)).
The individual factors of the likelihood function can be determined as
p(a++1 (.:+) = / p p( b (ia) db
where p(# b a)a) is given by marginalizing over the current belief b, as p(0 bet1 x1a)
p(11+1, bLt1, b, wi.c) db;
/ p(a,41, Lias by) p(b; maa) db,
/ p(1111, b b.) p(b: a1a) db,-
Linearization
To derive tractable approximation of the distribution p( b .1r)), we model the initial belief of the agent's state estimate p(b;) as Gaussian and linearize the joint dynamics function, leading to Gaussian approximation of the desired quantity, ie., p(# b Ø1a) se N (pe, Ex).
16

---

First, we apply first-order Taylor expansion of the joint dynamics around the observed state and the mean of the belief Hi (b) and the noises:
g( by, vt, wr 6.) ~ g ii 0,0,0) (b. -Me) Ju + ww + Jgh.,
where J. denotes the Jacobian of g w.r.t. evaluated at (æt, M4 0,0, To derive explicit representation of the Jacobians, insert the filtering and control law obtained by the Kalman filter and maximum entropy iLQG controller and find
f(r IT (T,,,, Et), v7)) (be,,, Te (bb Et), (1 wy)) f(r, Ue v²)
gl by, Dt. wr; Er)
= Be (be, uhi y1h
with
y: h( wt),
u, ne (br, Et) = Le B1.T) + m. + ūrr L.C..
(b,, u4, Mc) S(be, u4.0) + Ke(ye h(b;,, 0)),
leading to the equations
Vuf(m u4 Vb;
Jb =
DbBeebe ue Mr) + Vu (b,, Ue yt bb
V Vn faf Uz 0)
Vn.(br, uc 0) K, V.l(b,,, 0) V Vn (br, un 0)Li
V vef(xr ue 0) V fof Ue 0)
J.
Jw VnBi(ba, ue y: V vi
Jg Vul( u. 0) e Vuf(b; Un 0) tu
Vuf uc L f ue, 0) Lz
Propagating the Gaussian belief over the agent's state estimate, p(b; x1:)) through the linearized dynamics model (Eq. (4)) can be done by applying standard identities for linear transformations of Gaussian random variables [64] and gives
p( b
#11) N (Ju, Ex),
with g( by+1 gives the desired likelihood factor, while conditioning on #4-1 gives the belief statistic for the following time step (see Section 3.1
(b) 0, and Σ, = Jn J + J.J. + D.. + IeJ.. Marginalization over
17

---

E
Special case: full observability
the state e; is fully observable to the agent, the problem is simplified significantly. The control problem from the agent's 's perspective (Fig. ST] left) can be solved by applying iLQQ [ to the
observed states directly (Section? 2.2
Mc
Uy (
This also simplifies the IOC problem from the re- searcher's s perspective because evaluating the ap- proximate likelihood becomes straightforward as we do not need to marginalize over the agent's belief.
Figure S1: Left: decision network from the agent's perspective [following the notation used in [[]. The agent directly observes the state 2y. each time step, they perform control and incur cost ct. Right: probabilistic graphical model from the researcher's 's perspective, who observes a trajectory a i:T from an agent. The control signals u, are not directly observed.
Recall that the likelihood can be decomposed as
T-1
p(Z 0) p(zo
II p( (1.,
and the dynamics are given as
T++1 f(x,, Un vi) f( (ax, Ec), v().
We approximate the likelihood contribution at each time step
p(1++1 T1.1, 0) N Et),
with
Fr f( T (a 0),
Σ, I.Jl JgJT
where J. denotes the Jacobian of f w.r.t. evaluated at (æt,, π (x;, 0), 0):
J. (ar, un 0),
Jg = vuf(x, un gu, = Vuf(x, u. 0) Lt.
F
Implementation
We provide flexible framework for defining non-linear stochastic dynamical systems of the form introduced in Section 2. by implementing the dynamics, observation function, and cost function. The optimal estimation control methods based on iterative linearization are implemented using the automatic differentiation library jax (65). so that Jacobians and Hessians for the linearization of the dynamics and quadratization of the cost do not have be specified manually. Furthermore, the Jacobians needed for Gaaessinn approximation in the IOC likelihood (Section 3.2) are also computed using automatic differentiation. Gradient-basse
maximization of the log likelihood was performed using the L-BFGS-B scipy [66] from the jaxopt library [67]. The implementation is provided in the supplementary material and will be made public upon publication.
For computations, we used an Intel Xeon Platinum 9242 Processor, using core per run. The mean run times for computing maximum likelihood estimates (partially observable setting) were follows:
Reaching: 140
Navigation: 64
Pendulum: 31
CartPole: 115
18

---

G
Estimating controls for linearization
For every evaluation of the likelihood function described in Section 3.1 solve the forward control problem once by running the iLQG algorithm given the current parameter values. This computationally inefficient due the iterative procedure of iLQG, which requires multiple forward and backward passes. Instead of performing the iterative procedure, which computes the locally optimal control law {L m1:r-1} around a nominal trajectory {Ø1.T, -1,-1), then computes the new nominal trajectory given the current control law and so on, we realize that in the IOC setting, we already have observed the actual trajectory 21.7 performed by the agent. To make solving the forward control problem more efficient, we only compute the locally optimal control law once around this trajectory. This would require the control signals u1.7 to be given, but they are unobserved in our problem setting. To obtain proxy for the controls for the purposes of linearization, we solve for the controls given the trajectory I1T the system dynamics equation de+11 = f(æ,, Ue, 0) using the Gauss-Newton method for non-linear least squares implemented in jaxopt, i.c.,
we would naively need
ú arg maggax -f (zt, u. u.)))²
(11)
H Efficient gradient computation
In the case where the full state is not observable, one cannot linearize around the given trajectory as descibed in Section 3.3 The forward optimal control problem is commonly solved by starting with randomly initialized nominal trajectory and iterating between computing the locally optimal control law and linearization until convergence. To compute gradients, a naive approach would be to differentiate through the loop of the forward procedure. In experiments, found, however, that this approach is numerically unstable and does not yield correct gradients (see Fig. S2) Here, we provide two approaches to compute gradients in this case.
H.1 Differentiating through the last iteration
In this approach, one first solves the forward problem as usual to determine the linearization. One then executes one additional backward pass for the optimal controller, which one differentiate using automatic differentiation.
H.2 Applying the implicit function theorem
Determining the optimal linearization and controller is essentially fixed point problem. To dif- ferentiate through the optimal linearization and controller, we apply the implicit function theorem 668 69
We assume the forward equation determine the linearization is given by
Et+ f(x Ur O),
U, = te (x,, st, 0)
where O are the parameters we want to differentiate with respect to and is vector determining the optimal controller, such that can be determined via a backward pass with
8 1 = (w,, Uh 81, 0).
For the LQG controller, 8, would consist of the (vectorized) matrix characterizing the optimal value function.
The fixed point problem of jointly determining the optimal lincarization and controller can be formulated as determining X = [ui, 222 U2..... uT-1, Ør, $1..... ST] such that
X xX = | (æ1, 81, 0), (xx u1, 0), ( 82, 0), f(#T-1,UT-1 0), >a22 U2 82, i).... ...., ( ur, 8T, 0))
The optimal solution of the fixed point problem satisfies
K x(X (0), == X - nxX' (0), ( = =0,
19

---

where X (0) is a function yielding the optimal solution depending on 0, i.c., X R 7 (ntitt)) where s the number of elements of s,..
The implicit function theorem 168 gives, that for (Xo: Qo) satisfying oo 0n) = with the require- ment that x is continuously differentiable and the Jacobian 0gx (Xo, 0p) is square invertible matrix, then there exists function (-) on a neighborhood of O, such that X x*(0)) (0o) Xo. Furthermore,
for all 0 in this neighborhood, we have that xxX* (0). 0) and 0x (0) exists.
By applying the chain rule, the Jacobian 8X* (0) needs to satisfy
0 = 0i (0), 0) ( (0) 0gx(X* = (1 - (O), (0,,
(xn(("(()) 0)) is given by a sparse matrix, as each element of (), 0) only depends on few elements of X. The desired gradient ax** (0) (or vector-Jacobian products with it) can then be computed using linear system solvers [69].
H.3 Results with different gradient computation methods
We the MLE for the partially observable reaching task with different gradient computation methods. The results are shown in Fig. S2 While unrolling the loop led to numerical problems, so the estimates are far off the true values, all other methods led to quite similar results. The mean times were as follows:
Unrolled loop: 141
Differentiating through the last iteration: 103
Implicit differentiation: 243
Data-based linearization: 129
I Hyperparameters
Throughout the experiments, have used the following hyperparameters:
Table 2: Hyperparameters
Data set size (number of trajectories)
50
Number of datasets per evaluation
100
Number time steps (T)
50 (reaching, navigation, pendulum, light-dark),
200 (cartpole)
Optimizer
L-BFGS-B (scipy wrapper from jaxopt)
Maximum entropy temperature
10 10-6 (reaching, navigation),
10 3 (pendulum, cartpole),
10 5 (light-dark)
Optimizer restarts
50
J Tasks
J.1
Reaching task with biomechanical arm model
We implemented the non-linear biomechanical model for arm movements from Todorov and Li g and its partially observed version from Li and Todorov 43 which described in more detail in the PhD thesis of [70]. The dynamics describe the movement two-link arm, which can be controlled by applying torques the two joints. The task move the hand target location, as defined in the cost function
J = let - c²112 + [|eT +
|
20

---

dm
10-1
10
Unrolled loop
estimate
10-7
10
100
10a
10th
10
10
107
107
Implict diff estimate
10
100
10-1
16-1
107
107
Last Iteration
estimate
10a
10"1
105
10-4
10-1
10a
10
102
102
Data-based iin.
estimate
10
10
10-1
10
10:1
Figure S2: Maximum likelihood parameter estimates for the partially observable reaching task with different gradient computation methods
where ey is the position of the hand at the final time step in Cartesian coordinates, e* is the position of target, and ér the final velocity of the hand. That is, the task is to bring the hand the final time step (7 50) and minimize the final velocity while making as little control as possible along the way. The parameters C, and 6 trade off, how important the velocity and control parts the cost function are relative the main goal of reaching the target. Additionally, we parameterize the signal-dependent variability of the motor system with the parameter om, which defines how strongly the variability is scaled by the control magnitude.
target at
J.2 Navigation task
We implemented simple navigation task in which agent walks towards target. The state z; [tt M/s Or, wr] consists the position in the horizontal plane, the heading angle, and the speed in the heading direction. The dynamics
cos (i w
f(x, u e) x di
#1 T U1 (U2 + o. U2
which means that the agent controls the velocity of the heading angle and the acceleration in heading direction. There is doperol-de noise, whose strength is determined by the parameter o.- The observation model is
V(x k1 2 (y k2) tan 1 (y k2, kn)
h(w, w)
+ oow,
21

---

where k = [K1, ka] T is the position of the target and 0, determines the magnitude of the observation
noise. The cost function
T-1
= - k1)² (yr - k2)² + W, +
Σ ca
ue
with parameters determining the cost of controls (Ca) and the cost of the final velocity (cx). The time horizon was 7 = 50.
J.3 Classic control tasks
We evaluate method on two classic continuous control tasks. Specifically, we build upon the Pendulum and Cart Pole environments from the gym library [54]. Because these environments are not stochastic in their standard implementations and are typically used in infinite-horizon reinforcement learning setting, we made the following changes to the environments. To account for the finite-horizon horizon setting the task goal are applied only to the final time step, while control-dependent costs are applied at every time step along the trajectory. To make the problems stochastic, have added noise the dynamics and the observation function.
considering in this work, the state-dependent costs associated with
J.3.1 Pendulum
The task is to make pendulum, which is attached fixed point side, swing into upright position, i.e. to reach an angle 6 0. The pendulum starts at the bottom (0 = т) and can be controlled by applying torque to the free end of the pendulum. We added control-depente to the dynamics, where the parameter o. controls, how strongly standard normal noise is scaled by the magnitude of the torque. In addition to this motor noise parameter, the task has two other free parameters: the cost of controls ca and the cost of the final velocity c.. the partially observable version of the task, the agent receives the observation via the non-linear stochastic observation model
noise
sin Or, cos O,, 6.] + 0.1w; The time horizon was T = 50.
J.3.2 Cart pole
In the Cart Pole task, pole is attached cart that be moved left right by applying continuous force. In our version of the task, the goal is to move the cart from the horizontal position while balancing the pole an upright position. Again, we add control-dependent noise to the force, parameterized the strength of the linear control centrol-dependence dm- As above, the other parameters are the control cost c, and the cost of the final velocity Cu. In the partially observed version of the task, the agent receives a noisy observation with X, + wi. The time horizon was T 200.
J.4 Light-dark domain
We use a slightly modified version of the light-dark domain, which was originally introduced by Platt al. (55). We adapted the light-dark domain's 's motion model to include signal-dependent noise the control
Et+1 2; dt u, + 0.1 us
The standard deviation of the perceptual uncertainty varies with the horizontal distance the light source:
y: x. + of wt.
The cost function is composed of three terms. First, the agent should minimize the squared distance to the target p the final time step T. Second, the agent should minimize the squared control signals uy And finally, the agent should minimize the horizontal distance to the light source:
J [TT - p) M
u + c ( - 5)2
Einal
running cost
22

---

The mone-hoizzon was set to T 50.
We simulated the behavior of two agents in the light-dark domain in Fig. S3 observable version of iLQQ 43 takes the expected belief covariance into account for the computation of the policy. This agent another cost term that expresses a preference to be close to the light source does not change much about this behavior (top row). Second, the fully observable version of iLQG (8) computes the policy irrespective of the belief. To apply it in the partially observable setting, we simply combine it with an EKF for the state estimation. This agent does move towards the light source by default and as result, the belief uncertainty is higher. The agent only moves towards the light source before approaching the target if add the additional cost term 0. One can also see Fig. [S3 that the agent's s belief is more uncertain if it does not move to the light source before approaching the target. This results in a higher variability around the final position (inset plots in Fig. S3
First, the partially
towards the light source before approaching the target. Adding
C 0.0
0.1
1.0
Partially obs. iLQG
(Li & Todorov, 2007)
2.5
0.0
2.5
Fully obs. iLQG + EKF
(Todorov & Li, 2005)
2.5
0.0
-2.5
0
5
0
5
0
5
Figure S3: Trajectories and beliefs the light-dark domain. Partially observable (top) vs. fully observable (bottom) iLQG in the light dark domain with different values for the cost parameter which expresses an inherent desire to be near the light source. The circles indicate the agent's belief covariance (2 standard deviations). One also that the variability around the final position is higher for agents that do not move towards the light source before approaching the target (inset plots).
23

---

K
Additional results
The parameter estimates and values are provided in Fig.
Fig.
S5
Fig.
K.1
Results for all tasks in the partially observable setting
Cv
Qm
Gg
10-1
10
101
estimate
ours
10
10
10-1
10
10-
107
bascline
eatimate
10-1
10-
10
1011
10
10
100
10
true
true
true
Drue
Figure S4: Maximum likelihood parameter estimates for the Pendulum task (partially observable cases).
Om
0,
estimate
10
OUTS
10al
13rt
2
baseline
estimate
true
true
true
true
Figure S5: Maximum likelihood parameter estimates for the Cart Pole task (partially observable).
Cv
10
10a
estimate
30
ours
10th
10-4
10
10a
10a
baseline
eatimate
30-1
10
10
de
true
true
true
true
Figure S6: Maximum likelihood parameter estimates for the navigation task (partially observable).
24

---

K.2 Results for all tasks in the fully observable setting
Om
1011
1011
109
estimate
ours
10a2
10a2
10t1
10a
10"
10
10:1
10
baseline
estimate
10a2
10a2
10")
10"4
10"
10"
10
10
10
10
30
10
10
10
Figure S7: Maximum likelihood parameter estimates for the reaching task (fully observable).
Ca
Cy
mm
10a
16
estimate
ours
10-2
10.2
10a2
16
10a
10a
10
baseline
estimate
10a2
10-2
10-4
10
10-1
10-4
10
10-
10-7
10
10
10a
true
true
true
Figure S8: Maximum likelihood parameter estimates for the pendulum task (fully observable).
C
C
4m
10
101
estimate
10
ours
19-1
baseline
estimate
10a2
1011
x
10-2
10-1
10
2x
true
true
true
Figure S9: Maximum likelihood parameter estimates for the cart pole task (fully observable).
25

---

Qm
101
estimate
10-1
ours
10a1
10
10-1
baseline
estimate
10rt
10
10-9
10
10-4
10-1
10-1
10-1
10'
10-
10a
true
true
true
Figure S10: Maximum likelihood parameter estimates for the navigation task (fully observable).
26

---

K.3
Results for baseline with given control signals
Ca
Gm
10n1
10a1
10P
Reaching
estimate
10
10
10
10-3
10
10a1
10
10
10-7
30"1
10
Navigation
20
10
estimate
1011
10-2
10
10:2
10:1
10:2
10
10"
100
10-1
10-1
Pendulum
estimate
100
10'7
10a9
109
10
Cart Pole
estimate
10"
1012
10
102
10
20m
true
truc
truc
Figure S11: Maximum likelihood parameter estimates for the baseline with given control signals (fully observable)
27

---

K.4
Quantitative results for the light-dark domain
As for the other tasks, we generated 100 sets of parameters sampled from uniform distributions in the following ranges
horizontal target position pi (-1,1)
state-dependent cost parameter c: (10 1)
perceptual uncertainty a: 11)
and generated a dataset of 50 trajectories using iLQG 43 for each set of parameters. For each set, then performed inference using method and the baseline.
While the baseline accurately infers the target position, our method shows more variability in its estimates of the target position. The estimates of the cost term are comparable for both methods. However, the baseline completely fails to estimate the perceptual uncertainty, which our method manages to infer relatively accurately.
10a
10a
estimate
10-
ours
0.0
-0.5
-1.0
10a
10a
10
baseline
estimate
10a
10:1
10a2
10.2
102
10
10:
102
true
true
Figure $12: Maximum likelihood estimates for method (top) and the baseline (bottom) in the light-dark domain.
28


Reviews:
[{"review_id": "gvyUXvDylW", "reviewer": "Reviewer_TQks", "summary": "In this paper, the authors propose a method to infer an agent’s internal model in a Partially Observable Markov Decision Process (POMDP) when the agent’s actions are non observable. Using local linearization, the authors show how a closed form approximation of the likelihood function for state trajectories can be constructed to subsequently yield maximum likelihood estimates. They also show that when there are confounding factors that can lead to the same behavior, the proposed method is able to disambiguate the factors better compared to the baseline.\n\n**After author's rebuttal:** I appreciate the authors offering clarifications. My main concern was around the significance of the contribution, which I based on the references discussed in the paper since I haven't been working on this specific area myself. However, looking at the other reviews and the author's comments, it appears the paper does more than just relaxing a simple assumption or two. I also mentioned that if the inference mechanism is aware of the generative model, one would expect better estimation in general. I make this comment from a Bayesian perspective so if the observations are from a Gaussian distribution and we model it as such, we expect to learn the sensible parameters with enough data as opposed to an unaware model that assumes, say, a Laplace distribution. I would need to think more as to why this might not hold here, as I am still inclined to believe it does. In any case, based on the discussion of the contributions, I am changing my rating from 4 to 5.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper does a good job at laying down the groundwork for the problem, going over previous work and pointing at the potential shortcomings of existing methods. The motivating example in Fig. 1 works well and the paper’s structure follows naturally. The experiments are well-designed and the results are clearly discussed establishing the technique’s superior performance over the baseline.", "weaknesses": "However, in my opinion, the contribution lacks significance for acceptance at the venue. The problem formulation is slightly more general than existing work but only marginally so. The linearization to get approximate likelihood is clean but not novel. The results are not surprising since a technique that is aware of the generative assumptions is expected to lead to better posterior estimates. The disambiguation behavior follows from better estimation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our exposition of the problem, the experiments, and the results. However, we would like to ask the reviewer to substantiate the claim that “the contribution lacks significance for acceptance at the venue”. Looking at the publications at Neurips from recent years, IRL methods have been published that are targeted at inferring animal behavior completely excluding perception (partial observability due to sensory uncertainty), assuming the actions to be fully observed (the observed action is assumed to be identical to the planned and intended movement), and, therefore, cannot even accommodate the notion of actions being epistemic or pragmatic. Our problem formulation for partially observable systems with unobserved action signals, which the reviewer calls “slightly more general than existing work but only marginally so”, is motivated by a wide range of experiments in neuroscience and cognitive science. To the best of our knowledge, there had previously been no inverse optimal control method for these cases.\n\nWe also do not fully comprehend that it is not “surprising” that an inverse method that follows the generative assumptions of the forward problem works well, as no previous method has been available at all. Moreover, we argue that it is precisely this conceptual clarity that makes our method attractive and widely applicable. The better estimation mentioned in the review would not be possible with previous methods, as we show in our experiments.\n\nThe disambiguation between uncertainty and costs does not follow trivially but stems from using an inference method that accurately incorporates the sensing and acting uncertainties of the agent being modeled. If there are previously published methods that have all these properties, could you please point out IRL or IOC methods involving partial observability of the state, that can distinguish between epistemic and pragmatic actions, and allow for unobserved action signals?"}, {"review_id": "unOZH6IqCz", "reviewer": "Reviewer_EbpC", "summary": "This paper presents the new formalization of inverse optimal control on partially-overevable Markov decision processes. The authors argue that most existing works on inverse control or inverse reinforcement learning focus on fully-observable Markov decision processes. Their approach extends iterative linear quadratic Gaussian (iLQG) and Maximum causal entropy (MCE) reinforcement learning, introducing local mineralization to achieve tractable likelihood. The method is evaluated through simulations.\n", "questions": "1. Considering the numerous studies on inverse reinforcement learning and inverse optimal control without the assumption of linearization of the dynamical system and Gaussian approximations, a justification is needed in terms of applicability to robot learning. A comparison with methods without such assumptions should be included. Can the method handle more dynamic tasks such as walking?\n\n2. Clarify the difficulty of the problem in a more intuitive way to explain why this problem remains unsolved. The formulation is somewhat straightforward. Therefore, potential readers may wonder why the problem is still unsolved.", "limitations": "This is just a suggestion.\n\nRegarding POMDP in robot learning, exploring the relationship with world models (e.g., [1,2]) would be valuable.\n\n[1] Hafner, Danijar, et al. \"Dream to control: Learning behaviors by latent imagination.\" arXiv preprint arXiv:1912.01603 (2019).\n[2] Tadahiro Taniguchi, Shingo Murata, Masahiro Suzuki, Dimitri Ognibene, Pablo Lanillos, Emre Ugur, Lorenzo Jamone, Tomoaki Nakamura, Alejandra Ciria, Bruno Lara & Giovanni Pezzulo (2023) \"World models and predictive coding for cognitive and developmental robotics: frontiers and challenges,\" Advanced Robotics, 37:13, 780-806", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The formulation and derivation of the method are solid and well-motivated.\n\n*Weaknesses\n\n\n\n\n*Limitations\n", "weaknesses": "- The derived method seems natural, making it challenging to identify the novelty of the proposal.\n- The evaluation lacks sufficient quantitative and qualitative analysis, as it only covers simple settings and compares with the MCE approach ~~without using the proposed method~~.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for calling our formulation and derivations solid and well-motivated. In our answers to the specific questions below, we elaborate on the perceived lack of novelty and simplicity of our evaluations.\n\nWe are a bit unclear about what is meant by the purported weakness that \"The evaluation [...] compares with the MCE approach without using the proposed method.\" In our evaluation, we do use both the MCE method and our proposed method. Could you elaborate?\n\nAnswers to questions:\n\n1. We acknowledge that there are methods for IRL and IOC in settings with more complicated dynamics models such as walking. However, these methods typically work in deterministic (!), fully-observable (!) settings and assume the agent’s action signals to be observed (!) (i.e. most problems implemented in OpenAI gym and comparable frameworks). Here, we are motivated by problems relevant to cognitive science and neuroscience, where biological systems are often modeled as having noisy sensory and motor systems, resulting in stochastic, partially-observable problems with unobserved action signals. This complicates the inverse optimal control problem significantly. We present a method that explicitly models these factors and therefore achieves good results on a range of different problems, albeit with simpler dynamics compared to methods applied on deterministic fully-observed problems, which is to be expected. If a relevant scenario in robotics is of interest, it could be kinesthetic teaching or imitation learning. Our method e.g. allows taking into account the specific human signal-dependent motor variability or internal biomechanical cost.\n\n2. Thank you for raising this point. We agree that an IOC formulation that derives the generative model for a given forward problem and inverts it using probabilistic inference might seem straightforward. We argue that this is a virtue of our probabilistic problem formulation instead of a shortcoming. We have included a significance statement at the beginning of our answer to all reviewers. We will make sure to include a precise statement about what current methods in IRL and IOC cannot do that our method achieves. \n\nThanks for the suggested references about POMDPs in robot learning, which we happily will include. We agree that extending the proposed approach towards learning of world models is a promising direction, and we will include this in the discussion. However, they concern very different problems. Even if a world model is given, when observing a behaving agent, it is not clear what the internal belief states are, and what the internal cost functions are, particularly, when not observing the full description of the internally generated actions."}, {"review_id": "5Q4siH2jzm", "reviewer": "Reviewer_ff8k", "summary": "This paper is a strong contribution on the Inverse Optimal Control problem when actions cannot be observed. It is particularly interesting their modelling of the agent and the “researcher” observer. The mathematical depth is good and sound. The results may be enough for a theoretical paper. However, there are too many unknowns to be ready for acceptance, particularly some definitions like partial observability related to noise. Another issue is the clarity regarding the baseline used, is it self-programmed?, is it taken from previous literature?. It seems that it does not work for any of the problems tested.", "questions": "-\tThe description of the variables v_t and w_t could be improved\n-\tWhy there is a final cost that is independent on the action taken? Conversely to the previous time steps.\n-\tWhat is πt(xt) ? (section 2.2)\n-\tWhy stochastic policy has a capital Pi?\n-\tNot clear why the EKF is introduced.\n-\t“In the inverse optimal control problem, the goal is to estimate parameters θ ∈ Rp of the agent’s optimal control problem given the model and trajectory data. These parameters can include properties of the agent’s cost function, the sensory and control systems of the agent, or the system’s dynamics.” This is a definition proposed by the authors. Originally, IOC is to recover the cost function from expert demonstrations. Getting system parameters is called system identification.\n-\t“cost of final velocity cv” Is this related to the cost definition with the final cost at end time?\n-\t“observation noise” of what quantity? Experimenter, agent?\n-\t“Note that past methods based on MCE are usually limited to estimating cost functions, so that parameters such as the agent’s noise therefore cannot be inferred.” This sentence helps a lot to understand the authors approach.\n-\tIt is not clear why the dynamics of the agent state and its beliefs only depends on the state and the belief: “closed-form expression for p(xt+1, bt+1 | xt, bt).”\n-\t“we applied a baseline method based on the maximum causal entropy (MCE) approach [3].” Which method? Self-programmed or taken from the literature? Baseline text could be improved. In the end I am not sure how many baselines are being used.\n-\t“expressed as a non-quadratic cost function of the joint angles” Is this really true. Besides, while I understand that there is not enough space for everything on the paper, the description of the input output system would be nice to place it in the main paper so there is no need to go to the appendix to read that the arm is controlled with torque.\n-\tIt is not clear why the baseline cannot even recover the action cost properly. Would it work in the absence of noise?\n- The agent navigation dynamics can be simplified to use the turn rate as the control action:\n[dot{x}, dot{y}, dot{theta}]¨=  [cos(theta)w, sin(theta)w, u] \n\n", "limitations": "Limitations are sufficiently addressed. My only constructive suggestion is to mention that noise covariance estimation for non white-noise sources is a very important problem in control. So I would include it in the discussion.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "-\tThe mathematical depth is good.\n-\tInverse optimal control plus noise estimation is relevant for many applications; especially for motor control research.\n-\tResults may be enough for a theoretical paper.\n", "weaknesses": "-\tNarrative: While the contribution looks very promising, the explanation of the contribution in the introduction is not enough clear to understand what is the main focus of the paper. Furthermore, there is a need to jump forward and backward in the text to fully understand the details of the approach. Particularly, it is sometimes complicated to see if the authors are talking about the agent or the observer (“researcher”). Another example: I had to read the whole article to understand the title.\n-\tThe partial-observability may be controversial as it is defined. The idea of tracking the belief of the agent is interesting, but why not using directly the observed state? So then is it the noise that the algorithm is tackling as partial observability. Thus, then the complexity depends on the type of noise.\n-\tFollowing the previous comment, while the method may be used for any parameter estimation (as authors state) why only estimating the observation/motor noise? Btw, observation noise of the agent or the observer? From the text I assumed that the authors were doing system identification at the same time. But they are estimating the noise.\n-\tBaseline explanation needs further description for the sake of clarity.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the strong theoretical contribution of our work. The many detailed questions, which we answer below, will help us improve the clarity of our paper.\n\n[Noise variables] The noise variables $v_t$ and $w_t$ are both standard Gaussian random vectors. However, this does not mean that only additive Gaussian noise is possible. These variables can be subject to potentially non-linear transformations in the dynamics and observation functions, resulting, e.g., in signal-dependent noise models. Therefore, this accommodates both Weber-Fechner-type sensory uncertainty in humans and animals as well as signal-dependent noise of motor actions in humans and animals.\n\n[Final cost] This is a standard formulation in finite-horizon control problems (see e.g. Wikipedia on LQG). For time 1 up to T-1, the agent takes an action and receives a reward based on the current state and the action. At time T, the agent reaches the final state and can no longer perform actions. Therefore, the final cost depends only on the final state.\n\n[$\\pi_t(x_t)$] This is the agent’s policy function, which maps the current (estimate of the) state to the action. Note that later in the paper, we use a probabilistic policy, so that $\\pi_t$ also depends on a noise variable.\n\n[Capital pi] For the stochastic policy, we denote with $\\pi_t$ the policy function, depending besides the belief also on the noises. We use $\\Pi_t$ to denote the resulting policy distribution, just depending on the belief.\n\n[EKF] The agent has sensory uncertainty about the true state of the world (partially observable MDP), different from the RL setting. We use the EKF for modeling the belief dynamics of the agent, so it is used in our algorithm. Please refer to Section 3.2 to see how the concrete choices for belief dynamics (EKF) and policy (iLQG) are used in our IOC algorithm.\n\n[IOC vs. SI] You are right that commonly IOC is used for inferring parameters of the cost function given behavioral trajectory data. System identification, on the other hand, usually denotes the whole process of learning the dynamics of the system and includes determining controls for the data generation as well. Note, however, that in motor control, neuroscience, and cognitive science, the system is usually well characterized, e.g. muscle dynamics or the kinematic chain of a lib. As we consider the task of inferring the parameters of the system given a fixed set of trajectories, from our point of view, this corresponds best to the setting of IOC, which is why we labeled it this way.\n\n[Final velocity]: In our considered tasks, the system should be controlled to have close to zero velocity at the final time step. The parameter $c_v$ determines how high the cost for the velocity penalty at the final time step is. More details and precise formulas of the cost function parameterizations are provided in appendix J.\n\n[Obs. noise] We assume that the researcher can observe the state and the agent has noisy/partial state observations (Fig. 1 / section 3). Thus, the observation noise is regarding the agent.\n\n[State and belief dynamics] Generally, the state $x_{t+1} = f(x_t, u_t, v_t)$ in a POMDP can depend on all previous observations of the agent $y_{1:t-1}$, because the agent can choose an action based on all prior observations. One can simplify the problem by assuming that the agent’s action $u_t = \\pi(b_t, \\xi_t)$ is a function of the agent’s current belief (and perhaps some noise). The belief, in turn, is a function of the previous belief and observation $b_t = \\beta(b_{t-1}, y_{t-1})$. This allows us to write a system of states and beliefs, which only depends on the previous state and belief. We adapted this idea from van den Berg et al. (2011).\n\n[Which baseline] As previously stated, more details of the baseline can be found in appendix B. We are not aware of any previous approach that shares the same setting, i.e. partial observability of states and actions, and therefore, past implementations cannot be directly applied to the setting we consider (e.g. finite time horizon). Accordingly, we implemented the baseline ourselves. As there exist various approaches on how to choose a feasible approximation, we show that our implemented baseline works for the regarded tasks in the usual setting of IOC (fully-observability and given control signals) in appendix K3.\n\n[Cost] Good point, we should have mentioned that the reaching task is torque-controlled, which considerably complicates both the control and the IOC. Because the representation of the state consists of elbow and shoulder angles and the cost function depends quadratically on the hand position, which is a non-linear function of the angles, the cost is a non-quadratic function of the state. We will use the additional page of the camera-ready version to clarify this by expanding the description of the reaching task in the main text and adding some of the equations from the Appendix.\n\n[Baseline cost] The main reason why the baseline cannot recover the cost is that it assumes the agent’s action signals are known, which is essentially the assumption of all previous methods that we are aware of. When we provide the baseline with the action signals, it can recover cost parameters, but still struggles with noise parameters (see Appendix K.3). In the absence of noise, the baseline would work well, as the true actions could be exactly estimated based on the states.\n\n[Agent navigation dynamics] We politely disagree with the reviewer that the agent navigation dynamics can be simplified to the proposed form. The proposed model is actually very similar to the dynamics model we are using, but we additionally allow the agent to change the forward velocity $w$ by adding an acceleration to it, which is why we include it in the state. Please also keep in mind that we are using discrete time (difference) equations instead of continuous time (differential) equations, which also account for a difference in notation."}, {"review_id": "y2i4DfPQDM", "reviewer": "Reviewer_1CfE", "summary": "This paper introduces a probabilistic approach to inverse optimal control for partially-observable stochastic non-linear systems with unobserved action signals. It derives an approximate likelihood function for the model parameters by linearizing the system around the observed trajectories and tracking the agent's belief distribution. This method is demonstrated to accurately infer the parameters better than the baseline maximum causal entropy (MCE) approach on two classic control tasks and two human behavioral tasks. Additionally, it shows the ability to disentangle the influences of perceptual uncertainty and behavioral costs on information-seeking behavior sources of information-seeking behavior.", "questions": "Why is it necessary to assess the proposed method's performance on pendulum and cart pole tasks, despite their deterministic nature and lack of partial observability?", "limitations": "The authors have adequately acknowledged and addressed the limitations of their methods, which include the following: (1) The focus on tasks that can be well-solved by control methods based on linearization and Gaussian approximation (iLQG and EKF), which may not fully capture the complexity of naturalistic behavior, and (2) the concern that the method might not scale effectively to high-dimensional parameter spaces as optimization in a high-dimensional non-linear space can potentially lead to getting stuck in local minima.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The introduction gives a clear background, which is helpful for someone like me who is less familiar with this field. It provides a clear understanding of the research question and the existing approaches.\n- This paper proposes a new probabilistic approach for inverse optimal control in stochastic non-linear systems with missing control signals and partial observability, which outperforms the baseline MCE model and also provides an interpretable representation of the parameter space.\n- The proposed method is evaluated on different tasks, including two human behavioral tasks, which might inspire further investigation on how the method can be applied to neuroscience and cognitive science studies.", "weaknesses": "A more comprehensive investigation into the method's robustness under various conditions would strengthen the paper's contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive assessment of our work.\n\nMore comprehensive investigation into the method’s robustness: We agree that, as always, there is also in this manuscript room for even more evaluations. Figure 3 gives evaluations involving 100 datasets each for the considered problems. Additionally, appendix I, J, and K provide more evaluations. \n\nEvaluation on pendulum and cart pole: While the pendulum and cart pole tasks are deterministic and fully observable in their standard versions, we have implemented stochastic and partially observable versions of these tasks. While our main focus of our method is the application in cognitive science and motor science, we used these tasks to show that our method is applicable to this domain but can find applications also in other fields, such as robotics, where partial observability and stochasticity might also play a role.\n\nWe also agree that in principle much more naturalistic tasks are conceivable and indeed, over the last 15 years we have continuously contributed to establishing more naturalistic tasks in the field. However, over the last few years, probabilistic methods have found wide acclaim, including at NeurIPS, that do dynamic state inference for tasks that are more naturalistic but not fully unconstrained, such as navigation. Here, we provide not only inference of the belief state but additionally inference of the costs, and the uncertainties, which includes disambiguating pragmatic from epistemic actions. Thus, we think that this is a significant contribution to the field, it extends current analyses, and should be widely applicable in the field. \n\nIf these are the only two concerns with our paper, we would like to ask the reviewer politely whether increasing the score is appropriate."}, {"review_id": "iHG5Wv3WEe", "reviewer": "Reviewer_CoMo", "summary": "This paper targets for solving the inverse optimal control problem for partially-observable stochastic non-linear dynamics with no observation of the action. To estimate the parameters of the cost function in a stochastic non-linear system, the author first derives a likelihood function for the model parameters. They then approximate this likelihood by locally linearizing the system using a combination of iterative linear quadratic Gaussian (iLQG) and Extended Kalman filter (EKF) techniques. The proposed inverse method makes explicit assumptions about the dynamics of the control tasks, the agent's belief in a stochastic environment, and the structure of the cost function. The authors evaluate their approach on four simulation environments where structured noise is added to the dynamics of the environment to introduce stochasticity into the system dynamics. In comparison to the baseline, the proposed approach demonstrates better uncertainty estimation.", "questions": "In the reaching task and the navigation task, why is the goal position not inferred? \n\nIn the conclusion, the authors mentioned that their method may not be effective for high-dimensional parameter spaces. It would be beneficial for the community to know the accuracy of the uncertainty estimation as the number and dimension of parameters scale.\n\nIn the sensorimotor domain, do you always have a well-structured cost function that you can obtain beforehand? I.e. I think your approach would not work if the formulation of the behavior cost function like eq.8 is not given. \n", "limitations": "Limitations are well documented in the paper. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In general, I find this paper easy to follow and variables in the equations are well defined. The derivations are nicely structured. The details are well documented in the Appendix and the assumptions and limitations are explicitly considered. \n", "weaknesses": "I suggest the author clearly define their contribution in this paper, as the likelihood formulation of the inverse optimal control problem has been derived in another work, \\textit{Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System}. To enhance clarity, I recommend placing the likelihood formulation in the background section.\n\nClaiming that their approach can disentangle perceptual uncertainty and behavior costs might be too bold for two reasons: firstly, the author demonstrates this claim in a single simulation environment only, and secondly, the structure of perception uncertainty and behavior cost function is known. However, in reality, obtaining the structure of these elements is difficult.\n\nMy major concern pertains to the number of assumptions made in different aspects of the system to obtain the proposed closed-form solution. For instance, all noises added in the experiments are assumed to be Gaussian distributed. These assumptions render the proposed approach challenging to apply in real-world scenarios, and the experiments were solely conducted in simulations. In addition, potential solutions to these assumptions i.e. addressing multi-modals belief utilizing SMC, are only briefly mentioned in the conclusion section. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the generally favorable evaluation of our paper.\n\nWeaknesses:\n\nContribution: Yes, the likelihood formulation has previously been derived for linear systems in that paper. Here, we write it in a more general form, which is applicable beyond linear systems, which is why we kept it in Section 3. Importantly, we derive new algorithms to actually carry out the inference for the case of nonlinear dynamics, nonlinear observations, including much more general noise distributions. This was previously unavailable.\n\nDisentangling uncertainty and costs: One cannot build scientific models without making assumptions. For example, without any knowledge about parametrizations of perceptual uncertainty and parametrizations of the cost function, it is not possible to unambiguously identify these. We completely agree that obtaining the structure of perceptual uncertainty and behavioral cost from behavior is a very difficult problem. In fact, our current work is to our knowledge the only one for the case of nonlinear dynamics, nonlinear observations, interesting noises and unobserved actions. Therefore, we argue that explicitly stating the assumptions, under which conclusions about these factors are drawn, is a good thing. Here, we provide a framework for spelling out such assumptions in the form of parametric models and inferring their parameters. In many tasks in cognitive science and neuroscience, one has knowledge about some part of the model, e.g., the system dynamics or belief dynamics up to certain parameters but other quantities, e.g., costs and perceptual uncertainties are latent.\n\nAssumptions about the system: It is true that the noises v_t and w_t that go into the dynamics and observation function are Gaussian. However, they are transformed within these functions, resulting in non-linear signal-dependent noise models. This assumption is motivated by what is known about the sensorimotor system (e.g. Harris & Wolpert, 1995; Todorov and Jordan, 2002). Thus, very complex noise distributions can be handled by the current model.\n\nQuestions:\n\nWhy is the goal position not inferred? In reaching and navigation tasks, the goal position is usually known to both the agent and the experimenter, which is why there is no need to infer it. The (internal) cost and perceptual uncertainty, on the other hand, can usually not be set or determined by the experimenter. Further, we aimed to keep the parameters consistent across tasks, and therefore used a similar parameterization for all of them. In principle, the goal position could be inferred easily using our method as long as the problem remains uniquely identifiable.\n\nScaling with number of parameters: Thank you for your suggestion, this would be indeed interesting. However, we do not believe that there is a general scaling law, as the estimation depends on the specifics of the problem more than on the mere number of parameters.\n\nStructure of the cost function: In a motor control experiment, researchers typically have a good idea of the task being performed by the agent and potentially other factors influencing performance (cognitive, biomechanical effort costs, etc.), which can be formalized as parameters of a cost function. For example, the costs of movements may depend on path length, accelerations, or torques of movements.  If such a parametric cost function is not available, one would have to resort to more general function approximation methods, which can be more challenging to render interpretable."}, {"review_id": "ZPpnE8MEUG", "reviewer": "Reviewer_xp9a", "summary": "The authors propose an inverse optimal control method to handle the challenging case of inferring an internal model in a non-linear partially observable system, when the action sequence is not observed. Quantitative evaluation of the proposed method was shown for some classic control problems. The authors also demonstrated through an example the potential of their method in disentangling perceptual factors and behavioral costs.", "questions": "The authors describe their method as uniquely accounting for unobserved actions. But other Inverse Optimal Control methods allow for stochastic policies [e.g. Wu, Kwon et al 2020], in which actions can equivalently be interpreted as a latent intended action distorted by noise, or as a draw from a known policy. Aren’t these equivalent if the policy is optimized with action noise?\n\nThe authors should show a sanity check that their linearization successfully allows us to estimate actions near enough the ground truth.\n\nIt would be helpful to discuss more of the limitations from combining iLQG with EKF. Do the authors allow for controls that depend on a dynamic covariance, or just estimates of the mean? This was a bit unclear from their writing. It’s likely, given the techniques, that their method does not account for controllable uncertainty. Is this correct?\n\nI would like to see more evidence for the main claim, as suggested in the title, that the method disentangles perceptual uncertainty and behavioral costs. To show that taking into account the agent’s beliefs helps, it would be helpful to compute how uncertain the internal belief is while the agent moves towards the light before turning towards target, and plotting how it changes with subjective reward parameter c.", "limitations": "The authors have a thoughtful discussion of limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The written language is easy to understand, and well organized. This work is a novel combination of ideas from some well-established techniques. The method should be of particular interest since intended actions of other agents are often only partially observable.  \n\nThe value of the method is highlighted by a comparison with an alternative method that mistakenly attributes perceptual uncertainty to subjective preferences. The authors’ IOC is able to correctly infer that much of the action in this task is driven by exploration rather than exploitation.\n", "weaknesses": "It would be helpful to highlight more strongly that they can distinguish preferences from uncertainty. The authors rightly mention this virtue in the title, so I think it deserves more attention in the text. On the other hand, for that section, I would say that stronger evidence is needed, for example on a more complex task than the light and dark one. Uncertainty and cost are sometimes nonidentifiable: It could be too costly to correct a certain mistake, or it could be too uncertain to justify an optimal action, and these effects can be indistinguishable from the outside. However, when the uncertainty is dynamic while the cost function is static, these factors are dissociable. This should be addressed by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review, and specifically for highlighting the importance of disentangling costs from uncertainty. We will use the additional page of the camera-ready version to expand on this in the introduction to make it a more prominent feature of our work. The issue of potential non-identifiability of uncertainty and costs in certain cases is also a good point, that of course applies to each and every algorithm in IRL and IOC. We agree that in general, there can be cases like the one described in the review, where these factors cannot be distinguished. We will address this in the discussion. Our goal was to show that across several example scenarios, including those common in the motor control and cognitive science literature, where the costs and uncertainties can be disentangled, our method is able to do this while the baseline is not. Thus, our approach is a step towards starting to explore these issues of model identifiability.\n\nQuestions:\n\nUnobserved actions (in comparison to Wu, Kwon et al., 2020): In their PNAS paper, Wu et al. (2020) assume both the agent’s observations and the agent’s action signals to be fully known (see their figure 1B, where both the observation and the action are indicated as observed variables from the researcher’s point of view). In their NeurIPS paper, Kwon et al. (2020) assume the action signals to be known, but marginalize over the agent’s internal observations / belief (see their Figure 1 and algorithm 2). So, both of their formulations assume observed action signals, in contrast to our method, and additionally assume a stationary policy. But yes, they also use a stochastic policy in addition to the partial observability formulation - even if it does not correspond to the MCE controller widely used.\n\nSanity check of linearization: Thank you for your suggestion to show “a sanity check that their linearization successfully allows us to estimate actions near enough the ground truth”. Could you elaborate a bit further on what you precisely mean by this? The parameters that are estimated are close to ground truth, see figures 2 & 3 and appendix I, J, and K. The question asked about actions, though. We estimate actions only for determining the linearization points (and for the baseline) by using the non-linear system dynamics (appendix G). We do not claim that these estimates are very accurate as they ignore noise (and the baseline would perform better then) but are sufficiently close to yielding appropriate linearization points.\n\nLimitations of combining iLQG with EKF: It is true that the combination of iLQG and EKF results in controls based only on the mean of the belief. However, the partially-observable version of iLQG (Li & Todorov, 2007), which we use here, shows uncertainty-aware behavior (as we show in the light-dark domain), because the state-dependent covariance is taken into account during the computation of the control law. In general, an extension to control methods that are based on the covariance of the belief (e.g. belief-space iLQG, van den Berg et al., 2012) should be possible in our framework, by defining the belief dynamics based on a vector including the covariance. This extension is a fruitful idea for future work and will be added to the discussion. \n\n[More evidence that the method disentangles uncertainty and costs] Thanks for raising this point. We should have been clearer about the experiment in the light-dark domain. To illustrate sources of information-seeking behavior, we have created an additional figure that shows the agent’s belief for varying cost parameters (Figure 1 in the additional pdf). An agent who takes uncertainty of the belief into account during computation of the control law (e.g. partially observable iLQG; Li & Todorov, 2007) will move towards the light source before approaching the target. Adding another cost term (c) that expresses a preference to be close to the light source does not change much about this behavior (top row). An agent who computes the control law irrespective of the belief uncertainty (e.g. fully observable iLQG; Todorov & Li, 2005) will not move towards the light source per default. As a result, the belief uncertainty is higher. Only when we add an extra term in the cost function (c) to force the agent to go to the light source does the agent move to the right before approaching the target. These two sources of information-seeking behavior can lead to similar trajectories. To show that we can disentangle these factors, we simulated 100 random parameter sets (varying the target position $p$, perceptual uncertainty $\\sigma$ and cost $c$). In Appendix K.4, we show that our method can infer all of these parameters, while the baseline cannot infer the perceptual uncertainty. We will add the aggregated results of the light-dark problem to Fig. 3 of the final version of the paper."}, {"review_id": "OwA5BMDOHx", "reviewer": "Reviewer_McGM", "summary": "The paper introduces a new approach to inverse optimal control which is able to deal with partially observable systems in which action signals are not known. Most existing approaches only work in fully observable systems where actions are known. The paper introduces a probabilistic formulation for inverse optimal control and uses maximum likelihood to estimate the costs and parameters of the system. To make the likelihood tractable, they use local linearization similar to iLQG. The paper tests the approach on 4 tasks and shows improved performance over a maximum causal entropy-based baseline. \n", "questions": "1) My first question is about baselines. The authors use a maximum causal entropy-based baseline for the experiments. From my understanding, maximum causal entropy is mainly used for exploration in RL but it is not clear to me what is the IOC-based method used in MCE to estimate the cost function. Does it use something like ILQG or EKF?\n2) What functions are used to model f and \\beta in equation 2? From my understanding, the functional form should be known beforehand which would be quite limiting for a setting where this information is not known.\n", "limitations": "The authors have discussed all relevant limitations. ", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I am not familiar with this area of research and most of the concepts in the paper are new to me but I was still able to follow most of concepts introduced in the paper hence I think the writing is mostly coherent and fairly easy to follow. \n", "weaknesses": "I don't have any weaknesses to point, I mainly have questions which I will write below.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for rating our paper as easy to follow, even for a reader outside this specific area of research! The raised questions, which we answer below, will help us improve the accessibility of our paper even further.\nAs the reviewer stated, that he has limited familiarity with the field, we would like to refer to the significance statement that we included at the beginning of this rebuttal.\n\nQuestions:\n\n[Q1: Baselines] Yes, that is correct, MCE controllers add variability to the actions and are therefore often used for exploration. For the inverse problem, one usually assumes that the expert is not perfectly accurate and therefore an MCE controller is often used to model the variability of the agent. The MCE formulation for the inverse problem is also of a suitable mathematical form, making it efficiently applicable. As the IOC problem based on MCE cannot be solved exactly, there have been various approximations introduced, partly based on linearization [e.g., Levine and Koltun, 2012], making the application with iLQG straightforward. Specifically, in contrast to our method, there is no explicit model of partial observability, and action signals are assumed to be known. There has been very little work on IOC for partially-observable systems (see related work) and to the best of our knowledge, the EKF has not been used with the MCE formulation in IOC. Thanks for raising this point about the baseline, which we will clarify in the revised version.\n\n[Q2: Functional form of (belief) dynamics] It is true that the functional form of the system / belief dynamics is assumed to be known beforehand. This is typical in the application domains we have in mind, i.e. cognitive science and neuroscience, where researchers have quite accurate models of the sensorimotor system and are interested in inferring parameters of this system (e.g. costs, uncertainties, etc.). While more general function approximators, e.g. neural networks, which are also parametric models (albeit with more parameters) could be used for the dynamics, we are interested in examples with an interpretable (and therefore often low-dimensional) parameter space.\n"}, {"review_id": "4HBJFud6wn", "reviewer": "Reviewer_V7nV", "summary": "This paper proposes an algorithm for inverse optimal control for agents operating in a partially observable Markov decision process, using only state trajectories. Given state trajectory data, dynamics model and observation model, the algorithm estimates the parameters by three steps: first, a policy is estimated using iLQG; second, the belief dynamics is estimated with EKF; finally, filtering is done by linearizing belief propagation. The likelihood is then maximized by back-propagation. The authors tested the proposed algorithm on several synthetic problems and compare it with a baseline based on maximum causal entropy. The results show that the proposed approach can better estimate the unknown parameters from behaviors than the baseline.", "questions": "1. What is the assumption of known information made in the paper? It is not clearly specified in the current writing. From Algorithm 1, it seems that assuming only the dynamics and observation models, as well as state data trajectories. But in iLQG the cost is also needed, and in the EKF step the belief dynamics (i.e., beta_t) is needed. What are they in the experiments? The algorithm also uses the prior of belief. Is that assumed to be given too?\n\n2. How does the learning actually work? The paper mentions the usage of automatic differentiation. But how is that actually done and what is the computation graph involved?  Algorithm 1 requires multiple linearization steps in iLQG, EKF, and the filtering. In particular, iLQG is an iterative process, do you also backpropagate through them?\n\n3. While the paper supposes only state trajectories are given, it seems to make assumption on the knowledge of the belief dynamics, or what the belief of the agent is. It is not clear whether this is a stronger or a weaker assumption.\n\n4. The current experiments only consider rather small and synthetic problems. Can you include results on some more realistic datasets? In addition, what are the agents that generate the data in the experiment (I can only tell iLQG is used for the light-dark domain)? It would be also good to test the proposed algorithm the estimation of non-iLQG agents, since the proposed one is based on iLQG, so that we can know the generality of the proposal.", "limitations": "Yes, the authors made clear what most limitations are. It would be also good to point out the limitation due to limited empirical evaluations done in the paper (e.g., due to the synthetic, low-dim nature, the agent policy's form, the parametric structure, etc.)", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem studied here is relevant to community. I think especially that the authors consider the case without action information, which makes the setup more practical. The writing of the paper is easy to follow. The introduction is well motivated and the related work discussion is informative. The proposed idea is clean and is new to my knowledge. The authors discuss in details about the limitation of this work.", "weaknesses": "While the writing is easy to follow, it is unclear what the assumption is being made to use the proposed method. From Algorithm 1, the authors assume knowledge of the dynamics, observation models, and state trajectory, but the steps of Algorithm 1 requires more information than that, e.g., the cost is needed in iLQG. In addition, it is also unclear how the proposed algorithm actually operates; the authors describes how filtering and approximate likelihood (given all the parameters) is computed in details, but do not mention how the unknown parameters are actually estimated (the authors mention \"using gradient-based optimization\" with automatic differentiation, but it is unclear what the computational graph entails). As a result, while I get the high level idea, I do not think I fully understand the working details of the proposed scheme, and therefore it is hard to give it an accurate evaluation. I think the authors should compare more with recent IOC or IRL works, in the experiments, and consider more realistic datasets (rather than synthetic ones).  In addition, it would be more informative if the authors can better highlight their contribution in terms of the specific problem difficulties here (e.g., due to missing action).  Lastly, the proposed method is limited to low dim problems as the authors also point out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the generally positive assessment of our work. Below, we expand on some of the assumptions of our method. The questions raised about these assumptions will help us improve the clarity of our paper.\n\nRegarding the weakness point that a comparison with “more with recent IOC or IRL works” might be helpful, we want to highlight that the baseline we consider represents current state-of-the-art methods. Newer variants use mainly more expressive (deep) function approximations, which can learn more complicated cost functions in higher-dimensional spaces and with potentially unknown dynamics. For the setting we consider in our work, these methods, however, do not yield any advantage over the considered baseline let alone that they do not provide easily interpretable parameters that can easily be connected back to psychologically meaningful quantities.\n\nQuestions:\n\n[Q1+Q2 What are the assumptions and how does learning work?] For our method, we assume that we are given a parametric form of the system dynamics, belief dynamics, cost function and initial belief of the agent which all might depend on unknown parameters that we would like to infer. Further we assume a set of state trajectories. You are right that for evaluating the likelihood (and in iLQG), one needs the values of the parameters, e.g., for the cost function. In our IOC method, the goal is to maximize the likelihood w.r.t. these parameters. To do so, we use a gradient-based optimization approach, i.e., we start with some initial parameters and differentiate using automatic differentiation through the likelihood computation (Algorithm 1). \nIt is true that the iterative nature of iLQG significantly complicates the computation graph, because, naively, one would have to backpropagate through the iterative procedure, as correctly stated by the reviewer. We get around this by only linearizing once around the actually observed state trajectory (see Section 3.3), making the computation of the likelihood function and its gradients much more efficient.\nWe then use the computed gradient to adjust our previous estimate in the direction of the optimal one.\n\n[Q3 Knowledge of agent’s belief] We would like to highlight again, that we do not assume knowledge of the agent’s belief. As shown in Fig. 1, the agent’s belief is an unobserved variable from the researcher’s perspective. What we assume is that there is a parametric model that describes how the agent’s beliefs are generated from the previous belief and observation (i.e. the belief dynamics). The beliefs themselves remain latent and need to be marginalized over using our probabilistic belief tracking formulation. Therefore, this is a weaker assumption than requiring the beliefs (which are internal to the agent in general) to be known.\n\n[Q4 Small and synthetic experiments] We agree with the reviewer that our method does not focus on high-dimensional problems, but we would like to emphasize that the tasks we consider are highly relevant in cognitive science and motor control, where relatively low-dimensional models are adequate for moving from controlled experiments towards naturalistic behavior. For example, the non-linear reaching model we consider (or similar models) has been widely applied in the neuroscience literature for modeling reaching data [Liu & Todorov, 2007; Knill et al., 2011; Wochner et al., 2020] and constitutes therefore a “realistic” task. Similar optimal control models have been employed in human locomotion [Papadopoulos et al., 2016] navigation [Kessler et al., 2022] or ball catching [Belousov et al., 2016].\n\n[Q4 Non-iLQG agents] The probabilistic formulation of the IOC problem (Section 3.1) is sufficiently general that it should be applicable to a broad class of agents beyond iLQG. In this paper, we evaluate it for iLQG agents, as (1) it allows efficient computation via local linearization (Section 3.2) and (2) many problems relevant in cognitive science and sensorimotor neuroscience can be solved via iLQG. For efficient and successful application with non-iLQG agents we would assume that further adaptation is necessary work and we therefore leave it as future work. However, we are convinced that laying out this conceptual framework is already a substantial contribution to the cognitive science and sensorimotor neuroscience community.\n\nWe will make sure to include the mentioned limitation into our discussion in the revised version of the manuscript.\n"}]

Author rebuttal:
We would like to thank all reviewers for their generally positive reviews. In light of some of the questions and comments, we would like to clarify the current state of inverse reinforcement learning (IRL), inverse optimal control (IOC) when applied to human or animal behavior. Looking at the publications at Neurips from recent years, IRL methods have been published that are targeted at inferring animal behavior, which, however completely exclude perception (partial observability due to sensory uncertainty from the perspective of the agent), assume the actions to be fully observed (the action observed by the researcher may not be identical to the planned and intended movement and the control signal, e.g. acceleration, is not directly measure, e.g. position instead). Thus, this IRL work omits explicitly modeling perception, i.e. sensory uncertainty. Accordingly, this does not allow to estimate an observation function or perceptual noise and there is accordingly not even a notion of epistemic versus pragmatic actions.

In recent IOC work, methods typically work in deterministic, fully-observable settings and assume the agent’s action signals to be observed (i.e. most problems implemented in OpenAI gym and comparable frameworks). There is IOC work that explicitly assumes partial observability from the perspective of the agent, i.e. involving sensory noise. However, to the best of our knowledge, there is no current method that can accommodate the setting considered here, i.e. problems that are stochastic, partially observable (both the partial observability introduced by perception from the perspective of the agent as well as the partial observability of the true intended actions by the agent from the perspective of the researcher), non-Gaussian noise, which is modeled by passing Gaussian noise through the nonlinear functions describing dynamics and observations.  

We are motivated by problems relevant to neuroscience, motor control, cognitive science, and psychology, where biological systems are often modeled as having noisy sensory and motor systems, resulting in stochastic, partially-observable problems with unobserved action signals. For more naturalistic, sequential tasks, that are the frontier in current research, intrinsic beliefs and subjective costs are unknown. Moreover, when considering limb kinematics or muscle activations, dynamics are nonlinear, but usually well characterized, e.g. derived from first principles such as kinematics or measured empirically in separate experiments. Sensory uncertainty also involves nonlinearities, e.g. visual angles. Additionally, sensory noise and action noise are overwhelmingly non-Gaussian but well characterized, e.g. signal-dependent noise. 

This complicates the inverse optimal control problem tremendously. Here, we present a method that is applicable in this setting. To our knowledge, it is the first such method applicable to this setting and able to distinguish between epistemic and pragmatic actions. Therefore, we see broad applicability and utility in the present work from neuroscience to motor control and beyond.

Meta-review: This paper presents a new formalization of inverse optimal control in POMDPs. Using local linearization, a closed form approximation of the likelihood function for state trajectories is constructed to subsequently yield maximum likelihood estimates. This approach extends iterative linear quadratic Gaussian (iLQG) and Maximum causal entropy (MCE) reinforcement learning, introducing local mineralization to achieve tractable likelihood. The method is evaluated through simulations. The paper received overall mixed reviews. The reviewers appreciated the clarity of the paper, the novelty of the method, and the design of the experiments. One reviewer however think that the evaluation lacks sufficient quantitative and qualitative analysis. This opinion does not seem to be shared by other reviewers.
Decision: Accept (poster)