Title: Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costs
Venue: NeurIPS 2023
Accepted: yes
Keywords: inverse optimal control

probabilistic modeling

motor control

cognitive science
Abstract: Inverse optimal control can be used to characterize behavior in sequential decision-making tasks. Most existing work, however, is limited to fully observable or linear systems, or requires the action signals to be known. Here, we introduce a probabilistic approach to inverse optimal control for partially observable stochastic non-linear systems with unobserved action signals, which unifies previous approaches to inverse optimal control with maximum causal entropy formulations. Using an explicit model of the noise characteristics of the sensory and motor systems of the agent in conjunction with local linearization techniques, we derive an approximate likelihood function for the model parameters, which can be computed within a single forward pass. We present quantitative evaluations on stochastic and partially observable versions of two classic control tasks and two human behavioral tasks. Importantly, we show that our method can disentangle perceptual factors and behavioral costs despite the fact that epistemic and pragmatic actions are intertwined in sequential decision-making under uncertainty, such as in active sensing and active learning. The proposed method has broad applicability, ranging from imitation learning to sensorimotor neuroscience.

Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costs

Abstract

Inverse optimal control can be used to characterize behavior in sequential decisionmaking tasks. Most existing work, however, is limited to fully observable or linear systems, or requires the action signals to be known. Here, we introduce a probabilistic approach to inverse optimal control for partially observable stochastic non-linear systems with unobserved action signals, which unifies previous approaches to inverse optimal control with maximum causal entropy formulations. Using an explicit model of the noise characteristics of the sensory and motor systems of the agent in conjunction with local linearization techniques, we derive an approximate likelihood function for the model parameters, which can be computed within a single forward pass. We present quantitative evaluations on stochastic and partially observable versions of two classic control tasks and two human behavioral tasks. Importantly, we show that our method can disentangle perceptual factors and behavioral costs despite the fact that epistemic and pragmatic actions are intertwined in sequential decision-making under uncertainty, such as in active sensing and active learning. The proposed method has broad applicability, ranging from imitation learning to sensorimotor neuroscience.

1 Introduction

Inverse optimal control (IOC) is the problem of inferring an agent's cost function and other properties of their internal model from behavior. While IOC has been a fundamental task in artificial intelligence and machine learning, particularly reinforcement learning (RL) and robotics, it has widespread applicability in several scientific fields including behavioral economics, psychology, and neuroscience. For example, in cognitive science and sensorimotor neuroscience, optimal control models have explained key properties of behavior, such as speed-accuracy trade-offs [1] or the minimum intervention principle [2]. But, while researchers usually build an optimal control model and compare its predictions to behavior, certain parameters of the agent's internal processes are typically unknown. For example, an agent might have uncertainty about their perception or experience intrinsic costs of behavior. These parameters are different between individuals and inferring them from observed behavior can help to understand internal tradeoffs between behavioral goals, perceptual and cognitive

processes, and predict behavior under novel conditions. Applying IOC in these domains poses several challenges that make most previous methods not viable.

First, most IOC methods assume the agent's control signals to be known. This assumption, while convenient in simulations or robotics, where the control signals may be easily quantified, does not hold in many other real-world applications. In transfer learning or behavioral experiments, the control signals are internal quantities of an animal or human, e.g., neural activity or muscle activations, and are therefore not straightforwardly observable. Thus, we consider the scenario where a researcher has observations of the system's state only, i.e., measurements of behavior.

Second, most IOC methods model the variability of the agent using a stochastic policy in a maximum causal entropy formulation [MCE; 3]. Behavioral variability in biological systems, however, is known to arise from multiple distinct sources [4]. There is noise in the sensory system, which makes the state of the world partially observable, and in the motor system. In sensorimotor neuroscience, the uncertainty in the sensory and motor systems can be characterized quantitatively by formulating accurate models, which are helpful to understand behavioral variability [5].

Third, many IOC methods are based on matching feature expectations of the cost function between the model and data [3], and are thus not easily adapted to infer other model parameters. In a behavioral experiment, however, researchers are often interested in inferring the noise characteristics of the sensorimotor system or other properties of the agent's internal model besides the cost function [6].

Fourth, while the theory of linear-quadratic-Gaussian (LQG) control [7] is suited to deal with the issues above, many real-world systems are not captured by the LQG assumptions. First, the dynamics may be non-linear, e.g., in robotics and motor control when controlling joint angles in a kinematic chain. Second, the variability of the system may not be captured by normal distributions, e.g., in sensorimotor control, where the standard deviation of sensory and control signals scales with their means. While iterative methods for solving the optimal control problem such as iterative variants of LQG [8] exist, here we consider the corresponding inverse problem.

To address these issues, we adopt a probabilistic perspective. We distinguish between the control problem faced by the agent and the IOC problem faced by the researcher. From the agent's perspective, the problem consists of acting in a partially observable Markov decision process (POMDP; Fig. 1 A). We consider the setting of continuous states and controls, stochastic nonlinear dynamics, partial observations, and finite horizon. For this setting, there are efficient approximately optimal solutions to the estimation and control problem (see Section 2). The researcher, on the other hand, is interested in inferring properties of the agent's model and cost function. The IOC problem from their perspective can be formulated using a probabilistic graphical model (Fig. 1 B), in which the state of the system is observed, while variables internal to the agent are latent.

x

t

x

t

+1

x

t

+2

u

t

u

t

+1

y

t

y

t

+1

y

t

+2

c

t

c

t

+1

. . .

A

. . .

x

t

x

t

+1

x

t

+2

u

t

u

t

+1

y

t

y

t

+1

y

t

+2

b

t

b

t

+1

. . .

B

. . .

Here, we unify MCE models, which are agnostic regarding the probabilistic structure causing the observed stochasticity of the agent's policy, with IOC methods that involve an explicit observation model. First, we define the IOC problem where we allow for both: We employ an explicit observation model, but also allow the agent to have additional stochasticity through an MCE policy. Second, we provide a solution to the IOC problem in this setting by approximate filtering of the agent's state estimate via local linearization, which allows marginalizing over these latent variables and deriving an approximate likelihood function for observed trajectories given parameters (Section 3). By maximizing the approximate likelihood function, an estimate of the optimal parameters can be determined. Third, we evaluate our proposed method on two classic control tasks, pendulum and cart pole, and on two human behavioral tasks, navigation and a manual reaching (Section 4.2 - Section 4.3). Fourth, we show that our approach allows disentangling the influences of perceptual uncertainty and behavioral costs on information-seeking behavior in the light-dark domain (Section 4.4).

Related work

Inferring costs or utilities from behavior has been of interest for a long time in several scientific fields, such as behavioral economics, psychology, and neuroscience [10-12]. More specific to the problem formulation adopted here, estimating objective functions in the field of control was first investigated by Kalman [13] in the context of deterministic linear systems with quadratic costs. More recent formulations were developed first for discrete state and control spaces under the term inverse reinforcement learning [IRL; 14, 15], including formulations allowing for stochasticty in action selection [16]. In this line, the maximum entropy [ME; 17] and MCE formulation [3] gave rise to many new methods, e.g., for non-linear continuous systems via linearization [18] or importance sampling [19] for fully observable deterministic systems.

IOC methods for stochastic systems have been developed in the setting of affine control dynamics [20, 21]. Arbitrary non-linear stochastic dynamics in the infinite horizon setting have been approached using model-free deep MCE IRL [22, 23]. The latter approaches, however, do not yield interpretable representations, as the cost function is represented by a neural network. Further, past methods based on MCE are limited to estimating cost functions and cannot be used to infer other latent quantities, such as noises or subjective beliefs. The partially observable setting for IOC has previously been addressed for discrete state-action spaces [24] and continuous states with discrete actions [25]. Schmitt et al. [26] addressed systems with linear dynamics and continuous controls for a linear switching observation model. Other work has considered partial observability from the researcher's perspective, e.g., through occlusions [27, 28]. There are some IOC methods which are applicable to partially observable stochastic systems: In our previous work [29] we regarded LQG systems, while the work of Chen and Ziebart [30] can be used to estimate cost functions that depend on the state only. Non-linear dynamics in the infinite-horizon setting and the joint estimation of model parameters have been approached by Kwon et al. [31] by training a policy network as a function of the whole parameter space. This work, however, also assumes the control signals to be given and a stationary, deterministic policy.

Applications of IOC methods range from human locomotion [32] over spatial navigation [33], table tennis [34], to attention switching [35], and target tracking [36]. Other work has been aimed at inferring other properties of control tasks, e.g., the dynamics model [6], learning rules [37], or discount functions [38]. Several subfields of robotics including imitation and apprenticeship learning [39] as well as transfer learning [40] have also employed IOC.

2 Background

Before we introduce our probabilistic approach to inverse optimal control, we give an overview of the control and filtering problems faced by the agent and algorithms that can be used to solve it. For a summary of our notation in this paper, see Appendix A.

2.1 Partially observable Markov decision processes (POMDPs)

We consider a special case of POMDPs [41, 42], a discrete-time stochastic non-linear dynamical system (Fig. 1 A) with states x t ∈ R n following the dynamics equation x t +1 = f ( x t , u t , v t ) , where f is the dynamics function, u t ∈ R u are the controls and v t ∼ N (0 , I ) is v -dimensional Gaussian noise. We assume that the agent has only partial observations y t ∈ R m , following y t = h ( x t , w t ) , with h the stochastic observation function and w t ∼ N (0 , I ) w -dimensional Gaussian noise. While v t and w t are standard normal random variables, the system can incorporate general control- and statedependent noises through non-linear transformations within the dynamics function f and observation function h . The agent's goal is to minimize the expected cost over a time horizon T ∈ N , defined by

consisting of a final state cost c T ( x T ) and a cost at each time step c t ( x t , u t ) .

2.2 Iterative linear-quadratic Gaussian (iLQG)

The control problem from Section 2.1 can be solved approximately using iLQG [8, 43]. This method iteratively linearizes the dynamics and quadratizes the costs around a nominal trajectory, { ¯ x i , ¯ u i } i =1 ,...,T , with ¯ x i ∈ R n , ¯ u i ∈ R u , and computes the optimal linear control law, u t = π t ( x t ) = L t ( x t -¯ x t ) + m t + ¯ u 1: T for the approximated system. The quantities L t and m t are the control gain and offset, respectively, and determined through a backward pass for the current reference trajectory. In the following iteration, the determined optimal control law is used to generate a new reference trajectory and the process is repeated until the controller converges.

2.3 Maximum causal entropy (MCE) reinforcement learning

The goal of MCE RL is to minimize the expected cost as in Section 2.2, while maximizing the conditional entropy of the stochastic policy Π t ( u t | x t ) , i.e., to minimize E [ J ( x 1: T , u 1: T ) -∑ T -1 t =1 H (Π t ( u t | x t ))] . This formulation has been used to treat RL as probabilistic inference [44-46] and model the stochasticity of the agent in IRL [17, 3]. The objective of IRL is to maximize the likelihood of states and controls { x t , u t } t =1 ,...,N , induced by the maximum entropy policy. The resulting optimal policy is given by the distribution Π t ( u t | x t ) = exp( Q t ( x t , u t ) -V t ( x t )) , where Q t is the soft Q-function and V t the normalization [47]. For general dynamics and reward functions, it is not feasible to compute the soft Q-function exactly. Approximate solutions have been derived using linearization [18], importance sampling [19], or deep function approximation [23]. For linear dynamics and quadratic costs, the optimal policy is a Gaussian distribution Π t ( u t | x t ) = N ( u t ; L t x t , -H -1 t ) , where L t and H t result from the optimal LQG controller [48]. More detailed formulas are provided in Appendix B.

2.4 Extended Kalman filter (EKF)

Given the system defined in Section 2.1, the optimal filtering problem is to compute a belief distribution of the current state given past observations, i.e., p ( x t | y 1: t -1 ) . For linear-Gaussian systems, the solution is given in closed form and known as the Kalman filter [49]. In case of non-linear systems as in Section 2.1, a Gaussian approximation to the optimal belief can be computed using the extended Kalman filter (EKF) via b t +1 = f ( b t , u t , 0) + K t ( y t -h ( b t , 0)) , where b t ∈ R n denotes the mean of the Gaussian belief p ( x t | y 1 , . . . , y t -1 ) . The matrix K t denotes the Kalman gain for time t and is computed by applying the Kalman filter to the system locally-linearized around the nominal trajectory obtained by the approximate optimal control law of iLQG (Section 2.2).

3 Probabilistic inverse optimal control

Weconsider an agent acting in a partially observable Markov decision process (POMDP) as introduced in Section 2.1. We assume that the agent acts at time t based on their belief b t about the state of the system x t , which evolves according to b t +1 = β t ( b t , u t , y t ) . While the belief of the agent is defined commonly as a distribution over the true state, here we model b t as a finite-dimensional summary statistics of the distribution, i.e., b t ∈ R b . The function β t : R b × R u × R m → R b is called belief dynamics. We further assume that the agent follows a time-dependent policy π t : R b × R j → R u , i.e., u t = π t ( b t , ξ t ) , which can be stochastic with ξ t ∼ N (0 , I ) .

In the inverse optimal control problem, the goal is to estimate parameters θ ∈ R p of the agent's optimal control problem given the model and trajectory data. These parameters can include properties of the agent's cost function, the sensory and control systems of the agent, or the system's dynamics. More precisely, we assume that we are given a parametric form of the system dynamics, belief dynamics, cost function, and initial belief of the agent, which all might depend on the unknown parameters that we aim to infer. Further we assume a set of state trajectories. Importantly, we do not assume knowledge of the agent's belief. We follow a probabilistic approach to inverse optimal control, i.e., we consider the likelihood function

describing the probability of the observed trajectory data x 1: T := { x 1 , . . . , x T } given the parameters. For a set of trajectories, we assume them to be independent given the parameters, so that the likelihood factorizes into single trajectory likelihoods of the form in Eq. (1). In this equation, generally, each state x t +1 depends on all previous states x 1 , . . . , x t , because the agent's internal noisy observations and control signals are not accessible to the researcher (Fig. 1 B). Therefore, the Markov property does not hold from the researcher's perspective, rendering computation of the likelihood function intractable. To deal with this problem, we employ two key insights: First, the joint dynamical system of the states and the agent's belief is Markovian [50]. Second, by keeping track of the distribution over the agent's belief, i.e., by performing belief tracking [29], we can iteratively compute the individual factors of the likelihood function in Eq. (1). In our IOC method, the goal is to maximize the likelihood w.r.t. the parameters θ . To do so, we use gradient-based optimization with automatic differentiation to differentiate through the likelihood for computing the optimal parameters (Algorithm 1). An implementation of our algorithm is publicly available 2 .

We first introduce a general formulation of the IOC likelihood involving marginalization over the agent's internal beliefs in Section 3.1. Then, we show how to make the computations tractable by local linearization in Section 3.2. In Section 3.3, we provide details for suitable linearization points, which enables us to evaluate the approximate likelihood within a single forward pass.

3.1 Likelihood formulation

We start by defining a joint dynamical system of states and beliefs [50], in which each depends only on the state and belief at the previous time step and the noises. For that, we insert the policy into the dynamics and the policy and observation function into the belief dynamics, yielding the equation

For given values of x t and b t , this equation defines the distribution p ( x t +1 , b t +1 | x t , b t ) , as v t , w t , ξ t are independent of x t +1 and b t +1 . Importantly, with this formulation, the control signals are only implicitly regarded through the policy, as we assume them to be latent for the researcher. In Section 3.2 we will introduce an approximation via linearization, which leads to a closed-form expression for p ( x t +1 , b t +1 | x t , b t ) .

One can use this Markovian joint dynamical system to compute the likelihood factors for each time step [29]. To this end, we first rewrite the individual likelihood terms p ( x t +1 | x 1: t ) of Eq. (1) by marginalizing over the agent's belief at each time step, i.e.,

As the belief is an internal quantity of the agent and thus not observable to the researcher, we keep track of its distribution, p ( b t | x 1: t ) . For this, we rewrite

where we have exploited the fact that the joint dynamical system of states and beliefs is Markovian. The distribution p ( b t | x 1: t ) acts as a summary of the past states and can be computed by conditioning on the current state, i.e.,

After determining p ( b t | x 1: t ) , we can propagate it through the joint dynamical system to arrive at the distribution p ( x t +1 , b t +1 | x 1: t ) . To obtain the belief distribution of the following time step, p ( b t +1 | x 1: t +1 ) , we condition on the observed state x t +1 . To obtain the likelihood contribution, on the other hand, we marginalize out b t +1 . To summarize, starting with an initialization p ( b 0 ) , we can compute the individual terms p ( x t +1 | x 1: t ) of the likelihood by executing Algorithm 1 (Appendix C).

3.2 Tractable likelihood via local linearization

While the marginalization and propagating operations in the previous section can be done in closed form for linear-Gaussian systems, this is no longer feasible for non-linear systems. Therefore, we follow the approach of local linearization used in iLQG (Section 2.2) and the EKF (Section 2.4). For the belief statistics, we consider the mean of the agent's belief, i.e., b t = E [ x t | y 1 , . . . , y t -1 ] and initialize the distribution for the first time step as a Gaussian, p ( b 1 ) = N ( µ ( b ) 1 , Σ ( b ) 1 ) . We then approximate the distribution p ( x t +1 , b t +1 | x t , b t ) as a Gaussian by applying a first-order Taylor expansion of g .

To obtain a closed-form expression for g , which we can linearize, we model the agent's policy using iLQG (Section 2.2) and the belief dynamics using the EKF (Section 2.4). This choice leads to an affine control and belief given b t , making linearization of p ( x t +1 , b t +1 | x t , b t ) straightforward. To allow for additional stochasticity in the agent's policy, we use the MCE formulation (Section 2.3). For linearized dynamics, the MCE policy is given by a Gaussian distribution, so that π t ( b t , ξ t ) = L t ( b t -¯ x 1: T ) + m t + ¯ u 1: T -˜ H t ξ t , with ˜ H t the Cholesky decomposition of H t , and can be marginalized out in closed form.

The approximations we have introduced allow us to solve the integral in Eq. (4) in closed form by applying standard equations for linear transformations of Gaussians, resulting in

the Jacobian of g w.r.t. · , evaluated at ( x t , µ ( b ) t , 0 , 0 , 0) . Under this Gaussian approximation, both remaining operations of Algorithm 1 (Appendix C) can also be performed in closed form. A more detailed derivation and representation of these formulas can be found in Appendix D. If the agent has full observations of the system's state, the inverse optimal control problem is simplified significantly (see Appendix E). Details about the implementation are provided in Appendix F.

3.3 Data-based linearization

The forward optimal control problem is commonly solved by starting with a randomly initialized nominal trajectory and iterating between computing the locally optimal control law and linearization until convergence. To compute the likelihood in the inverse problem, we can take a more efficient approach by linearizing directly around the given trajectory x 1: T . We then need to perform only one backward pass to compute an approximately optimal control law given the current parameters, and a forward pass to compute an approximately optimal filter. This, in particular, allows efficient computation of the gradient of the likelihood function for the optimization procedure. As we assume the controls to be unobservable, but they are needed for the linearization, we compute estimates of the controls by minimizing the squared difference of the noiseless predicted states and the actual states (see Appendix G). Note that these estimated controls are only used for the linearization, but are not used as observed controls in the IOC likelihood itself. In the case where the full state is not observable, we cannot linearize around the trajectory. For these cases, we propose two approaches to compute gradients based on implicit differentiation and differentiating only through the last iteration. As this setting is not the main focus of this paper, details of these approaches are provided in Appendix H.

4 Experiments

We evaluated our method on two classic control tasks, i.e., Pendulum and Cart Pole, and two human behavioral tasks, manual reaching and navigation. To evaluate the accuracy of the parameter estimates obtained by our method and to compare it against a baseline, we computed absolute relative errors per parameter, i.e., | ( θ -ˆ θ ) /θ | . This metric makes averages across parameters on different scales more interpretable compared to other metrics such as root mean squared errors. For each task, we simulated 100 sets of parameters from a uniform distribution in logarithmic space. For each set of parameters, we simulated 50 trajectories. We then maximized the log likelihood using gradient-based optimization with automatic differentiation [L-BFGS algorithm; 51]. See Appendix I for a summary of the hyperparameters of our experiments.

b) Log likelihood of parameters

d) Maximum likelihood parameter estimates

a) Data simulated with

ground truth parameters

c) Data simulated with maximum

likelihood parameter estimates

Low cost, low noise

High cost, high noise

ours

baseline

All tasks we consider have four free parameters: control cost c a , cost of final velocity c v , motor noise σ m , and observation noise σ o of the agent. In the fully observable case, we leave out the observation noise parameter and only infer the three remaining parameters. For concrete definitions of the parameters in each specific task, see Appendix J.

4.1 Baseline method

For a comparison to previously proposed methods, we applied a baseline method based on the maximum causal entropy (MCE) approach [3]. As for this approach, control signals of the observed trajectories are required, we use the estimates of the controls that we determine in our proposed method for the data-based linearization (Section 3.3). Note that the baseline, representative for applicable past IOC methods, does not have an explicit model of partial observability. Further note that past methods based on MCE are limited to estimating cost functions, so that parameters such as the agent's noise cannot be inferred. For the specific MCE linearization-based baseline we consider, it is actually straightforward to maximize the likelihood with respect to the noise parameters, which enables us to evaluate noise estimates. To show that this approach constitutes a suitable baseline, in

Appendix K.3, we provide results for the case where the true control signals are known and there is no partial observability. More details of the baseline are provided in Appendix B.

4.2 Evaluation on manual reaching task

We evaluate the method on a reaching task with a non-linear two-joint biomechanical arm model, which has been applied to reaching movements in the sensorimotor neuroscience literature [e.g., 52, 53]. The agent's goal is to move its arm towards a target at position e ⋆ by controlling the torque to the two joints. This objective is expressed as a non-quadratic cost function of the joint angles,

since the final position and velocity of the hand e T , ˙ e T are non-linear functions of the joint angles. See Appendix J.1 for details.

We use a fully observable [8] and a partially observable version of the task [43]. Fig. 2 A shows simulations from the model with two different parameter settings. Evaluating the likelihood function for a grid of two of the parameters (Fig. 2 B) confirms that it has its maxima close to the true parameter values. Simulated data using the maximum likelihood estimates look indistinguishable from the ground truth data (Fig. 2 C).

In Fig. 2 D, we show maximum likelihood estimates and true values for repeated runs with different random parameter settings. The parameter estimates of our method closely align with the true parameter values, showing that we can successfully recover the parameters from data. The baseline method, in contrast, shows considerably worse performance, in particular for estimating noises due to the lacking explicit representation of partial observability. Importantly, even when the true control signals are provided, the noise parameter estimates of the baseline are not well estimated (Appendix K.3). Estimates for the fully observable case are provided in Appendix K.2. The median absolute relative errors of our method were 0.11, while they were 0.93 for the baseline. The influence of missing control signals and of the lack of an explicit observation model in the baseline can be observed by comparing the results to the fully observable case and the case of given control signals in Appendix K.2 and Appendix K.3.

4.3 Quantitative evaluation on other tasks

To show that our method works for a range of different tasks, we evaluated it on the three other tasks (navigation, pendulum and cart pole). In the navigation task, we consider an agent navigating to a target under non-linear dynamics while receiving noisy observations from a non-linear observation model. To reach the target, the agent can control the angular velocity of their heading direction and the acceleration with which they move forward. The agent observes noisy versions of the distance to the target and the target's bearing angle. We provide more details about the experiment in Appendix J.2. Maximum likelihood parameter estimates for the navigation task are shown for the partially observable case in Fig. S6 and for the fully observable case in Fig. S10. As for the reaching task, our method provides parameter estimates close to the true ones, while the estimates of the baseline deviate for many trials. Median absolute relative errors of our method were 0.31, while they were 1.99 for the baseline (Fig. 3).

The two classic control tasks (Pendulum and Cart Pole) are based on the implementations in the gym library [54]. Because these tasks are neither stochastic nor partially observable in their standard formulations, we introduce noise on the dynamics and turn them into partially observable problems by defining a stochastic observation function (see Appendix J.3). In Appendix K, we show the parameter estimates for the Pendulum (Fig. S4) and for the Cart Pole (Fig. S5) for the partially observable case, while Fig. S8 and Fig. S9 show the fully observable case, respectively. One can observe that the results are qualitatively similar to the ones in the reaching and navigation tasks, showing that our method provides accurate estimates of the parameters. Median absolute relative errors of our method were 0.12 and 0.41, while for the baseline they were 2.21 and 3.82 (Fig. 3).

fully obs.

partially obs.

10

3

10

2

10

1

10 0

10 1

10 2

10 3

absolute relative error

fully obs.

partially obs.

fully obs.

partially obs.

fully obs.

partially obs.

ours

baseline

A

Reaching

B

Navigation

C

Pendulum

D

Cart Pole

4.4 Information-seeking behavior in the light-dark domain

Finally, we investigate the ability of our method to disentangle sources of information-seeking behavior. In the light-dark domain [55], an agent moves in a 2D space and receives noisy measurements of its position, whose standard deviation depends on the horizontal distance from a light source:

where σ governs the amount of perceptual uncertainty. This task is a common test for informationseeking behavior, because it requires the agent to move towards the light source and at the same time away from the target to reduce the uncertainty about its position relative to the target. When this uncertainty has been reduced, the agent can approach the target (see Appendix J.4 for details). The agent's goal is to reach the target's position p at the final time step, while minimizing control effort u 2 t :

Different from the original problem formulation, we consider the case in which the agent may have an additional inherent desire to be close to the light, parameterized by c . We recover the original cost function [55] for c = 0 , but we can represent agents that seek light more than necessary to reduce uncertainty for reaching the goal state with c > 0 . Accordingly, both the reduction of perceptual uncertainty and state-dependent running cost could encourage the agent to move towards the light source before approaching the target. For an external observer, e.g., an experimenter, observing an agent moving towards the light might not reveal the different potential sources for this behavior. We now ask if it is possible to disentangle these two factors using our proposed IOC algorithm.

A

Final state cost

Perceptual

uncertainty

Running costs

Ground

truth

Our

estimates

Baseline

estimates

B

C

x [m]

x [m]

x [m]

y [m]

y [m]

y [m]

5

.0

.5

.0

.5

.0

0

5

0

5

0

5

We simulated 100 trajectories of an iLQG agent [partially observable version, 43]

with no inherent desire to be near the light source ( c = 0 ) and some perceptual uncertainty ( σ = 0 . 2 ) depending on the distance to the light source. The agent first moves towards the light source and then reaches the target (Fig. 4 A). We inferred the parameters σ , p , and c using our method and the baseline. Both methods recover the target position. Our method in addition infers values close to the true c , and σ . It therefore correctly attributes the information-seeking behavior of the agent to the perceptual uncertainty (Fig. 4 B). The baseline method, however, does not infer the correct

perceptual uncertainty. It instead attributes the agent's information-seeking behavior to an inherent desire to be in the right part of the room in the running cost (Fig. 4 C). This highlights the importance for IOC in partially observable domains to probabilistically take the agent's belief into account, if one is interested in inferring the correct cognitive mechanisms. For a quantitative evaluation of the maximum likelihood estimates in the light-dark domain, see Appendix K.4.

5 Conclusion

In this paper, we introduced a new IOC method for partially observable systems with stochastic non-linear dynamics and missing control signals. Using a probabilistic approach, we formulate the IOC problem as maximizing the likelihood of the observed states given the parameters. As the exact evaluation of the likelihood for a general non-linear model is intractable, we developed an efficient approximate likelihood by linearizing the system locally around the given trajectories, as in popular approaches such as the EKF or iLQG. By maintaining a distribution that tracks the agent's belief, an approximate likelihood can be evaluated in closed form within a single forward pass and efficiently optimized. Our proposed formulation is able to incorporate multiple sources of the stochasticity of the agent, reconciling the theory of past MCE IOC algorithms [e.g., 3] and approaches where the agent's stochasticity stems from an explicit stochastic observation and control model [29].

We evaluated our method on two stochastic variants of classic control tasks, pendulum and cart pole, and on two human behavioral tasks, a reaching and a navigation task. In comparison to an MCE baseline, we have found our method to achieve lower estimation errors across all tasks. Further, it successfully inferred noise parameters of the system, which was not possible with the baseline. In the light-dark domain, we were able to infer costs and perceptual uncertainty parameters, two different causes of apparent information-seeking behavior that could lead to qualitatively similar trajectories. This means that our method is a first step towards distinguishing pragmatic from epistemic controls. To further investigate this, it would be fruitful to examine belief-space planning methods that explicitly include the agent's belief covariance in the policy [56].

The limitations of our method are mainly due to the linearization of the dynamical system and the Gaussian approximations involved in the belief tracking formulation of the likelihood function. In more complex scenarios with non-Gaussian belief distributions, e.g., multimodal beliefs, the method will likely produce inaccurate results. This problem could be addressed by replacing the closedform Gaussian belief by particle-based methods [57]. Further, we focused on tasks which could be solved well by control methods based on linearization and Gaussian approximation (iLQG and EKF), motivated by their popularity in applications in cognitive science and neuroscience. Forward problems that cannot be solved using iLQG are probably not directly solvable using our inverse method. While, in principle, our method is also applicable to other forward control methods that compute differentiable policies, it is an empirical question whether linearizing these policies leads to accurate approximate likelihoods and parameter estimates. A further limitation of our method is that it requires parametric models of the dynamics and noise structure. While missing parameters can be determined using our method, in the case of completely unknown dynamics a model-free approach to IOC would be more suitable. Lastly, while we have shown that inference is feasible, the results probably do not scale to high-dimensional parameter spaces. One reason for this is that optimization in a high-dimensional non-linear space can potentially get stuck in local minima. This problem could be relieved by using more advanced optimization methods. A further, more fundamental, concern with higher-dimensional parameter spaces is that identifiability issues and ambiguous solutions arise. However, our probabilistic approach with a closed-form likelihood opens up the possibility of using Bayesian methods to investigate the identifiability of model parameters [58]. For future work exploring the relationship to methods for learning world models in POMDPs might also be a fruitful direction [59, 60].

Our method provides a tool for researchers, e.g., in sensorimotor domains, to model sequential behavior by inferring an agent's subjective costs and internal uncertainties. This will enable answering novel scientific questions about how these quantities are affected by different experimental conditions, how they deviate from intended task goals and provided task instructions, or how they vary between individuals. This is particularly relevant to a computational understanding of naturalistic behavior [61-63], for which subjective utilities are mostly unknown.

Acknowledgments and Disclosure of Funding

We gratefully acknowledge the computing time on the high-performance computer Lichtenberg at the NHR Centers NHR4CES at TU Darmstadt, and financial support by the project 'Whitebox' funded by the Priority Program LOEWE of the Hessian Ministry of Higher Education, Science, Research and Art. We thank Fabian Kessler for helpful discussions and comments and an anonymous reviewer of a previous version of the manuscript for surmising that the present method could not infer information-seeking behavior.

References

Christopher M Harris and Daniel M Wolpert. Signal-dependent noise determines motor planning. Nature , 394(6695):780-784, 1998.

Emanuel Todorov and Michael I Jordan. Optimal feedback control as a theory of motor coordination. Nature neuroscience , 5(11):1226-1235, 2002.

Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning , 2010.

A Aldo Faisal, Luc PJ Selen, and Daniel M Wolpert. Noise in the nervous system. Nature reviews neuroscience , 9(4):292-303, 2008.

Daniel M Wolpert and Zoubin Ghahramani. Computational principles of movement neuroscience. Nature neuroscience , 3(11):1212-1217, 2000.

Matthew Golub, Steven Chase, and Byron Yu. Learning an internal dynamics model from control demonstration. In International Conference on Machine Learning , pages 606-614, 2013.

Brian D O Anderson and John B Moore. Optimal control: linear quadratic methods . Prentice-Hall, 1990.

Emanuel Todorov and Weiwei Li. A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In American Control Conference , pages 300-306. IEEE, 2005.

Mykel J Kochenderfer, Tim A Wheeler, and Kyle H Wray. Algorithms for decision making . MIT press, 2022.

Frederick Mosteller and Philip Nogee. An experimental measurement of utility. Journal of Political Economy , 59(5):371-404, 1951.

Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econometrica , 47(2):263-292, 1979.

Konrad Paul Körding and Daniel M Wolpert. The loss function of sensorimotor learning. Proceedings of the National Academy of Sciences , 101(26):9839-9842, 2004.

R E Kalman. When Is a Linear Control System Optimal? Journal of Basic Engineering , 86(1):51-60, 1964.

Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning , volume 1, 2000.

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning , volume 5, 2004.

Constantin A Rothkopf and Christos Dimitrakakis. Preference elicitation and inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference (ECML PKDD) , pages 34-48. Springer, 2011.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence , volume 23, pages 1433-1438, 2008.

Sergey Levine and Vladlen Koltun. Continuous inverse optimal control with locally optimal examples. In International Conference on Machine Learning , pages 475-482, 2012.

Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 182-189, 2011.

Navid Aghasadeghi and Timothy Bretl. Maximum entropy inverse reinforcement learning in continuous state spaces with path integrals. In International Conference on Intelligent Robots and Systems , pages 1561-1566, 2011.

Weiwei Li, Emanuel Todorov, and Dan Liu. Inverse optimality design for biological movement systems. World Congress of the International Federation of Automatic Control (IFAC) , 44(1):9662-9667, 2011.

Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning , pages 49-58. PMLR, 2016.

Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. IQ-learn: Inverse soft-Q learning for imitation. In Advances in Neural Information Processing Systems , volume 34, pages 4028-4039, 2021.

JD Choi and Kee-Eung Kim. Inverse reinforcement learning in partially observable environments. Journal of Machine Learning Research , 12:691-730, 2011.

Júnior AR Silva, Valdir Grassi, and Denis Fernando Wolf. Continuous deep maximum entropy inverse reinforcement learning using online POMDP. In IEEE International Conference on Advanced Robotics , volume 19, pages 382-387, 2019.

Felix Schmitt, Hans-Joachim Bieg, Dietrich Manstetten, Michael Herman, and Rainer Stiefelhagen. Exact maximum entropy inverse optimal control for modeling human attention switching and control. In IEEE International Conference on Systems, Man, and Cybernetics , pages 2807-2813, 2016.

Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In European Conference on Computer Vision , pages 201-214. Springer, 2012.

Kenneth Bogert, Jonathan Feng-Shun Lin, Prashant Doshi, and Dana Kulic. Expectation-maximization for inverse reinforcement learning with hidden data. In International Conference on Autonomous Agents & Multiagent Systems , pages 1034-1042, 2016.

Matthias Schultheis, Dominik Straub, and Constantin A Rothkopf. Inverse optimal control adapted to the noise characteristics of the human sensorimotor system. In Advances in Neural Information Processing Systems , volume 34, pages 9429-9442, 2021.

Xiangli Chen and Brian Ziebart. Predictive inverse optimal control for linear-quadratic-Gaussian systems. In Artificial Intelligence and Statistics , pages 165-173, 2015.

Minhae Kwon, Saurabh Daptardar, Paul R Schrater, and Xaq Pitkow. Inverse rational control with partially observable continuous nonlinear dynamics. In Advances in Neural Information Processing Systems , volume 33, pages 7898-7909, 2020.

Katja Mombaur, Anh Truong, and Jean-Paul Laumond. From human to humanoid locomotion - an inverse optimal control approach. Autonomous Robots , 28(3):369-383, 2010.

Constantin A Rothkopf and Dana H Ballard. Modular inverse reinforcement learning for visuomotor behavior. Biological cybernetics , 107(4):477-490, 2013.

Katharina Muelling, Abdeslam Boularias, Betty Mohler, Bernhard Schölkopf, and Jan Peters. Learning strategies in table tennis using inverse reinforcement learning. Biological Cybernetics , 108(5):603-619, 2014.

Felix Schmitt, Hans-Joachim Bieg, Michael Herman, and Constantin A Rothkopf. I see what you see: Inferring sensor and policy models of human real-world motor behavior. In AAAI Conference on Artificial Intelligence , 2017.

Dominik Straub and Constantin A Rothkopf. Putting perception into action with inverse optimal control for continuous psychophysics. Elife , 11:e76635, 2022.

Zoe Ashwood, Nicholas A. Roy, Ji Hyun Bak, and Jonathan W Pillow. Inferring learning rules from animal decision-making. In Advances in Neural Information Processing Systems , volume 33, pages 3442-3453, 2020.

Matthias Schultheis, Constantin A Rothkopf, and Heinz Koeppl. Reinforcement learning with nonexponential discounting. In Advances in Neural Information Processing Systems , volume 35, pages 3649-3662. Curran Associates, Inc., 2022.

Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics , 7(1-2):1-179, 2018.

Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(7), 2009.

Karl Johan Åström. Optimal control of Markov processes with incomplete state information I. Journal of Mathematical Analysis and Applications , 10:174-205, 1965.

Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence , 101(1-2):99-134, 1998.

Weiwei Li and Emanuel Todorov. Iterative linearization methods for approximately optimal control and estimation of non-linear stochastic system. International Journal of Control , 80(9):1439-1453, 2007.

Hilbert J Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as a graphical model inference problem. Machine learning , 87(2):159-182, 2012.

Marc Toussaint. Robot trajectory optimization using approximate inference. In International Conference on Machine Learning , pages 1049-1056, 2009.

Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909 , 2018.

Adam Gleave and Sam Toyer. A primer on maximum causal entropy inverse reinforcement learning. arXiv preprint arXiv:2203.11409 , 2022.

Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning , pages 1-9, 2013.

R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering , 82(1):35-45, 1960.

Jur Van Den Berg, Pieter Abbeel, and Ken Goldberg. LQG-MP: Optimized path planning for robots with motion uncertainty and imperfect state information. The International Journal of Robotics Research , 30 (7):895-913, 2011.

Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software (TOMS) , 23(4):550-560, 1997.

Arne J Nagengast, Daniel A Braun, and Daniel M Wolpert. Optimal control predicts human performance on objects with internal degrees of freedom. PLoS Computational Biology , 5(6):e1000419, 2009.

David C Knill, Amulya Bondada, and Manu Chhabra. Flexible, task-dependent use of sensory feedback to control hand movements. Journal of Neuroscience , 31(4):1219-1237, 2011.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Robert Platt, Russell Louis Tedrake, Leslie P. Kaelbling, and Tomas Lozano-Perez. Belief space planning assuming maximum likelihood observations. In Proceedings of Robotics: Science and Systems , 2010. doi: 10.15607/RSS.2010.VI.037.

Jur Van Den Berg, Sachin Patil, and Ron Alterovitz. Motion planning under uncertainty using iterative local optimization in belief space. The International Journal of Robotics Research , 31(11):1263-1278, 2012.

Arnaud Doucet, Nando De Freitas, and Neil James Gordon. Sequential Monte Carlo methods in practice . Springer, 2001.

Luigi Acerbi, Wei Ji Ma, and Sethu Vijayakumar. A framework for testing identifiability of bayesian models of perception. In Advances in Neural Information Processing Systems , volume 27, 2014.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 , 2019.

Tadahiro Taniguchi, Shingo Murata, Masahiro Suzuki, Dimitri Ognibene, Pablo Lanillos, Emre Ugur, Lorenzo Jamone, Tomoaki Nakamura, Alejandra Ciria, Bruno Lara, et al. World models and predictive coding for cognitive and developmental robotics: frontiers and challenges. Advanced Robotics , pages 1-27, 2023.

John W Krakauer, Asif A Ghazanfar, Alex Gomez-Marin, Malcolm A MacIver, and David Poeppel. Neuroscience needs behavior: correcting a reductionist bias. Neuron , 93(3):480-490, 2017.

Paul Cisek and Alexandre Pastor-Bernier. On the challenges and mechanisms of embodied decisions. Philosophical Transactions of the Royal Society B: Biological Sciences , 369(1655):20130479, 2014.

Cory T Miller, David Gire, Kim Hoke, Alexander C Huk, Darcy Kelley, David A Leopold, Matthew C Smear, Frederic Theunissen, Michael Yartsev, and Cristopher M Niell. Natural behavior is the language of the brain. Current Biology , 32(10):R482-R493, 2022.

Christopher M Bishop. Pattern recognition and machine learning . Springer, 2006.

Roy Frostig, Matthew James Johnson, and Chris Leary. Compiling machine learning programs via high-level tracing. Systems for Machine Learning , 4(9), 2018.

Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods , 17(3):261-272, 2020.

Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López, Fabian Pedregosa, and Jean-Philippe Vert. Efficient and modular implicit differentiation. arXiv preprint arXiv:2105.15183 , 2021.

Steven George Krantz and Harold R Parks. The implicit function theorem: history, theory, and applications . Springer Science & Business Media, 2002.

Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López, Fabian Pedregosa, and Jean-Philippe Vert. Efficient and modular implicit differentiation. In Advances in Neural Information Processing Systems , volume 35, pages 5230-5242, 2022.

Weiwei Li. Optimal control for biological movement systems . PhD thesis, UC San Diego, 2006.

Appendix

A Notation

and V θ t the normalization,

B MCEIRL Baseline

Maximum causal entropy inverse reinforcement learning (MCE IRL) [47] can be formulated as maximizing the likelihood

with

Here, Q θ t is the soft Q-function at time t , given by

For arbitrary systems, computing the soft Q-function exactly is infeasible, therefore common methods apply approximations such as linearization [18], importance sampling [19], or deep function approximation [23]. For the case of linear dynamics and quadratic reward, the optimal policy is given by a Gaussian distribution Π t ( u t | x t ) = N ( u t ; L t x t , -H -1 t ) , where L t is the controller gain and H t a matrix resulting from the computation of the LQG controller [48]. As the tasks we consider can be well solved by linearizing the dynamics locally, we choose an approximation by linearization and use the optimal MCE controller for the linearized dynamics to compute the likelihood function for the parameter set θ .

To apply this baseline to the setting where control signals are missing, we use the estimates of the controls as we determine in our proposed method for the data-based linearization (Section 3.3 and Appendix G).

We can compute the approximate likelihood function by performing the following steps:

Estimate the missing control signals using Eq. (11)

Linearize the system as described in Section 3.3

Compute the MCE policy for the linearized system, i.e., compute Q θ t and V θ t

Compute the likelihood (in log space) using Eq. (10)

To maximize the (log) likelihood efficiently, one needs to compute the gradient of the likelihood function, which is straightforwardly achieved by backpropagating the gradient using automatic differentiation in step 4.

Note that there is no explicit model of partial observability and we use point estimates for the control signals, therefore the likelihood in Eq. (10) decomposes in independent factors given states and controls. In contrast, in our approach, when incorporating partial observability and missing controls, one has to compute approximate likelihood contributions within a forward pass.

C Algorithm to compute an approximate likelihood for IOC

Algorithm 1 Approximate likelihood computation

Output: Approximate likelihood of parameters p ( x 1: T | θ ) Input: Parameters θ , Data x 1: T , Model f, h

1: Determine the policy π using iLQG

2: Determine the belief dynamics β using the EKF

3: for t in { 1 , . . . , T -1 } do

4: Compute p ( x t +1 , b t +1 | x 1: t ) using Eq. (4)

5: Update p ( b t +1 | x 1: t +1 ) using Eq. (5)

6: Obtain p ( x t +1 | x 1: t ) using Eq. (3)

7: end for

D Inverse optimal control derivations

We start with defining the joint dynamics of states and estimates, which returns the next state and estimate as a function of the previous state and estimate and the noises,

To do so, we insert the policy and observation function into the system and belief dynamics, giving

The individual factors of the likelihood function can be determined as

where p ( x t +1 , b t +1 | x 1: t ) is given by marginalizing over the current belief b t as

Linearization

To derive a tractable approximation of the distribution p ( x t +1 , b t +1 | x 1: t ) , we model the initial belief of the agent's state estimate p ( b 1 ) as a Gaussian and linearize the joint dynamics function, leading to a Gaussian approximation of the desired quantity, i.e., p ( x t +1 , b t +1 | x 1: t ) ≈ N ( µ t , Σ t ) .

First, we apply a first-order Taylor expansion of the joint dynamics g around the observed state x t and the mean of the belief µ ( b ) t and the noises:

where J · denotes the Jacobian of g w.r.t. · , evaluated at ( x t , µ ( b ) t , 0 , 0 , 0) .

To derive an explicit representation of the Jacobians, we insert the filtering and control law obtained by the Kalman filter and maximum entropy iLQG controller and find

with

leading to the equations

Propagating the Gaussian belief over the agent's state estimate, p ( b t | x 1: t ) through the linearized dynamics model (Eq. (4)) can be done by applying standard identities for linear transformations of Gaussian random variables [64] and gives

with µ t = g ( x t , µ ( b ) t , 0 , 0 , 0) and Σ t = J b Σ ( b ) t J T b + J v J T v + J w J T w + J ξ J T ξ . Marginalization over b t +1 gives the desired likelihood factor, while conditioning on x t +1 gives the belief statistic for the following time step (see Section 3.1).

E Special case: full observability

If the state x t is fully observable to the agent, the problem is simplified significantly. The control problem from the agent's perspective (Fig. S1, left) can be solved by applying iLQG [8] to the observed states directly (Section 2.2)

This also simplifies the IOC problem from the researcher's perspective because evaluating the approximate likelihood becomes straightforward as we do not need to marginalize over the agent's belief.

Recall that the likelihood can be decomposed as

and the dynamics are given as

We can approximate the likelihood contribution at each time step as

where J · denotes the Jacobian of f w.r.t. · , evaluated at ( x t , π ( x t , 0) , 0) :

F Implementation

We provide a flexible framework for defining non-linear stochastic dynamical systems of the form introduced in Section 2.1 by implementing the dynamics, observation function, and cost function. The optimal estimation control methods based on iterative linearization are implemented using the automatic differentiation library jax [65], so that Jacobians and Hessians for the linearization of the dynamics and quadratization of the cost do not have to be specified manually. Furthermore, the Jacobians needed for linear-Gaussian approximation in the IOC likelihood (Section 3.2) are also computed using automatic differentiation. Gradient-based maximization of the log likelihood was performed using the L-BFGS-B scipy [66] from the jaxopt library [67]. The implementation is provided in the supplementary material and will be made public upon publication.

For computations, we used an Intel Xeon Platinum 9242 Processor, using 1 core per run. The mean run times for computing maximum likelihood estimates (partially observable setting) were as follows:

Reaching: 140 s

Navigation: 64 s

Pendulum: 31 s

CartPole: 115 s

with

x

t

x

t

+1

x

t

+2

u

t

u

t

+1

c

t

c

t

+1

. . .

. . .

x

t

x

t

+1

x

t

+2

u

t

u

t

+1

. . .

. . .

G Estimating controls for linearization

For every evaluation of the likelihood function as described in Section 3.1, we would naively need to solve the forward control problem once by running the iLQG algorithm given the current parameter values. This is computationally inefficient due to the iterative procedure of iLQG, which requires multiple forward and backward passes. Instead of performing the iterative procedure, which computes the locally optimal control law { L 1: T -1 , m 1: T -1 } around a nominal trajectory { ¯ x 1: T , ¯ u 1 T -1 } , then computes the new nominal trajectory given the current control law and so on, we realize that in the IOC setting, we already have observed the actual trajectory x 1: T performed by the agent. To make solving the forward control problem more efficient, we only compute the locally optimal control law once around this trajectory. This would require the control signals u 1: T to be given, but they are unobserved in our problem setting. To obtain a proxy for the controls for the purposes of linearization, we solve for the controls given the trajectory x 1: T in the system dynamics equation x t +1 = f ( x t , u t , 0) using the Gauss-Newton method for non-linear least squares as implemented in jaxopt , i.e.,

H Efficient gradient computation

In the case where the full state is not observable, one cannot linearize around the given trajectory as descibed in Section 3.3. The forward optimal control problem is commonly solved by starting with a randomly initialized nominal trajectory and iterating between computing the locally optimal control law and linearization until convergence. To compute gradients, a naive approach would be to differentiate through the loop of the forward procedure. In our experiments, we found, however, that this approach is numerically unstable and does not yield correct gradients (see Fig. S2). Here, we provide two approaches to compute gradients in this case.

H.1 Differentiating through the last iteration

In this approach, one first solves the forward problem as usual to determine the linearization. One then executes one additional backward pass for the optimal controller, which one can differentiate using automatic differentiation.

H.2 Applying the implicit function theorem

Determining the optimal linearization and controller is essentially a fixed point problem. To differentiate through the optimal linearization and controller, we apply the implicit function theorem [68, 69].

We assume the forward equation to determine the linearization is given by

where θ are the parameters we want to differentiate with respect to and s t is a vector determining the optimal controller, such that it can be determined via a backward pass with

For the LQG controller, s t would consist of the (vectorized) matrix characterizing the optimal value function.

The fixed point problem of jointly determining the optimal linearization and controller can be formulated as determining X := [ u 1 , x 2 , u 2 , . . . , u T -1 , x T , s 1 , . . . , s T ] such that

The optimal solution of the fixed point problem satisfies

where X ∗ ( θ ) is a function yielding the optimal solution depending on θ , i.e., X ∗ : R p → R ( n + u + s ) t , where s is the number of elements of s t .

The implicit function theorem [68] gives, that for ( X 0 , θ 0 ) satisfying κ ( X 0 , θ 0 ) = 0 with the requirement that κ is continuously differentiable and the Jacobian ∂ θ κ ( X 0 , θ 0 ) is a square invertible matrix, then there exists a function X ∗ ( · ) on a neighborhood of θ 0 such that X ∗ ( θ 0 ) = X 0 . Furthermore, for all θ in this neighborhood, we have that κ ( X ∗ ( θ ) , θ ) = 0 and ∂X ∗ ( θ ) exists.

By applying the chain rule, the Jacobian ∂X ∗ ( θ ) needs to satisfy

∂ 1 η ( X ∗ ( θ ) , θ )) is given by a sparse matrix, as each element of η ( X, θ ) only depends on few elements of X . The desired gradient ∂X ∗ ( θ ) (or vector-Jacobian products with it) can then be computed using linear system solvers [69].

H.3 Results with different gradient computation methods

We ran the MLE for the partially observable reaching task with different gradient computation methods. The results are shown in Fig. S2. While unrolling the loop led to numerical problems, so the estimates are far off the true values, all other methods led to quite similar results. The mean run times were as follows:

Unrolled loop: 141 s

Differentiating through the last iteration: 103 s

Implicit differentiation: 243 s

Data-based linearization: 129 s

I Hyperparameters

Throughout the experiments, we have used the following hyperparameters:

J Tasks

J.1 Reaching task with biomechanical arm model

We implemented the non-linear biomechanical model for arm movements from Todorov and Li [8] and its partially observed version from Li and Todorov [43], which are described in more detail in the PhD thesis of Li [70]. The dynamics describe the movement of a two-link arm, which can be controlled by applying torques to the two joints. The task is to move the hand to a target location, as defined in the cost function

10

4

10

3

10

2

10

1

Unrolled loop

estimate

ca

10

3

10

2

10

1

10

2

10

1

10 0

m

10

1

10 0

10 1

10 2

o

10

4

10

3

10

2

10

1

Implicit diff

estimate

10

3

10

2

10

1

10

2

10

1

10 0

10

1

10 0

10 1

10 2

10

4

10

3

10

2

10

1

Last iteration

estimate

10

3

10

2

10

1

10

2

10

1

10 0

10

1

10 0

10 1

10 2

10

3

10

1

true

10

4

10

3

10

2

10

1

Data-based lin.

estimate

10

3

10

2

10

1

true

10

3

10

2

10

1

10

2

10

1

10 0

true

10

2

10

1

10 0

10 0

10 2

true

10

1

10 0

10 1

10 2

where e T is the position of the hand at the final time step in Cartesian coordinates, e ⋆ is the position of the target, and ˙ e T is the final velocity of the hand. That is, the task is to bring the hand to the target at the final time step ( T = 50 ) and minimize the final velocity while making as little control as possible along the way. The parameters c v and c a trade off, how important the velocity and control parts of the cost function are relative to the main goal of reaching the target. Additionally, we parameterize the signal-dependent variability of the motor system with the parameter σ m , which defines how strongly the variability is scaled by the control magnitude.

J.2 Navigation task

We implemented a simple navigation task in which an agent walks towards a target. The state x t = [ x t , y t , θ t , ω t ] T consists of the position in the horizontal plane, the heading angle, and the speed in the heading direction. The dynamics are

which means that the agent controls the velocity of the heading angle and the acceleration in heading direction. There is control-dependent noise, whose strength is determined by the parameter σ m . The observation model is

where k = [ k 1 , k 2 ] T is the position of the target and σ o determines the magnitude of the observation noise. The cost function is

with parameters determining the cost of controls ( c a ) and the cost of the final velocity ( c v ). The time horizon was T = 50 .

J.3 Classic control tasks

We evaluate our method on two classic continuous control tasks. Specifically, we build upon the Pendulum and Cart Pole environments from the gym library [54]. Because these environments are not stochastic in their standard implementations and are typically used in an infinite-horizon reinforcement learning setting, we made the following changes to the environments. To account for the finite-horizon setting we are considering in this work, the state-dependent costs associated with the task goal are applied only to the final time step, while control-dependent costs are applied at every time step along the trajectory. To make the problems stochastic, we have added noise on the dynamics and the observation function.

J.3.1 Pendulum

The task is to make a pendulum, which is attached to a fixed point on one side, swing into an upright position, i.e. to reach an angle θ = 0 . The pendulum starts at the bottom ( θ = π ) and can be controlled by applying a torque to the free end of the pendulum. We added control-dependent noise to the dynamics, where the parameter σ m controls, how strongly a standard normal noise is scaled by the magnitude of the torque. In addition to this motor noise parameter, the task has two other free parameters: the cost of controls c a and the cost of the final velocity c v . In the partially observable version of the task, the agent receives the observation via the non-linear stochastic observation model y t = [ sin θ t , cos θ t , ˙ θ t ] T +0 . 1 w t . The time horizon was T = 50 .

J.3.2 Cart pole

In the Cart Pole task, a pole is attached to a cart that can be moved left or right by applying a continuous force. In our version of the task, the goal is to move the cart from the horizontal position x = 0 to x = 1 while balancing the pole in an upright position. Again, we add control-dependent noise to the force, parameterized the strength of the linear control-dependence σ m . As above, the other two parameters are the control cost c a and the cost of the final velocity c v . In the partially observed version of the task, the agent receives a noisy observation with y t = x t + w t . The time horizon was T = 200 .

J.4 Light-dark domain

We use a slightly modified version of the light-dark domain, which was originally introduced by Platt et al. [55]. We adapted the light-dark domain's motion model to include signal-dependent noise on the control

The standard deviation of the perceptual uncertainty varies with the horizontal distance to the light source:

The cost function is composed of three terms. First, the agent should minimize the squared distance to the target p at the final time step T . Second, the agent should minimize the squared control signals u t . And finally, the agent should minimize the horizontal distance to the light source:

The time-horizon was set to T = 50 .

We simulated the behavior of two agents in the light-dark domain in Fig. S3. First, the partially observable version of iLQG [43] takes the expected belief covariance into account for the computation of the policy. This agent moves towards the light source before approaching the target. Adding another cost term c that expresses a preference to be close to the light source does not change much about this behavior (top row). Second, the fully observable version of iLQG [8] computes the policy irrespective of the belief. To apply it in the partially observable setting, we simply combine it with an EKF for the state estimation. This agent does not move towards the light source by default and as a result, the belief uncertainty is higher. The agent only moves towards the light source before approaching the target if we add the additional cost term c > 0 . One can also see in Fig. S3 that the agent's belief is more uncertain if it does not move to the light source before approaching the target. This results in a higher variability around the final position (inset plots in Fig. S3).

2.5

0.0

2.5

Partially obs. iLQG

(Li & T odorov, 2007)

c = 0.0

c = 0.1

c = 1.0

0

5

2.5

0.0

2.5

Fully obs. iLQG + EKF

(T odorov & Li, 2005)

0

5

0

5

O

K Additional results

The parameter estimates and true values are provided in Fig. S4, Fig. S5, Fig. S6.

K.1 Results for all tasks in the partially observable setting

10

3

10

2

10

1

ours

estimate

ca

10

3

10

2

10

1

10

1

10 0

m

10

1

10 1

o

10

3

10

2

10

1

true

10

3

10

2

10

1

baseline

estimate

10

3

10

2

10

1

true

10

3

10

2

10

1

10

1

10 0

true

10

1

10 0

10

1

10 1

true

10

1

10 1

10

3

10

1

ours

estimate

ca

10

3

10

1

10 1

10

1

10 0

m

10

2

10

1

10 0

o

10

3

10

1

true

10

3

10

1

baseline

estimate

10

3

10

1

10 1

true

10

3

10

1

10 1

10

1

10 0

true

10

1

10 0

10

2

10

1

10 0

true

10

2

10

1

10 0

K.2 Results for all tasks in the fully observable setting

10

3

10

2

10

1

ours

estimate

ca

10

3

10

2

10

1

10

2

10

1

10 0

m

10

3

10

2

10

1

true

10

3

10

2

10

1

baseline

estimate

10

3

10

2

10

1

true

10

3

10

2

10

1

10

2

10

1

10 0

true

10

2

10

1

10 0

10

3

10

2

10

1

ours

estimate

ca

10

3

10

2

10

1

10

1

10 0

m

10

3

10

2

10

1

true

10

3

10

2

10

1

baseline

estimate

10

3

10

2

10

1

true

10

3

10

2

10

1

10

1

10 0

true

10

1

10 0

10

3

10

1

ours

estimate

ca

10

3

10

1

10 1

10

1

10 0

m

10

3

10

1

true

10

3

10

1

baseline

estimate

10

3

10

1

10 1

true

10

3

10

1

10 1

10

1

10 0

true

10

1

10 0

K.3 Results for baseline with given control signals

K.4 Quantitative results for the light-dark domain

As for the other tasks, we generated 100 sets of parameters sampled from uniform distributions in the following ranges

horizontal target position p : ( -1 , 1)

state-dependent cost parameter c : (10 -2 , 1)

perceptual uncertainty σ : (10 -2 , 1)

and generated a dataset of 50 trajectories using iLQG [43] for each set of parameters. For each set, we then performed inference using our method and the baseline.

While the baseline accurately infers the target position, our method shows more variability in its estimates of the target position. The estimates of the cost term c are comparable for both methods. However, the baseline completely fails to estimate the perceptual uncertainty, which our method manages to infer relatively accurately.

1.0

0.5

0.0

0.5

1.0

ours

estimate

p

10

3

10

2

10

1

10 0

c

10

2

10

1

10 0

1

0

1

true

1.0

0.5

0.0

0.5

1.0

baseline

estimate

10

2

10 0

true

10

3

10

2

10

1

10 0

10

2

10

1

10 0

true

10

2

10

1

10 0

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm for inverse optimal control for agents operating in a partially observable Markov decision process, using only state trajectories. Given state trajectory data, dynamics model and observation model, the algorithm estimates the parameters by three steps: first, a policy is estimated using iLQG; second, the belief dynamics is estimated with EKF; finally, filtering is done by linearizing belief propagation. The likelihood is then maximized by back-propagation. The authors tested the proposed algorithm on several synthetic problems and compare it with a baseline based on maximum causal entropy. The results show that the proposed approach can better estimate the unknown parameters from behaviors than the baseline.\n\nSTRENGTHS:\nThe problem studied here is relevant to community. I think especially that the authors consider the case without action information, which makes the setup more practical. The writing of the paper is easy to follow. The introduction is well motivated and the related work discussion is informative. The proposed idea is clean and is new to my knowledge. The authors discuss in details about the limitation of this work.\n\nWEAKNESSES:\nWhile the writing is easy to follow, it is unclear what the assumption is being made to use the proposed method. From Algorithm 1, the authors assume knowledge of the dynamics, observation models, and state trajectory, but the steps of Algorithm 1 requires more information than that, e.g., the cost is needed in iLQG. In addition, it is also unclear how the proposed algorithm actually operates; the authors describes how filtering and approximate likelihood (given all the parameters) is computed in details, but do not mention how the unknown parameters are actually estimated (the authors mention \"using gradient-based optimization\" with automatic differentiation, but it is unclear what the computational graph entails). As a result, while I get the high level idea, I do not think I fully understand the working details of the proposed scheme, and therefore it is hard to give it an accurate evaluation. I think the authors should compare more with recent IOC or IRL works, in the experiments, and consider more realistic datasets (rather than synthetic ones).  In addition, it would be more informative if the authors can better highlight their contribution in terms of the specific problem difficulties here (e.g., due to missing action).  Lastly, the proposed method is limited to low dim problems as the authors also point out.\n\nQUESTIONS:\n1. What is the assumption of known information made in the paper? It is not clearly specified in the current writing. From Algorithm 1, it seems that assuming only the dynamics and observation models, as well as state data trajectories. But in iLQG the cost is also needed, and in the EKF step the belief dynamics (i.e., beta_t) is needed. What are they in the experiments? The algorithm also uses the prior of belief. Is that assumed to be given too?\n\n2. How does the learning actually work? The paper mentions the usage of automatic differentiation. But how is that actually done and what is the computation graph involved?  Algorithm 1 requires multiple linearization steps in iLQG, EKF, and the filtering. In particular, iLQG is an iterative process, do you also backpropagate through them?\n\n3. While the paper supposes only state trajectories are given, it seems to make assumption on the knowledge of the belief dynamics, or what the belief of the agent is. It is not clear whether this is a stronger or a weaker assumption.\n\n4. The current experiments only consider rather small and synthetic problems. Can you include results on some more realistic datasets? In addition, what are the agents that generate the data in the experiment (I can only tell iLQG is used for the light-dark domain)? It would be also good to test the proposed algorithm the estimation of non-iLQG agents, since the proposed one is based on iLQG, so that we can know the generality of the proposal.\n\nLIMITATIONS:\nYes, the authors made clear what most limitations are. It would be also good to point out the limitation due to limited empirical evaluations done in the paper (e.g., due to the synthetic, low-dim nature, the agent policy's form, the parametric structure, etc.)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
The paper introduces a new approach to inverse optimal control which is able to deal with partially observable systems in which action signals are not known. Most existing approaches only work in fully observable systems where actions are known. The paper introduces a probabilistic formulation for inverse optimal control and uses maximum likelihood to estimate the costs and parameters of the system. To make the likelihood tractable, they use local linearization similar to iLQG. The paper tests the approach on 4 tasks and shows improved performance over a maximum causal entropy-based baseline.

STRENGTHS:
I am not familiar with this area of research and most of the concepts in the paper are new to me but I was still able to follow most of concepts introduced in the paper hence I think the writing is mostly coherent and fairly easy to follow.

WEAKNESSES:
I don't have any weaknesses to point, I mainly have questions which I will write below.

QUESTIONS:
1) My first question is about baselines. The authors use a maximum causal entropy-based baseline for the experiments. From my understanding, maximum causal entropy is mainly used for exploration in RL but it is not clear to me what is the IOC-based method used in MCE to estimate the cost function. Does it use something like ILQG or EKF?
2) What functions are used to model f and \beta in equation 2? From my understanding, the functional form should be known beforehand which would be quite limiting for a setting where this information is not known.

LIMITATIONS:
The authors have discussed all relevant limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an inverse optimal control method to handle the challenging case of inferring an internal model in a non-linear partially observable system, when the action sequence is not observed. Quantitative evaluation of the proposed method was shown for some classic control problems. The authors also demonstrated through an example the potential of their method in disentangling perceptual factors and behavioral costs.\n\nSTRENGTHS:\nThe written language is easy to understand, and well organized. This work is a novel combination of ideas from some well-established techniques. The method should be of particular interest since intended actions of other agents are often only partially observable.  \n\nThe value of the method is highlighted by a comparison with an alternative method that mistakenly attributes perceptual uncertainty to subjective preferences. The authors’ IOC is able to correctly infer that much of the action in this task is driven by exploration rather than exploitation.\n\nWEAKNESSES:\nIt would be helpful to highlight more strongly that they can distinguish preferences from uncertainty. The authors rightly mention this virtue in the title, so I think it deserves more attention in the text. On the other hand, for that section, I would say that stronger evidence is needed, for example on a more complex task than the light and dark one. Uncertainty and cost are sometimes nonidentifiable: It could be too costly to correct a certain mistake, or it could be too uncertain to justify an optimal action, and these effects can be indistinguishable from the outside. However, when the uncertainty is dynamic while the cost function is static, these factors are dissociable. This should be addressed by the authors.\n\nQUESTIONS:\nThe authors describe their method as uniquely accounting for unobserved actions. But other Inverse Optimal Control methods allow for stochastic policies [e.g. Wu, Kwon et al 2020], in which actions can equivalently be interpreted as a latent intended action distorted by noise, or as a draw from a known policy. Aren’t these equivalent if the policy is optimized with action noise?\n\nThe authors should show a sanity check that their linearization successfully allows us to estimate actions near enough the ground truth.\n\nIt would be helpful to discuss more of the limitations from combining iLQG with EKF. Do the authors allow for controls that depend on a dynamic covariance, or just estimates of the mean? This was a bit unclear from their writing. It’s likely, given the techniques, that their method does not account for controllable uncertainty. Is this correct?\n\nI would like to see more evidence for the main claim, as suggested in the title, that the method disentangles perceptual uncertainty and behavioral costs. To show that taking into account the agent’s beliefs helps, it would be helpful to compute how uncertain the internal belief is while the agent moves towards the light before turning towards target, and plotting how it changes with subjective reward parameter c.\n\nLIMITATIONS:\nThe authors have a thoughtful discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets for solving the inverse optimal control problem for partially-observable stochastic non-linear dynamics with no observation of the action. To estimate the parameters of the cost function in a stochastic non-linear system, the author first derives a likelihood function for the model parameters. They then approximate this likelihood by locally linearizing the system using a combination of iterative linear quadratic Gaussian (iLQG) and Extended Kalman filter (EKF) techniques. The proposed inverse method makes explicit assumptions about the dynamics of the control tasks, the agent's belief in a stochastic environment, and the structure of the cost function. The authors evaluate their approach on four simulation environments where structured noise is added to the dynamics of the environment to introduce stochasticity into the system dynamics. In comparison to the baseline, the proposed approach demonstrates better uncertainty estimation.\n\nSTRENGTHS:\nIn general, I find this paper easy to follow and variables in the equations are well defined. The derivations are nicely structured. The details are well documented in the Appendix and the assumptions and limitations are explicitly considered.\n\nWEAKNESSES:\nI suggest the author clearly define their contribution in this paper, as the likelihood formulation of the inverse optimal control problem has been derived in another work, \\textit{Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System}. To enhance clarity, I recommend placing the likelihood formulation in the background section.\n\nClaiming that their approach can disentangle perceptual uncertainty and behavior costs might be too bold for two reasons: firstly, the author demonstrates this claim in a single simulation environment only, and secondly, the structure of perception uncertainty and behavior cost function is known. However, in reality, obtaining the structure of these elements is difficult.\n\nMy major concern pertains to the number of assumptions made in different aspects of the system to obtain the proposed closed-form solution. For instance, all noises added in the experiments are assumed to be Gaussian distributed. These assumptions render the proposed approach challenging to apply in real-world scenarios, and the experiments were solely conducted in simulations. In addition, potential solutions to these assumptions i.e. addressing multi-modals belief utilizing SMC, are only briefly mentioned in the conclusion section.\n\nQUESTIONS:\nIn the reaching task and the navigation task, why is the goal position not inferred? \n\nIn the conclusion, the authors mentioned that their method may not be effective for high-dimensional parameter spaces. It would be beneficial for the community to know the accuracy of the uncertainty estimation as the number and dimension of parameters scale.\n\nIn the sensorimotor domain, do you always have a well-structured cost function that you can obtain beforehand? I.e. I think your approach would not work if the formulation of the behavior cost function like eq.8 is not given.\n\nLIMITATIONS:\nLimitations are well documented in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a probabilistic approach to inverse optimal control for partially-observable stochastic non-linear systems with unobserved action signals. It derives an approximate likelihood function for the model parameters by linearizing the system around the observed trajectories and tracking the agent's belief distribution. This method is demonstrated to accurately infer the parameters better than the baseline maximum causal entropy (MCE) approach on two classic control tasks and two human behavioral tasks. Additionally, it shows the ability to disentangle the influences of perceptual uncertainty and behavioral costs on information-seeking behavior sources of information-seeking behavior.\n\nSTRENGTHS:\n- The introduction gives a clear background, which is helpful for someone like me who is less familiar with this field. It provides a clear understanding of the research question and the existing approaches.\n- This paper proposes a new probabilistic approach for inverse optimal control in stochastic non-linear systems with missing control signals and partial observability, which outperforms the baseline MCE model and also provides an interpretable representation of the parameter space.\n- The proposed method is evaluated on different tasks, including two human behavioral tasks, which might inspire further investigation on how the method can be applied to neuroscience and cognitive science studies.\n\nWEAKNESSES:\nA more comprehensive investigation into the method's robustness under various conditions would strengthen the paper's contributions.\n\nQUESTIONS:\nWhy is it necessary to assess the proposed method's performance on pendulum and cart pole tasks, despite their deterministic nature and lack of partial observability?\n\nLIMITATIONS:\nThe authors have adequately acknowledged and addressed the limitations of their methods, which include the following: (1) The focus on tasks that can be well-solved by control methods based on linearization and Gaussian approximation (iLQG and EKF), which may not fully capture the complexity of naturalistic behavior, and (2) the concern that the method might not scale effectively to high-dimensional parameter spaces as optimization in a high-dimensional non-linear space can potentially lead to getting stuck in local minima.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is a strong contribution on the Inverse Optimal Control problem when actions cannot be observed. It is particularly interesting their modelling of the agent and the “researcher” observer. The mathematical depth is good and sound. The results may be enough for a theoretical paper. However, there are too many unknowns to be ready for acceptance, particularly some definitions like partial observability related to noise. Another issue is the clarity regarding the baseline used, is it self-programmed?, is it taken from previous literature?. It seems that it does not work for any of the problems tested.\n\nSTRENGTHS:\n-\tThe mathematical depth is good.\n-\tInverse optimal control plus noise estimation is relevant for many applications; especially for motor control research.\n-\tResults may be enough for a theoretical paper.\n\nWEAKNESSES:\n-\tNarrative: While the contribution looks very promising, the explanation of the contribution in the introduction is not enough clear to understand what is the main focus of the paper. Furthermore, there is a need to jump forward and backward in the text to fully understand the details of the approach. Particularly, it is sometimes complicated to see if the authors are talking about the agent or the observer (“researcher”). Another example: I had to read the whole article to understand the title.\n-\tThe partial-observability may be controversial as it is defined. The idea of tracking the belief of the agent is interesting, but why not using directly the observed state? So then is it the noise that the algorithm is tackling as partial observability. Thus, then the complexity depends on the type of noise.\n-\tFollowing the previous comment, while the method may be used for any parameter estimation (as authors state) why only estimating the observation/motor noise? Btw, observation noise of the agent or the observer? From the text I assumed that the authors were doing system identification at the same time. But they are estimating the noise.\n-\tBaseline explanation needs further description for the sake of clarity.\n\nQUESTIONS:\n-\tThe description of the variables v_t and w_t could be improved\n-\tWhy there is a final cost that is independent on the action taken? Conversely to the previous time steps.\n-\tWhat is πt(xt) ? (section 2.2)\n-\tWhy stochastic policy has a capital Pi?\n-\tNot clear why the EKF is introduced.\n-\t“In the inverse optimal control problem, the goal is to estimate parameters θ ∈ Rp of the agent’s optimal control problem given the model and trajectory data. These parameters can include properties of the agent’s cost function, the sensory and control systems of the agent, or the system’s dynamics.” This is a definition proposed by the authors. Originally, IOC is to recover the cost function from expert demonstrations. Getting system parameters is called system identification.\n-\t“cost of final velocity cv” Is this related to the cost definition with the final cost at end time?\n-\t“observation noise” of what quantity? Experimenter, agent?\n-\t“Note that past methods based on MCE are usually limited to estimating cost functions, so that parameters such as the agent’s noise therefore cannot be inferred.” This sentence helps a lot to understand the authors approach.\n-\tIt is not clear why the dynamics of the agent state and its beliefs only depends on the state and the belief: “closed-form expression for p(xt+1, bt+1 | xt, bt).”\n-\t“we applied a baseline method based on the maximum causal entropy (MCE) approach [3].” Which method? Self-programmed or taken from the literature? Baseline text could be improved. In the end I am not sure how many baselines are being used.\n-\t“expressed as a non-quadratic cost function of the joint angles” Is this really true. Besides, while I understand that there is not enough space for everything on the paper, the description of the input output system would be nice to place it in the main paper so there is no need to go to the appendix to read that the arm is controlled with torque.\n-\tIt is not clear why the baseline cannot even recover the action cost properly. Would it work in the absence of noise?\n- The agent navigation dynamics can be simplified to use the turn rate as the control action:\n[dot{x}, dot{y}, dot{theta}]¨=  [cos(theta)w, sin(theta)w, u]\n\nLIMITATIONS:\nLimitations are sufficiently addressed. My only constructive suggestion is to mention that noise covariance estimation for non white-noise sources is a very important problem in control. So I would include it in the discussion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the new formalization of inverse optimal control on partially-overevable Markov decision processes. The authors argue that most existing works on inverse control or inverse reinforcement learning focus on fully-observable Markov decision processes. Their approach extends iterative linear quadratic Gaussian (iLQG) and Maximum causal entropy (MCE) reinforcement learning, introducing local mineralization to achieve tractable likelihood. The method is evaluated through simulations.\n\nSTRENGTHS:\n+ The formulation and derivation of the method are solid and well-motivated.\n\n*Weaknesses\n\n\n\n\n*Limitations\n\nWEAKNESSES:\n- The derived method seems natural, making it challenging to identify the novelty of the proposal.\n- The evaluation lacks sufficient quantitative and qualitative analysis, as it only covers simple settings and compares with the MCE approach ~~without using the proposed method~~.\n\nQUESTIONS:\n1. Considering the numerous studies on inverse reinforcement learning and inverse optimal control without the assumption of linearization of the dynamical system and Gaussian approximations, a justification is needed in terms of applicability to robot learning. A comparison with methods without such assumptions should be included. Can the method handle more dynamic tasks such as walking?\n\n2. Clarify the difficulty of the problem in a more intuitive way to explain why this problem remains unsolved. The formulation is somewhat straightforward. Therefore, potential readers may wonder why the problem is still unsolved.\n\nLIMITATIONS:\nThis is just a suggestion.\n\nRegarding POMDP in robot learning, exploring the relationship with world models (e.g., [1,2]) would be valuable.\n\n[1] Hafner, Danijar, et al. \"Dream to control: Learning behaviors by latent imagination.\" arXiv preprint arXiv:1912.01603 (2019).\n[2] Tadahiro Taniguchi, Shingo Murata, Masahiro Suzuki, Dimitri Ognibene, Pablo Lanillos, Emre Ugur, Lorenzo Jamone, Tomoaki Nakamura, Alejandra Ciria, Bruno Lara & Giovanni Pezzulo (2023) \"World models and predictive coding for cognitive and developmental robotics: frontiers and challenges,\" Advanced Robotics, 37:13, 780-806", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a method to infer an agent’s internal model in a Partially Observable Markov Decision Process (POMDP) when the agent’s actions are non observable. Using local linearization, the authors show how a closed form approximation of the likelihood function for state trajectories can be constructed to subsequently yield maximum likelihood estimates. They also show that when there are confounding factors that can lead to the same behavior, the proposed method is able to disambiguate the factors better compared to the baseline.\n\n**After author's rebuttal:** I appreciate the authors offering clarifications. My main concern was around the significance of the contribution, which I based on the references discussed in the paper since I haven't been working on this specific area myself. However, looking at the other reviews and the author's comments, it appears the paper does more than just relaxing a simple assumption or two. I also mentioned that if the inference mechanism is aware of the generative model, one would expect better estimation in general. I make this comment from a Bayesian perspective so if the observations are from a Gaussian distribution and we model it as such, we expect to learn the sensible parameters with enough data as opposed to an unaware model that assumes, say, a Laplace distribution. I would need to think more as to why this might not hold here, as I am still inclined to believe it does. In any case, based on the discussion of the contributions, I am changing my rating from 4 to 5.\n\nSTRENGTHS:\nThe paper does a good job at laying down the groundwork for the problem, going over previous work and pointing at the potential shortcomings of existing methods. The motivating example in Fig. 1 works well and the paper’s structure follows naturally. The experiments are well-designed and the results are clearly discussed establishing the technique’s superior performance over the baseline.\n\nWEAKNESSES:\nHowever, in my opinion, the contribution lacks significance for acceptance at the venue. The problem formulation is slightly more general than existing work but only marginally so. The linearization to get approximate likelihood is clean but not novel. The results are not surprising since a technique that is aware of the generative assumptions is expected to lead to better posterior estimates. The disambiguation behavior follows from better estimation.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}