SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To obtain a closed-form expression for  $g$ , which we can linearize, we model the agent's policy using iLQG (Section 2.2) and the belief dynamics using the EKF (Section 2.4). This choice leads to an affine control and belief given  $\mathbf{b}_t$ , making linearization of  $p(\pmb{x}_{t+1}, \mathbf{b}_{t+1} \mid \pmb{x}_t, \mathbf{b}_t)$  straightforward. To allow for additional stochasticity in the agent's policy, we use the MCE formulation (Section 2.3). For linearized dynamics, the MCE policy is given by a Gaussian distribution, so that  $\pi_t(\mathbf{b}_t, \pmb{\xi}_t) = L_t(\mathbf{b}_t - \bar{\pmb{x}}_{1:T}) + \mathbf{m}_t + \bar{\pmb{u}}_{1:T} - \tilde{H}_t \pmb{\xi}_t$ , with  $\tilde{H}_t$  the Cholesky decomposition of  $H_t$ , and can be marginalized out in closed form.
The approximations we have introduced allow us to solve the integral in Eq. (4) in closed form by applying standard equations for linear transformations of Gaussians, resulting in
$$
p \left(\boldsymbol {x} _ {t + 1}, \mathbf {b} _ {t + 1} \mid \boldsymbol {x} _ {1: t}\right) \approx \mathcal {N} \left(\mu_ {t}, \Sigma_ {t}\right), \tag {6}
$$
with  $\mu_t = g(\pmb{x}_t, \mu_t^{(b)}, 0, 0, 0)$  and  $\Sigma_t = \mathbb{J}_{\mathbf{b}} \Sigma_t^{(b)} \mathbb{J}_{\mathbf{b}}^T + J_{\mathbf{v}} \mathbb{J}_{\mathbf{v}}^T + \mathbb{J}_{\mathbf{w}} \mathbb{J}_{\mathbf{w}}^T + \mathbb{J}_{\pmb{\xi}} \mathbb{J}_{\pmb{\xi}}^T$ , where  $\mathbb{J}_{\bullet}$  denotes the Jacobian of  $g$  w.r.t.  $\bullet$ , evaluated at  $(\pmb{x}_t, \mu_t^{(b)}, 0, 0, 0)$ . Under this Gaussian approximation, both remaining operations of Algorithm 1 (Appendix C) can also be performed in closed form. A more detailed derivation and representation of these formulas can be found in Appendix D. If the agent has full observations of the system's state, the inverse optimal control problem is simplified significantly (see Appendix E). Details about the implementation are provided in Appendix F.
3.3 Data-based linearization
The forward optimal control problem is commonly solved by starting with a randomly initialized nominal trajectory and iterating between computing the locally optimal control law and linearization until convergence. To compute the likelihood in the inverse problem, we can take a more efficient approach by linearizing directly around the given trajectory  $x_{1:T}$ . We then need to perform only one backward pass to compute an approximately optimal control law given the current parameters, and a forward pass to compute an approximately optimal filter. This, in particular, allows efficient computation of the gradient of the likelihood function for the optimization procedure. As we assume the controls to be unobservable, but they are needed for the linearization, we compute estimates of the controls by minimizing the squared difference of the noiseless predicted states and the actual states (see Appendix G). Note that these estimated controls are only used for the linearization, but are not used as observed controls in the IOC likelihood itself. In the case where the full state is not observable, we cannot linearize around the trajectory. For these cases, we propose two approaches to compute gradients based on implicit differentiation and differentiating only through the last iteration. As this setting is not the main focus of this paper, details of these approaches are provided in Appendix H.
We evaluated our method on two classic control tasks, i.e., Pendulum and Cart Pole, and two human behavioral tasks, manual reaching and navigation. To evaluate the accuracy of the parameter estimates obtained by our method and to compare it against a baseline, we computed absolute relative errors per parameter, i.e.,  $|(\theta - \hat{\theta}) / \theta|$ . This metric makes averages across parameters on different scales more interpretable compared to other metrics such as root mean squared errors. For each task, we simulated 100 sets of parameters from a uniform distribution in logarithmic space. For each set of parameters, we simulated 50 trajectories. We then maximized the log likelihood using gradient-based optimization with automatic differentiation [L-BFGS algorithm; 51]. See Appendix I for a summary of the hyperparameters of our experiments.
a) Data simulated with ground truth parameters
b) Log likelihood of parameters
c) Data simulated with maximum likelihood parameter estimates
d) Maximum likelihood parameter estimates
Figure 2: IOC for non-linear reaching. A Simulated trajectories for eight targets. Increasing the control cost and the motor noise affects the trajectories, since reaching the target becomes less important and variability increases. B IOC log likelihood for control cost  $c_{a}$  and motor noise  $\sigma_{m}$ . The maximum likelihood estimate (pink cross) is close to the ground truth parameters (black dot). C Simulated trajectories using the MLEs from B. The simulations are visually indistinguishable from the ground truth data. D True parameters plotted against maximum likelihood estimates. Top row: our method, bottom row: MCE baseline. The columns contain the four different model parameters (control cost  $c_{a}$ , velocity cost  $c_{v}$ , motor noise  $\sigma_{m}$ , observation noise  $\sigma_{o}$ ).
All tasks we consider have four free parameters: control cost  $c_{a}$ , cost of final velocity  $c_{v}$ , motor noise  $\sigma_{m}$ , and observation noise  $\sigma_{o}$  of the agent. In the fully observable case, we leave out the observation noise parameter and only infer the three remaining parameters. For concrete definitions of the parameters in each specific task, see Appendix J.
4.1 Baseline method
For a comparison to previously proposed methods, we applied a baseline method based on the maximum causal entropy (MCE) approach [3]. As for this approach, control signals of the observed trajectories are required, we use the estimates of the controls that we determine in our proposed method for the data-based linearization (Section 3.3). Note that the baseline, representative for applicable past IOC methods, does not have an explicit model of partial observability. Further note that past methods based on MCE are limited to estimating cost functions, so that parameters such as the agent's noise cannot be inferred. For the specific MCE linearization-based baseline we consider, it is actually straightforward to maximize the likelihood with respect to the noise parameters, which enables us to evaluate noise estimates. To show that this approach constitutes a suitable baseline, in
Appendix K.3, we provide results for the case where the true control signals are known and there is no partial observability. More details of the baseline are provided in Appendix B.
4.2 Evaluation on manual reaching task
We evaluate the method on a reaching task with a non-linear two-joint biomechanical arm model, which has been applied to reaching movements in the sensorimotor neuroscience literature [e.g., 52, 53]. The agent's goal is to move its arm towards a target at position  $\mathbf{e}^{\star}$  by controlling the torque to the two joints. This objective is expressed as a non-quadratic cost function of the joint angles,
$$
J = \left\| \mathbf {e} _ {T} - \mathbf {e} ^ {*} \right\| ^ {2} + c _ {v} \left\| \dot {\mathbf {e}} _ {T} \right\| ^ {2} + c _ {a} \sum_ {t = 1} ^ {T - 1} \left\| \mathbf {u} _ {t} \right\| ^ {2}, \tag {7}
$$
since the final position and velocity of the hand  $\mathbf{e}_T$ ,  $\dot{\mathbf{e}}_T$  are non-linear functions of the joint angles. See Appendix J.1 for details.
We use a fully observable [8] and a partially observable version of the task [43]. Fig. 2 A shows simulations from the model with two different parameter settings. Evaluating the likelihood function for a grid of two of the parameters (Fig. 2 B) confirms that it has its maxima close to the true parameter values. Simulated data using the maximum likelihood estimates look indistinguishable from the ground truth data (Fig. 2 C).
In Fig. 2 D, we show maximum likelihood estimates and true values for repeated runs with different random parameter settings. The parameter estimates of our method closely align with the true parameter values, showing that we can successfully recover the parameters from data. The baseline method, in contrast, shows considerably worse performance, in particular for estimating noises due to the lacking explicit representation of partial observability. Importantly, even when the true control signals are provided, the noise parameter estimates of the baseline are not well estimated (Appendix K.3). Estimates for the fully observable case are provided in Appendix K.2. The median absolute relative errors of our method were 0.11, while they were 0.93 for the baseline. The influence of missing control signals and of the lack of an explicit observation model in the baseline can be observed by comparing the results to the fully observable case and the case of given control signals in Appendix K.2 and Appendix K.3.
4.3 Quantitative evaluation on other tasks
To show that our method works for a range of different tasks, we evaluated it on the three other tasks (navigation, pendulum and cart pole). In the navigation task, we consider an agent navigating to a target under non-linear dynamics while receiving noisy observations from a non-linear observation model. To reach the target, the agent can control the angular velocity of their heading direction and the acceleration with which they move forward. The agent observes noisy versions of the distance to the target and the target's bearing angle. We provide more details about the experiment in Appendix J.2. Maximum likelihood parameter estimates for the navigation task are shown for the partially observable case in Fig. S6 and for the fully observable case in Fig. S10. As for the reaching task, our method provides parameter estimates close to the true ones, while the estimates of the baseline deviate for many trials. Median absolute relative errors of our method were 0.31, while they were 1.99 for the baseline (Fig. 3).
The two classic control tasks (Pendulum and Cart Pole) are based on the implementations in the gym library [54]. Because these tasks are neither stochastic nor partially observable in their standard formulations, we introduce noise on the dynamics and turn them into partially observable problems by defining a stochastic observation function (see Appendix J.3). In Appendix K, we show the parameter estimates for the Pendulum (Fig. S4) and for the Cart Pole (Fig. S5) for the partially observable case, while Fig. S8 and Fig. S9 show the fully observable case, respectively. One can observe that the results are qualitatively similar to the ones in the reaching and navigation tasks, showing that our method provides accurate estimates of the parameters. Median absolute relative errors of our method were 0.12 and 0.41, while for the baseline they were 2.21 and 3.82 (Fig. 3).
Figure 3: Evaluation across tasks. Absolute relative errors (log scale) for different tasks. Our method consistently outperforms the MCE baseline.
4.4 Information-seeking behavior in the light-dark domain
Finally, we investigate the ability of our method to disentangle sources of information-seeking behavior. In the light-dark domain [55], an agent moves in a 2D space and receives noisy measurements of its position, whose standard deviation depends on the horizontal distance from a light source:
$$
\boldsymbol {y} _ {t} = \boldsymbol {x} _ {t} + \sigma | x _ {t, 1} - 5 | \boldsymbol {w} _ {t}, \tag {8}
$$
where  $\sigma$  governs the amount of perceptual uncertainty. This task is a common test for information-seeking behavior, because it requires the agent to move towards the light source and at the same time away from the target to reduce the uncertainty about its position relative to the target. When this uncertainty has been reduced, the agent can approach the target (see Appendix J.4 for details). The agent's goal is to reach the target's position  $\mathbf{p}$  at the final time step, while minimizing control effort  $u_{t}^{2}$ :
$$
J = \underbrace {\left(\boldsymbol {x} _ {T} - \mathbf {p}\right) ^ {2}} _ {\text {f i n a l c o s t}} + \underbrace {\sum_ {t = 1} ^ {T - 1} \frac {1}{2} \boldsymbol {u} _ {t} ^ {2} + c \left(x _ {t , 1} - 5\right) ^ {2}} _ {\text {r u n n i n g c o s t}}. \tag {9}
$$
Different from the original problem formulation, we consider the case in which the agent may have an additional inherent desire to be close to the light, parameterized by  $c$ . We recover the original cost function [55] for  $c = 0$ , but we can represent agents that seek light more than necessary to reduce uncertainty for reaching the goal state with  $c > 0$ . Accordingly, both the reduction of perceptual uncertainty and state-dependent running cost could encourage the agent to move towards the light source before approaching the target. For an external observer, e.g., an experimenter, observing an agent moving towards the light might not reveal the different potential sources for this behavior. We now ask if it is possible to disentangle these two factors using our proposed IOC algorithm.
We simulated 100 trajectories of an iLQG agent [partially observable version, 43]
with no inherent desire to be near the light source ( $c = 0$ ) and some perceptual uncertainty ( $\sigma = 0.2$ ) depending on the distance to the light source. The agent first moves towards the light source and then reaches the target (Fig. 4 A). We inferred the parameters  $\sigma$ ,  $\mathbf{p}$ , and  $c$  using our method and the baseline. Both methods recover the target position. Our method in addition infers values close to the true  $c$ , and  $\sigma$ . It therefore correctly attributes the information-seeking behavior of the agent to the perceptual uncertainty (Fig. 4 B). The baseline method, however, does not infer the correct
Figure 4: Light-dark domain. Cost maps and perceptual uncertainty map plotted with true parameters (A) and inferred parameters using our method (B) and baseline (C). Mean trajectories with start point (circle) and target (cross) are shown in orange.
perceptual uncertainty. It instead attributes the agent's information-seeking behavior to an inherent desire to be in the right part of the room in the running cost (Fig. 4 C). This highlights the importance for IOC in partially observable domains to probabilistically take the agent's belief into account, if one is interested in inferring the correct cognitive mechanisms. For a quantitative evaluation of the maximum likelihood estimates in the light-dark domain, see Appendix K.4.
In this paper, we introduced a new IOC method for partially observable systems with stochastic non-linear dynamics and missing control signals. Using a probabilistic approach, we formulate the IOC problem as maximizing the likelihood of the observed states given the parameters. As the exact evaluation of the likelihood for a general non-linear model is intractable, we developed an efficient approximate likelihood by linearizing the system locally around the given trajectories, as in popular approaches such as the EKF or iLQG. By maintaining a distribution that tracks the agent's belief, an approximate likelihood can be evaluated in closed form within a single forward pass and efficiently optimized. Our proposed formulation is able to incorporate multiple sources of the stochasticity of the agent, reconciling the theory of past MCE IOC algorithms [e.g., 3] and approaches where the agent's stochasticity stems from an explicit stochastic observation and control model [29].
We evaluated our method on two stochastic variants of classic control tasks, pendulum and cart pole, and on two human behavioral tasks, a reaching and a navigation task. In comparison to an MCE baseline, we have found our method to achieve lower estimation errors across all tasks. Further, it successfully inferred noise parameters of the system, which was not possible with the baseline. In the light-dark domain, we were able to infer costs and perceptual uncertainty parameters, two different causes of apparent information-seeking behavior that could lead to qualitatively similar trajectories. This means that our method is a first step towards distinguishing pragmatic from epistemic controls. To further investigate this, it would be fruitful to examine belief-space planning methods that explicitly include the agent's belief covariance in the policy [56].
The limitations of our method are mainly due to the linearization of the dynamical system and the Gaussian approximations involved in the belief tracking formulation of the likelihood function. In more complex scenarios with non-Gaussian belief distributions, e.g., multimodal beliefs, the method will likely produce inaccurate results. This problem could be addressed by replacing the closed-form Gaussian belief by particle-based methods [57]. Further, we focused on tasks which could be solved well by control methods based on linearization and Gaussian approximation (iLQG and EKF), motivated by their popularity in applications in cognitive science and neuroscience. Forward problems that cannot be solved using iLQG are probably not directly solvable using our inverse method. While, in principle, our method is also applicable to other forward control methods that compute differentiable policies, it is an empirical question whether linearizing these policies leads to accurate approximate likelihoods and parameter estimates. A further limitation of our method is that it requires parametric models of the dynamics and noise structure. While missing parameters can be determined using our method, in the case of completely unknown dynamics a model-free approach to IOC would be more suitable. Lastly, while we have shown that inference is feasible, the results probably do not scale to high-dimensional parameter spaces. One reason for this is that optimization in a high-dimensional non-linear space can potentially get stuck in local minima. This problem could be relieved by using more advanced optimization methods. A further, more fundamental, concern with higher-dimensional parameter spaces is that identifiability issues and ambiguous solutions arise. However, our probabilistic approach with a closed-form likelihood opens up the possibility of using Bayesian methods to investigate the identifiability of model parameters [58]. For future work exploring the relationship to methods for learning world models in POMDPs might also be a fruitful direction [59, 60].
Our method provides a tool for researchers, e.g., in sensorimotor domains, to model sequential behavior by inferring an agent's subjective costs and internal uncertainties. This will enable answering novel scientific questions about how these quantities are affected by different experimental conditions, how they deviate from intended task goals and provided task instructions, or how they vary between individuals. This is particularly relevant to a computational understanding of naturalistic behavior [61-63], for which subjective utilities are mostly unknown.
We gratefully acknowledge the computing time on the high-performance computer Lichtenberg at the NHR Centers NHR4CES at TU Darmstadt, and financial support by the project "Whitebox" funded by the Priority Program LOEWE of the Hessian Ministry of Higher Education, Science, Research and Art. We thank Fabian Kessler for helpful discussions and comments and an anonymous reviewer of a previous version of the manuscript for surmising that the present method could not infer information-seeking behavior.
[1] Christopher M Harris and Daniel M Wolpert. Signal-dependent noise determines motor planning. Nature, 394(6695):780-784, 1998.
[2] Emanuel Todorov and Michael I Jordan. Optimal feedback control as a theory of motor coordination. Nature neuroscience, 5(11):1226-1235, 2002.
[3] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning, 2010.
[4] A Aldo Faisal, Luc PJ Selen, and Daniel M Wolpert. Noise in the nervous system. Nature reviews neuroscience, 9(4):292-303, 2008.
[5] Daniel M Wolpert and Zoubin Ghahramani. Computational principles of movement neuroscience. Nature neuroscience, 3(11):1212-1217, 2000.
[6] Matthew Golub, Steven Chase, and Byron Yu. Learning an internal dynamics model from control demonstration. In International Conference on Machine Learning, pages 606-614, 2013.
[7] Brian D O Anderson and John B Moore. Optimal control: linear quadratic methods. Prentice-Hall, 1990.
[8] Emanuel Todorov and Weiwei Li. A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In American Control Conference, pages 300-306. IEEE, 2005.
[9] Mykel J Kochenderfer, Tim A Wheeler, and Kyle H Wray. Algorithms for decision making. MIT press, 2022.
[10] Frederick Mosteller and Philip Nogee. An experimental measurement of utility. Journal of Political Economy, 59(5):371-404, 1951.
[11] Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. *Econometrica*, 47(2):263-292, 1979.
[12] Konrad Paul Körding and Daniel M Wolpert. The loss function of sensorimotor learning. Proceedings of the National Academy of Sciences, 101(26):9839-9842, 2004.
[13] R E Kalman. When Is a Linear Control System Optimal? Journal of Basic Engineering, 86(1):51-60, 1964.
[14] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume 1, 2000.
[15] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, volume 5, 2004.
[16] Constantin A Rothkopf and Christos Dimitrakakis. Preference elicitation and inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference (ECML PKDD), pages 34-48. Springer, 2011.
[17] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, volume 23, pages 1433-1438, 2008.
[18] Sergey Levine and Vladlen Koltun. Continuous inverse optimal control with locally optimal examples. In International Conference on Machine Learning, pages 475-482, 2012.
[19] Abbas, J., and Jan Peters. Relative entropy inverse reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 182-189, 2011.