Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System

1 Introduction . Computational level theories of behavior strive to answer the questions , why a system behaves the way it does and what the goal of the system ’ s computations is . Such goals can be formalized based on the reward hypothesis . In the words of Richard Sutton , the reward hypothesis assumes , that “ goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal ” [ 1 ] . Thus , to understand human sensorimotor behavior , it is essential to characterize its goals and purposes quantitatively in terms of costs and benefits . But particularly in every-day tasks , the cost and benefits underlying behavior are unknown . Stochastic optimal control allows formulating behavioral goals in terms of a cost function for tasks involving sequential actions under action variability , uncertainty in the internal model , and delayed rewards . The solution to the optimization problem entailed in the cost function is a sequence of ∗Both authors contributed equally to this paper . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) actions , which is then compared to human movements . While early models optimized costs related to deterministic kinematics of a movement to a target [ 2 ] , task goals were subsequently formalized as costs on the variance of stochastic movements ’ endpoint distances to a goal target [ 3 ] . Importantly , although [ 3 ] considered open-loop control , it revealed the importance of modeling the specific variability of human movements , which increases linearly with the magnitude of the control signal [ 4 ] . As the neuronal control signal increases , so does its variability , leading to optimal movements trading off between achieving task goals and reducing the expected impact of movement variability . Including sensory feedback in stochastic optimal control leads to a computationally much more intricate problem , which can be formulated as a partially observable Markov decision process ( POMDP ) and is intractable in general . One of the few tractable cases is the linear-quadratic Gaussian ( LQG ) setting , where dynamics are linear , costs are quadratic , and variability is additive and Gaussian , leading to sensory inference of the state and control to be decoupled [ 5 ] . However , not only is human movement variability signal dependent , but additionally the uncertainty of sensory signals increases linearly with the magnitude of the stimulus , a phenomenon known as Weber ’ s law [ 6 ] . Todorov [ 7 ] extended the LQG case by introducing stochastic optimal feedback control with signaldependent noise , which allows the specification of noise models in line with what is known about the human sensorimotor system . This model [ 8 ] has been able to explain a broad range of phenomena in sensorimotor control [ 9 , 10 ] , including linear movement trajectories , smooth velocity profiles , speed-accuracy tradeoffs , and corrections of errors only if they influence attaining the behavioral goal . Particularly incorporating signal-dependent noise has been crucial in explaining experimental data , ranging from how corrections of movements during action execution depend on feedback and task goals [ 11 ] , that movements consider sensory uncertainty and temporal delays in real-time [ 12 ] , that movement plans in novel environments are reoptimized based on the learning of internal models to minimize implicit motor costs and maximize rewards [ 13 ] , and many others [ 14 , 15 , 16 , 17 ] . Explaining human behavior in these studies usually starts by hypothesizing the cost function describing a task , obtaining the optimal feedback controller , and comparing simulated trajectories to those observed experimentally . This line of inquiry , therefore , utilizes similarity of trajectories to quantify the degree of optimality in human behaviors . In some cases [ 14 ] , trajectories are simulated from the model to check for robustness with respect to changes in the model parameters . If our goal is to use optimal control models to infer such quantities , which often can not be measured independently , from behavior , it would instead be desirable to invert the problem and find those parameter settings which are consistent with the observed trajectories . While such inverse methods have been developed both in the field of reinforcement learning to infer the rewards being optimized by an agent [ 18 , 19 , 20 ] as well as in optimal control for the LQG case [ 21 , 22 , 23 ] , this is currently not possible for the noise characteristics of the human sensorimotor system . Here , we introduce a probabilistic formulation of inverse optimal feedback control under signaldependent noise in the tradition of rational analysis [ 24 , 25 ] . Our starting point is the forward problem introduced in [ 8 ] . We formulate the inference problem faced by an agent as a POMDP and distinguish it from the inference problem of an experimenter observing the agent . We proceed by deriving the likelihood of a sequence of observed states and provide an approximation to the non-Gaussian uncertainty due to the signal-dependent noise . First , this allows recovering the cost function underlying the agent ’ s behavior from observed behavioral data . Second , we extend the inference of the cost function to the case in which the state variables are only partially observable to the experimenter , e.g. , when only measuring the position of the agent ’ s movement . Third , we show through simulated and experimental data that the cost functions can indeed be recovered . Fourth , the probabilistic formulation allows recovering the agent ’ s belief during the experiment as well as the experimenter ’ s uncertainty about the inferred belief . Related work Inferring the cost functions underlying an agent ’ s behavior has long been of interest in different scientific fields ranging from economics [ 26 ] and psychology [ 27 ] to neuroscience [ 28 ] and artificial intelligence , particularly reinforcement learning [ 18 , 19 , 29 , 20 ] . Inverse Reinforcement Learning ( IRL ) specifically addresses the question of inferring the cost function being optimized [ 18 , 19 , 29 ] or approximately optimized [ 30 ] by an agent , with more recent approaches employing deep neural networks [ 31 , 32 ] . Some work has particularly addressed sensorimotor behavior [ 33 , 34 , 35 , 36 ] and extensions to the partially observable setting have been developed [ 37 ] . More relevant to the present study are computational frameworks that invert Bayesian models of perception and decision making to infer beliefs and costs of the agent . While this literature is extensive , exemplary studies for trial-based actions include cognitive tomography [ 38 ] and inference of the cost function in sensorimotor learning [ 39 ] . Extensions to sequential tasks have also been proposed , particularly considering active perception [ 40 , 41 ] , real-world behavior [ 42 , 43 ] , and general formulations [ 44 , 45 ] . Very similarly , work on Bayesian theory of mind uses highly related computational models , which also allow for the subjective beliefs of the agent to be different from the observer ’ s [ 46 , 47 ] . In the context of inverse optimal control , related work has considered different problem settings , e.g. , deterministic MDPs without additive noise and full observability [ 48 ] . More specifically in the LQR and LQG domain , [ 21 ] is concerned with finding the cost matrices and noise covariances given a known system with controller and Kalman gain . Similarly , [ 49 ] infers the above-mentioned cost matrices in the LQG setting based on observations but additionally takes constraints into account . The extension by [ 23 ] infers the terminal cost and the state cost function together with an exponential discount factor in the LQR setting . A different line of work has been concerned with estimating the dynamics , state sequence , and delay of internal LQG models from neural population activity [ 22 , 50 ] under the assumption that an agent might not know the dynamics , e.g. , of a brain-computer interface . Other work [ 51 ] is concerned with learning a control policy from states , observations , and controls . Our approach is different from previous research in two important ways . First , while most other approaches take different quantities such as the filter or controller gains or the agent ’ s observations or controls as given , we consider the setting that is typical in a behavioral experiment : The filter and controller as well as the agent ’ s observations are internal to the experimental subject and can not be observed . Instead , we assume that only trajectories x1 : T are observed . Second , other approaches to inverse optimal control in the LQG setting do not involve signal-dependent noise , which we address in this paper . 2 Background : LQG with sensorimorotor noise characterisitics . We model a human subject as an agent in a partially-observable environment as introduced by Todorov [ 7 ] and depicted in Fig . 1 A . For this we consider a discrete-time linear dynamical system with state xt ∈ Rm and control ut ∈ Rp with both control-independent and control-dependent noises xt+1 = Axt +But + V ξt + c∑ i=1 εitCiut . ( 1 ) The noise terms ξt ∈ Rm and it ∈ R are standard Gaussian random vectors and variables , respectively , resulting in control-independent noise with covariance V V T and control-dependent noise having covariance ∑ i Ciutu T t C T i . The agent receives an observation yt ∈ Rk from the observation model yt = Hxt +Wωt + d∑ i=1 itDixt . ( 2 ) The noise terms ωt ∈ Rk and εt ∈ R are again standard Gaussian , so that the covariance of the state-independent observation noise is WWT , while for the state-dependent observation noise it is∑ iDixtx T t D T i . All matrices of the linear dynamical system can in principle be time-varying , but we leave out the time indices for notational simplicity . The objective of the agent is to choose ut to minimize a quadratic cost function , J ( u1 : T ) = E x1 : T [ T∑ t=1 xTt Qtxt + u T t Rtut ] . ( 3 ) While the original LQG problem without control- and state-dependent noises can be solved exactly by determining an optimal linear filter and controller independently [ 5 ] , this separation principle is no longer applicable in the case considered here . Todorov [ 7 ] introduced an approximate solution method in which the optimal filters Kt and controllers Lt are iteratively determined in an alternating fashion , leaving the respective other one constant . The resulting optimal filter which minimizes the expected cost , is of the form x̃t+1 = Ax̃t +But +Kt ( yt −Hx̃t ) + Eηt , ( 4 ) where ηt is a standard Gaussian random vector and represents internal estimation noise . The optimal linear control law can be formulated as ut = −Ltx̃t . ( 5 ) The equations for determining the matrices Lt and Kt are given in Appendix B . For a detailed derivation the reader is referred to [ 7 ] . 3 Inverse optimal control . In this paper , we consider the inverse problem , i.e. , we observe an agent who is acting optimally in an agent-experiment loop ( Fig . 1 B ) according to the model of Section 2 , and want to infer properties of the agent ’ s perceptual and action processes , which are represented by parameters θ . In the examples in this paper , we have treated all matrices except the subjective control costs ( R ) and parameters of the task objective ( Q ) as given . This choice is motivated by the fact that the cost function is usually the least understood quantity in a behavioral experiment , while sensorimotor researchers often have quite accurate models for the dynamics in the tasks they are studying and for subjects ’ noise characteristics . In principle , however , our probabilistic formulation of the inverse optimal control problem allows inferring parameters θ of any of the matrices of the system by evaluating the likelihood function w.r.t . those parameters . Given a set of N independent trajectories { x1 : T } i=1 : N , each of length T , we can infer θ by maximizing the product of their likelihoods p ( x1 : T | θ ) , each decomposing as p ( x1 : T | θ ) = pθ ( x1 ) T−1∏ t=1 pθ ( xt+1 | x1 : t ) . ( 6 ) In the following , we drop the explicit dependency of the parameters θ . The graphical model from the agent ’ s point of view ( Fig . 1 A ) is structurally identical to that from the experimenter ’ s perspective ( Fig . 1 C ) . But , since we as experimenter observe the true states x1 : T instead of the agent ’ s noisy observations y1 : T , the usual Markov property does not hold and each xt generally depends on all previous states x1 : t−1 via the agent ’ s estimates and actions . To efficiently compute the likelihood factors p ( xt |x1 : t−1 ) , we track our belief about the agent ’ s belief p ( x̃t |x1 : t ) , which gives a sufficient statistic for the history . This approach allows propagating our uncertainty about the agent ’ s beliefs and actions over time and estimating the agent ’ s belief . To compute the likelihood function for some value of θ , we first determine the control and filter gains Lt and Kt using the iterative method introduced by Todorov [ 7 ] . We then compute an approximate likelihood factor p ( xt | x1 : t−1 ) for each time step in the following way ( see Algorithm 1 ) : First , we determine the distribution p ( xt+1 , x̃t+1 | xt , x̃t ) , which describes the joint evolution of xt and x̃t ( Section 3.1 ) . Second , we combine it with the belief distribution p ( x̃t | x1 : t ) , yielding p ( xt+1 , x̃t+1 | x1 : t ) ( Section 3.2 ) . As this step can not be done in closed-form due to the signaldependent noise , we introduce a Gaussian approximation of this quantity . Third , marginalizing over x̃t+1 gives the desired likelihood factor p ( xt+1 |x1 : t ) , while conditioning on the observed true states xt+1 gives the statistic of the history p ( x̃t+1 | x1 : t+1 ) , which we use for computing the likelihood factor of the following time step . In Section 3.3 , we extend this procedure to the setting where the state xt is only partially observed as a noisy linearly transformed version ot ( Fig . 1 D ) . Algorithm 1 : Approximate Likelihood Computation Result : Approximate log likelihood of parameters θ Input : Parameters θ , Data { xi1 : T } i=1 : N , Model Lt , Kt ← Approximate optimal controller and filter using the method of Todorov [ 7 ] ; initialize p ( x̃0 | x0 ) as the experimenter ’ s initial belief of the agent ’ s belief ; for each trajectory x1 : T from { xi1 : T } i=1 : N do for t← 0 to T − 1 do Compute p ( xt+1 , x̃t+1 | x1 : t ) using Eq . ( 10 ) ; Marginalize over x̃t+1 to get p ( xt+1 | x1 : t ) ; Condition on xt+1 to get p ( x̃t+1 | x1 : t+1 ) using Eq . ( 11 ) ; end end return ∑N i=1 log p ( x i 1 ) + ∑T t=2 log p ( x i t | xi1 : t−1 )

Summary: This paper formulates an inverse optimal control solution for a control system with partial observability and noise properties characteristic of the human sensorimotor system - that is, control signal-dependent motor noise, and state-dependent observation noise. The contribution is that this formulation can enable an experimenter to estimate parameters of the cost function of a control system (e.g. a human’s motor control system) under assumptions that that control system acts optimally, has full knowledge of the system dynamics (an accurate internal model), and has a functional form similar to that of the proposed formulation (a variant of the LQG case). Through a series of experiments comparing their inverse model to simulated reaching data, real macaque reaching data, simulated saccadic eye movements, and simple non-biological control problems, the authors show that their method can accurately estimate unknown cost function parameters to reverse-engineer an approximate model of the control system in question.