Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System

1 Introduction . Computational level theories of behavior strive to answer the questions , why a system behaves the way it does and what the goal of the system ’ s computations is . Such goals can be formalized based on the reward hypothesis . In the words of Richard Sutton , the reward hypothesis assumes , that “ goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal ” [ 1 ] . Thus , to understand human sensorimotor behavior , it is essential to characterize its goals and purposes quantitatively in terms of costs and benefits . But particularly in every-day tasks , the cost and benefits underlying behavior are unknown . Stochastic optimal control allows formulating behavioral goals in terms of a cost function for tasks involving sequential actions under action variability , uncertainty in the internal model , and delayed rewards . The solution to the optimization problem entailed in the cost function is a sequence of ∗Both authors contributed equally to this paper . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) actions , which is then compared to human movements . While early models optimized costs related to deterministic kinematics of a movement to a target [ 2 ] , task goals were subsequently formalized as costs on the variance of stochastic movements ’ endpoint distances to a goal target [ 3 ] . Importantly , although [ 3 ] considered open-loop control , it revealed the importance of modeling the specific variability of human movements , which increases linearly with the magnitude of the control signal [ 4 ] . As the neuronal control signal increases , so does its variability , leading to optimal movements trading off between achieving task goals and reducing the expected impact of movement variability . Including sensory feedback in stochastic optimal control leads to a computationally much more intricate problem , which can be formulated as a partially observable Markov decision process ( POMDP ) and is intractable in general . One of the few tractable cases is the linear-quadratic Gaussian ( LQG ) setting , where dynamics are linear , costs are quadratic , and variability is additive and Gaussian , leading to sensory inference of the state and control to be decoupled [ 5 ] . However , not only is human movement variability signal dependent , but additionally the uncertainty of sensory signals increases linearly with the magnitude of the stimulus , a phenomenon known as Weber ’ s law [ 6 ] . Todorov [ 7 ] extended the LQG case by introducing stochastic optimal feedback control with signaldependent noise , which allows the specification of noise models in line with what is known about the human sensorimotor system . This model [ 8 ] has been able to explain a broad range of phenomena in sensorimotor control [ 9 , 10 ] , including linear movement trajectories , smooth velocity profiles , speed-accuracy tradeoffs , and corrections of errors only if they influence attaining the behavioral goal . Particularly incorporating signal-dependent noise has been crucial in explaining experimental data , ranging from how corrections of movements during action execution depend on feedback and task goals [ 11 ] , that movements consider sensory uncertainty and temporal delays in real-time [ 12 ] , that movement plans in novel environments are reoptimized based on the learning of internal models to minimize implicit motor costs and maximize rewards [ 13 ] , and many others [ 14 , 15 , 16 , 17 ] . Explaining human behavior in these studies usually starts by hypothesizing the cost function describing a task , obtaining the optimal feedback controller , and comparing simulated trajectories to those observed experimentally . This line of inquiry , therefore , utilizes similarity of trajectories to quantify the degree of optimality in human behaviors . In some cases [ 14 ] , trajectories are simulated from the model to check for robustness with respect to changes in the model parameters . If our goal is to use optimal control models to infer such quantities , which often can not be measured independently , from behavior , it would instead be desirable to invert the problem and find those parameter settings which are consistent with the observed trajectories . While such inverse methods have been developed both in the field of reinforcement learning to infer the rewards being optimized by an agent [ 18 , 19 , 20 ] as well as in optimal control for the LQG case [ 21 , 22 , 23 ] , this is currently not possible for the noise characteristics of the human sensorimotor system . Here , we introduce a probabilistic formulation of inverse optimal feedback control under signaldependent noise in the tradition of rational analysis [ 24 , 25 ] . Our starting point is the forward problem introduced in [ 8 ] . We formulate the inference problem faced by an agent as a POMDP and distinguish it from the inference problem of an experimenter observing the agent . We proceed by deriving the likelihood of a sequence of observed states and provide an approximation to the non-Gaussian uncertainty due to the signal-dependent noise . First , this allows recovering the cost function underlying the agent ’ s behavior from observed behavioral data . Second , we extend the inference of the cost function to the case in which the state variables are only partially observable to the experimenter , e.g. , when only measuring the position of the agent ’ s movement . Third , we show through simulated and experimental data that the cost functions can indeed be recovered . Fourth , the probabilistic formulation allows recovering the agent ’ s belief during the experiment as well as the experimenter ’ s uncertainty about the inferred belief . Related work Inferring the cost functions underlying an agent ’ s behavior has long been of interest in different scientific fields ranging from economics [ 26 ] and psychology [ 27 ] to neuroscience [ 28 ] and artificial intelligence , particularly reinforcement learning [ 18 , 19 , 29 , 20 ] . Inverse Reinforcement Learning ( IRL ) specifically addresses the question of inferring the cost function being optimized [ 18 , 19 , 29 ] or approximately optimized [ 30 ] by an agent , with more recent approaches employing deep neural networks [ 31 , 32 ] . Some work has particularly addressed sensorimotor behavior [ 33 , 34 , 35 , 36 ] and extensions to the partially observable setting have been developed [ 37 ] . More relevant to the present study are computational frameworks that invert Bayesian models of perception and decision making to infer beliefs and costs of the agent . While this literature is extensive , exemplary studies for trial-based actions include cognitive tomography [ 38 ] and inference of the cost function in sensorimotor learning [ 39 ] . Extensions to sequential tasks have also been proposed , particularly considering active perception [ 40 , 41 ] , real-world behavior [ 42 , 43 ] , and general formulations [ 44 , 45 ] . Very similarly , work on Bayesian theory of mind uses highly related computational models , which also allow for the subjective beliefs of the agent to be different from the observer ’ s [ 46 , 47 ] . In the context of inverse optimal control , related work has considered different problem settings , e.g. , deterministic MDPs without additive noise and full observability [ 48 ] . More specifically in the LQR and LQG domain , [ 21 ] is concerned with finding the cost matrices and noise covariances given a known system with controller and Kalman gain . Similarly , [ 49 ] infers the above-mentioned cost matrices in the LQG setting based on observations but additionally takes constraints into account . The extension by [ 23 ] infers the terminal cost and the state cost function together with an exponential discount factor in the LQR setting . A different line of work has been concerned with estimating the dynamics , state sequence , and delay of internal LQG models from neural population activity [ 22 , 50 ] under the assumption that an agent might not know the dynamics , e.g. , of a brain-computer interface . Other work [ 51 ] is concerned with learning a control policy from states , observations , and controls . Our approach is different from previous research in two important ways . First , while most other approaches take different quantities such as the filter or controller gains or the agent ’ s observations or controls as given , we consider the setting that is typical in a behavioral experiment : The filter and controller as well as the agent ’ s observations are internal to the experimental subject and can not be observed . Instead , we assume that only trajectories x1 : T are observed . Second , other approaches to inverse optimal control in the LQG setting do not involve signal-dependent noise , which we address in this paper . 2 Background : LQG with sensorimorotor noise characterisitics . We model a human subject as an agent in a partially-observable environment as introduced by Todorov [ 7 ] and depicted in Fig . 1 A . For this we consider a discrete-time linear dynamical system with state xt ∈ Rm and control ut ∈ Rp with both control-independent and control-dependent noises xt+1 = Axt +But + V ξt + c∑ i=1 εitCiut . ( 1 ) The noise terms ξt ∈ Rm and it ∈ R are standard Gaussian random vectors and variables , respectively , resulting in control-independent noise with covariance V V T and control-dependent noise having covariance ∑ i Ciutu T t C T i . The agent receives an observation yt ∈ Rk from the observation model yt = Hxt +Wωt + d∑ i=1 itDixt . ( 2 ) The noise terms ωt ∈ Rk and εt ∈ R are again standard Gaussian , so that the covariance of the state-independent observation noise is WWT , while for the state-dependent observation noise it is∑ iDixtx T t D T i . All matrices of the linear dynamical system can in principle be time-varying , but we leave out the time indices for notational simplicity . The objective of the agent is to choose ut to minimize a quadratic cost function , J ( u1 : T ) = E x1 : T [ T∑ t=1 xTt Qtxt + u T t Rtut ] . ( 3 ) While the original LQG problem without control- and state-dependent noises can be solved exactly by determining an optimal linear filter and controller independently [ 5 ] , this separation principle is no longer applicable in the case considered here . Todorov [ 7 ] introduced an approximate solution method in which the optimal filters Kt and controllers Lt are iteratively determined in an alternating fashion , leaving the respective other one constant . The resulting optimal filter which minimizes the expected cost , is of the form x̃t+1 = Ax̃t +But +Kt ( yt −Hx̃t ) + Eηt , ( 4 ) where ηt is a standard Gaussian random vector and represents internal estimation noise . The optimal linear control law can be formulated as ut = −Ltx̃t . ( 5 ) The equations for determining the matrices Lt and Kt are given in Appendix B . For a detailed derivation the reader is referred to [ 7 ] . 3 Inverse optimal control . In this paper , we consider the inverse problem , i.e. , we observe an agent who is acting optimally in an agent-experiment loop ( Fig . 1 B ) according to the model of Section 2 , and want to infer properties of the agent ’ s perceptual and action processes , which are represented by parameters θ . In the examples in this paper , we have treated all matrices except the subjective control costs ( R ) and parameters of the task objective ( Q ) as given . This choice is motivated by the fact that the cost function is usually the least understood quantity in a behavioral experiment , while sensorimotor researchers often have quite accurate models for the dynamics in the tasks they are studying and for subjects ’ noise characteristics . In principle , however , our probabilistic formulation of the inverse optimal control problem allows inferring parameters θ of any of the matrices of the system by evaluating the likelihood function w.r.t . those parameters . Given a set of N independent trajectories { x1 : T } i=1 : N , each of length T , we can infer θ by maximizing the product of their likelihoods p ( x1 : T | θ ) , each decomposing as p ( x1 : T | θ ) = pθ ( x1 ) T−1∏ t=1 pθ ( xt+1 | x1 : t ) . ( 6 ) In the following , we drop the explicit dependency of the parameters θ . The graphical model from the agent ’ s point of view ( Fig . 1 A ) is structurally identical to that from the experimenter ’ s perspective ( Fig . 1 C ) . But , since we as experimenter observe the true states x1 : T instead of the agent ’ s noisy observations y1 : T , the usual Markov property does not hold and each xt generally depends on all previous states x1 : t−1 via the agent ’ s estimates and actions . To efficiently compute the likelihood factors p ( xt |x1 : t−1 ) , we track our belief about the agent ’ s belief p ( x̃t |x1 : t ) , which gives a sufficient statistic for the history . This approach allows propagating our uncertainty about the agent ’ s beliefs and actions over time and estimating the agent ’ s belief . To compute the likelihood function for some value of θ , we first determine the control and filter gains Lt and Kt using the iterative method introduced by Todorov [ 7 ] . We then compute an approximate likelihood factor p ( xt | x1 : t−1 ) for each time step in the following way ( see Algorithm 1 ) : First , we determine the distribution p ( xt+1 , x̃t+1 | xt , x̃t ) , which describes the joint evolution of xt and x̃t ( Section 3.1 ) . Second , we combine it with the belief distribution p ( x̃t | x1 : t ) , yielding p ( xt+1 , x̃t+1 | x1 : t ) ( Section 3.2 ) . As this step can not be done in closed-form due to the signaldependent noise , we introduce a Gaussian approximation of this quantity . Third , marginalizing over x̃t+1 gives the desired likelihood factor p ( xt+1 |x1 : t ) , while conditioning on the observed true states xt+1 gives the statistic of the history p ( x̃t+1 | x1 : t+1 ) , which we use for computing the likelihood factor of the following time step . In Section 3.3 , we extend this procedure to the setting where the state xt is only partially observed as a noisy linearly transformed version ot ( Fig . 1 D ) . Algorithm 1 : Approximate Likelihood Computation Result : Approximate log likelihood of parameters θ Input : Parameters θ , Data { xi1 : T } i=1 : N , Model Lt , Kt ← Approximate optimal controller and filter using the method of Todorov [ 7 ] ; initialize p ( x̃0 | x0 ) as the experimenter ’ s initial belief of the agent ’ s belief ; for each trajectory x1 : T from { xi1 : T } i=1 : N do for t← 0 to T − 1 do Compute p ( xt+1 , x̃t+1 | x1 : t ) using Eq . ( 10 ) ; Marginalize over x̃t+1 to get p ( xt+1 | x1 : t ) ; Condition on xt+1 to get p ( x̃t+1 | x1 : t+1 ) using Eq . ( 11 ) ; end end return ∑N i=1 log p ( x i 1 ) + ∑T t=2 log p ( x i t | xi1 : t−1 )

Summary: This paper presents a theoretical framework for cost function estimation for linear, quadratic, Gaussian systems with action- and state-dependent noises.  Algorithms for estimating the probabilities of latent variables and observed signals are derived for both the cases of complete and partial observation of the task variables by the experimenter. Then a gradient-free optimizer BOYAQA is used to obtain the maximum likelihood estimate of the parameters of the objective function. The performance was examined with 1D reaching simulation and real arm reaching and saccadic eye movement experimental data.