Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System
Authors: Matthias Schultheis
Dominik Straub
Constantin A. Rothkopf
Venue: NeurIPS 2021 Poster (2021)
Keywords: motor control
optimal feedback control
inverse optimal control
signal-dependent noise

Abstract: Computational level explanations based on optimal feedback control with signal-dependent noise have been able to account for a vast array of phenomena in human sensorimotor behavior. However, commonly a cost function needs to be assumed for a task and the optimality of human behavior is evaluated by comparing observed and predicted trajectories. Here, we introduce inverse optimal control with signal-dependent noise, which allows inferring the cost function from observed behavior. To do so, we formalize the problem as a partially observable Markov decision process and distinguish between the agent’s and the experimenter’s inference problems.  Specifically, we derive a probabilistic formulation of the evolution of states and belief states and an approximation to the propagation equation in the linear-quadratic Gaussian problem with signal-dependent noise. We extend the model to the case of partial observability of state variables from the point of view of the experimenter. We show the feasibility of the approach through validation on synthetic data and application to experimental data. Our approach enables recovering the costs and benefits implicit in human sequential sensorimotor behavior, thereby reconciling normative and descriptive approaches in a computational framework.

Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System
Matthias Schultheis'
Dominík Straub'
Centre for Cognitive Science
Centre for Cognitive Science
Technical University of Darmstadt
Technical University Darmstadt
Darmstadt, Germany
Darmstadt, Germany
matthias.schultheis duddarmstadd.de
Constantin A. Rothkopf
Centre for Cognitive Science
Technical University of Darmstadt Darmstadt, Germany
constantin. resttanin.rothkoo@@@eee hetpaain..ookkoppe@uudustttttttt
Abstract
Computational level explanations based on optimal feedback control with signal- dependent noise have been able to account for vast array of phenomena in human sensorimotor behavior. However, commonly a cost function needs to be assumed for a task and the optimality of human behavior is evaluated by comparing observed and predicted trajectories. Here, we introduce inverse optimal control with signal- dependent noise, which allows inferring the function from observed behavior. To do so, we formalize the problem as partially observable Markov decision pro- cess and distinguish between the agent's and the experimenter's inference problems. Specifically, we derive a probabilistic formulation of the evolution of states and be- lief states and an approximation the propagation equation in the quaadaiq Gaussian problem with signal-depender noise. We extend the model to the case of partial observability of state variables from the point of view of the experimenter. We show the feasibility of the approach through validation on synthetic data and application to experimental data. Our approach enables recovering the costs and benefits implicit in human sequential sensorimotor behavior, thereby reconciling normative and descriptive approaches computational framework.
1 Introduction
Computational level theories of behavior strive to answer the questions, why a system behaves the way does and what the goal of the system's computations Such goals can be formalized based on the reward hypothesis. In the words of Richard Sutton, the reward hypothesis assumes, that "goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of received scalar signal" [1]. Thus, to understand human sensorimotor behavior, it is essential to characterize its goals and purposes quantitatively in terms of costs and benefits. But particularly in every-day tasks, the cost and benefits underlying behavior are unknown.
Stochastic optimal control allows formulating behavioral goals in terms of function for tasks involving sequential actions under action variability, uncertainty in the internal model, and delayed rewards. The solution to the optimization problem entailed in the cost function is sequence of
Both authors contributed equally this paper.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

actions, which is then compared to human movements. While early models optimized costs related to deterministic kinematics of movement to a target [2]. task goals were subsequently formalized as costs on the variance of stochastic movements' endpoint distances to a goal target [3]. Importantly, although [3] considered open-loop control, revealed the importance of modeling the specific variability of human movements, which increases linearly with the magnitude of the control signal [4]. As the neuronal control signal increases, so does its variability, leading to optimal movements trading off between achieving task goals and reducing the expected impact of movement variability,
Including sensory feedback in stochastic optimal control leads to a computationally much more intricate problem, which can be formulated as a partially observable Markov decision process (POMDP) and is intractable in general. One of the few tractable cases is the anear-quadratic Gaussian (LQG) setting, where dynamics are linear, costs are quadratic. and variability is additive and Gaussian, leading to sensory inference of the state and control to be decoupled [5]. However, not only is human movement variability signal dependent, but additionally the uncertainty of sensory signals increases linearly with the magnitude of the stimulus, a phenomenon known as Weber's s law [6].
Todorov [7] extended the LQG case by introducing stochastic optimal feedback control with signal- dependent noise, which allows the specification of noise models in line with what known about the human sensorimotor system. This model [8] has been able to explain a broad range of phenomena in sensorimotor control [9, 10], including linear movement trajectories, smooth velocity profiles. speed-accuracy tradeoffs, and corrections of errors only if they influence attaining the behavioral goal. Particularly incorporating
noise has been crucial in explaining experimental data, ranging from how corrections of movements during action execution depend on feedback and task goals [11], that movements consider sensory uncertainty and temporal delays in real-time [12], that movement plans in novel environments are reoptimized based on the learning of internal models to minimize implicit motor costs and maximize rewards [13], and many others [14, 15, 16, 17].
Explaining human behavior in these studies usually starts by hypothesizing the cost function describ- ing a task, obtaining the optimal feedback controller, and comparing simulated trajectories those observed experimentally. This line of inquiry, therefore, utilizes similarity of trajectories to quantify the degree of optimality in human behaviors. In some cases [14], trajectories are simulated from the model to check for robustness with respect to changes in the model parameters. If our goal is to use optimal control models to infer such quantities, which often cannot be measured independently, from behavior, it would instead be desirable to invert the problem and find those parameter settings which are consistent with the observed trajectories. While such inverse methods have been developed both in the field of reinforcement learning to infer the rewards being optimized by an agent [18, 19, 20] as well as optimal control for the LQG case [21, 22, 23), this is currently not possible for the noise characteristics of the human sensorimotor system.
Here, we introduce a probabilistic formulation of inverse optimal feedback control under signal- dependent noise in the tradition of rational analysis [24, 25]. Our starting point is the forward problem introduced in [8]. We formulate the inference problem faced by an agent as a POMDP and distinguish it from the inference problem of an experimenter observing the agent. We proceed by deriving the likelihood of sequence of observed states and provide an approximation to the non non-Gaussian uncertainty due to the signal-dependen noise. First, this allows recovering the cost function underlying the agent's behavior from observed behavioral data. Second, we extend the inference of the cost function to the case in which the state variables are only partially observable to the experimenter, e.g., when only measuring the position of the agent's 'sss movement. Third, we show through simulated and experimental data that the cost functions can indeed be recovered. Fourth, the probabilistic formulation allows recovering the agent's 's belief during the experiment as well as the experimenter's 's uncertainty about the inferred belief.
Related work
Inferring the cost functions underlying an agent's behavior has long been of interest in different
scientific fields ranging from economics [26] and psychology [27] to neuroscience [28] and artificial
intelligence, particularly reinforcement learning [18, 19, 29, 20]. Inverse Reinforcement Learning (IRL) specifically addresses the question of inferring the cost function being optimized [18, 19, 29] or approximately optimized [30] by an agent, with more recent approaches employing deep neural networks [31, 32]. Some work has particularly addressed sensorimotor behavior [33, 34, 35, 36) and extensions to the partially observable setting have been developed [37].
2

---

More relevant to the present study are computational frameworks that invert Bayesian models of perception and decision making infer beliefs and costs of the agent. While this literature is extensive, exemplary studies for trial-based actions include cognitive tomography [38] and inference of the cost function sensorimotor learning [39]. Extensions to sequential tasks have also been proposed, particularly considering active perception [40, 41], real-world behavior [42, 43], and general formulations [44, 45J. Very similarly, work on Bayesian theory of mind uses highly related computational models, which also allow for the subjective beliefs of the agent to be different from the observer's [46, 47J.
In the context of inverse optimal control, related work has considered different problem settings, e.g., deterministic MDPs without additive noise and full observability [48]. More specifically in the LQR and LQG domain, [21] is concerned with finding the cost matrices and noise covariances given a known system with controller and Kalman gain. Similarly, [49] infers the above-mentioned cost matrices in the LQG setting based on observations but additionally takes constraints into account. The extension by [23] infers the terminal cost and the state cost function together with an exponential discount factor in the LQR setting. A different line of work has been concerned with estimating the dynamics, state sequence, and delay of internal LQG models from neural population activity [22, 50] under the assumption that agent might not know the dynamics, e.g.. of a brain-computer interface. Other work [51] is concerned with learning a control policy from states, observations, and controls.
Our approach is different from previous research in two important ways. First, while most other approaches take different quantities such as the filter or controller gains or the agent's observations or controls as given, we consider the setting that is typical in a behavioral experiment: The filter and controller as well as the agent's observations are internal to the experimental subject and cannot be observed. Instead, we assume that only trajectories æ1.r are observed. Second, other approaches to inverse optimal control in the LQG setting do not involve signal-dependent noise, which we address in this paper.
2 Background: LQG with sensorimorotor noise characterisitics
We model human subject as an agent in a partially-observable environment as introduced by Todorov [7] and depicted in Fig. 1 A. For this we consider a discrete-time linear dynamical system with state 2; RT and control u, E I2" with both control-independent and control-dependent noises
Ft+1 = Axx + Bu: + VE. +
C,u,
(1)
The noise terms & RM and cj R are standard Gaussian random vectors and variables, respectively, resulting in control- -nneeeneend noise with covariance VVT and control-dependent noise having covariance E, Cuuu? CT. The agent receives observation y: RR. from the observation model
y: = Ha; + Ww + Σ & D..
(2)
The noise terms W, Re and E R are again standard Gaussian, so that the covariance of the iate-inneeendent observation noise is WWT, while for the state-dependent observation noise E, D DJ. All matrices of the linear dynamical system can in principle be time-varyee varyveva but we leave out the time indices for notational simplicity. The objective of the agent is to choose u, to minimize a quadratic cost function,
J (wnrr) = E FIT
Σ x Qux + ut R,us
While the original LQG problem without control- and state-dependent noises can be solved exactly by determining optimal linear filter and controller independently [5], this separation principle is no longer applicable in the case considered here. Todorov [7] introduced an approximate solution method in which the optimal filters K, and controllers L, iteratively determined in alternating fashion, leaving the respective other one constant. The resulting optimal filter which minimizes the expected cost, of the form
L = AĀ, + Bu, + K, H + ET::
(4)
3

---

A
B
observation
belief state
Agent
reward
action
MK
Experiment
siate
Agent's model
Agent-experiment loop
Experimenter's model
Experimenter's latent model
Figure 1: The agent-experimenter loop and its formalization POMDP together with the inference problems from the agent's point of view and from the experimenter's point of view.
where 7: is standard Gaussian random vector and represents internal estimation noise. The optimal linear control law can be formulated as
Ut = LL
(5)
The equations for determining the matrices Le and K2 are given in Appendix B. For a detailed derivation the reader is referred to [7].
3 Inverse optimal control
In this paper, we consider the inverse problem, i.e., we observe an agent who is acting optimally in an agent-experiment loop (Fig. B) according to the model of Section 2, and want infer properties of the agent's 's perceptual and action processes, which are represented by parameters θ. In the examples in this paper, we have treated all matrices except the subjective control costs (R) and parameters of the task objective (Q) as given. This choice is motivated by the fact that the cost function is usually the least understood quantity in behavioral experiment, while sensorimotor researchers often have quite accurate models for the dynamics in the tasks they are studying and for subjects' noise characteristics. In principle, however, our probabilistic formulation of the inverse optimal control problem allows inferring parameters 0 of any of the matrices of the system by evaluating the likelihood function w.r.t. those parameters. Given of N independent trajectories } I each of length T, we can infer 0 by maximizing the product of their likelihoods p(æ1:r ()), each decomposing as
T-1
p( 0) Po (3x1 II Po (#++1 .1:))
In the following, we drop the explicit dependency of the parameters 0. The graphical model from the agent's point of view (Fig. A) is structurally identical to that from the experimenter's perspective (Fig. C). But, since we as experimenter observe the true states #1.7 instead of the agent's 's noisy observations y1.7, the usual Markov property does not hold and each æ; generally depends on all previous states 21.t-1 via the agent's 's estimates and actions. To efficiently compute the likelihood factors p(æ; 11:1-1), we track our belief about the agent's belief p( æ1:t), which gives sufficient statistic for the history. This approach allows propagating our uncertainty about the agent's 's beliefs and actions over time and estimating the agent's belief.
To compute the likelihood function for some value of 0, we first determine the control and filter gains L, and K, using the iterative method introduced by Todorov [7]. We then compute an approximate likelihood factor p(æ; 1 for each time step in the following way (see Algorithm 1):
First, we determine the distribution p(#++1, Øe+1 Ee, æt), which describes the joint evolution of He and ã; (Section 3.1). Second, we combine it with the belief distribution p(ă, #1:)), yielding p( Lt+1 11:e) (Section 3.2). As this step cannot be done in closed-form due to the signal- dependent noise, we introduce a Gaussian approximation of this quantity. Third, marginalizing over ē(+1 gives the desired likelihood factor p (42411 T1:4), while conditioning the observed true states 2z+1 gives the statistic of the history p [+ 21:++) which we use for computing the likelihood factor of the following time step.
In Section 3.3, we extend this procedure to the setting where the state a, is only partially observed as noisy linearly transformed version o (Fig. 1 D).

---

Algorithm 1: Approximate Likelihood Computation
Result: Approximate log likelihood of parameters 0
Input: Parameters 9, Data (=irr)==::N Model
Lt, Kr f Approximate optimal controller and filter using the method Todorov [7];
initialize p(@o Φo) as the experimenter's initial belief of the agent's belief;
for each trajectory 21.r from { h-r:h do do
for f 0
Compute p (1 Æt+1 (1:1) using Eq. (10);
Marginalize over 0141 to get p( s1;e);
Condition on #++1 to get p(0++1 (1 using Eq. (11);
end end
return Ee ==
log logp(z))
Et=z p( x
3.1 Joint dynamics of states and estimates
In this section, derive the joint dynamics of states and estimates, specifying the distribution p(0141, ēt+1 Dt, )t To do so, we build on work by Van Den Berg et al. [52], who introduced this idea for the standard LQG case in the context of planning, and extend it to the model with state- and action-dependent noises as considered in Section 2. First, we substitute the control in the state update (I) with its law (5), giving
Ax BL, + VE,- Σ C,LL
Et+1
and rewrite the filter update equation (4) as Ør+1 (A - BL, + Ke(y:- HT,) + Evi
= -A- - BL, - K2 + K,Hæ, + K,W + KG Σ CDaa, + Em.
(8)
In the last equation, have again inserted the control law (5), then the observation model (2), and rearranged terms. Equations (7) and (8) give us a representation of æ, and ã, which only depends on states estimates from the previous time step. Stacking both equations together specifies the distribution p(œe+1, Øe+1 Lt, Tt), with
C+++ T1+1
A K.(H + Ei (D)))
-(B 1B+ L.
x A - BI K.H
D Zt
V
0
0
& Wi
=
+
K.W
E
?
d
(Fa M. Σ &D))ar + (f.
Σ
C,C,,. + G.S.,
(9)
where S ~ NO0 I). For a detailed definition of the matrices Fú, F., Me, M, see Appendix C.I.
3.2 Approximate propagation
We obtain the distribution p(æ;+1, 0 [E1:t) by propagating p( (æ1:t) through the joint dynamics model (9). But, since the latter involves a product Gaussian random variables & and ã,, the result- ing distribution is no longer Gaussian. To make likelihood computation tractable, we approximate it by a Gaussian using moment matching. This allows us to maintain an approximate Gaussian belief' about the agent's belief and gives
approximation of the likelihood function in Eq. (6).
First, we assume that our belief of the agent's belief time step given by a Gaussian distribution p(ă, æ1=e) = . (Maja, Σapx). To approximately propagate p(ž; x1:t) and the observation through Eq. (9), we compute the mean and variance the resulting distribution via moment matching
(see Appendix E) and obtain the approximation
[
[āz] P
Dun
Epz Σ
p(x T1+1 xrxe) ~ N/
put
E =
(10)
Ø++1
Din
5

---

with
Aut1 = Fr + Fipeales
Ee+1 Me Mc Dara D Mr + +E i=1
D
C.M, + DM of + + GG".
i=1
Marginalizing over Te41 gives approximation of the likelihood factor of time step & + 1, p( Ø1.t) s ((zz Ewx). On the other hand, conditioning observation a++1 gives the belief of the agent's belief for the following time step, p(āc+1 x1:++1) =N (palee Eggle with
Aaje = Pa + Egg S-1 Jaa (2 - pe), Enje = EBg Eep D-I Eus.
(11)
We initialize p(oo wo) with the initial belief of the agent.
3.3 Partial observability from the observer's point view
In practice, we often do not have access to the full state in the model, e.g., if there are unmeasured quantities of the physical world such as velocity and acceleration when using a tracking system which only provides measurements of position in time, or if we have latent variables in our model representing internal states of the observed agent. Furthermore, measurements might noisy, c.g., due to the use of imprecise tracking hardware. We therefore consider the case where the state is partially observable for both the agent, i.e., the subject in the experiment, and the observer, i.e., the experimenter. In this case, assume that the experimenter observes linear transformation R of the state 2, with additive Gaussian noise, i.e.,
O = Sa. + U@
(12)
where 0, is standard Gaussian random vector, resulting in the distribution p(o, N (Sx,, UUT). The resulting Bayesian network is shown in Fig. I D. We can again formulate joint dynamical system æt and ot with additional observations OG, resulting in
xx,)
T++1 T++1
A = K.(H + Ei DD)
-(B & A- BL, K.H -S(B E d.)
V
0
0
07
It _t
+
0
K,W
K2
W
0
1,++1
SA
SV
0
0
U]
(F M, Σ
e D.)ar, + (p, Σ e.C.M.) + G.G.,
(13)
with matrices F,, Ft. Mr. M, defined accordingly (for definitions see Appendix C.2). Note that this equation structurally the same as for the fully-observable case (Eq. (9)) and have overloaded the matrix definitions to highlight that both can be treated similarly.
The likelihood of an observed trajectory decomposes as p(o1. ()) (o(o)) IT=e Po 01:-11). For computing the factors p(o: 01:-11), we follow structurally the same steps as for the fully- observable case, but now p(z,,, Øt 01:)) serves sufficient statistic of the history: We first assume the distribution p(x,, H, (1..) be Gaussian distributed and approximately propagate through the joint dynamics model p(#++1, 2z++, Or+1 Lt, Z..) (Eq. (13)) by computing the mean and variance. We marginalize the resulting Gaussian approximation of p( # Or+1 @1 over 1 0x+1, yielding the likelihood factors p(0r+1 (1:)). On the other hand, conditioning
the observation or gives the history statistic p(z,, T; 01:)) for the following time step. All steps are very similar to the fully- oully-observable case, but a more detailed description is given in Appendix D.
3.4 Parameter inference
In the previous sections, we have provided an algorithm for computing an approximate likelihood of the parameters 0 given set of observed trajectories (æi-r)i=1.N- To determine the optimal parameters, we maximize the likelihood, giving us a point estimate OMLE of the true parameters. As one has to solve the control problem (determining L, and K2) by iterative procedure [7] for every

---

A Simulated data
B Likelihood function
C Balief tracking
posiioo
velocity
acceleration
position
10
5
167
.25
25
.25
NLL MLE True
time [s]
experise for's oésovallon aparte beler
estraate of boicc acoetracters Lezarrailyy
D Mean simulated data
E RRSE for pairs
F RMSE per parameter
G RMSE # trials
RMSE KL
position
RMSE
10
70"
RMSE
10"
0.08
22
time [s]
161
1071
number of trials
Figure 2: Validation on synthetic reaching data. A Simulated trajectories of the reaching task. B Negative log likelihood for different combinations of the parameters given data and the resp. third parameter. C Belief tracking: During a reaching movement, the agent has a belief about the position, velocity, and acceleration (green dotted lines). The experimenter observes noisy version of the agent's actual movements (blue) and computes an estimate of the agent's belief (orange dashed lines with shaded region representing 2 SD). D Mean trajectories with RMSE 0.016 (MLE), 0. 0.5, and 1.0. E RMSE of the MLEs for all parameters parameter). The colors are the same as the previous subplot. F Distribution over 10 repetitions of RMSE for each parameter across different values for the other parameters, G RMSE and KL divergences (between empirical distributions of true trajectories and simulated ones based on the MIEE) with 0.2 and 0.8 quantiles for different numbers of trajectories with parameters as in (A).
pairwise grids (for one value respective third
likelihood evaluation, computing gradients of the likelihood (although possible) not very efficient. We instead use log likelihood based on a quadratic approximation. Our implementation in jax [54] is available on github."
the robust gradient-free optimizer BOBYQA? [53], which minimizes the negative
Evaluation and applications
4.1 Validation on synthetic reaching data
We apply the introduced method to recover parameters in a single-joint reaching task with control- dependent noise and 5-dimensional state space (details in Appendix F.1). The goal is bring the hand to a target while minimizing control effort. The cost function has three parameters: (i) v. the cost of the velocity the final time step. (ii) f, the cost of the acceleration at the final timestep, (iii) the cost of actions at each timestep. Simulated data for the parameters r = 10 50-5, v = 0.2, f 0.02 are shown in Fig. Visual inspection of the likelihood function (Fig. B) shows that the maximum likelihood estimate (MLE) is very close to the true parameter values.
Once we have obtained the MLE, belief of the agent's 's belief p (1:t). As an example, we simulated = 20 trajectories {our) I:N with T 30 from partially observed version of the reaching task used above in which we only observe the position and treat velocity and acceleration as latent variables. Fig. 2 C shows our approximate belief about the agent's belief for the MLE parameters, together with the true agent's belief. Note that we can recover the agent's 's belief of state, velocity, and acceleration quite accurately from noisy observations of the position only.
perform belief tracking, i.e., computing our approximate
Python implementation under GNU GPI. available at PyPT httpp::/ggit thw.bb..ttttttt/////ee/eee/... http:////w...o...aaaaaaaaaaaaaaa
7

---

A Log likelihood
B Trajectories
V
position
velocity
acceleration
10"
10
Data
.05
.25
2.5
10
-2.5
10
10
10
10
10
Model
.25
2.5
16"
.05
NLL
10
0
-2.5
MLE
x
10"
25 time [s]
25
10
10
10'
time [s]
time [s]
Figure 3: Application to real reaching data. A Negative log likelihoods for the three model parameters. B Reaching trajectories (velocities and accelerations computed with finite differences) and simulated data from the model with MLE parameters.
For the evaluation of parameter estimates, we compute their root mean squared errors (RMSEs) in logarithmic space. The effect of estimation errors on the resulting trajectories is illustrated in Fig. 2 D, where we simulated trajectories in Fig. 2 A with different mean parameter errors. To show that method yields good parameter estimates over a range of different parameter settings, we perform maximum likelihood estimations (MLEs) of all three parameters for different true parameter values of which two were chosen from a pairwise grid while the third one was left as in Fig. 2 A. In this analysis, used 100 simulated trajectories and 10 repetitions each. In Fig. 2 E, which shows the resulting RMSEs for different combinations of true parameters on pairwise grids, we demonstrate that the RMSEs are small over wide range of parameter values. The RMSEs (Fig. F) across different values for the respective other two parameters and 10 repetitions were 2.4 x 10 (r), 2.1 x 10-² (v) and 3.1 10 (f). We compared the RMSES obtained by our estimation method to the ones of two baseline approaches. A first baseline is obtained by running a version of the algorithm without signal-dependent noise (i.e., using the basic LQG during inference). for which the likelihood can be evaluated exactly in closed-form. The results on the reaching problem (Section 4. in terms of RMSE of the parameter estimates are worse by roughly two orders of magnitude (RMSE of 1.766 vs 0.027). A second, stronger baseline is obtained by setting the additive noise in the standard LQG to the average noise magnitude of simulated trajectories. In this case, the RMSEs are still worse the
by roughly order of magnitude (RMSE of 0.702 0.027). Note that the information average noise level would not be readily available for real data without knowing the true parameters and therefore constitutes strong baseline. A plot visualizing the results of the basclines for each parameter is provided in Appendix G.1.1.
Finally, we investigated the influence of the number of samples by evaluating estimates of trajectories as in Fig. A for different numbers of trials (Fig. 2 G). As expected, more trials increase the accuracy of the estimates and lead to lower error. We estimated the convergence rate by fitting line to the log-log plot and obtained 0.78 for the RMSE. An analysis using the Kullback-Leibler divergence between the empirical trajectory distributions of the true and maximum likelihood parameters as an additional evaluation metric can be found in Appendix 1.1
We perform a similar analysis for generated partially observed trajectories in which we only observe the position and treat velocities, accelerations. and forces as latent variables. The results are qualita- tively very similar and are therefore presented in Appendix G.1.3. An additional empirical evaluation of the impact of the moment matching approximation is given in Appendix G.1.4
4.2 Application to real reaching movements
To show the applicability to real data, we apply our method to reaching trajectories from a previously published experiment [55], in which a rhesus monkey had to perform center-out reaching movements and hold its hand at the target to receive a reward. Since the data contains only measurements of position (velocity and acceleration are computed using finite differences). we use the partially observable version of the reaching model described in Section 1. treating velocity and acceleration as latent variables. The approximate likelihood functions with respective ML.Es of the three parameters are shown in Fig. A, indicating that can determine the parameter set for which the trajectories are
8

---

most likely. Fig. 3 shows the given trajectories together with simulations using our model with the MLE parameters. We observe that the inferred parameters produce simulated data that convincingly look like the real data and provides smooth estimates of the latent velocity and acceleration profiles.
4.3 Application to eye movements
We also apply our method to model of saccadic eye movements which presented by Creve- coeur and Kording [56]. This model captures fixating one's eyes an initial point and then performing saccade to fixate another point. A cost parameter (r) is used to trade off the cost of the movement and the deviation from the target. As this model is an LQG model with control- dependent noises, it directly allows the applica- tion our method for recovering the parameter. Fig. 4 A shows simulated trajectories represent- ing typical eye movements encountered in the experiments. The MLEs based simulated data for range of 20 different parameter val- ues (100 repetitions each) are shown in Fig. 4 B. Except for very few outliers, the estimated pa- rameters very close to the true parameters.
A Saccadic eye movements
B Parameter estimates
Angle
reddet
-10
Control
250
10
10'
10"
time [ms]
Figure 4: Saccadic eye movements. Generated trajectories. Median and quantiles of MLEs for a range of true parameter values of r.
4.4 Application to random problems
To demonstrate that the inference method works wide range of problems defined according to the model definition (Section 2), we evaluate it on randomly generated problems with 5-dimensional state-space space and two-dimensional action space. Detailed information on the generation procedure given in Appendix F.3. Each model has two parameters (ri and r2) which again represent cost of control effort in each of the two movement directions. For each model, sampled true value parameter 72 from a uniform random distribution and estimated both parameters jointly by maximizing the approximate likelihood. In Fig. 5 A show the errors for range of parameter values 71 for different random models. The median and quantiles for the results 2000 random problems shown in Fig. 5 B. One observe that the estimates are generally very close the true parameters. The results for the other parameter 72 are basically identical since the problem is symmetric w.r.t. the parameters, but include the results in Appendix G.2.
Conclusion
In this paper, we investigated the inverse optimal control problem under signal-dependent noise. We formalized the problem as POMDP and introduced a first method for inferring cost parameters of an agent in a quaara- control problem with signal-dependent noise. Numerical simulations show
A Individual random seed
B Aggregated
Seed
Seed
Seed
Seed
Seed 2000
10a
10
10
10"
16
107
10
Figure 5: Random problems. A MLEs for range of parameter values of r for different random problems and different values of r2 (color). Aggregated results (median, percentiles) for 2000 random problems.

---

that accurate inference of cost parameters given synthetic data is feasible in random control problems, simulated arm movements, and simulated eye movements. Additionally, the method can be used to probabilistically infer the belief of the considered agent. Furthermore, the method was applied to real data from macaque monkey performing reaching movements. The inferred parameters reproduce reaching data in simulation that convincingly agrees with the original data and provides smooth estimates of the latent velocity and acceleration profiles. More recent and more general methods for optimal control in high-dimensional continuous domains exist, but while some of these may provide interpretable non-linear features, they consider deterministic MDPs without noise and assume full observability [48], others relying function approximation through neural networks including GANs, are useful in engineering applications but may not provide a computational level explanation of behavior [31, 32]. Taken together, our method does not require designing cost function and testing for similarity of simulated trajectories with experimental data, but allows inferring the cost functions directly from behavior, thereby reconciling normative and descriptive approaches to human sensorimotor behavior.
Limitations and Future work The proposed algorithms are based on stochastic optimal feedback control with linear dynamics, quadratic cost functions, and signal-dependen noise. As such, the first limitation lies the restriction to problems with linear dynamics. An extension to non-linear dynamics could be achieved by linearizing the dynamics locally or more generally by using a framework that iteratively linearizes the dynamics at each time step, e.g., iLQG [57]. Similarly, while quadratic cost functions allow modeling wide range of costs and benefits within sensorimotor control, certain cost functions such as exponential discounting may be more cumbersome to accommodate.
While the presented method was able to recover cost functions in the considered problems, higher- dimensional parameter spaces will likely pose difficulties in finding unique point estimates of parameters. This problem could be addressed by using appropriate structured prior distributions over parameters. A fully Bayesian treatment could be realized by using Markov chain Monte Carlo involving our likelihood model. Further research should similarly investigate the limits of the Gaussian approximation of the likelihood. the cases considered here, the approximation of the likelihood function appeared to be unbiased up to an RMSE of approximately 3 x 10-2, however, approximating distributions by simpler ones may introduce systematic biases and noise. Possible extensions could resort to using particle filters albeit at higher computational cost.
Another issue regarding gigher-dimensionll parameter spaces is that evaluation of the likelihood function becomes quite expensive by determining the optimal controller and optimal filter iteratively. Even if this procedure takes only one second on a common PC for the reaching task, optimization in high-dimenn -igh--imensional spaces requires larger number of function evaluations, which renders inference costly. By relying on the iterative procedure, also becomes difficult to compute gradients, which may prevent the use of efficient gradient- based solvers. While for our applications a solver based on quadratic approximations was efficient, one could resort to Bayesian optimization.
Future work in the area of robotics may explore applying the inferred cost functions in the training of visuomotor policies [58, 59] in humanoid robots with reinforcement learning [60]. Possible applications include utilizing the inferred cost functions in apprenticeship learning [61, 62, 63]. in which a policy is learned from demonstrations of a potentially suboptimal demonstrator or teacher. Similarly, applications may also include transfer learning [64, 65, 66). in which learned policies or cost functions are transferred to related tasks.
Finally, characterizing individual human subjects by analyzing their behavior may in principle be used with negative societal impact. In the context of scientific investigations of human sensorimotor control within cognitive science and neuroscience, only anonymized behavioral data for the understanding of the human mind and brain are employed.
Acknowledgments and Disclosure of Funding
We acknowledge support from the Lichtenberg high performance computing facility of the TU Darmstadt, the projects 'The Adaptive Mind' and "The Third Wave of Al' funded by the Excellence Program and the project 'WhiteBox' funded by the Priority Program LOEWE of the Hessian Ministry of Higher Education, Science, Research and Art.
10

---

References
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[2] Tamar Flash and Neville Hogan. The coordination of arm movements: experimentally confirmed mathematical model. Journal of nearoscience, 617):1688-1733, 1985.
[3] Christopher Harris and Daniel M Wolpert. Signal-dependent noise determines motor planning. Nature, 394(6695):780-784, 1998.
[4] Richard A Schmidt, Howard Zelaznik, Brian Hawkins, James S Frank, and John T Quinn Jr. Motor-output variability: a theory for the accuracy of rapid motor acts. Psychological review, 86(5):415, 1979.
[5] Robert F Stengel. Optimal control and estimation. Courier Corporation, 1994.
[6] Gustav Theodor Fechner. Elemente der psychophysik, volume 2. Breitkopf Härtel, 1860.
[7] Emanuel Todorov. Stochastic optimal control and estimation methods adapted to the noise characteristies of the sensorimotor system. Neural computation, 17(5):1084 1108, 2005.
[8] Emanuel Todorov and Michael Jordan. Optimal feedback control as a theory of motor coordination. Nature neuroscience, ((11)):222-12::)) 2002.
[9] Reza Shadmehr and John Krakauer. A computational neuroanatomy for motor control. Experimental brain research, 185(3):359-38) 2008.
[10] David W Franklin and Daniel M Wolpert. Computational mechanisms of sensorimotor control. Neuron, 72(3):425-44,, 2011.
[11] Joseph Y Nashed, Frédéric Crevecocur, and Stephen H Scott. Influence of the behavioral goal and environmental obstacles on rapid foedback responses. Journal of neurephysiology. 108(4):999 1009, 2012.
[12] Frédéric Crevecoeur, Douglas Munoz, and Stephen Scott. Dynamic multisensory integration: so- matosensory speed trumps visual accuracy during feedback control. Journal of Neuroscience, 36(33): 8598-8611, 2016.
[13] Jun Izawa, Tushar Rane, Opher Donchin, and Reza Shadmehr. Motor adaptation as a process of reopti- mization. Journal of Neuroscience, 28(1 11:28333 2891, 2008
[14] Sang-Hoon Yeo, David W Franklin, and Daniel M Wolpert. When optimal feedback control is not enough: Feedforward strategies are required for optimal control with active sensing. PLoS computational biology. 12(12)e10051 190, 2016.
[15] Ame J Nagengast, Daniel A Braun, and Daniel M Wolpert. Risk-sensitive optimal feedback control accounts for sensorimotor behavior under uncertainty. PLoS computational biology, 6(7):c100085) 2010.
[16] Vincent Ethier. David Zee, and Reza Shadmehr: Changes in control of saccades during gain adaptation. Journal of Neuroscience, 139991139999-3399, 2008.
[17] Jöm Diedrichsen. Optimal tesk-dependent changes of bimanual feedback control and adaptation. Current Biology, 17(19):1675- 1679, 2007.
[18] Andrew Ng. Stuart Russell, and others. Algorithms for inverse reinforcement learning. Ieml, volume 1, page 2, 2000.
[19] Brian Ziebart, Andrew L Maas. J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement lcarning. In Aaai, volume 8, pages 1433-1438. Chicago. IL, USA, 2008.
[20] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49- 58. PMLR, 2016.
[21] M Cody Priess. quadratic gaussian control with application to biological systems analysis. Dynamic Systems and Control Conference, volume 46209, page V003T42A004. American Society Mechanical Engineers, 2014.
Jongeun Choi, and Clark Radcliffe. The inverse problem of continuous-time linear
[22] Matthew Golub, Steven Chase, and Byron Yu. Learning internal dynamics model from control demonstration. In International Conference Machine Learning, pages 606-616 PMLR, 2013.
11

---

[23] Haitham El-Hussieny and Jee-Hwan Ryu. Inverse discounted-based lqr algorithm for learning human movement behaviors. Applied Intelligence, 1500 2019.
[24] Herbert A Simon. A behavioral model of rational choice. The quarterly journal economics, 69(1): 99-118, 1955.
[25] John R Anderson. Is human cognition adaptive? Behavioral and Brain Sciences, 14())471-483, 1991.
[26] Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263-292, 1979.
[27] Frederick Mosteller and Philip Nogee. An experimental measurement of utility. Journal Political Economy, 59(5):371-404 1951.
[28] Matthew Chalk, Gasper Tkacik, and Olivier Marre. Inferring the function performed by a recurrent neural network. Plas one, 16(4))e0248940, 2021.
[29] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proceedings of the Fourleenth International Conference Artificial Intelligence and Statistics, pages 182-189, IMLR Workshop and Conference Proceedings, 2011.
[30] Constantin A Rothkopf and Christos Dimitrakakis. Preference elicitation and inverse reinforcement Icarning. In Joint European conference on machine learning and knowledge discovery in databases, pages 34-48. Springer, 2011.
[31] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference Learning Representations, 2018.
[32] Alex James Chan and Mihaela van der Schaar. Scalable hayesian inverse reinforcement learning. In International Conference on Learning Representations. 2020.
[33] Katja Mombaur, Anh Truong, and Jean-Pau Paun Laumond. From human to humanoid locomotion an inverse optimal control approach. Autonomous robots, 28(3):369-33) 383, 2010.
[34] Constantin A Rothkopf and Dana H Ballard, Modular inverse reinforcement learning for visuomotor behavior. Biological cybernetics, 107(4) 2013.
[35] Katharina Muelling, Abdeslam Boularias, Betty Mohler, Berhhard Schölkopf, and Jan Peters. Learning strategies in table tennis using inverse reinforcement learning. Biological cybernetics, 108(5)x603-19, 2014.
[36] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Where do you think you't re going? inferring beliefs about dynamics from behavior. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 1461-1472, 2018.
[37] Choi and Kee-Eung Kim. Inverse reinforcement learning partially observable environments. Journal of Machine Learning Research, 12:691-730, 2011.
[38] Neil MT Houlsby, Ferenc Huszár, Mohammad M Ghassemi, Gergö Orbán, Daniel M Wolpert, and Máté Lengyel. Cognitive tomography reveals complex, task-independent mental representations. Current Biology, 23(21):2169 2175, 2013.
[39] Konrad Paul Körding and Daniel M. Wolpert. The loss function of sensorimotor learning. Proceedings of the National Academy of Sciences, 101((6):8839-9))) 99922 2004.
[40] Sheeraz Ahmad and Angela Yu. Active sensing as bayes-optimal optimal sequential decision-makinn In Proceedings of the Mienty-Ninthh Conference on Uncertainty in Artificial Intelligence, pages 12 21, 2013.
[41] David Hoppe and Constantin A Rothkopf. Multi-step planning of eye movements in visual search. Scientific reports, 91):1-12, 2019.
[42] Boris Belousov, Gerhard Neumann, Constantin Rothkopf, and Jan R Peters. Catching hcuristics are optimal control policics. Advances in neural information processing systems, 29: 10:1122-4434 2016.
[43] Felix Schmitt, Inferring sensor and policy models of human worl-wrrd motor behavior. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Bieg, Michael Herman, and Constantin A Rothkopf. see what you see:
12

---

[44] Jean Daunizeau, Hanneke EM Den Ouden, Matthias Pessiglione, Stefan J Kiebel, Klaas E Stephan, and Karl J Friston. Observing the observer (i): meta-bayesian models of learning and decision-making.. PloS one, 15554,,15554 2010.
[45] Zhengwei Wu, Minhae Kwon, Saurabh Daptardar, Paul Schrater. and Xaq Pitkow. Rational thoughts in neural codes. Proceedings of the National Academy of Sciences, 117(47):29311 -293200 2020.
[46] Chris L. Baker, Rebecca Saxe, and Joshua B. Tenenbaum. Action understanding inverse planning. Cognition, 113(3):329-39, 2009, Publisher: Elsevier.
[47] Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. Online bayesian goal inference for boundedly rational planning agents, Advances in Neural Information Procexsing Systems, 33, 2020.
[48] Sergey Levine and Vladlen Koltun. Continuous inverse optimal control with locally optimal examples. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 482, 2012.
[49] Ahmed Ramadan, Jongeun Choi, and Clark Radcliffe. Inferring human subject motor control intent using inverse mpc. In 2016 American Control Conference (ACC), pages 5791-5796. IGEE, 2016.
[50] Matthew D Golub, M Yu Byron, and Steven M Chase. Internal models for interpreting neural population activity during sensorimotor control. Elife, 4: 2015.
[51] Xiangli Chen and Brian Ziebart. Predictive inverse optimal control for liner-quaadratic-gaussin systems. In Artificial Intelligence and Statistics, pages 165-173. PMLR, 2015.
[52] Jur Van Den Berg, Picter Abbeel, and Ken Goldberg. Lqg-mp: Optimized path planning for robots with motion uncertainty and imperfect state information. The International Journal of Robotics Research, 30 (7):895-913, 2011.
[53] Coralia Cartis, Jan Fiala, Benjamin Marteau, and I. Lindon Roberts. Improving the flexibility and robustness of model-based derivative-free optimization solvers. ACM Transactions on Mathematical Software (TOMS). 45(3): 1-1,,,442222000 2019.
[54] Roy Frostig, Matthew James Johnson, and Chris Leary. Compiling machine learning programs via high-level tracing. Systems for Machine Learning, 2018
[55] Robert D Flint, Erie W Lindherg, Luke R Jordan, Lee E Miller, and Mare W Slutzky. Accurate decoding of reaching movements from field potentials the absence of spikes. Journal of nearal engineering, 9(4): 046006, 2012.
[56] Frederic Crevecncur and Konrad Kording. Saccadic suppression as a perceptual consequence of efficient sensorimotor estimation. Elife. 6:e25073, 2017.
[S7] Emanuel Todorov and Weiwei Li. A generalized iterative lag method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pages 300-306. IEEE, 2005.
[58] Sergey Levine and Vladlen Koltun, Guided policy search. In International conference on machine learning, pages 1-9, PMLR, 2013.
[59] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-337, 2016.
[60] Peters, S Vijayakumar, and S Schaal. Reinforcement learning for humanoid robotics. In 3rd JEEE-RAS International Conference on Humanoid Robots (ICHR 2003), pages 1-20. VDI/DD-GMA, 2003.
[61] Pieter Abheel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Procee-
ings of the twenty- first international conference on Machine learning, page 1, 2004.
[62] Jomathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29:4565-4573, 2016.
[63] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbcel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 1112))1--79, 2018.
[64] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7). 2009.
13

---

[65] Alessandro Lazarie. Transfer in reinforcement learning: framework and survey. In Reinforcement Learning, pages 143-173. Springer, 2012.
[66] Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint 200.020009.77888 2020.
14


Reviews:
[{"review_id": "u0lCHoZndKI", "reviewer": "Reviewer_VUPJ", "summary": "The authors present a method for recovering cost functions from observed state trajectories through inverse optimal control with signal-dependent noise for linear systems and quadratic costs. \nThey formulate a POMDP and distinguish between the agent’s and the experimenter's inference problems since the agent does not have access to the experimenter's hidden states.\nThey derive the joint dynamics of the state and estimate by stacking the individual transition functions.\nThe resulting probabilistic belief transition function for the linear-quadratic Gaussian problem with signal-dependent noise is approximated through moment matching since it involves a product of two Gaussians (which does not result in a Gaussian).\nIn a similar fashion to stacking the agent and experimenter dynamics, the authors suggest additionally stacking the observer dynamics and solving the transition jointly for the case that they don’t have access to the full state.\nThe approach is validated on synthetic and experimental data and applied to human sequential sensorimotor behavior.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The problem is interesting and the solution is appropriate for a limited number of use cases.\n- A concise background and description of LQG. Although this is nearly 1:1 from [1] I believe it is necessary to follow the notation and problem setting.\n- Visualizations are intuitive and easy to understand\n- Focus on methods that increase interpretability of behavior\n- Good discussion of limitations in the last section\n\nWeaknesses:\n- It would have been good to more clearly state which matrices are given and what quantities are inferred.\nIntroducing an algorithm before section 3.1 would be helpful to make this more clear.\n- Similarly, it would have been good to detail more closely how K_t and L_t are iteratively determined.\n- I believe a more complete discussion about the impact of the moment matching assumption is needed. This could be done either theoretically or empirically through Monte-Carlo simulation and discussion of an intuitive example.\n- The related work section does a good job referring to previous work, but overall I feel like it is too dated. There have been many more recent works in this domain over the last 5 years.\n- “While more recent and more general methods for optimal control in high-dimensional continuous domains exist, they rely on function approximation methods, which is useful in engineering applications but may not provide a computational level explanation of behavior” I feel that this argument needs to be substantiated since similar approaches in the continuous inverse optimal control setting exist which could potentially be adapted to the signal-dependent noise setting [2]. These do not necessarily rely on NN function approximation, allow for interpretable non-linear features, and are computationally tractable.\n- Overall the applicability of the approach is very limited. While the authors hint at a more capable integration with iLQG, I had hoped to see this in this paper already.\n- The evaluation is nicely visualized but more than 1-dimensional experiments are needed. While the authors state that the approach does not scale to higher dimensions, I would have liked to see a dimensionality of at least 3.\n- The evaluation lacks comparison to any baselines. No ablations are done. Additionally to related work, it would be good to ablate how well the costs be inferred through more primitive versions of the algorithm (e.g. no noise assumption, no signal-dependent noise, etc.)\n\nOverall, I believe the work is interesting and timely but needs improvements, focusing on the evaluation. I would also like to encourage the authors to increase the scope and applicability of the work.\n\nTypos:\n“an which in general is intractable”\n“requires a progressively larger number function evaluations”\n\n[1] Todorov, Emanuel. \"Stochastic optimal control and estimation methods adapted to the noise characteristics of the sensorimotor system.\" Neural computation 17.5 (2005): 1084-1108.\n\n[2] Levine, Sergey, and Vladlen Koltun. \"Continuous inverse optimal control with locally optimal examples.\" arXiv preprint arXiv:1206.4617 (2012).\n\nEDIT: I thank the authors for addressing my concerns with respect to the impact of the moment matching approximation and incorporating two baselines. Nonetheless, I believe that using a single experiment is not sufficient for this conference. Due to the limited experiments and limited applicability of the method (without a simple iLQG integration, which should not be a huge effort), I believe this paper needs additional work and would like to keep the current rating as is.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "nPo5SLCvR_f", "reviewer": "Reviewer_9TgB", "summary": "This paper formulates an inverse optimal control solution for a control system with partial observability and noise properties characteristic of the human sensorimotor system - that is, control signal-dependent motor noise, and state-dependent observation noise. The contribution is that this formulation can enable an experimenter to estimate parameters of the cost function of a control system (e.g. a human’s motor control system) under assumptions that that control system acts optimally, has full knowledge of the system dynamics (an accurate internal model), and has a functional form similar to that of the proposed formulation (a variant of the LQG case). Through a series of experiments comparing their inverse model to simulated reaching data, real macaque reaching data, simulated saccadic eye movements, and simple non-biological control problems, the authors show that their method can accurately estimate unknown cost function parameters to reverse-engineer an approximate model of the control system in question.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper takes a previous formulation by Todorov (2007) of the control problem faced by an agent with noise properties similar to that of the human sensorimotor system (and who acts optimally under full knowledge of its own dynamical system), and instead considers the inverted problem of how to discover the cost function parameters of this agent. The authors use probabilistic methods to derive this formulation and provide an algorithm for computing the approximate likelihood of the control parameters. The main contribution is thus to combine inverse optimal control in the LQG setting with the signal-dependent sensory and motor noise modelled by Todorov.\nThe authors validate that their method works in a series of well conceived experiments using primate reaching and saccade movement simulators, real primate psychophysical reaching data (previously published) as well as a classic control problem. The method does seem to work well, based on the data presented.\nThe surrounding literature seems well reviewed, and I agree that this method could in theory be applied to estimating control properties in psychophysical motor experiments, if of course, the objective is still to estimate what trade-offs a human or non-human primate is making when they move a single part of their body in a highly constrained environment (under the assumption of optimality, and that the agent controls using the same state-space basis as the model assumes). Psychology and neuroscience experiments moving towards more naturalistic movements and environments will pose a major problem for this form of control work, as the state-space grows and becomes more difficult to explicitly specify. \nThe point of modelling a separation between the agent’s beliefs and those of the experimenter was a little lost on me, as it’s not clear to me what exact scientific application this could have.\n\n*EDIT:*\nI thank the authors for clarifying their motivation for modelling a separation between the agent's and experimenter's beliefs, and on reflection I agree this contribution is a valuable one. I have increased my overall score accordingly, and would recommend that the authors attempt to make the points they have listed here in their response clearer in the paper if possible. Thanks also for the clear response to my note about the limitations of these traditional frameworks for assessing motor behaviour. I agree that acknowledging this point in the limitations of the paper would be helpful for readers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "H9v_nxOycVj", "reviewer": "Reviewer_QXkU", "summary": "This paper presents an algorithm to infer cost parameters of LQG problems given trajectories. The effectiveness of the algorithm is demonstrated on simulated arm and eye movements as well as random control problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n\n1. The proposed method is demonstrated on a wide range of tasks, including real world dataset of a rhesus monkey reaching for objects.\n\n2. The proposed method is able to infer the parameters under partially observed trajectories.\n\n3. Latent velocity and acceleration can also be estimated using the proposed approach.\n\nCon:\n\n1. Only linear dynamics and quadratic const are considered. Although the limitation is discussed.\n\nQuestions:\n\n1. It is not clear to me if the dynamics is assumed to be known (i.e, A, B matrix). If it is known, then how is the dynamics for the monkey estimated?\n\n2. How important is the assumption that A, B is constant? Can we work with a time varying A, B in the current framework. This will be important to extend this to iLQG settings.\n\n3. An important application of inferring the cost parameters is using it to predict behaviors that is not in the dataset or generate controllers that can mimic the behaviors of the agent. Any thoughts on this?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They are well discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "96L-TyiMSkU", "reviewer": "Reviewer_ueSj", "summary": "This paper presents a theoretical framework for cost function estimation for linear, quadratic, Gaussian systems with action- and state-dependent noises. \nAlgorithms for estimating the probabilities of latent variables and observed signals are derived for both the cases of complete and partial observation of the task variables by the experimenter. Then a gradient-free optimizer BOYAQA is used to obtain the maximum likelihood estimate of the parameters of the objective function.\nThe performance was examined with 1D reaching simulation and real arm reaching and saccadic eye movement experimental data.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe method is straight forward applications of dynamic Bayesian inference and maximum likelihood estimate, but the formulation of inference by the agent and by the experimenter as graphical models as presented in Figure 1 is eye-opening.\n\nQuality:\nThe mathematical derivation is solid and clear.\nIn the 1D reaching simulation, why isn't the end-point position error included in the cost function? Is it supposed to be zero cost as long as the end point is within a target range?\nThe case of center-out arm reaching clearly violates the linear assumption. The method seems to be working nevertheless, and the reason why would be worth discussing.\n\nClarity:\nProbably not all the readers are familiar with the impact of Harris & Wolpert paper, it may worth more intuitive explanation.\nIf I understand correctly, the method assumes that the experimenter knows the dynamics model (A, B, V, Ci, H, W, Di). It is worth mentioning how they can be estimated\n\nSignificance:\nAlthough the theory was derived under linear, quadratic, Gaussian assumptions, its applicability with real arm and eye movement data is encouraging.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "These are well considered and documented.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a theoretical framework for estimating cost functions in linear, quadratic, Gaussian systems with action- and state-dependent noises. Algorithms for inference are derived for the complete and partial observation cases and distinguish between the agent’s and the experimenter's inference tasks. The approach is validated on synthetic and experimental data and applied to human sequential sensorimotor behavior.

Whilst the proposed approach is a simple applications of dynamic Bayesian inference and maximum likelihood estimate, the proposed inference formulation was considered original. The theory was derived under linear, quadratic, Gaussian assumptions, but the approach is demonstrated to work well in real-world applications, such as real arm and eye movement data. The clarity of the paper could be improved, especially w.r.t. to some literature.




Decision: Accept (Poster)