Inverse decision-making using neural amortized Bayesian actors
Authors: Dominik Straub
Tobias F. Niehues
Jan Peters
Constantin A. Rothkopf
Venue: ICLR 2025 Poster (2025)
Keywords: Bayesian actor models
perception and action
cognitive science
Bayesian inference
inverse modeling
Primary area: applications to neuroscience & cognitive science
TL;DR: We developed an efficient Bayesian inference methods for priors, uncertainties, and costs from behavior by amortizing Bayesian actor models using neural networks.

Abstract: Bayesian observer and actor models have provided normative explanations for many behavioral phenomena in perception, sensorimotor control, and other areas of cognitive science and neuroscience. They attribute behavioral variability and biases to interpretable entities such as perceptual and motor uncertainty, prior beliefs, and behavioral costs. However, when extending these models to more naturalistic tasks with continuous actions, solving the Bayesian decision-making problem is often analytically intractable. Inverse decision-making, i.e. performing inference over the parameters of such models given behavioral data, is computationally even more difficult. Therefore, researchers typically constrain their models to easily tractable components, such as Gaussian distributions or quadratic cost functions, or resort to numerical approximations. To overcome these limitations, we amortize the Bayesian actor using a neural network trained on a wide range of parameter settings in an unsupervised fashion. Using the pre-trained neural network enables performing efficient gradient-based Bayesian inference of the Bayesian actor model's parameters. We show on synthetic data that the inferred posterior distributions are in close alignment with those obtained using analytical solutions where they exist. Where no analytical solution is available, we recover posterior distributions close to the ground truth. We then show how our method allows for principled model comparison and how it can be used to disentangle factors that may lead to unidentifiabilities between priors and costs. Finally, we apply our method to empirical data from three sensorimotor tasks and compare model fits with different cost functions to show that it can explain individuals' behavioral patterns.

Published as a conference paper at ICLR 2025
INVERSE DECISION-MAKING USING NEURAL AMOR- TIZED BAYESIAN ACTORS
Dominik Straub".. .1.2, Tobias F. Niehues" ,1,2, Jan Peters & Constantin A. Rothkopf
:2.3 ,2,3
Institute of Psychology, Technical University Darmstadt
2 Centre for Cognitive Science, Technical University of Darmstadt
3 Department of Computer Science, Technical University of Darmstadt Hessian Center for Artificial Intelligence
[firstname, lasttnmme stu-darmstadt...
ABSTRACT
Bayesian observer and actor models have provided normative explanations for many behavioral phenomena in perception, sensorimotor control, and other areas of cognitive science and neuroscience. They attribute behavioral variability and biases to interpretable entities such as perceptual and motor uncertainty, prior beliefs, and behavioral costs. However, when extending these models to more naturalistic tasks with continuous actions, solving the Bayesian decision-making problem often analytically intractable. Inverse decision-makinn i.e. performing inference over the parameters of such models given behavioral data, is computa- tionally even more difficult. Therefore, researchers typically constrain their models to easily tractable components, such as Gaussian distributions or quadratic cost functions, or resort to numerical approximations. To overcome these limitations, amortize the Bayesian actor using neural network trained wide range parameter settings in an unsupervised fashion. Using the pre-trained neural network enables performing efficient gradient-based Bayesian inference of the Bayesian actor model's 's parameters. We show on synthetic data that the inferred posterior distributions are in close alignment with those obtained using analytical solutions where they exist. Where no analytical solution is available, we recover posterior distributions close the ground truth. We then show how our method allows for principled model comparison and how it can be used to disentangle factors that may lead to unidentifiabilities between priors and costs. Finally, we apply our method to empirical data from three sensorimotor tasks and compare model fits with different cost functions to show that can explain individuals' behavioral patterns.
INTRODUCTION
Explanations of human behavior based on Bayesian observer and actor models have been widely successful, because they structure the factors influencing behavior into interpretable components
(Ma) 2019). They have explained a wide range of phenomena perception (Weiss et al., 2002 Frnst
& Banks 2002 Wei & Stocker 2015 Kersten et al. 2004), motor control (Körding & Wolpert 2004
Todorov & Jordan 2002), other domains of cognitive science (Grifliths Tenenbaum 2006 Xu &
Tenenbaum 2007), and neuroscience (Behrens et al.) 2007 Berkes et [011)) Bayesian observer
models assume that an actor receives uncertain sensory information about the world. This sensory information is uncertain because of ambiguity of the sensory input noise in neural responses Kersten et al. 2004). To obtain a belief about the state of the world, the actor fuses this information with prior knowledge according world, they also act in it. In Bayesian actor models, this is expressed as the minimization of cost function, which expresses the actor's goals and constraints. An optimal actor should minimize this cost function while taking their belief about the state of the world into account. However, there is
Bayes' rule. But humans do not only form beliefs about the
Both authors contributed equally this paper.

---

Published as a conference paper at ICLR 2025
not only uncertainty in perception but also in action outcomes, e.g. due to the inherent variability of the motor system (Van Beers et al. (200). If an actor wants to perform the best action possible, this uncertainty should also be incorporated into the pocision-making process by integrating over the distribution of action outcomes (Trommershäuser et al. 2008).
One problem, which makes solving the Bayesian decision-making problem hard in practice, is that the expected cost is often not analytically tractable, because it involves integrals over the posterior distribution and the action distribution. Consequently, optimization of the expected cost is also often intractable. Certain special cases, especially Gaussian distributions and quadratic cost functions, admit analytical solutions. Applications of Bayesian models have typically made use of these assumptions. However, empirical evidence shows that the human sensorimotor system does not conform to these assumptions. Noise in the motor system depends on the force produced (Harris) & Wolpert 1998; Todorov & Jordan 2002) and variability in the sensory system follows a similar signal-dependenee known as Weber's 's law (Weber 183T). Cost functions other than quadratic costs have been shown to be required in sensorimotor tasks (Körding& & Wolpert 2004 Sims 2015). If we want to model what is known about the human sensorimotor system or treat more naturalistic task settings, we need to incorporate these non- -an-Gassiian distributions and non-quadratic costs.
Another challenge is that Bayesian actor models often have free parameters. One option is to set these parameters by hand, e.g. often assuming that subjects use the identical prior and cost function defined by the experiment, as in ideal observer analysis, or by matching prior distributions to the statisties of the natural environment. Predictions of the Bayesian model are then compared to behavioral data to assess optimality. This practice is problematic, because priors and costs may not be known to the experimenter beforehand and can be idiosyncratic individual participants. For example, an actor's cost function might not only contain task goals, e.g. hitting a target, but also internal costs. These can include cognitive factors such as computational resources (Lewis) ett al., 2014: Lieder & Griffiths| 2020; Gershman et al. 2015), but also more generally cognitive and physiological factors including biomechanical effort (Hoppe
Rothkopi 2016; Straub & Rothkopf 2022). An actor's prior distribution might neither match the statistics of the task at hand nor those of the natural environment (Feldman 2013). J. In the spirit of rational analysis (Simon) [955] Anderson 1991 Gershman et al. 2015). this has motivated researchers to invert Bayesian actor models, T.e. to use Bayesian actor models as statistical models of behavior and perform Bayesian inference over the free parameters from behavior. This approach has come to be known in different application areas under different names, including cognitive tomography (Houlsby et al. 2013), inverse reinforcement learning (Rothkopf & Ballard 2013; Muelling et al.) 2014) and inverse optimal control (Kwon et al. 2020 [Schultheis et al., 2021))
analysis (Aitchison et al.) 2013)) ,)
However, because the forward problem of computing optimal Bayesian actions for a given perception and decision- decision-making problem is computationally expensive, the inverse decision making problem, only numerical solutions are available, is prohibitively expensive and makes computing gradients with respect to the parameters for efficient optimization or sampling infeasible.
Here, we address these issues by providing new method for inverse decision decision-making in sensorimotor tasks with continuous actions. Such tasks are widespread in cognitive science, psychology, and neuroscience and include so-called production, reproduction, magnitude estimation, and adjustment tasks. First, we formalize such tasks with Bayesian networks, both from the perspective of the researcher and from the perspective of the participant. Second, we approximate the solution of the Bayesian decision-maie making problem with neural network, which is trained in an unsupervised fashion using the decision problem's 's cost function as a stochastic training objective. Third, using the pre-trained neural network stand-in for the Bayesian actor within statistical model enables efficient Bayesian inference of the Bayesian actor model's parameters given observed behavior. Fourth, we show on simulated datasets that the posterior distributions obtained using the neural network recover the ground truth parameters very closely to those obtained using the analytical solution for various typical response patterns like undershots or regression the mean behavior. Fifth, we show how posterior distributions over the actor's ' internal parameters can be used to perform principled model comparison. Sixth, identifiability problems between priors and costs of Bayesian actor models are investigated, which can now be resolved based on our proposed method. Finally, we apply our method to human behavioral data from three different experiments and show that the inferred cost functions explain the previously mentioned typical behavioral patterns not only in synthetically generated but also empirically observed data.
2

---

Published as a conference paper at ICLR 2025
pls (m)--pmnaa 22 p15)
B
orggin BMf .M
Elfle Xil
stammius
A Bayesian Deciisin-mmainne problem from the perspective of the subject. The subject needs to find the optimal action a" based the sensory measurement of the state of the world Combined with a cost function E(r, 8) and the action distribution r Lognormal(a, J,), they want to minimize cost function under their belief about the state of the world which yields the optimal action a* = arg min, Ep(s m) [Entr a) [e(r, s)]] B Bayesian inference problem about the subject's parameters from the perspective of the researcher. The researcher solves the inverse decision-making problem, i.e. they want to infer the posterior distribution p(0 'D) over the parameters of the subject's 's perception- action system and cost function given a dataset D {st, r ...... n1 of stimuli 8; and responses ry from n trials. To make inference of the posterior over 0 feasible, we use neural network as an approximator for the optimal action a".
Figure 1:
RELATED WORK
Inferring priors and costs from behavior has been problem of interest in cognitive science for many decades. In psychophysics, for example, signal detection theory is an early example of an application of Bayesian observer model used to estimate sensory uncertainty and criterion, which encompasses prior beliefs and a particular cost function (Green et al. 1966). Psychologists and behavioral economists have developed methods to measure the subjective utility function from economic decisions (Terrsy & Kahneman 1992). More recently, Bayesian actor models have been used within statistical models infer parameters of the observer's likelihood function (Girshick et al. 2011). the prior (Stocker & Simoncelli 2006; Girshick et al. 2011 Sohn & Jazayeri | [021]), and the cost function (Körding Wolpert 2004 Sims 2015 Sohn & Jazayeri 2021). The inference methods are often bespoke tools for the specific model considered in study. They are also typically limited discrete decisions and cannot be applied continuous actions.
There are two notable exceptions. AAerbi et al. (2014) presented an inference framework for Bayesian observer models using mixtures of Gaussians. While their approach only takes perceptual uncertainty into account in the decision-making process, we assume that the agent also considers action variability here. Furthermore, instead of only inverted Gaussian mixture cost functions, method allows for arbitrary, parametric cost functions that easily interpretable. Neupärt] & Rothkopf (2021) introduced the idea of approximating Bayesian decision-making with neural networks. They trained neural networks in supervised fashion using dataset of numerically optimized actions. We extend this approach in three ways. First, we train the neural networks directly on the cost function of the Bayesian decision-making problem without supervision, overcoming the necessity for computationally expensive numerical solutions. Second, in addition to cost function parameters and motor variability, we also infer priors and sensory uncertainty. Finally, we leverage the differentiability of neural networks in order to apply efficient maudient-based Bayesian inference methods, allowing us to draw thousands of samples from the posterior over parameters in a few seconds.
Our method is also related to amortized and likelihood- free inference (Fengler et al. 2021 Greenberg et al.) 2019; Govindarajan et al. 2022 Radev et al. 2020). A conceptual difference is that we amortize the solution of the Bayesian decision decision-making problem faced by subject instead of the inference process itself. This allows to solve the Bayesian inference problem from the perspective of a researcher using efficient gradient- based inference techniques, without the need to use amortized or likelihood-fres inference.
3

---

Published as a conference paper at ICLR 2025
2 BACKGROUND: BAYESIAN DECISION-MAKING
We start with the standard formulation of Bayesian decision theory Berger 1985). illustrated in Fig. A. An actor receives stochastic observation m generated from latent state according to generative model p(m s). Since the actor has no direct access to the true value of the state s, they need infer it by combining prior distribution p(s) with the likelihood p(m s) using Bayes' rule p(s] m) & p(m 8) p(s). This Bayesian inference process describes the actor's perception. Based on their perception, the actor's goal to perform an optimal action a", which represents the intended motor response. This is commonly framed as a decision decision-making problem based on a cost function E(a, s). The optimal action a" for the actor is the action a that minimizes the expected cost under the posterior distribution
a = arg min / E(a, s) p(s [m) ds.
Often, the loss is not defined directly in terms of the performed action a, but terms the expectation over some stochastic version of it r ~ p(r a), modeling variability in responses with the same intended action a. Taking the expectation over the posterior distribution and the response distribution lets us expand Eq. (1)
/ s) a) p(s mm) dr ds,
a" a"= arg min
which changes the problem conceptually from computing an estimate of latent variable to performing an action that is subject to motor noise.
3 METHOD
The Bayesian decision- decision-making problem in Section 2 describes the situation faced by a subject performing task and the optimal solution it. From the researcher's perspective, we now want to infer the parameters of the subject's perception-action system. For example, we might be interested in the subject's perceptual uncertainty and their prior belief about possible target location, their action variability, and the cost of extending effort for motor responses. These parameters constitute parameter vector Formally, we want compute the posterior p(0 D) for a set D of behavioral data, as shown in Fig. [B.
Our proposed method consists of two parts. First, we approximate the optimal solution of the Bayesian decision-making problem with a neural network (Section 3.1). A forward-pass of the neural network is very fast compared to the computation of numerical solutions to the original Bayesian macciion-making problem, and gradients of the optimal action w.r.t. the parameters of the model (uncertaintics, priors, costs) can be efficiently computed. In the second part of our method, this allows us to utilize the neural network within statistical model of an actor's behavior to perform inference about model parameters (Section 3.2
3.1 AMORTIZING BAYESIAN DECISION DACISION-MAKING USING NEURAL NETWORKS
Because the Bayesian decision-making problem stated in Eq. T is intractable for general cost functions, we approximate using a neural network a" Ri (f, m). which takes the parameters of the Bayesian model 0 and the observed variable m input and is parameterized by learnable parameters vi. It can then be used as a stand-in for the computation of the optimal action a* in down-stream applications of the Bayesian actor model, in our case perform inference about the Bayesian actor's 's parameters.
3.1.1 UNSUPERVISED TRAINING
We train the neural network in an unsupervised fashion by using the cost function of the decision- making problem implicitly solves the Bayesian decision- making problem.
unsupervised stochastic training objective. After training, the neural network
Specifically, we use the expected posterior loss
L ((bb Ep(s m) [Ep(r (o(0m)) [e(r, s)]]
4

---

Published as a conference paper at ICLR 2025
a training objective. Because the inner expectation depends the parameters of the neural network, w.r.t. which we want to compute the gradient, we need to apply the reparameterization trick (Kingma & Welling 2014) and instead take the expectation
[( = pp(s m) Epte) [ (r, s)]]
over distribution p(c) that does not depend on &ú where transformation g. For example, in the perceptual decision-makinn model used later (Section [4.]). p(r a) is log-normal with scale parameter o,, can sample No, 1) and use the reparame- terization g(a, e) = exp (log(a) + EO,).
(fr (0,m) :.) c) with some appropriate
Now, we can easily evaluate the gradient of the objective using Monte Carlo approximation of the two expectations,
K K ΣV V Colorr sk),
1
VoL(+)
which makes possible to train the network using any variant of stochastic gradient descent. In other words, the loss function used to train the neural network that approximates the optimal Bayesian decision-maker is simply the loss function of the underlying Bayesian decision-makinn problem. All that we require is model in which it is possible to draw samples from the posterior distribution 8k ~ p(s m) and the response distribution Tn ~ p(r a). This allows us to train the network in an unsupervised fashion, i.e. we only need a training data set consisting of parameters @ and sensory inputs m. without the need to solve for the optimal actions beforehand. The procedure is summarized in Algorithm[] [T See Section D.I for the prior distributions used generate parameters during training. We used the RMSProp optimizer with learning rate of 10-4, batch size of 256, and N = M 128 Monte Carlo samples per evaluation of the stochastic training objective. The networks were trained for 500,000 steps, and we assessed convergence using an evaluation set of analytically or numerically solved optimal actions (see Section D.2)
3.1.2 NETWORK ARCHITECTURE
We used multi-layer perceptron with 4 hidden layers and 16, 64, 16, 8 nodes in the hidden layers, respectively. We used swish activation functions at the hidden layers (Elfwing et al. 2018). As the final layer, we used linear function with output y E R3, followed by non- non-linearity a" = sy + ya), where m is the observation received by the subject. This particular non-linearity is motivated by the functional form of the analytical solution of the Bayesian decision- making problem for the quadratic cost function as a function of m and 0 (Section C.I and serves as an inductive bias (Section D.3).
3.2 BAYESIAN INFERENCE OF MODEL PARAMETERS
The graphical model in Fig. B illustrates the generative model of behavior in an experiment. Our goal is infer p(0 D). where D {si, r3
In...,n] is dataset of stimuli and responses. We assume that for every stimulus s presented to the subject, they receive a stochastic measurement m pmm s). They then solve the Bayesian decision problem given above, i.c. they decide on an action a. We used the neural network a' 2 f(m, 0) to approximate the optimal action. The chosen action then corrupted by action variability yield a response r p(r a). To sample from the researcher's posterior distribution over the subject's model parameters p(0 (D), we use the Hamiltonian Monte Carlo algorithm NUTS (Hoffman et al. 20144). This gradient-bases inference algorithm can be used because the neural network is differentiable with respect to the parameters O and sensory input m. The procedure is summarized in Algorithm
3.3
IMPLEMENTATION
The method was implemented in jax (Frostig et al., 2018). using the packages equinox (Kidger & Garcia )021) for neural networks and numpyro (Phan et al. 2019) for probabilistic modeling and inference. Our implementation is publicly available at https:::///. //gi thub cwwccooooooooeeee whothkoofii h naba Our software package enables the user to define new parametric families of cost functions and train neural networks to approximate the decision-makin about its parameters. Training a neural network for 500,000 steps took 10 minutes, and drawing 20,000 posterior samples for typical dataset with 60 trials took 10 seconds.
problem and perform Bayesian inference
5

---

Published as a conference paper at ICLR 2025
A
plos)
r=s
Ho
ours
anelytical
true
p(aa)
C
1.4
Ho
O0
pla,)
0.15
MSE
0.005
0.0025
0.000
0.0000
0.05
onslytical
m0
anstytical
snalytical Scal
0.150
0.15
a, Figure 2: A Simulated data from the Bayesian actor model with quadratic cost function are shown as a scatter plot. The posterior predictive distribution p( s, r) (mean and 94% CI) obtained using our method shown as blue line with shaded region. B Posterior distributions of parameters obtained using the analytical solution or the neural network to compute optimal actions. The top plot of each column shows the respective marginal posterior distribution for each parameter. Ground truth parameter values are shown for comparison. C Evaluation (MSE between posterior mean and ground truth) for multiple runs with uniformly sampled ground truth parameters.
RESULTS
We evaluate our method on a perceptual decision-making task with log-normal prior, likelihood and action distribution (Section 4.1. which we later combine with several cost functions. For certain cost. functions, this Bayesian decision- makiinn problem is analytically solvable. We evaluate our method's posterior distributions against those obtained when using the analytical solution for the optimal action (Section 4.2). Our evaluations allowed us to find possible identifiability problems between prior and cost parameters inherent Bayesian actor models, which analyze in more detail (Section 443). Finally, we apply our method to real data from a sensorimotor task performed by humans and show that it explains variability and biases in the data (Section 4.5
4.1 PERCEPTUAL DECISION MAKING MODEL
We now make the decision-making problem more concrete by introducing a log-normal model for the perceptual and action uncertainties. This model is applicable to many different tasks involving perception action, with wide range of stimuli, such as time (Yi, 2009 Roberts (006), space (Longo & Lourenco| 2007), sound (Sun al. 2012), numerosity (Roberts) 2006 Longo & Lourenco 2003), and different motor actions such as throwing a ball (Willey & Liu 2018). shooting a hockey puck (Neupärtl et al. 2020), producing certain force (Onneweer et al. 2016).
Sensory measurement We assume that the observer's sensory measurements are generated from log-normal distribution m ~ Lognormal(m 8s s, This assumption is motivated by Weber's law, i.e. that the variability scales linearly with the mean (Weber 1831).
Assuming a log-normal prior Lognormal on) and log- nor-normal likelihood m Lognormal(s,, o), the posterior is p(s m) Lognormal ppost), with
Posterior distribution
past (
In In mnm o2
Mpost exp
(3)
This can be shown by using the equations for Gaussian conjugate priors for a Gaussian likelihood in logarithmic space and then converting back to the original space.
Response distribution As a probability distribution for the variability responses r given an intended action a, we again use log-normal distribution ~ Lognormal(7 o,). This assumption
motivated by the idea of signal-dependent noise in actions (Sutton & Sykes 1967 Schmidt et al.) 1979; Harris & Wolpert 1998).

---

Published as a conference paper at ICLR 2025
A
B
C
Behavior
ours true
a
90
0.2
r-5
0.004
response
6
6
0.25
0.002
0.000
4
- 2.5
MSE
Ho
MSE
2
9.8
0.004
0
2
0.002
0.000
stimulus
0.25
none Ho
0, Op
[A0 α
Fixed peremeters
Figure 3: A Simulated behavior from the asymmetric quadratic cost (Eq. (4)) The responses exhibit undershots, which could be due to the prior or due to the cost. B Posteriors distributions: for each pair of parameters, the plot shows the contours of the 94%-HDI The pairwise posterior between prior mean No and cost asymmetry parameter a shows strong correlation. C MSE between ground truth and posterior mean when fixing no parameters, the cost asymmetry parameter a or the prior parameter Poo Fixing one of the confounding variables results in an improvement of accuracy in the inference of the other variable. The remaining parameters maintain their accuracy.
4.2 EVALUATION USING SYNTHETIC DATA
We first evaluate our method on a case for which we know the analytical solution for the optimal action: the quadratic cost function (see Section C.T derivation). We generated a dataset of 60 pairs of stimuli 8; and responses using the analyfical solution for the optimal action with ground truth parameters #40 1.5, ao = 0.15, a = 0.2, o, 0.1. The simulated data are shown Fig. A with characteristic pattern of signal-dependet increase in variability, values and an undershot for higher stimulus values due to the prior. We then computed posterior distributions for the model parameters using the analytical solution for the optimal action and using the neural network. We drew 20,000 samples from the posterior distribution in 4 chains, each with 5,000 warmup steps. Note that, because the concrete values of and og are unidentifiable even when using the analytical solution for the optimal action (see Section F.I we kept a fixed to its true value during inference. This was done evaluate our method on version of the model, for which the analytical solution as a gold standard produces reliable results. Fig. 2 recover the true parameters, and the contours of both posteriors align well. The posterior predictive distribution generated from the neural network posterior reproduces the pattern of variability and bias in the data (shaded region in Fig. A).
overshot for low stimulus
B shows that both versions
To ensure and quantitatively assess that the method works for a wide range of parameter settings, we then simulated 100 sets of parameters sampled uniformly (see Section D.I for the choice of prior distributions). For each set of parameters, we simulated dataset consisting of 60 trials. We then computed posterior distributions for each dataset in two ways: using the analytical solution for the optimal action and using the neural network approximate the optimal action. In both cases, drew 20,000 samples (after 5,000 warm-up steps) from the posterior distribution in chains and assessed convergence by checking that the R-hat statistic (Gelman & Rubin 1992) below 1.05. The mean squared errors between the posterior mean and the ground truth parameter value in Fig- show that the inference method using the neural network recovers the ground truth parameters just as well as the analytical version. Fig. F.3 A&B additionally show the error as a function of the ground truth parameter value and Table shows the results from Fig. 2 C numerically.
4.3 LIMITS OF IDENTIFIBILITY OF COSTS AND PRIORS
We can now apply our method new cost functions, for which analytical solutions are not available. For example, we consider an asymmetric version of quadratic cost (AsymQC, see Table E. that penalizes overshooting target more strongly then undershooting or vice versa:
A(r, 8) = 2/a 1(r- - s)((r 8)² with 1 (xx) = {
if x x
else
7

---

Published as a conference paper at ICLR 2025
This cost function introduces another souree of biases besides perceptual priors: people might under- shoot either because of a prior belief that favors closer targets or because of a cost for overshooting (see Fig. 3
Using our method, we can turn to investigating whether we can tease these different sources of biases apart. Fig. A shows a pattern of behavior with undershot with ground truth parameters 44o = 4.47, 0 = 0.17, a = 0.22, or = 0.23 and a = 0.29. Fig. 3 B shows that the ground truth parameters are contained with in the posterior distribution. However, the undershot can either be attributed to a subject trying to avoid overshoots, or to biased perception due to low prior mean. This difficult to disentangle, and leads to correlated posterior for the cost parameter a and the prior mean Nl (Fig. 3 B). Over 100 simulated datasets with range of different ground truth parameter values, the MSE between the inferred posterior mean and the ground truth is higher when both Jig and are unknown. Once we fix one of the confounding parameters at their true value and exclude them from the set of inferred parameters, we observe an increase in accuracy of the inferred parameters (see Fig. 3 C).
Nevertheless, this unidentifiability is a property of the model itself and not shortcoming of the inference method with the neural network approximation of the optimal action. In fact, our method opens up the possibility to investigate these properties of Bayesian actor models in the first place. To demonstrate this, repeated this analysis for quadratic cost function with quadratic effort cost term (see Section [:2 for which we have derived an analytical solution) and showed that the posteriors obtained using the neural network match those obtained with the analytical solution (see Fig. B).
4.4 DISENTANGLING PRIORS AND COSTS
The result in Section 3 might sound disappointing. If we do not know the subject's prior or cost, cannot easily infer them both from a simple experiment. Fortunately, the situation can be remedied by experimental design, we will now show in simulations. If we introduce different levels of perceptual uncertainty, e.g. by decreasing the contrast of the target, we can disentangle the effects of priors and costs. The intuition is as follows: At different levels of perceptual uncertainty, the actor's prior has a different influence on their posterior. The cost function, however, does not affect the shape of the posterior distribution itself, but only the optimal action under the posterior distribution. Having multiple different levels of prior or cost in an experiment should therefore resolve the unidentifiability, as also proposed by Wei & Hahn (2024).
We simulated experiment following this intuition. We set the effort parameter of the quadratic cost with quadratic effort to B = 0.9, and the parameters the log-normal sensorimotor model to 14 = 1.5, du .2, and o, 0.15. We chose two levels of perceptual uncertainty (0.1, 0.2) and simulated 45 trials for each level (see Fig. A). With this experimental design, both prior mean Ho and effort cost 3 can be estimated with good precision (see Fig. 4B, magenta curves).
A
B p(pO)
5
MA 1i (64,02)
K pe p
0.2
0.8
2
0
0.6
0
5
Figure 4: Disentangling priors and costs using different levels of perceptual uncertainty. A Data simulated with two different levels of perceptual uncertainty a.. B Inferred posterior distributions (94% Cls) for an experiment with 45 trials of each of two different levels of perceptual uncertainty (purple) and with 90 trials of one level of perceptual uncertainty (shades of blue).
To rule out that just one of the two levels of perceptual uncertainty would have been enough to disentangle prior and cost in this example, we simulated an experiment with an equivalent number of trials in total for each individual level of perceptual uncertainty. Compared to the experiment with two different levels, the posterior distributions showed higher uncertainty in Pg and 3, and stronger correlation between the two parameters (see Fig. 4 B, blue curves).
8

---

Published as a conference paper at ICLR 2025
A
Bean Bag Throwing #(t, 4)
Puck Sliding
Force Reproduction a(e, 43
Response r
Data
PPC mean 94% Data
2-
B
Model Comparison Inferred Prior
Stimulus
Stimulus 94. 94.0% HDI
Stimulus s 94.0% HDI
94.0% HDI
Symmetric Symmettic +Elfort
Asymmetric
PSIS-LOO AELPD
10
1004
AsymmetricQuadratic
5-
AbsskuaaConQuusdrrtiffffftt
InvertedGaussian
AsymOG
GCGE
Linex
ADGE
Linex
Asym
GOGE
ACGE
AC
ACDE
Asymicc
AC
GOOE
(G oG Lines
QuadraticCost
AbsoluteCost
Figure 5: Data of exemplary subjects from three different tasks. The tasks were throwing bean bags at target (Willey & Liu 2018), sliding puck to a target (Neupärtl et al. (2200) and producing a force of certain magnitude (Onneweer et al. 2016), from left to right. Rows show different aspects of the data. A Data of the subject with mean and 94% confidence interval of posterior predictive distributions along with qualitative plot of the cost function with the inferred mean parameters (thick) and the inferred 94%-HDI bounds (light background) over the response r with fixed stimulus in the upper left panel. B Inferred means and 94%-CI C1 for prior mean μ40 for range of different cost functions, grouped by their functional form as symmetric, symmetric plus effort term or asymmetric. The cost functions are given explicitly in Table E.I C Model comparison based on the differences in PSIS-LOO estimates of the expected log point-wise predictive density (ELPD) of different cost functions the best scoring cost function (lower is better). Error bars denote standard error.
4.5 INFERENCE OF HUMAN COSTS AND PRIORS
The intended use of our method is to serve as a toolbox for modeling behavior that allows for comparison of different modeling choices in Bayesian actor models. Here, we evaluate seven different parametric cost functions (see Table F. empirical data from three different tasks. The tasks used our evaluation are a bean bag (BB) throwing task (Willey & Liu 2018), a puck sliding (PU) task (Neupärtl et al. 2020) and force reproduction (FOR) task (Onneweer et al.) 2016). In the BB task, participants were asked to throw a bean bag at five different target distances from 3 to feet (0.9 to 3.4 meters) with 2 feet increments (0.6 meters). In the PU task, participants needed to press a button for a certain amount of time, which resulted in the distance traveled by a simulated puck on screen. The virtual distances screen corresponded to 1.0 to 5.0 meters and were uniformly sampled. The participants received visual feedback by seeing how far the puck travelled onscreen. In the FOR task, participants needed produce forces haptic manipulator from 10 to 160 newtons with 30 newtons increments. They received verbal feedback from the instructor and visual feedback about the applied force.
We fit our model to the data from individual subjects and evaluate it with posterior predictive checks and model comparison over different cost functions, as shown in Fig. 5 To illustrate the effects of different cost functions on the inferred prior belief, we also visualize the inferred posteriors over Ho given different cost functions. For each run, we drew 20,000 posterior samples after 5,000 warm-up steps in four chains. We assessed convergence by checking that the R-hat statistic (Gelman & Rubin 1992) was below 1.05.
Fig. 5 shows an exemplary subject from each task, overall showing different qualitative patterns of behavior. The subject from the BB task has a tendency to overshoot for near targets and undershoots for far targets. This tendency is explained by the inferred prior means μl. which are close to the actual mean of targets in the experiment at 2.15 meters. The model comparison shows a slight advantage for the asymmetric cost function. The subject from the PU task has a slight tendency to undershoot the target. The inferred prior means are very uncertain and the model comparison is inconclusive, indicat- ing that the data from this experimental condition is not sufficient to tell different cost functions apart.

---

Published as a conference paper at ICLR 2025
Finally, the subject in the FOR task produces the target force quite accurately. In this data set, different cost functions can lead to greatly differing inferences about the subject's 's prior mean. The model comparison shows that the Linex cost function provides the worst fit, while the absolute cost with quadratic effort best explains the data. These results illustrate how our method can be used in future work for extensive analyses of different cost functions across large variety of tasks and subjects.
DISCUSSION
First, we have presented new method for Bayesian inference about the parameters of Bayesian actor models. Computing optimal actions in these models is intractable for general cost functions and, therefore, previous work has often focused on cost functions with analytical solutions, derived custom tools for specific tasks. We propose an unsupervised training scheme for neural networks to approximate Bayesian actor models with general parametric cost functions. The approach is extensible to cost functions with other functional forms suited to particular tasks. Second, performing inference about the parameters of Bayesian actor models given behavioral data is computationally very expensive because each evaluation of the likelihood requires solving the makiingn problem. By plugging in the neural network approximation, we can perform efficient inference. Third, a very large number of tasks involve continuous responses, including economic rreision-making ( ('how much would you wager in a bet?'), ), psychophysical production ('hit the target'), ), magnitude reproduction ("reproduce the duration of a tone'), sensorimotor tasks ('reproduce the force you felt'), and cross- modality matching ( ('adjust a sound to appear as loud as the brightness of this light"). Therefore we see very broad applicability of our method in the behavioral sciences including cognitive science, psychology, neuroscience, sensorimotor control, and behavioral economics. Fourth, over the last few years, a greater appreciation of the necessity for experiments with continuous responses has developed (Yoo et al., (22TT) Such decision problems are closer to natural environments than discrete forced-choier decisions, which have historically been the dominant approach. We provide a statistical method for analyzing continuous responses, which allows for quick and exhaustive exploration of the model space spanned by design choices for the Bayesian actor model (e.g. cost functions), and show this on empirical data from three different tasks. Thereby, our method aims to advance scientific progress in our understanding of the internal processes and drivers of human behavior in perceptual decision decision-making problems. Finally, although the optimality assumption has been common in perceptual and economic decision decision-making, motor control, and other cognitive tasks, it has been repeatedly disputed (Rahnev & Denison, 2018). In fact, people are not always optimally tuned the statistics of the task at hand or act only to fulfill the instructed task. This is precisely the motivation for inverse decision-making methods. Instead of postulating optimality, these methods infer under which assumptions the behavior would have been optimal, and thereby conceptually reconcile normative and descriptive models of decision-maiee makinn while capturing behavior quantitatively.
Once we move beyond quadratic cost functions, identifiability issues between prior and cost pa- rameters can arise. As recognized in other work as well (Acerbi et al. 2014, Sohn & Jazayeri 2021), these identifiability issues in Bayesian models have implications for how experiments should be designed. We have shown that, when either priors or costs are known, the identifiability issue vanishes. Furthermore, we have shown in simulations that multiple levels of perceptual uncertainty can disentangle the behavioral effects of priors and costs. Based on these results, we recommend experiments with multiple conditions that employ perceptual uncertainties of different magnitudes, between which one can assume either priors or costs to stay fixed, in order to disentangle their effects. Our methodology should prove particularly useful for investigating task configurations that lead to or avoid such unidentifabilliiies and can be used to perform inference give data from such experiments.
Limitations The strongest limitation we see with the present approach is that we require a model of the perceptual problem that allows drawing samples from the observer's 's posterior distribution, which works for stimuli that can be described well by these assumptions, e.g. magnitude- like stimuli with log-normal distributions considered in our experiments. Ideally, we would also like to apply the method to perceptual stimuli for which these assumptions do not hold (e.g- circular variables). We a potential to extend our method by learning an approximate posterior distribution together with the action network. This idea is closely related to loss-calibrated inference [Lacoste-Julien et al.) 2011 Kušmierczyk al. 2019), approach that learns variational approximations to posterior distributions adapted to loss functions. We will explore this connection more in future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Nils Neupärtl for initial work on this project idea (Neupärtl & Rothkopp 2021). We acknowledge the suggestion by an anonymous NeurIPS reviewer try unsupervised training. We thank Chéla Willey and Zili Liu for sharing their beanbag throwing data (Willey & Liu 2018), Bram Onneweer for sharing the force reproduction data (Onneweer al., 20T6), and Fabian Tatai for his help in analyzing the puck sliding data (Neupärtl et al.] 2020). This research was supported by the European Research Council (ERC; Consolidator Award "ACTOR" project number ERC-CoG- 101045783), by the "The Adaptive Mind", funded by the Excellence Program of the Hessian Ministry of Higher Education, Science, Research and Art, and the Hessian research priority program LOEWE within the project "WhiteBox". We gratefully acknowledge the computing time provided to on the high-pun perfforrrrmeeec computer Lichtenberg at the NHR Centers NHR4CES at TU Darmstadt.
ETHICS STATEMENT
Methods for inferring beliefs and costs from behavioral data have potential for both positive and negative societal impact. We see great benefit for behavioral research from methods that provide estimates of people's perceptual, motor, and cognitive properties from continuous action data, particularly in clinical upplications where such tasks are easier for patients than classical forced- choice tasks. On the other hand, methods to infer people's 's uncertainties and costs could potentially be used in harmful ways, especially if they are used without the subjects' consent. The inference methods presented here are applicable to controlled experiments, and are far from scenarios with harmful societal outcomes. Still, applications of these methods should of course take place with the subjects' informed consent and the oversight of ethical review boards.
REPRODUCIBILITY STATEMENT
We have described our methods in detail in Section 3 Specifically, the training procedure (Sec- tion B.I.Ip, network architecture (Section 3.1.2). Bayesian inference method (Section 3.2). and implementation (Section 3.3) are described in individual sections. In the appendix, we also provide pseudo-code version of the algorithms (Section A detailed graphical models describing the prob- abilistic models (Section B). and analytical derivations for optimal actions of two cost functions (Section C Additional details about the prior distributions used for training the neural networks, and for performing the inference and evaluation are provided in Section [D]] and about the evaluation dataset in Section D.2 We provide an implementation of our methods with instructions for setup and running the code at htttt::///ggtt //github com wwtom//ttoooofaaae naba
REFERENCES
Luigi Acerbi, Wei Ji Ma, and Sethu Vijayakumar. A framework for testing identifiability of bayesian models of perception. Advances in neural information processing systems, 27, 2014.
Laurence Aitchison, Dan Bang, Bahador Bahrami, and Peter E Latham. Doubly bayesian analysis of confidence in perceptual Decision-making PLoS computational biology, 11(10)e 2015.
John R Anderson. Is human cognition adaptive? Behavioral and brain sciences, 14(3):471-4855 1991.
Timothy EJ Behrens, Mark W Woolrich, Mark E Walton, and Matthew FS Rushworth. Learning the value of information in an uncertain world. Nature neuroscience, (09):1214-12211 2007.
James Berger. Statistical Decision Theory and Bayesian Analysis. Springer Science & Business Media, 1985.
Pictro Berkes, Gergö Orbán, Máté Lengyel, and J6zsef Fiser. Spontaneous cortical activity reveals hallmarks an optimal internal model of the environment. Science, (311333)) 2011.
Stanislas Dehaene. The neural basis of the Weber -Fechner law: a logarithmic mental number line. TRENDS in Cognitive Sciences, 7(4): 2003.

---

Published as a conference paper at ICLR 2025
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Marc O Ernst and Martin S Banks. Humans integrate visual and haptic information in statistically optimal fashion. Nature, (1t 415(6870):49-433, 2002.
Jacob Feldman. Tuning your priors to the world. Topics in cognitive science, 5(1):13-34, 2013.
Alexander Fengler, Lakshmi N Govindarajan, Tony Chen, and Michael Frank. Likelihood approxi- mation networks (lans) for fast inference of simulation models cognitive neuroscience. Elife, 10: e65074, 2021.
Roy Frostig, Matthew James Johnson, and Chris Leary, Compiling machine learning programs via high-level tracing. Systems for Machine Learning. 4(9), 2018.
Andrew Gelman and Donald B Rubin, Inference from iterative simulation using multiple sequences. Statistical science, 7(4):457-472, 1992.
Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 34966255) 2015.
Ahna R Girshick, Michael S Landy, and Eero P Simoncelli. Cardinal rules: visual orientation perception reflects knowledge of environmental statistics. Nature neuroscience, 14(7):926-932. 2011.
Lakshmi Narasimhan Govindarajan, Jonathan S Calvert, Samuel R Parker, Minju Jung. Radu Darie, Priyanka Miranda, Elias Shaaya, David A Borton, and Thomas Serre. Fast inference of spinal neuromodulation for motor control using amortized neural networks. Journal of neural engineering, 19(5):050377, 2022.
David Marvin Green, John A Swets, et al. Signal detection theory and psychophysics, volume 1. Wiley New York, 1966.
David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood-free inference. International Conference on Machine Learning, pp. 2404-2414. PMLR, 2019.
Thomas Griffiths and Joshua B Tenenbaum. Optimal predictions in everyday cognition. Psycholog- ical science, 77(9)7677773,, 2006.
Christopher M Harris and Daniel M Wolpert. Signal-dependent noise determines motor planning. Nature, 394(6695),780-78. 1998.
Matthew D Hoffman, Andrew Gelman, al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15 15(1):1593-1623, 2014.
David Hoppe and Constantin A Rothkopf. Learning rational temporal eye movement strategies. Proceedings of the National Academy of Sciences, 113(29):8332-832) 2016.
Neil MT Houlsby, Ferenc Huszár, Mohammad M Ghassemi, Gergö Orbán, Daniel M Wolpert, and
Máté Lengyel. Cognitive tomography reveals complex, task-independent mental representations.
Current Biology, 23(21):2169-2177 2013.
Daniel Kersten, Pascal Mamassian, and Alan Yuille. Object perception as bayesian inference. Annu. Rev. Psychol., 55:271-304 2004.
Patrick Kidger and Cristian Garcia. Equinox: neural networks in JAX via callable PyTrees and filtered transformations. Differentiable Programming workshop at Neural Information Processing Systems 2021, 2021.
Diederik Kingma and Max Welling. 1rXiv=1312.61 2014.
Auto-encoding variational bayes.
arXiv preprint
12

---

Published as a conference paper at ICLR 2025
Konrad P Körding and Daniel M Wolpert. Bayesian integration in sensorimotor learning. Nature, 427(69711:244 247, 2004.
Tomasz Kušmierczyk, Joseph Sakaya, and Arto Klami. Variational bayesian decision-making for continuous utilities. Advances Neurul Information Processing Systems, 32, 2019.
Minhae Kwon, Saurabh Daptardar, Paul R Schrater, and Xaq Pitkow. Inverse rational control with partially observable continuous nonlinear dynamics. Advances in neural information processing systems, 33:7898-788 7909, 2020.
Konrad Paul Körding and Daniel M. Wolpert. The loss function of sensorimotor learning. Proceedings of the National Academy of Sciences, 101(6)):839--942 2004.
Simon Lacoste-Julien, Ferenc Huszár, and Zoubin Ghahramani. Approximate inference for the loss-calibrated bayesian. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 416-424. JMLR Workshop and Conference Proceedings, 2011.
Richard L Lewis, Andrew Howes, and Satinder Singh. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science, 6(2):279-31) 2014.
Falk Lieder and Thomas L Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use limited computational resources. Behavioral and brain sciences, 43:e1, 2020.
Matthew R. Longo and Stella F. Lourenco. Spatial attention and the mental number line: Evidence for characteristic biases and compression. Neuropsychologia, 45(7):1400 1407, 2007.
Wei Ma. Bayesian decision models: A primer. Neuron, 104(1):164-175, 2019.
Katharina Muelling, Abdeslam Boularias, Betty Mohler, Bernhard Schölkopf, and Jan Peters. Learn- ing strategies in table tennis using inverse reinforcement learning. Biological cybernetics, 108: 603-619, 2014.
Nils Neupärtl and Constantin A Rothkopf. Inferring perceptual decision making parameters from behavior in production and reproduction tasks. arXiv preprint arXiv:2112.15521 2021.
Nils Neupärtl, Fabian Tatai, and Constantin A Rothkopf. Intuitive physical reasoning about ob- jects' masses transfers to visuomotor decision task consistent with newtonian physics. PLoS Computational Biology, 16(10)ee007773, 2020.
Bram Onneweer, Winfred Mugge, and Alfred C. Schouten. Force reproduction error depends on force level, whereas the position reproduction error does not. IEEE Transactions on Haptics, 9(1): 54-61, 2016. doi: 10. 10.10/TTHHH2015255779999
Du Phan, Neeraj Pradhan, and Martin Jankowiak. Composable effects for flexible and accelerated probabilistic programming in numpyro. arXiv preprint 19t. 2019.
Stefan T Radev, Ulf K Mertens, Andreas Voss, Lynton Ardizzone, and Ullrich Köthe. Bayesflow: Learning complex stochastic models with invertible neural networks. IEEE transactions on neural networks and learning systems, 13(4)):452-1466 2020.
Dobromir Rahnev and Rachel Denison. Suboptimality in perceptual decision making. Behavioral and brain sciences, 41:e223, 2018.
William A. Roberts. Evidence that pigeons represent both time and number on logarithmic scale. Behavioural Processes, 72(3):207-214, 2006.
Constantin A Rothkopf and Dana H Ballard. Modular inverse reinforcement learning for visuomotor behavior, Biological cybernetics, 107:477-490, 2013.
Richard A Schmidt, Howard Zelaznik, Brian Hawkins, James S Frank, and John T Quinn Jr. Motor- output variability: a theory for the accuracy rapid motor acts. Psychological review, 86(5):415, 1979.
13

---

Published as a conference paper at ICLR 2025
Matthias Schultheis, Dominik Straub, and Constantin A Rothkopf. Inverse optimal control adapted to the noise characteristics the human sensorimotor system. Advances in Neural Information Processing Systems, 34:9429-944 9444,,442222 2021.
Herbert Simon. A behavioral model of rational choice. The quarterly journal of economics, pp. 99-118, 1955.
Chris R Sims. The cost of misremembering: Inferring the loss function in visual working memory. Journal of vision, 15(3):2-2, 2015.
Hansem Sohn and Mchrdad Jazayeri. Validating model-based bayesian integration using prior-cost metamers. Proceedings of the National Academy of Sciences, 118025):e0215311118 2021.
Alan A Stocker and Eero P Simoncelli. Noise characteristics and prior expectations in human visual speed perception. Nature neuroscience, 9()): -58, 2006.
Dominik Straub and Constantin A Rothkopf. Putting perception into action with inverse optimal control for continuous psychophysics. Elife, 11:e76635, 2022.
John Z. Sun, Grace Wang, Vivek K Goyal, and Lav R. Varshney. A framework for Bayesian optimality of psychophysical laws. Journal of Mathematical Psychology, 56(6))495-50, 2012.
GG Sutton and K Sykes. The variation of hand tremor with force in healthy subjects. The Journal of physiology, 191(3):699-711, 1967.
Emanuel Todorov and Michael I Jordan. Optimal feedback control as a theory of motor coordination. Nature neuroscience, 5(1 12:66)222 2002.
Julia Trommershäuser, Laurence Maloney, and Michael Landy. Decision making, movement planning and statistical decision theory. Trends in cognitive sciences, 12(8):291-29)) 202 2008.
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297-323, 1992.
Robert J. Van Beers, Patrick Haggard, and Daniel M. Wolpert. The Role of Execution Noise in Movement Variability. Journal of Neurophysiology. 9102:1050-1063, 2004.
Ernst Heinrich Weher. Pulsu, resorptione, auditu et tactu: Annotationes anatomicae et physiolog- icae. Koehler, 1831.
Xue-Xin Wei and Michael Hahn. The identifiability of bayesian models of perceptual decision. Journal of Vision, 27(10):572-572, 2024.
Xue-Xin Wei and Alan A Stocker. A bayesian observer model constrained by efficient coding can explain' bayi-ba percepts. Nature neuroscience, 18(10):1509-1511 2015.
Yair Weiss, Eero Simoncelli, and Edward Adelson. Motion illusions as optimal percepts. Nature neuroscience, 5(6):598-604, 2002.
Chéla R Willey and Zili Liu. Long-term motor learning: Effects of varied and specific practice. Vision research, 152:10-16, 2018.
Fei Xu and Joshua B Tenenbaum. Word learning as bayesian inference. Psychological Review, 114 (2):245, 2007.
Linlin Yi. Do rats represent time logarithmically or linearly? Behavioural Processes, 81(2):274-279 2009.
Seng Bum Michael Yoo, Benjamin Yost Hayden, and John M Pearson. Continuous decisions. Philosophical Transactions of the Royal Society B, 3,3761811)220110666 2021.
14

---

Published as a conference paper at ICLR 2025
A ALGORITHM
Algorithm Train neural network to approximate Bayesian actor model
Output: Trained neural network (0, m) arg min, Ep(s m) Eprr [e(r, s)]] Researcher's prior distribution for training the neural network p(0), Subject's perceptual model po (m s) and po()),
Input:
Subject's response model po(r a),
Subject's cost function (T, s)
1: while test error desired test error do
2: Draw 0 ~ p(0)
3: Compute expected loss [(v) pots m) pp(c) 1Lo (r,
2k=1 E lo 3k)
K
4: Perform gradient descent step with & (()) ~ K N Σ Σn=1
Velo(rn, 8k)
5: Evaluate error of optimal actions on an evaluation set (see Section D.2
6: end while
Algorithm 2 Bayesian inference about the parameters of a Bayesian actor model
Output:
Samples from posterior distribution p(0) 'D)
Input:
Researcher's prior distribution p(0).
Data D {ss,
L,...,
Subject's perceptual model pe (m s) and po(s),
Subject's 'sss response model pe (r a.,
Subject's cost function (r.
1:
Obtain pre-trained neural network f (0, m) from Algorithm|
2: Sample from the posterior p(0 'D) defined by the graphical model Fig.
C using NUTS
B PERCEPTUAL DECISION-MAKING MODEL
A
B
d+
Ocast
m
8
m
a
I.....
NN
140
Figure B.1: The graphical models from Fig. [ with specific parameters for the log-normal perceptual Docision-making model described in Section 4.1 The parameters represent the perceptual uncertainty a, the motor variability a, and the log-normal prior defined by (Joo oo). The parameter vector Ocnat defined by the employed cost function. E.g., for quadratic cost without any free parameters, Oena {Ø}. or for costs function incorporating cost parameters in equations equation (quation/6amn and equation 4. it is Boost = {3} and 0cost = {a), respectively. Subject view of the task. B Model from the researcher's perspective.
C DERIVATIONS OF OPTIMAL ACTIONS
C.I QUADRATIC COST
One cost function for which the Bayesian decisionnmmaiing problem under a nor-normal observation model and log-normal response model (Section 4.1 can be solved in closed form is the quadratic
15

---

Published as a conference paper at ICLR 2025
function. In that case, can write the expected loss
E (r, s)] = E (s r)
IE |² 2E (s] +
assuming independence between and r.
Using the moment-generating function of the log-normal distribution, we can evaluate these expecta- tions as
Rot 2 c
of 3 c² 22
E [8(r, s)] c² 2 2e upot
By differentiating with respect to
[((r, s)] 2620 2 e to
da
and setting to zero, we obtain the optimal action
(oposs - -30
a Mpost exp
Inserting the posterior mean (Fq.
we obtain
In Mu
In m
1
(o 30
exp
post
exp
2
(5)
Tim
1
oin o
(0 2
30
1400
exp
2
C.2 QUADRATIC COST WITH QUADRATIC ACTION EFFORT
We now consider a class of cost functions of the form
E(r, = 3(x - -r)² + (1 33
This allows us to write the expected cost as
E [e(r s)] E [3(s - r)2 (1 :3)r?] BE
[(8 r) + (1 BYYT [T-2]
Using the previously obtained result of Section C.1 we can write this as (((r, s)] (1g0a 2a
(e
F Inat + 22 + 3)e 2nnat2o?
2e Mooar
Differentiating
s)] 20² a 23 po + ((o))
E ((((
Du
and setting to zero as well as inserting the posterior mean (Eq. 3 yields
a = Mpus cxp (2 (o)
-30,)
opest
30
= B exp
wi In p4 +
m um
2
2
with wi
22 and Wm G
o
16

---

Published as a conference paper at ICLR 2025
D HYPERPARAMETERS AND OTHER METHODS DETAILS
D.1 PARAMETER PRIOR DISTRIBUTIONS
We use relatively wide priors generate the training data for the neural networks ensure that they accurately approximate the optimal action over wide range of possible parameter values:
o ~ 4(0.01,, 0.5)
o, U(0.01,0.5) 0.5)
do (.0.01.0.5)
Ho (.01,,
During inference, we use narrower, but still relatively uninformed prior distributions, to avoid the regions of the parameter space on which the neural network has been trained:
Half-Normal (.25)
oy Half-Normal (.25)
Half-Normal .25)
((00
To evaluate the inference procedure, we use parameters sampled from priors, which correspond the actual parameter values that we would in expect in a behavioral experiment:
a 0,0
o, U((011 25)
σo U((01,00 0, 25)
No ((2.0,5.0)
In contrast to the sensorimotor parameters, we kept the same priors for cost parameters during training of the neural network, inference and evaluation:
Quadratic cost with lincar effort: 3 14(0.5, 1.0)
Quadratic cost with quadratic effort: / ~ (0.55 1.0)
Asymmetric quadratic cost: (1(0.,,0.))
D.2 EVALUATION DATASET
To assess convergence of the neural networks, generated evaluation dataset consisting of 100,000 parameter sets and optimal solutions of the Bayesian decision-making problem for each cost function. If analytical solution was available (e.g. quadratic cost), computed the optimal action analytically. there was no analytical solution known to us, we solved the Bayesian decision- making problem numerically. Specifically, computed Monte Carlo approximations of the posterior expected loss
L(a)
K Epps m) [pir a)) [f(r, s)]]
((ro, 8k)
A= n=
with = K 10, .000 and used the BFGS optimizer (implemented in jcc. sscipy) to solve for the optimal action a".
17

---

Published as a conference paper at ICLR 2025
D.3 INDUCTIVE BIAS FOR THE NEURAL NETWORK
We know the closed-form solution for the optimal action a' as function of the parameters 0 for the quadratic cost function (Section C.T). Therefore, assume that the parametric form of the optimal action as function of the sensory measurement m will not be substantially different for other cost functions, although the specific dependence on the parameters will vary. We rewrite the optimal action for the quadratic loss (Eq. ) as
a" (0) + f0))).
To allow for additive biases as well, we have added an additive term, which is zero for the quadratic cost function.
E COST FUNCTIONS USED IN MODEL COMPARISON
The different cost functions in the evaluation on empirical data from Section 4.5 are shown more detail in Table E.I Parameters 3 and are free parameters in the respective cost functions and are also inferred along with the other latent sensorimotor variables.
Functional Type
Name
Cost Function E(r, 8)
Absolute Cost (AC)
r
Symmetric
Quadratic Cost (QC)
(r s)2
Inverted Gaussian (IG)
- exp
{r-
23
Symmetric with effort
Absolute Cost Quadratic Effort (ACQE)
8|r 8 + (1 - -3)p²
Quadratic Cost Quadratic Effort (QCQE)
B(r 8)² + (1 -3rr
Asymmetric Quadratic (AsymQC)
2/a 1(r 8)((r =s) 2 (exp {a(r 8)} a(r - s) 1)
Asymmetric
Linex
Table E.I: Different cost functions used in fitting our model to empirical data in Section 4.5
The heaviside function 1(x) is given by
x 2
1 1xx)
0
else
18

---

Published as a conference paper at ICLR 2025
F ADDITIONAL RESULTS
F.1 INFERENCE WITH BOTH PERCEPTUAL AND PRIOR UNCERTAINTY AS FREE PARAMETERS
During evaluation of our method, found identifiability issues inherent to Bayesian actor models between the prior uncertainty do and the sensory variability σ. In order to produce the same behavior as the one observed, only the ratio do needs the joint posterior of the two parameters in Fig. F.1 A) the absolute magnitude of each of the two parameters does not substantially influence the resulting behavior and thus the solution to the true values conditioned makes sense, since the resulting behavior is largely shaped by how much influence prior and sensory information have, which is weighted by on and Ø, respectively.
inferred correctly (see diagonal correlation in
the observed behavior is underdetermined problem. This intuitively
We were able to considerably improve accuracy the inference of these two confounding parameters by fixing one of them their true value (see Fig. B.) Therefore, decided to keep the perceptual uncertainty a fixed when probing our method since this corresponds to psychophysically measuring prior applying our method and fixing it at the measure value.
A
6
10
MSE
MSE
Ho
enalytical
true
0.50
0.25
0.00
Hi
MSE
d
MSE
dr
0.0050
0.50
0.01
0.0025
8 0.25
0.00
0.0000
3
.55
5
3
0.5
none
none
0r
0 Hg
00
Fixed parameters
Figure F.I: A Posteriors with the analytical model versus the neural network model for an arbitrary. synthetically generated data set that captures the identifiability problem between a and ao quite clearly. Our method approximates the posterior with the analytical solution quite well and mimics its behavior regarding the uncertainty over different parameters. The pairwise posterior between prior uncertainty do and perceptual uncertainty shows strong correlation. B MSE when fixing parameters, the perceptual uncertainty do or the prior parameter a. Fixing one of the confounding variables results in considerable improvement of accuracy in the inference of the non-fixed parameter. The prior parameter Ao maintains its accuracy while the errors for the response variability o, also slightly decrease.
19

---

Published as a conference paper at ICLR 2025
F.2 IDENTIHIABILITY OF QUADRATIC COST WITH QUADRATIC ACTION EFFORT
Additionally, we consider a cost function that incorporates the cost of the effort of actions. It is more costly throw& ball longer distance due the force needed to produce the movement, is more effortful to press a button for longer duration. This can be achieved using a weighted sum of the squared distance the target stimulus and the square of the response:
{(r, s) 3(s r)² + (1
33r
(6)
This cost function also has another source of biases besides perceptual priors: people might undershoot stimuli larger distances more (see Fig. R:2). An analytical solution the optimal action for this cost function is derived in Section
Using our method, we can turn to investigating whether we can tease these different sources of biases apart. Fig. F.2 shows pattern of behavior with an undershot with ground truth parameters 840 2.95, σo 0.19, 0.14, 0.23, 0.72. The posterior distribution Fig. F.2 B shows that the undershot can either be attributed to a subject trying to avoid the mental or physical strain of larger effort, to biased perception due low prior mean. This is difficult to disentangle, and shows in a correlated posterior for the effort cost parameter / and the prior mean μ4o- Over 100 simulated datasets with range of different ground truth parameter values, the MSE between the inferred posterior mean and the ground truth is high when both Ao and are unknown. Once we fix one of the confounding parameters at their true value and exclude them from the set of inferred parameters, we observe considerable increase in accuracy of the inferred parameters (see Fig. F2 C). Fig. C&D show the error as a function of the ground truth parameter value and Table
shows the results from Fig. C numerically.
A
B
C
Behavior
B
0o
ours
MSE
MSE
r=5
enelytical
0.04
true
0.005
6
0.3
0.02
response
0.00
0.000
4
6
0.2
MSE
Ho
MSE
5.0
0.0050
0
0.0025
0.6
0.0
0.0000
stimulus
4
2.
.3
2
3
.75
1.0
Ho
H4
14
0g
B
Fixed parameters
Figure F.2: A Simulated behavior from the quadratic cost function with effort cost (Eq. (6)). The responses exhibit undershots, which could be due to the prior or due the cost. B Posteriors with the analytical model versus the neural network model. For each pair of parameters, the plot shows the contours of the 94%-HDI. Our method approximates the posterior with the analytical solution well. The pairwise posterior between prior mean μ4 and effort cost parameter 3 shows strong correlation. C MSE between ground truth and posterior mean when fixing no parameters, the effort parameter or the prior parameter Ao. Fixing of the confounding variables results in an improvement of accuracy in the inference of the other variable. The remaining parameters maintain their accuracy.
20

---

Published as a conference paper at ICLR 2025
F.3 ACCURACY INFERENCE WITH AMORTIZED OPTIMAL ACTIONS
F.3.1 COMPARISON OF POSTERIOR MEANS AND STANDARD DEVIATIONS
To assess the accuracy of the inference with amortized optimal actions, we used the data sets also shown Fig- and Fig.BC.. We compared means and standard deviations of the posteriors obtained with the analytical solutions for the optimal actions a² with the neural network approximation for the subject's s decision-making. The neurally amortized inference accurately produces very similar posteriors to the ones obtained using the analytical solution, shown in Fig. F3
Manalyticll - -un
anaaltical - Gnn
Ho
0g
Ho
do
Gr
0.2
0.025
0.000
0.0
-0.025
2.5
5.0 0.1
0.2
0.1
0.2
2.5
5.0 0.1
0.2
0.1
0.2
ground
ground truth
ground
ground
ground truth
ground
truth
truth
truth
truth
Manulytccl -hin
analyyccll Om
Ho
do
Hg
dg
0.5
0.05
0.0
0.00
0.5
1.0
2.5 5.0
0.1 0.2
0.1 0.2
0.5 1.0
2.5 5.0
0.1 0.2
0.1 0.2
ground
ground truth
ground truth
ground truth
ground
ground truth
ground truth
ground truth
truth
truth
Figure F.3; Error between posteriors as function of ground truth parameter values. For each simulated data set in our evaluation (Fig. C and Fig. 3 C), we computed the difference between posterior means and standard deviations for posterior distributions obtained using analytical optimal actions and our neural network approximations. Difference between posterior means for quadratic cost. B Difference between posterior standard deviations for quadratic cost. C Difference between posterior means for quadratic cost with quadratic effort. D Difference between posterior standard deviations for quadratic cost with quadratic effort.
21

---

Published as a conference paper at ICLR 2025
F.3.2 MEAN SQUARED ERRORS
We additionally summarize the mean squared errors visualized in Fig. 2 (Table F.1 (Table F22 and Fig. F (Table F3
Fig.
MSE
Parameter
analytical
NN
μ4
0.10
0.10
3.98 x 10-1
3.77 x 10-1
0r 6.23 X 10 6.13 x 10
Table F.1: Summary of results for quadratic cost function from Fig. C Each entry shows the mean squared error (MSE) between the posterior means and ground truth parameters, obtained using the analytical neural network optimal action.
MSE
Parameter
analytical
NN
110
0.74
0.77
00
3.63 10
4.08 x 10
Or
4.87 X 10
4.86 x 10
8
8.97 x 10-4 8.97 x 10-3
Table F.2: Summary of results for quadratic cost with quadratic effort from B Each entry shows the mean squared error (MSE) between the posterior means and ground truth parameters, obtained using the analytical vs. neural network optimal action.
Parameter
MSE (NN)
0.34
3.87 x 10-4
Or
5.05 x 10-4
3.23 x 10-2
Table F3:: Summary of results for asymmetric quadratic cost Fach entry shows the squared
error (MSE) between the posterior means and ground truth parameters, obtained using the neural network optimal action.
22


Reviews:
[{"review_id": "CHrpkDBoiw", "reviewer": "Reviewer_bwX5", "summary": "This paper introduces a method for performing Bayesian inference on the parameters of Bayesian observer-actor models, particularly suited for scenarios where Bayesian decision-making can be computationally intractable. The approach leverages a neural network to amortize the decision-making process of the subject by training the network to minimize the expected task-relevant cost with respect to the posterior over latent states and the action distribution. This setup allows for efficient, gradient-based inference of parameters from behavioral data. The authors validate their approach on synthetic data, highlighting its effectiveness and also discuss identifiability issues with recommendations to mitigate them. They further illustrate the method's applicability to human behavioral data.", "questions": "- In figure 2A, could you clarify what is $r^{\\ast}$? Should it be $a^{\\ast}$ instead?\n\n- The top left panel in figure 2B is missing a label. Should it be $\\sigma_0$?\n\n- In figure 2C, it would be useful to include separate x-axis labels for the analytical and nn cases.\n\n- The caption for figure 3B uses $\\beta$ as cost asymmetry parameter, but all figure labels use $\\alpha$. Are they the same?\n\n- In figures 3B and 3C, it would be helpful to make the ranges of the axes the same in all panels. \n\n- Figure 4b is difficult to follow, including a legend would be very helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper address an important bottleneck in inverse decision-making by amortizing the agent's behavior using a neural network. This enables efficient Bayesian inference over the subject's behavioral model parameters. The experiments on synthetic data validate the approach through comparison with analytical solutions. The discussion on identifiability and the experiment design recommendations add valuable practical insights.", "weaknesses": "In this work, the proposed approach aims to infer what a subject’s decisions were optimal for. However, there is still an assumption of optimal behavior, which may not always hold in real-world scenarios. Factors such as suboptimal learning or changing task demands can lead to deviations from optimality. Even if these deviations could potentially be reframed as an alternative optimality criterion, doing so would introduce additional identifiability challenges. It would be beneficial  to discuss the limitations of this assumption.\n\nAnother potential limitation is that the use of the reparameterization trick requires a specific form of action distribution, which may restrict the model’s adaptability to diverse datasets and tasks where this distributional form does not apply.\n\nFinally, the presentation could be improved. Several figures lack clear labels and legends, making them difficult to interpret, and acronyms are introduced without prior definition. A revised presentation with attention to these details would enhance the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V1egZ4TdhT", "reviewer": "Reviewer_QV14", "summary": "The paper addresses the challenge of using Bayesian models to infer decision-making parameters (inverse decision making) from behavioral data, especially for tasks involving continuous actions where traditional Bayesian methods struggle with computational intractability. \n\nThe authors propose a new method where a pre-trained neural network, trained unsupervisely, was used to approximate an actor model’s parameter. The gradient-based Bayesian inference makes the method relatively efficient. This approach shows promising alignment with analytical solutions where they exist and effectively models human behavioral data in various sensorimotor tasks.", "questions": "How scalable is the current approach? What are the computational requirements for training the neural networks for more complex cognitive reasoning tasks?\n\nCould the authors provide more details about the choice of network architecture and hyperparameters?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper provides an innovative approach by using a neural network to approximate the Bayesian model for inverse inference, which traditionally faces computational intractability issues. Their neural network method, trained in an unsupervised manner, enables efficient inference of decision-making parameters without relying on closed-form solutions or restrictive assumptions (like Gaussian distributions or quadratic costs).\n\nClear problem formulation and motivation.", "weaknesses": "The authors mentioned that their method could be applicable to a large number of tasks involving continuous responses, including economic decision-making, psychophysical production and crossmodality matching. However, the authors only tested their method on sensorimotor tasks. Testing methods on a diverse set of tasks involving continuous responses would significantly strengthen the paper.\n\nThe authors acknowledge that this method is currently constrained to relatively straightforward perceptual models. Extending it to more complex tasks (such as those involving circular variables or advanced cognitive reasoning) remains a limitation in its current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yLqKL8eWaD", "reviewer": "Reviewer_nJXd", "summary": "This paper considers an important and fundamental problem of inferring priors and costs from behavior. Typically, the inverse decision-making problem is intractable. Therefore, the author approximate the solution with a neural network, and show that the ground truth can be recovered well on simulated dataset. The author also explore the human behavioral data.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is clear, and the core ideas are well articulated. \n2. This paper introduces a novel approach for Bayesian inference about the parameters of Bayesian actor models.", "weaknesses": "The most significant concern is the lack of experimental advancements. This work only presents experimental results from numerical simulations and some simple human behavioral dataset, where simple MLP is able to recover posterior distributions. Presumably, the algorithm proposed by the author will face several challenge when we have to due with high-dimensional input.\n1. It might be hard to train $f_{\\psi} (\\theta, m)$ when $\\theta$ contains more than 100M parameters.\n2. The HMC might not have the property of rapid mixing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)