Inverse decision-making using neural amortized Bayesian actors

Authors: Dominik Straub, Tobias F. Niehues, Jan Peters, Constantin A. Rothkopf
Primary area: applications to neuroscience & cognitive science
Keywords: Bayesian actor models, perception and action, cognitive science, Bayesian inference, inverse modeling

Abstract:
Bayesian observer and actor models have provided normative explanations for many behavioral phenomena in perception, sensorimotor control, and other areas of cognitive science and neuroscience. They attribute behavioral variability and biases to interpretable entities such as perceptual and motor uncertainty, prior beliefs, and behavioral costs. However, when extending these models to more naturalistic tasks with continuous actions, solving the Bayesian decision-making problem is often analytically intractable. Inverse decision-making, i.e. performing inference over the parameters of such models given behavioral data, is computationally even more difficult. Therefore, researchers typically constrain their models to easily tractable components, such as Gaussian distributions or quadratic cost functions, or resort to numerical approximations. To overcome these limitations, we amortize the Bayesian actor using a neural network trained on a wide range of parameter settings in an unsupervised fashion. Using the pre-trained neural network enables performing efficient gradient-based Bayesian inference of the Bayesian actor model's parameters. We show on synthetic data that the inferred posterior distributions are in close alignment with those obtained using analytical solutions where they exist. Where no analytical solution is available, we recover posterior distributions close to the ground truth. We then show how our method allows for principled model comparison and how it can be used to disentangle factors that may lead to unidentifiabilities between priors and costs. Finally, we apply our method to empirical data from three sensorimotor tasks and compare model fits with different cost functions to show that it can explain individuals' behavioral patterns.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730608180432

thread_type:
official_review

<thread_object>
creator: reviewer_njxd

<content>
summary:
This paper considers an important and fundamental problem of inferring priors and costs from behavior. Typically, the inverse decision-making problem is intractable. Therefore, the author approximate the solution with a neural network, and show that the ground truth can be recovered well on simulated dataset. The author also explore the human behavioral data.

strengths:
1. The writing is clear, and the core ideas are well articulated. 
2. This paper introduces a novel approach for Bayesian inference about the parameters of Bayesian actor models.

weaknesses:
The most significant concern is the lack of experimental advancements. This work only presents experimental results from numerical simulations and some simple human behavioral dataset, where simple MLP is able to recover posterior distributions. Presumably, the algorithm proposed by the author will face several challenge when we have to due with high-dimensional input.
1. It might be hard to train $f_{\psi} (\theta, m)$ when $\theta$ contains more than 100M parameters.
2. The HMC might not have the property of rapid mixing.

questions:
Please refer to the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711700591

thread_type:
official_review

<thread_object>
creator: reviewer_qv14

<content>
summary:
The paper addresses the challenge of using Bayesian models to infer decision-making parameters (inverse decision making) from behavioral data, especially for tasks involving continuous actions where traditional Bayesian methods struggle with computational intractability. 

The authors propose a new method where a pre-trained neural network, trained unsupervisely, was used to approximate an actor model’s parameter. The gradient-based Bayesian inference makes the method relatively efficient. This approach shows promising alignment with analytical solutions where they exist and effectively models human behavioral data in various sensorimotor tasks.

strengths:
The paper provides an innovative approach by using a neural network to approximate the Bayesian model for inverse inference, which traditionally faces computational intractability issues. Their neural network method, trained in an unsupervised manner, enables efficient inference of decision-making parameters without relying on closed-form solutions or restrictive assumptions (like Gaussian distributions or quadratic costs).

Clear problem formulation and motivation.

weaknesses:
The authors mentioned that their method could be applicable to a large number of tasks involving continuous responses, including economic decision-making, psychophysical production and crossmodality matching. However, the authors only tested their method on sensorimotor tasks. Testing methods on a diverse set of tasks involving continuous responses would significantly strengthen the paper.

The authors acknowledge that this method is currently constrained to relatively straightforward perceptual models. Extending it to more complex tasks (such as those involving circular variables or advanced cognitive reasoning) remains a limitation in its current form.

questions:
How scalable is the current approach? What are the computational requirements for training the neural networks for more complex cognitive reasoning tasks?

Could the authors provide more details about the choice of network architecture and hyperparameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742238361

thread_type:
official_review

<thread_object>
creator: reviewer_bwx5

<content>
summary:
This paper introduces a method for performing Bayesian inference on the parameters of Bayesian observer-actor models, particularly suited for scenarios where Bayesian decision-making can be computationally intractable. The approach leverages a neural network to amortize the decision-making process of the subject by training the network to minimize the expected task-relevant cost with respect to the posterior over latent states and the action distribution. This setup allows for efficient, gradient-based inference of parameters from behavioral data. The authors validate their approach on synthetic data, highlighting its effectiveness and also discuss identifiability issues with recommendations to mitigate them. They further illustrate the method's applicability to human behavioral data.

strengths:
This paper address an important bottleneck in inverse decision-making by amortizing the agent's behavior using a neural network. This enables efficient Bayesian inference over the subject's behavioral model parameters. The experiments on synthetic data validate the approach through comparison with analytical solutions. The discussion on identifiability and the experiment design recommendations add valuable practical insights.

weaknesses:
In this work, the proposed approach aims to infer what a subject’s decisions were optimal for. However, there is still an assumption of optimal behavior, which may not always hold in real-world scenarios. Factors such as suboptimal learning or changing task demands can lead to deviations from optimality. Even if these deviations could potentially be reframed as an alternative optimality criterion, doing so would introduce additional identifiability challenges. It would be beneficial  to discuss the limitations of this assumption.

Another potential limitation is that the use of the reparameterization trick requires a specific form of action distribution, which may restrict the model’s adaptability to diverse datasets and tasks where this distributional form does not apply.

Finally, the presentation could be improved. Several figures lack clear labels and legends, making them difficult to interpret, and acronyms are introduced without prior definition. A revised presentation with attention to these details would enhance the paper.

questions:
- In figure 2A, could you clarify what is $r^{\ast}$? Should it be $a^{\ast}$ instead?

- The top left panel in figure 2B is missing a label. Should it be $\sigma_0$?

- In figure 2C, it would be useful to include separate x-axis labels for the analytical and nn cases.

- The caption for figure 3B uses $\beta$ as cost asymmetry parameter, but all figure labels use $\alpha$. Are they the same?

- In figures 3B and 3C, it would be helpful to make the ranges of the axes the same in all panels. 

- Figure 4b is difficult to follow, including a legend would be very helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732185819718

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to Reviews

comment:
We would like to thank all reviewers for their reviews. In light of some of the questions and comments, we would like to clarify the conceptual framework of our method: we are not just solving the decision-making problem faced by an agent in a task, but instead, we are taking a so-called doubly-Bayesian approach. This means that we frame both the perceptual problem of a participant in an experiment and the statistical problem of a researcher analyzing the observed behavior of a participant from a Bayesian perspective. In other words, we perform Bayesian inference on the free parameters of Bayesian decision-making models.

Second, the researcher performs Bayesian inference about the parameters of the subject’s internal model and cost function. Thus, our paper deals with the much more difficult problem of inference over the decision-making problem faced by an agent. This is comparable to the difference between solving the reinforcement learning problem versus solving the inverse reinforcement learning problem in sequential decision-making.

Third, inverse modeling does not imply that subjects are optimal from the perspective of the researcher who is analyzing behavioral data. Instead, behavior can be highly suboptimal from the perspective of the researcher and could even be fully random, and this would still be expressible through internal uncertainties and cost functions. Put differently, subjects are always optimal with respect to a certain cost function. However, the cost function a researcher assumes can be vastly different from the cost function a participant implicitly employs, and this is further distorted by perceptual and motor noise and potentially false beliefs. This is the fundamental premise of inverse modeling, which, therefore, fourth, fundamentally reconciles normative (“optimal versus suboptimal”) and descriptive (“data fitting”) models.

Fifth, without revealing our identity, we have contributed over the last 20 years to quite a number of algorithms for inverse modeling precisely to address “real-world scenarios”, i.e., finding the internal beliefs and uncertainties and internal costs and benefits in naturalistic behavior! To pick one specific example, computing Bayes-optimal actions is intractable for more general, structured cost functions, as they are relevant in naturalistic settings. Previous work has often focused on quadratic costs, which admit analytical solutions for certain posteriors, or derived custom tools tailored to a specific task. Here, we propose a new unsupervised training scheme for neural networks to approximate Bayesian actor models with general parametric cost functions, which broadens the range of possible cost functions, e.g., to include subjective internal costs, allowing us to identify resource-rational decisions.

Sixth, performing inference about the parameters of Bayesian actor models given behavioral data is computationally very expensive because each evaluation of the likelihood requires solving the Bayesian actor’s decision-making problem. By utilizing the neural network approximation, we can perform efficient inference in the inner inference loop. Finally, a very large number of tasks involving continuous responses across fields ranging from economic decision-making (‘how much would you wager in the following bet?’) to psychophysical production (‘hit the target’), magnitude reproduction tasks (‘reproduce the duration of the tone you just heard with a button press’) and sensorimotor tasks (‘reproduce the force you just felt with the robotic manipulandum’) or even cross-modality matching tasks (‘adjust a sound to appear as loud as the brightness of this light’).

Therefore, we see very broad applicability of our method in the behavioral sciences including cognitive science, psychology, neuroscience, and behavioral economics. We will update the final version of the paper by more clearly stating the conceptual framework and methodology.
Therefore, in our view, this paper represents a significant step beyond the state of the art in the modeling of decision-making in the fields of computational cognitive science, computational neuroscience, and economics.

We would like to ask the reviewers to additionally see the individual responses we provide below. If our responses can clarify some of the questions posed in the reviews and lead the reviewers to appreciate the value of our method, we would very much appreciate it if the reviewers consider increasing the score.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigate a new method for inverse decision-making in sensorimotor tasks with continuous actions.
The proposed approach aims to infer what are a subject’s optimal decisions, with the assumption of optimal human behavior. 
As mentioned by reviewers, the parameterization trick requires a specific form of action distribution, which will restrict the model’s adaptability to diverse online optimization tasks.
The AC is very familiar with Bayesian optimization theory and algorithms, but not psychology. It might be a good paper for a psychology conference instead of an AI paper.