Computational and neural mechanisms underlying the influence of action affordances on value learning

Domain: neuroscience
Published: 2026-01-08
Editorial outcome: major_revision

Abstract
When encountering a novel situation, an intelligent agent must identify which actions are most beneficial for interacting with that environment. One purported mechanism for narrowing down the scope of possible actions is action affordance. Here, we examine the neuro-computational mechanisms accounting for how action affordance shapes value-based learning in a novel environment by combining a novel behavioral task with computational modeling of human behavioral and fMRI data. Our findings indicate that action affordance operates as an independent system that concurrently guides action-selection alongside value-based decision-making rather than simply exerting an initial or persistent bias on value-driven choices. These two systems engage in a competitive process to determine final action selection, governed by a dynamic metacontroller. We find that the pre-supplementary motor area and anterior cingulate cortex play a central role in exerting metacontrol over the two systems. Meanwhile, the posterior parietal cortex integrates the predictions from these two controllers to determine which action to select. As a result, the action-selection process dynamically takes into account both the expected value and the appropriateness of particular actions for a given scenario.

Introduction
In order to interact successfully with situations as they occur in the world, humans and other animals need to select particular actions from a very large set of possible actions based on which actions are most appropriate to the situation. One fundamental guiding principle for action selection is that actions should be selected based on their expected value, that is by how much a particular action might increase an individual’s access to rewards, or decrease potential exposure to aversive outcomes1–3. A large literature has shed light on the neural and computational underpinnings of value-based action selection, including reinforcement-based mechanisms for learning which actions to select based on the future expected rewards they engender4–6. However, when encountering a stimulus for the first time in a particular context, the value of that stimulus is largely unknown and the brain needs to have a strategy to reduce the extremely large set of possible actions that could be selected to a tractable set of possible actions that could form the basis of subsequent trial and error learning. One putative mechanism for this is visual affordance7–9. Action affordances are features of stimuli in the world that suggest the appropriateness of particular actions, which can make the action selection problem more tractable. For example, a computer keyboard might suggest pressing or poking actions, a watering can might suggest a grabbing or clenching action, a pair of chopsticks might suggest a pinching action.

It has previously been found that affordance automatically potentiates particular actions that are compatible with properties an object or scene presented8,10–12. Particularly, it has been suggested that the selection of visually guided actions is supported directly by affordance and that this affordance mechanism is in turn biased by other decision variables such as the value of choice options13,14. However, the role that affordances might play in actually guiding learning during value-based decision-making is essentially unknown. A natural hypothesis is that affordance acts as an initial prior on value learning by, for example, subtly inflating the value of the afforded action during initial choice behavior so as to guide exploration, or alternatively, affordance might act as a constant yet moderate tug on action selection, persistently increasing the likelihood that an affordance-based action is selected in spite of the effects of value learning. Yet another possibility is that affordance-based choice and value-based choice operate as independent systems, competing for access to behavior. As we will see, our findings rather surprisingly support this latter possibility and suggest a role for a dynamic arbitration between independent expert systems implementing affordance and value-based choice.

To investigate how affordance influences decision-making, we designed a novel behavioral task that examines conditions in which the most rewarding action aligns with an affordance-compatible action and those in which they are incongruent. The incongruent conditions, in particular, capture a fundamental aspect of real-world action learning: the need to override intuitive action affordances when adapting to novel tools, interfaces, or environments. Many everyday tasks-such as learning to use chopsticks or mastering a musical instrument-require individuals to suppress immediate, affordance-driven responses in favor of newly learned actions that lead to the desired outcome.

To systematically explore the mechanisms underlying this process, we employed a computational model-driven approach. We specified a series of computational models to capture different ways in which affordance might influence value learning and tested these models against human behavior. Additionally, we used functional MRI (fMRI) to measure brain activity and identify the neural correlates of these computational processes.

By examining how participants adjust their motor responses in response to probabilistic rewards, our study provides valuable insights into the cognitive mechanisms supporting this type of behavioral adaptation. Just as real-world learning involves trial and error, feedback processing, and reinforcement learning, our task allows us to isolate and analyze these processes in a controlled setting. Thus, the overarching goal of this study is to investigate how action affordances interact with value-based action learning across behavioral, computational, and neural levels.

Results
Effects of affordance on reaction times and action selection
We first ascertained to what extent did the affordance properties of a stimulus influence response reaction times (RTs). We expected that choice of an action compatible with the dominant action affordance for an object would be associated with shorter RTs than choice of an affordance incompatible action regardless of the presence of ongoing value learning10–12. RTs were defined as the interval between the stimulus onset and initiation of movement, and were indeed found to be significantly shorter when making affordance-compatible actions compared to affordance-incompatible actions in both the behavioral and fMRI data even when participants were freely exploring the environment for finding the most rewarding hand gesture (Fig. 2a, f; for details on the reaction time measurement, see Methods). Moreover, the difference in RT remained consistent throughout. The average RT difference between the first 5 trials and the last 5 trials was not significantly different in either dataset (t(18)=-1.329, p=0.20 in the behavioral and t(29)=0.7592, p=0.45 in the fMRI data). We also found that the RTs were influenced by value-learning, in that participants were faster to respond to stimuli with higher expected value than stimuli with lower expected value (Supplementary Fig. 1). Thus, to disambiguate the effects of affordance on RTs from other possible confounding variables such as experimental condition, action type, choosing the most-rewarding action, and the progress of learning across trials, all of which are associated with value learning, we conducted mixed-effect linear regression which included each of these effects as potential confounding covariates. Even after accounting for these confounds, the effect of affordance on RT remained significant (Supplementary Table 1; β = −0.026, z = −2.66, p≤0.01 for selecting affordance-compatible actions in the behavioral experiment, and β = −0.029, z = −3.02, p≤0.01 in the fMRI experiment, which indicates that RTs for affordance-compatible actions were about 2.7% shorter). These results not only confirm that our experimental paradigm and stimulus set reliably induce affordance-related response preparation effects, but also reveal that the reaction time effect due to stimulus-response compatibility persists regardless of the effects of value-learning.

Next, we aimed to examine the effect of affordance on which action was chosen on a given trial. Specifically, we hypothesized that the affordance associated with a particular object would bias choices in favor of the afforded action, independently of the expected value of that action. To test for this, we analyzed the initial responses participants made to each object, as those are the actions not affected by value learning. The probability of selecting affordance-compatible actions as the initial response was significantly higher than chance (1/3) in both datasets (Fig. 2b, g). Furthermore, those initial actions were biased toward the afforded action that each object was selected to confer based on the initial affordance ratings we previously obtained in a separate sample (Supplementary Table 2; χ2 test; null hypothesis was the probability distribution of choosing each action type independent of the affordance of objects calculated using the choice data; χ2(2, N=304)=9.33, p≤0.01 for pinch, χ2(2, N=304)=16.02, p≤0.001 for clench, χ2(2, N=304)= 21.12, p≤10−4 for poke in the behavioral experiment; χ2(2, N=240) = 17.91, p≤0.001 for pinch, χ2(2, N=240) = 16.18, p≤0.001 for clench, χ2(2, N=240) = 16.54, p≤0.001 for poke in the fMRI experiment). Moreover, the initial selection bias remained constant throughout the task for all new objects introduced over the course of the experiment (Supplementary Figs. 2a and 2c for details). This observation rules out an alternative explanation regarding the initial selection bias which posits that participants might have gradually inferred that the structure of the task is such that affordance-compatible actions are the most-rewarding action in half of the trials, and thus, the baseline probability for selecting affordance-compatible action would be 1/2 rather than 1/3. According to this hypothesis, the initial selection bias toward affordance-actions would be expected to increase as the task progresses as participants become more aware of the task structure with increasing experience. However, the actual data contradict this notion by showing that the initial selection bias remained stable and did not increase as the task progressed in both the behavioral and fMRI studies.

To further ensure that the observed initial bias toward affordance-compatible actions was not merely driven by prior familiarity or semantic associations with the presented objects, we examined participants’ familiarity ratings for all stimuli used in the behavioral and fMRI experiments. This analysis was designed to rule out the possibility that the effect reflected only conceptual knowledge about objects, rather than the physical properties inferred from their visual features. The familiarity scores varied substantially across items (66.45 ± 24.62 on a 0−100 scale, Supplementary Fig. 3). However, the difference in the probability of selecting affordance-compatible actions on the initial trial between unfamiliar objects (familiarity score < 66.45) and familiar objects (familiarity score ≥ 66.45) was not significant (t(18)=1.529, p=0.144 in the behavioral, t(29)=0.001, p=0.996 in the fMRI). These results indicate that the initial bias toward affordance-compatible actions cannot be explained by familiarity or semantic factors alone and instead arises from the physical affordances implied by the objects’ visual characteristics.

On top of that, we observed significant positive correlations across participants between these two distinct affordance-related bias effects on choice. We computed the degree of RT bias as the additional reaction time for executing affordance-incompatible actions relative to affordance-compatible actions (RT effect = RTincomp − RTcomp / RTcomp) for each participant and compared it to the frequency of choosing affordance-compatible actions throughout the task. Significant correlations were observed between the two metrics in both datasets (Fig. 2c, h), and moreover, the RT effect positively correlated with the frequency of selecting the affordance-compatible action as the initial response to each object (Supplementary Figs. 2b, d; Pearson r = 0.41, p = 0.08, df=17 in the behavioral, Pearson r = 0.58, p≤0.001, df=28 in the fMRI experiment). These findings suggest that both forms of affordance-related biases on choice behavior are related and have a shared substrate.

Affordance influences value learning
The pronounced action selection bias toward the afforded action also influenced value-learning and contributed to the choice accuracy difference between congruent and incongruent conditions. To be specific, selection of the most rewarding action for each object was significantly greater in the congruent than in the incongruent conditions (t(18)=2.50, p≤0.05 in the behavioral, t(29)=3.25, p≤0.01 in the fMRI experiment). However, the bias toward selecting actions based on affordance was most evident in the early phase of the interaction with each specific object and diminished across subsequent trials involving that specific object. As demonstrated in Fig. 2d, i, statistical tests on the difference in choice accuracy between congruent and incongruent conditions in each trial revealed that the affordance effect was significant on early trials within a block but became less pronounced as learning progressed in both experiments.

Because the exploration of choice options is biased by action affordance, participants are likely to need more trials to experience positive outcomes in incongruent than in congruent conditions. Therefore, the choice accuracy difference between congruent and incongruent conditions might be due to an affordance bias operating on the choice process rather than reflecting the effects of impaired value learning.

To analyze the influence of affordance on learning further, we examined the choice accuracy difference between congruent and incongruent conditions as a function of the number of rewarded trials previously encountered. The analysis indicates that learning slopes were actually steeper in the incongruent conditions, as evidenced by a decreasing pattern in the difference between the two learning slopes from each condition (Fig. 2e, j). Mixed-effect GLM analyses confirmed that the choice accuracy difference between the conditions is decreasing and that the incongruent condition has a steeper slope than the congruent condition (Fixed-effect coefficients for the gradient of the choice accuracy difference ΔP: β = − 0.010, z = −3.16, p≤0.01 in the behavioral, β = −0.009, z = −2.97, p≤0.01 in the fMRI experiment; Supplementary Table 3).

As previously mentioned, two possible explanations might account for this effect: first, it is possible that participants go through a longer exploration phase between receiving rewards in the incongruent conditions, which might support counterfactual learning during exploration, leading to steeper learning15,16. In such scenarios, using a reinforcement learning (RL) model that incorporates negative learning signals to update the values of unchosen actions in the trials without rewards would be sufficient to account for the learning effect, as such a model could capture counterfactual learning. Alternatively, it is plausible that behavioral adaptation based on reward history is more sensitive in the incongruent conditions. This could result from having a higher learning rate in incongruent conditions, or it could be due to the exertion of a higher level of cognitive control on incongruent conditions resulting in decisions that are more heavily reliant on learned values17–19.

As we will see, the computational modeling we implement supports the notion that cognitive control is allocated to balance the influence of affordance and value-learning in order to govern task performance (see below).

We also plotted the learning slope in high and low conditions as the function of the number of rewarded trials previously encountered. However mixed-effect GLM analyses revealed there is no statistically significant decrease in the difference between high and low conditions (Fixed-effect coefficients for the gradient of the choice accuracy difference ΔP: β = − 0.006, z = − 1.30, p = 0.20 in the behavioral, β = −0.010, z = −0.66, p = 0.51 in the fMRI experiment; Supplementary Fig. 4).

Dynamic meta-level control merging affordance-based and value-based decision-making best explains behavior
To investigate the underlying computational mechanism responsible for these behavioral effects, we implemented various computational models, fit those to participant’s behavioral data and then performed a formal model comparison (see Methods for details; Fig. 3a). We applied an RL model to capture value learning. We then modeled the degree of affordance assigned to each action for each object using the affordance-compatibility scores for the hand gestures for each object provided by the participants themselves after the completion of the value-based choice task.

We also tested the possibility that value learning could be supported by Bayesian inference rather than RL (Bayes model)20. Although Bayesian learning offers a normative framework for optimal belief updating, we found it less well-suited for capturing the flexible and sometimes suboptimal behavioral patterns observed in our task. We implemented several Bayesian variants, including models where affordance was incorporated either as a prior or as a bias influencing belief updates. However, these Bayesian models consistently showed poorer fits to the data compared to RL models (See Fig. 3), which allowed for more expressive adaptation to feedback. Given the greater flexibility of RL and its empirical success in modeling value-based choice behavior, we focused our subsequent modeling efforts on RL-based models.

One simple way in which affordance might influence choice is via a constant action-selection bias–essentially providing a constant push toward choosing the afforded action on each trial over and above other considerations such as value. Another possibility is that affordance acts as an initial prior operating on the initial value of the afforded action. If the afforded action is presumed to have a higher initial value than the other actions, this could produce a bias toward choosing that action more often at the beginning of a block. We implement both of these possible biases in separate RL models, either as a bias in the decision variable (bias model), or as a bias in the initial values assigned to an action before the onset of reinforcement-learning (prior model).

In addition, based on the observation that the steeper learning slope in incongruent conditions (Fig. 2e, j) could be captured by cognitive control resulting in a greater focus on value in the incongruent conditions, we explored models inspired by the concept of a mixture of experts21. In this class of models, we assumed that two different decision-making systems are concurrently making predictions about the appropriate action for a given object. The first system is an affordance-based decision-making system, which simply makes choices in a manner proportional to the degree of affordance attached to particular actions, while the second system is a value-based decision-making system, by which actions get selected based on their learned expected values. Additionally, we assumed a meta-level controlling mechanism that arbitrates between the two systems and mediates the influence of the two systems in selecting actions. Four candidate arbitration mechanisms were tested.

The first model assumes that the outputs from the two systems are mixed with a fixed weight, which is conceptually similar to the bias model (Fixed arbitration model). We also tested a model that assumed affordance acts as a cost that makes the outcome from selecting affordance-incompatible actions less rewarding, thereby hindering selection of affordance-incompatible actions. This component was added on top of the fixed arbitration model (Cost model)22. The second type of arbitration model gave a boost in control to the value-based decision-making system, when the two systems made conflicting predictions, which mainly happened in incongruent conditions (Conflict-based arbitration model)23–25. The third model type assigned a greater degree of control to the system that had the lower level of prediction errors, or a higher level of reliability in its predictions, so that participants could minimize the uncertainty in predicting the values of each action. The reliabilities of each system were estimated using the absolute value of reward prediction errors (RPE) or affordance prediction errors (APE), which is the difference between the outcome and the affordance compatibility of the chosen action (Reliability-based arbitration model)21,26,27.

The last type of model allowed for a larger influence from a system that had higher expected outcomes when the decision maker followed that particular system in making a decision18,22. By doing so, the decision maker can maximize the outcome they can collect (see Methods for details). The performance of each system, or the expected outcome by following the specific decision-making system to a given object, was estimated using a method called inverse propensity scoring28,29, which was implemented in a form of delta rule supported by performance prediction errors (PPE) within each system. (Performance-based arbitration model; see Methods for details).

As illustrated in Fig. 3b, h, the performance-based arbitration model was found to best explain the actual choice data in terms of group-level log model evidence. Additionally, the Bayesian model selection results showed that the posterior model frequency and protected exceedance probabilities were in agreement with the above model comparison results (Supplementary Fig. 5). It is noteworthy that the task design used in this study had sufficient power to differentiate between the various decision-making models tested. For example, the performance-based arbitration model could be well recovered when the actual data generative process was based on itself. When the performance-based model provided the best fit for a given individual’s data, the probability of the performance-based model being the true generative model of the data was found to be 0.89 (Supplementary Table 4; Methods).

In addition, the variables extracted from the best performing model suggest that the congruency between affordance and value determines how the meta-level controller weighs each system. Specifically, the arbitration weight on value-based decision-making was found to be higher in incongruent conditions (Fig. 3c, i; t(18)=2.28, p≤0.05 in the behavioral; t(29)=3.59, p≤0.01 in the fMRI data). We also observed that how often rewards were given on recent trials increased the arbitration weight toward value-based decision-making22. In the high conditions where rewards were given more frequently, the weight on the value-based decision-making system was higher (Supplementary Figs. 6a, f; t(18)=2.53, p≤0.05 in the behavioral study; t(29)=3.77, p≤0.01 in the fMRI study). Moreover, those trials with less weight on value-based decision-making were also trials in which the actual choice was compatible with the afforded action more often (Fig. 3d, j). We also observed that those participants who exhibited higher accuracy in selecting the most rewarding hand gestures were the individuals who also had a higher average arbitration weight assigned to the value-based decision-making system (Supplementary Figs. 6b, g; Pearson r = 0.62, p≤0.01, df=17 in the behavioral, Pearson r = 0.65, p≤0.001, df=28 in the fMRI experiment).

Furthermore, through model simulations utilizing the estimated parameters, we observed that the performance-based arbitration model could very closely replicate behavioral patterns found in the real choice data. In the simulated data, initial action selection was biased toward affordance-compatible actions and this bias was consistent across the blocks (Fig. 3e, k; Supplementary Fig. 8) as found in the real data described earlier. The simulated learning curves also exhibited a close correspondence with the actual learning curves in each condition, and the choice accuracy gap between congruent and incongruent conditions showed a tendency to be larger in the earlier trials of each object, which was consistent with the actual data (Fig. 3f, l; Fixed-effect coefficients for the gradient of the choice accuracy difference ΔP: β = − 0.001, z = − 2.26, p≤0.05 in the behavioral, β = −0.001, z = −1.50, p = 0.13 in the fMRI data simulation; Supplementary Table 5).

It is notable that, apart from the performance-based arbitration model, simulations using other types of arbitration model did not show an initial action selection bias or a gradual decrease in the choice accuracy gap between congruent and incongruent conditions (Fig. 3e, f, k, l; Supplementary Fig. 9). Although the cost and the conflict-based arbitration models could exhibit several properties of the actual choice data with specific ranges of free parameters, simulations using the fitted parameters could not reproduce the patterns from actual choices (Supplementary Fig. 9). For example, the choice accuracy gap was increasing in the cost model, which is the opposite pattern from the real data. The conflict-based arbitration model did not show a statistically significant bias toward affordance compatible actions in the initial trial. Moreover, in this model, the simulated choice accuracy in incongruent condition was even better than, or not statistically significantly worse than, that in congruent condition, which contradicts the actual data (P(correct|incongruent) − P(correct|congruent) of the conflict-based arbitration model; t(189) = − 3.12, p≤0.01 in the behavioral data simulation, t(299) = −0.69, p = 0.49 in the fMRI data simulation). The fixed and the reliability-based arbitration models could reproduce the initial action selection bias, but the gradient of the choice accuracy gap between congruent and incongruent conditions was marginal compared to that from the performance-based arbitration model (Supplementary Table 5).

Additionally, when model-simulated learning curves were plotted as a function of the number of rewarded trials similar to Fig. 2e, j, the performance-based and reliability-based arbitration models could reproduce the real behavioral patterns, but not the fixed arbitration model (See Fig. 3g, m; Supplementary Fig. 10 and Supplementary Table 6). For example, the difference between the congruent and incongruent conditions in terms of frequency of choosing correct actions as a function of the number of rewarded trials was only decreasing in the simulated data using the performance-based and the reliability-based arbitration models, but not statistically significant in the fixed arbitration model’s simulated choices (Fixed-effect coefficients for the gradient of the choice accuracy difference ΔP: performance-based arbitration model’s β = −0.007, z = −6.64, p≤0.001 in the behavioral, β = −0.008, z = −8.81, p≤0.001 in the fMRI data simulation; reliability-based arbitration model’s β = −0.003, z = −3.08, p≤0.01 in the behavioral, β = −0.004, z = −4.28, p≤0.01 in the fMRI data simulation; fixed arbitration model’s β = −0.001, z = −0.91, p = 0.36 in the behavioral, β = 0.001, z = 0.62, p = 0.54 in the fMRI data simulation).

Nonetheless, the steeper learning slope we observed might be due to a potentially higher learning rate in the incongruent conditions. To explore this possibility, we fit the data with a fixed-arbitration model that incorporates two distinct learning rate parameters, one for incongruent conditions and another for congruent conditions. However, our analysis revealed no significant difference in learning rates between congruent and incongruent conditions (t(18) = −1.16, p = 0.26 in the behavioral, t(29) = 0.73, p = 0.47 in the fMRI data). Additionally, the model with two distinct learning rates exhibited lower log evidence compared to the single learning rate model and was unable to replicate the decreasing performance gap observed between congruent and incongruent conditions (see Supplementary Fig. 11). Moreover, we could eliminate the possibility that the presence of counterfactual learning alone is sufficient to demonstrate the steeper learning slope. With the implementation of counterfactual learning through the updating of values for unchosen actions in unrewarded trials, it is clear that counterfactual learning on its own falls short of replicating the actual behavioral data (see Methods and Supplementary Fig. 12). These results collectively indicate that the observed steeper learning slope in incongruent conditions is due to a dynamic arbitration mechanism that boosts the better working system for a given situation, rather than being a result of counterfactual learning in unrewarded trials or the effects of a larger learning rate in the incongruent conditions.

In addition, the non-decreasing difference between high and low conditions in terms of the frequency of choosing correct actions as a function of the number of rewarded trials could be captured by all tested arbitration models (Supplementary Fig. 13, Supplementary Table 7 and Supplementary Note 2).

Consequently, through rigorous model comparisons and simulations, we identified performance-based arbitration as the best candidate model for explaining the behavioral data, both quantitatively and qualitatively.

To further rule out alternative explanations, we additionally implemented and tested a series of simpler models. These included two passive-decay models in which the affordance bias decreased over time according to either an exponential or hyperbolic decay function, as well as a restricted arbitration model in which the arbitration weight was determined solely by the reliability of the value-based decision system. All of these models were significantly outperformed by the full arbitration models that incorporate dynamic performance or reliability tracking across both systems (see Supplementary Fig. 14). Moreover, simulations from these simplified models failed to capture a key behavioral feature: the decreasing gap between congruent and incongruent conditions in the frequency of correct action selection as a function of the number of rewarded trials. In both the decay-based and the value-reliability-only models, this gap failed to decrease significantly (see Supplementary Fig. 15). These findings suggest that neither a passive transition away from affordance-driven behavior nor an arbitration process based only on value reliability is sufficient to explain the observed data. Rather, a dynamic, bidirectional arbitration mechanism-tracking the performance or reliability of both systems-is necessary to reproduce the behavioral effects.

Another question that arises from our findings is the temporal dynamics between affordance-based and value-based processes. We considered behavioral approaches to explore this question. A speeded decision-making paradigm, for instance, could reveal whether one system dominates earlier in the decision process. If value information emerges later than affordance-based processes, then performance in incongruent conditions should be poorer during speeded decisions compared to slower ones, even post-learning. Conversely, if affordance-based processes come online later, participants should perform better in speeded decisions and worse in slower ones in incongruent conditions. Alternatively, if a central arbitration mechanism governs the integration of these processes, no significant performance differences would be expected across speeded and non-speeded decisions in incongruent conditions.

Since our study did not employ a speeded decision-making paradigm, we instead conducted supplementary analyses to compare trials with quick versus slow reaction times. GLM analyses on incongruent trials revealed that both choosing correct actions and affordance-compatible actions were associated with shorter RTs. Notably, the coefficient for choosing affordance-compatible actions was significant only in the behavior-only study, but not in the fMRI study (Supplementary Table 8; β = − 0.034, z = − 2.47, p≤0.05 for selecting affordance-compatible actions in the behavioral experiment, and β = −0.026, z = −1.57, p = 0.116 in the fMRI experiment). While performance was better in fast RT trials, slow RT trials were not consistently biased toward affordance-based decisions across studies. Therefore, we cannot definitively conclude that affordance acts as a consistent bias for value-based decisions or vice versa.

Next, we explored the neural implementation of affordance-based action-selection and its influence on value-based action-selection by utilizing computational variables from this model in the analysis of the fMRI data.

Neural correlates of affordance, value-learning, and action selection
We conducted a GLM analysis that included the chosen action’s affordance-compatibility score, action value and action selection probability from the performance-based model as parametric regressors to identify the regions associated with affordance and value-based decision-making respectively, as well as to uncover an action selection region responsible for integrating the predictions from these two systems to guide action-selection (GLM1; See Methods for details).

First of all, we found that the affordance-compatibility of the chosen action on each trial was encoded in the higher-level ventral visual stream such as V3 and V4 in the left occipital lobe which is the region that has been suggested to be responsible for affordance-perception or object recognition and processing physical properties such as shape and size that are necessary for hand gesture control (Fig. 4a)30–33. The chosen action value from the performance-based arbitration model was found in medial prefrontal cortex (mPFC) which is the region that has been implicated in the encoding of learned value (Fig. 4b)34–36.

Regions correlating with the action selection probability of the chosen action, which is the integration of the predictions from value-based and affordance-based decision-making systems were located in the posterior parietal cortex (PPC) near the parieto-occipital junction (POJ), ventral premotor cortex (PMv), and the posterior superior temporal sulcus (pSTS) all of which are parts of the cortical grasping circuit (Fig. 4c)33,37–39. Notably, these identified regions are responsible for not only performing the hand gesture itself but also integrating different sensory modalities related to manual actions40.

To ensure that regions correlating with the probability of the chosen action were not merely reflecting the effects of action execution vigor, we conducted an identical GLM analysis but incorporated RT as an additional parametric regressor41. These regions remained significant (at an uncorrected threshold of p < 0.001) suggesting that the regions are associated with the weighted sum of affordance-based and value-based decision-making systems rather than merely reflecting movement vigor (Supplementary Fig. 16).

Moreover, we extracted coordinates of the right-hand joints from the recorded hand videos and included them in the GLM as a parametric regressor in order to control for any potential confounding effects arising from the effects of hand movements per se (See Methods for details). While we found action selection related signals in various regions of the cortical grasping circuit, the actual motor implementation of the hand gesture was correlated with activity in the left primary motor cortex, left premotor cortex, left primary somatosensory cortex, and bilateral inferior parietal lobule (IPL) (Supplementary Figs. 17c-f).

Another GLM analysis using the second best fitting reliability-based arbitration model was conducted as well, which identified mostly overlapping regions for each of the cognitive variables (GLM2; See Supplementary Figs. 18a-c, and 16).

Neural implementation of the meta-level controller
Next, we probed for brain regions responsible for the meta-level computations involved in arbitrating between value and affordance-based decision-making systems. To achieve this, we utilized arbitration-related variables extracted from the performance model as parametric regressors in a GLM (GLM3; See Methods for details).

As illustrated in Fig. 4d, e, we found that the mPFC, rostral anterior cingulate cortex (rACC), posterior division of middle temporal gyrus (pMTG), and IPL to be associated with the performances of both systems. The posterior division of superior temporal gyrus (pSTG), dorso and ventro lateral prefrontal cortex (dlPFC, vlPFC), mid-cingulate cortex (MCC), orbitofrontal cortex (OFC), precuneous cortex (PCUN), and hippocampus (HPC) were correlated with the performance of value-based decision-making but not with the performance of affordance-based decision-making.

In addition, we observed that the signal corresponding to the difference in performance between the two decision-making systems (Perfaff − Perf q), which determines the arbitration weight, was present in the pre-supplementary motor area (preSMA), lateral prefrontal cortex (lPFC), and insular cortex (IC) (Fig. 4f). We also found that the IPL and PCUN were regions correlated with the difference in performance between the affordance and value-based decision-making systems. These regions have been shown to be associated with the implementation of cognitive control18,42–45. Interestingly, our findings indicate that the regions identified were less activated when the value-based decision-making system was allocated more weight.

Similar to the RPE signals (Supplementary Fig. 17a), the variables responsible for updating arbitration weights, which are also outcome-dependent prediction error signals, were found in the striatum (Fig. 4g). Specifically, the PPE of the value-based decision-making system was prominently encoded in the ventral striatum, while the PPE of the affordance-based decision-making system was found in the dorsal striatum.

Additionally, we attempted to identify changes in functional connectivity between the action selection region and other identified regions (chosen value, chosen affordance, performance of value, or performance of affordance) when there is a competition between actions, and presumably weighing more on the value-based choice for the final decision. However, we did not find a statistically significant change in functional connectivity. (For additional details, see Note 4 in the Supplementary Materials.)

In addition to investigating neural correlates for the performance-based arbitration model, we also tested for regions correlating with reliability-based arbitration which was the second best fitting model in our model comparison (GLM4). We also found clear neural correlates of reliability-based arbitration signals (Supplementary Fig. 17b and Supplementary Figs. 18d-g). In particular, we found the regions that correlate with the difference between the reliabilities of the two systems largely coincide with those identified using the performance-based arbitration model, particularly the preSMA. It is noteworthy that the average correlation between the arbitration variables of the two models were small (r = 0.196 in the fMRI data and r = 0.159 in the behavioral data). Therefore, the most plausible explanation is that each model captures distinct variance in activity in the preSMA. These findings indicate that the brain keeps track not only of the performance of the different systems but also of their reliability, suggesting that both variables might ultimately be taken into account during the arbitration process.

Better task performance is linked to a more robust representation of arbitration variables in meta-level controller regions
Next we explored the relationship between individual differences in task performance and representation of the arbitration variables. We found that individuals with higher task performance demonstrated more robust representations of the arbitration variables such as the performances of the two systems and the difference in performance between the two systems in brain regions involved in encoding arbitration-related variables. To be specific, we found positive correlations between each participant’s propensity to choose the most rewarding actions and the extent to which BOLD activity in the identified arbitration regions can be explained by our model-derived arbitration variables.

For instance, in those participants who tend to be more accurate in choosing the correct actions, the computational variable corresponding to the difference in performance between the two decision-making systems provided a better account of BOLD activity in the arbitration regions shown in Fig. 4f (r = 0.47, p≤0.01; Fig. 5a, See Methods for details). Moreover, in those participants who achieved higher accuracy in choosing the most rewarding actions, brain regions involved in representing the computational variables corresponding to the performance of each individual system were more correlated with the model-estimated performance variables (r = 0.38, p≤0.05 for the performance of the affordance-based system; r = 0.43, p≤0.05 for the performance of the value-based system; Fig. 5a). However, the strength of the neural representation of the within system computational variables produced by each system relevant for decision-making did not show statistically significant correlations with the participants’ accuracy (r = 0.23, p = 0.22 for the affordance; r = 0.08, p = 0.68 for the value; Fig. 5a). These findings, suggest that more robust neural representations of performance-based arbitration are associated with better behavioral performance on the task, providing additional evidence in support of the role for an arbitration process in mediating effective interactions between value-based and affordance-based action selection (Analogous results were also found for the reliability-based arbitration model; see Supplementary Fig. 19).

We also found that those individuals with a stronger behavioral affordance effect exhibited increased activation of performance-based arbitration regions when executing affordance-incompatible actions. Such individuals likely have to exert stronger behavioral control to suppress the prepotent affordance compatible action compared to individuals with a lower overall affordance effect in their behavior. To test for this, we looked at the relationship between the overall proportion of affordance compatible actions chosen by each individual and activation on trials where the affordance incompatible action was chosen compared to when it was not chosen (GLM5; See Methods for details). A significant correlation was observed in performance-based arbitration regions (Perfaff − Perf q regions in Fig. 4f; r = 0.59, p≤0.001; Fig. 5b). A similar effect was found when examining the relationship between the reaction time increase when executing affordance-incompatible actions compared to affordance compatible reaction times, and activity in those areas (r = 0.38, p≤0.05; Fig. 5b; See Methods for details). These findings suggest that regions involved in arbitration including the preSMA, have an important role in suppressing the prepotent affordance response, especially in those individuals who have a stronger overall bias toward affordance effects42.

Methods
Participants
We recruited 21 and 32 healthy participants for the behavioral-only experiment and the fMRI experiment respectively. No statistical method was used to predetermine sample size. 2 participants were excluded from the behavioral and another 2 from the fMRI experiment because they didn’t complete the study. Therefore, data from the remaining 19 (8 females, 18 right-handed; 1 ambidextrous, 18 ~24 years: 6; 25 ~34 years: 10; 35 ~44 years: 1; 45 years or above: 2) and 30 (13 females, 28 right-handed; 2 ambidextrous, 18 ~24 years: 12; 25 ~34 years: 12; 35 ~44 years: 4; 45 years or above: 2) were used for the analyses. Before taking part in the experiment, all participants were assessed to ensure that those with the history of neurological or psychiatric illness were excluded. All participants provided their informed consent. The study was approved by the Institutional Review Board of the California Institute of Technology (IR20-1033), which covered both the behavioral and fMRI experiments. Participants were compensated on an hourly basis ($20 per hour for behavioral experiments and $30 per hour for fMRI experiments), with an additional performance-based bonus of up to $5.

Stimuli
To implement the paradigm, we began by creating a set of stimuli. The stimulus set consisted of various object images, each of which was designed to engender a specific hand configuration among four hand movement classes known to be employed during object interaction: pinch, clench, poke, and palm (e.g., a button-shaped object that affords poking)90. We first obtained 1000 images of around 900 unique objects and built a set of visual stimuli by superimposing those object images onto a human silhouette image. We designed the stimuli in this way to ensure that the visual stimuli retain information about the object’s size.

Then we used those edited photographs in an online task on Amazon Mechanical Turk (M-Turk). Each image was displayed 4 times during the task to collect annotations on the suitability, or affordance-compatibility score, of 4 hand movements to the object displayed. Within a trial, the image with the human silhouette was displayed for 2s and the object’s zoomed-in image was shown for 1s along with a text of one of the 4 hand gestures. Then while the zoomed-in image was on the screen for additional 8s, M-Turk participants responded how suitable it is to pinch, clench, poke or palm the object shown by sliding a bar ranging from “Very unsuitable” to “Very suitable”. Additionally, we asked about their familiarity with the object 4 times. The affordance-compatibility and familiarity scores were both converted to 0 to 100 scale later. Throughout the task, 4 catch questions were used to check the participants’ attention.

We recruited 227 participants online, but only 160 of them were included in the data analysis as we excluded those that didn’t pass at least 2 catch trials. Each of the participants was asked to annotate 50 objects. On average, each object was annotated by 7.2 individuals.

Then we chose objects based on their 0−100 affordance-compatibility scores. For example, if an objects’ average affordance-compatibility score of pinch was > 50, and the mean score of pinch was significantly larger than the mean scores of other actions (independent t-tests, p < 0.05 uncorrected), the object was chosen as a pinch-affording object. By doing so we obtained 21 pinch-, 102 clench- and 16 poke-compatible objects. Then we randomly selected 16 items from each set, yielding 48 stimuli for the main task (Fig. 1a).

Task
We designed a variant of a 3-armed bandit task in which the available actions are natural pinch, clench, or poke gestures made by the right hand, which yields binary outcomes. We used 48 visual object stimuli for the behavioral experiment and 24 object stimuli (a subset of the 48 stimuli) for the fMRI experiment. On each trial, one of the stimuli showing an object with a human silhouette was displayed, and a participant mimed one of the three hand gestures while maintaining their wrist position on the provided wrist pad (Fig. 1a, e). The images used here (including the human silhouettes) were identical in form to the selected sub-set of images initially presented in the online survey. Participants were provided with instructional videos showcasing example hand movements corresponding to each gesture type to ensure that they had a clear understanding of the hand gestures under consideration. Participants were instructed that the reward probability of each hand motion is different for each object, and that they need to figure out the most rewarding hand gesture for each object by trying different hand motions (See Supplementary Figs. 20, 21 and 22 for the details). Each category of hand postures corresponded to actions with different reward probabilities, and an object was shown in a trial to evoke one of the three hand gesture affordances. There were 4 experimental conditions in this paradigm: congruent high, congruent low, incongruent high, and incongruent low conditions. In a congruent trial, an action that was consistent with the presented stimulus’ affordance had a high reward probability, whereas in an incongruent trial, the opposite applied. Each hand position had a reward probability of 0.8, 0.2, or 0.2 in the high condition and 0.4, 0.1, or 0.1 in the low condition.

The task consisted of a 2 day experiment for the behavioral-only study and a 1 day experiment for the fMRI study, with 76~84 trials (on average 80 trials) per block, and 6 blocks per day, totaling 960 trials for the behavioral study and 480 trials for the fMRI study. In each block, 4 distinct objects were displayed using the event-related design. Each object corresponded to one of the 4 conditions and was shown for an average of 20 trials (ranging from 19 to 21 trials). All objects were displayed once in every 4 trials in random order. Everyone in the behavioral-only and fMRI experiment was shown the same set of 48 or 24 objects respectively, but the associations between experimental conditions and stimuli was randomized across participants. For the counterbalancing, at most two objects within a block had the same affordance and at most two objects within a block had the same most-rewarding response. Also, the two congruent condition objects within a block had different affordances and the two incongruent condition objects had different affordances and different correct responses from each other.

Each trial lasted between 7 and 11.5s (9.25s on average). The trial timing is detailed in Fig. 1b. Each trial started with a jittered fixation cross (1-2.5s), followed by the stimulus display. The stimulus was shown at most for 4s. After a hand gesture was recognized, there was another jittered fixation cross (1-2.5s) and a jittered reward feedback (1-2.5s). Following the reward feedback, another fixation cross was displayed for the duration of the difference between 4s and the reaction time plus the processing time for response registration. Eye-tracking data were also collected during both the behavioral and fMRI experiments but is not analyzed herein.

Following the main task, the behavioral and fMRI participants did the same online survey that the M-Turk participants completed to annotate affordance-compatibility scores and familiarity for each object. However, in this phase, the participants annotated only the 48 (or 24 in the fMRI case) stimuli that were used for the main experiment.

Response decoding
In the behavioral experiment, participants’ hand gestures were tracked by a consumer webcamera (30 FPS) and decoded in real-time using a computer vision algorithm, Openpose91–94, and a fully connected neural network (FCNN) that classifies such movement, frame by frame. For the real-time processing of the video stream, frames were first resized to 192×108 pixels and then inputted into BODY_25 model of Openpose with a net resolution setting -1×112 and an output resolution -1×80. The FCNN classified a participant’s hand position in a frame using an 81-dimensional vector that was created based on Openpose estimations of the 2-dimensional coordinates of 21 key points on the right hand (4 from each finger and 1 from the wrist), as well as the confidence scores of the estimations for each key point.

Specifically, the x and y coordinates of each key point were centered using the wrist location, and then scaled based on the x-directional distance between the left-most and right-most key points, as well as the y-directional distance between the bottom-most and the top-most key points in the current frame. The 81-dimensional input to the FCNN was then built using those centered and scaled x, y coordinates of each key point except for the wrist (20×2 dimensions), their sum of squares representing the distances between each key point and the wrist (20 dimensions), and confidence scores of the 21 key point estimations (21 dimensions).

The neural network classifier comprised of 3 hidden layers (128, 64, and 32-dimensions, respectively, from low to high layers) and a 4-dimensional softmax output layer in which each unit was associated with one of four hand gestures: pinch, clench, poke, or palm. The deep network classifier was trained using videos of hand motions that were labeled frame-by-frame. The training videos featured hand gestures of 2 males and 2 females, and the total duration of the videos were 40240 frames. We used MLPClassifier function from Sklearn 0.23.1 with lbfgs optimizer and learning rate 10−5.

A response in a trial was registered as a valid action only when all frames for 500ms were classified with a probability >95% into one specific hand gesture. To ensure consistency in the starting hand position across trials, it was required that each trial begins with a hand position classified as the palm position by the deep network. The classifier classified responses with 93.2% accuracy and misclassified responses were manually corrected after the experiment (and we then used the correctly decoded action in the subsequent analyses—those ~ 7% errors trials would have produced outcomes with differing probabilities to that intended according to the experimental design, but this variation would not have been noticeable to the participants and was fully accounted for in subsequent data analysis involving the computational models). Reaction times were measured by calculating the time difference between the stimulus onset and the frame where the hand gesture was initiated from the resting palm position using the recorded videos of hand gestures.

In the fMRI experiment, decoding of participant’s actions was found to be much less reliable with the machine-learning algorithm we used successfully in the behavioral study, because participant’s hand position was occluded by the MRI bore. Consequently, we resorted instead to manually decoding each gesture in real-time. Participants’ hand gestures were monitored by the experimenter using a low light USB camera (See3CAM_CU30, 40 FPS), and the footage was displayed in the control room in real-time. The experimenter manually but rapidly categorized the movements into one of 3 hand gesture categories (pinch, clench, or poke) thereby enabling participants to interact with the task in real-time. The registered responses were double-checked after each experiment by replaying the recorded video stream, revealing that <0.1% of the responses were mistakenly classified during the task performance phase. These error trials were corrected prior to the subsequent data analysis to reflect participant’s intended responses. RTs were calculated using the identical method used for analyzing RTs in the behavioral data.

Computational modeling of behavior
We tested 9 different models to identify the computational mechanism that best explains the behavioral data. The models were fit to each subject data separately using the Computational Behavioral Modeling (CBM) toolkit cbm_lap function which calculates model evidence and likelihood of the data using Laplace approximation and estimates parameters using maximum-a-posteriori (MAP) estimation95. We used mean 0 and variance 6.25 Gaussian priors for calculating the parameter estimates. Because CBM assumes the parameters to be normally distributed, we applied transformation functions to the parameters: a sigmoid function to model parameters which ranges are between 0 and 1, or an exponential function to model positive parameters. We did not utilize hierarchical parameter estimation in this study and relied instead on estimating individuals subject-level parameters. This was done because while hierarchical Bayesian inference for parameter estimation is especially useful in situations where participant-level data are sparse, its utility diminishes when there are sufficient data points for reliable individual parameter estimation95,96. In our case, each subject completed 480 or 960 trials, allowing for reliable estimation of individual subject-level parameters. In additional analyses we found that the hierarchical approach introduced additional noise from other participants into the individual subject-level estimates resulting in less accurate rather than more accurate parameter estimates. For example, in parameter recovery analysis using 100 times simulated data, the average correlation between recovered parameters of Performance-based arbitration model using the hierarchical Bayesian inference was 0.73 but without the hierarchical inference 0.76. This resulted in failures to reproduce the actual behavioral patterns in the posterior predictive checks (See Supplementary Fig. 26). In stead, we performed the CBM outputs to do the Bayesian model comparison. The models were developed solely based on the behavioral data and later tested on the fMRI data.

In order to model affordance perception, we employed the affordance-compatibility score derived from the post-task survey of each participant, which was scaled from 0 to 1. We used individually derived affordance-compatibility scores to fit the model to the behavioral experiment data. However, in the case of fitting the model to each fMRI participant, we used the average scores of annotations from all participants as such a strategy showed a better fit to the data. Notably, the choice of using either individual scores or average scores did not affect the results of the model comparison for either dataset.

Reinforcement learning model
The reinforcement learning (RL) model is the simplest model we tested that doesn’t consider the effect of affordance, but only models value learning. Action values Q for each object are initialized at 0.

Q(s, a) = 0 (1)
0

And the probability of choosing an action a given a stimulus s is a softmax function of action value

P(a|s) = softmax(βQ(s, a) + b ) = P exp(βQ(s, a) + b ) / Σ exp(βQ(s, a') + b )

[where β > 0 is an inverse temperature parameter and b 2 R models action selection bias toward the action independent to the stimulus shown. The action value Q(s, a) for the chosen action a to the stimulus s at trial t is updated based on the delta rule

RPE = r − Q(s, a)

Q(s, a) Q(s, a) + α × RPE

where the reward prediction error (RPE) is the difference between the reward r (0 or 1 depending on the realized outcome) at the trial t and the action value. 0 ≤ α ≤ 1 is a learning rate parameter. There were 5 free parameters (β, α, b pinch, b clench and b poke) in this model.

A Bayesian learning model
In this model, instead of using the reinforcement learning rule, value learning is modeled as Bayesian inference20. The model tracks the probability of reward of each action given an object using a beta prior and a binomial likelihood. Specifically, the prior on the reward probability of an action a to an object s is

P(R=1|s, a) (cid:3) Beta(α a, β a), β a = α a (1 − γ) / γ

where α a > 0 and 0 < γ < 1 are free parameters and the prior mean is γ. Given a scenario where an object s shown n times, with an action a being chosen n a times, resulting in r a positive outcomes, the posterior reward probability of the action a to the object s is

P n(R=1|s, a) (cid:3) Beta(α a + r a, β a + n a − r a), n = n a

which posterior mean is α a + r a / α a + β a + n a.

We then assumed affordance elicits its effect as a bias in the action selection. We used 0-to-1 scaled affordance-compatibility score of action a to object s from the post-task survey as affordance-compatibility score Aff(s, a) and used the softmax function,

P t(a|s) = softmax(β(Q t(s, a) + k × Aff(s, a)) + b a)

as the action selection policy where Q t(s, a) is the posterior mean of P(R=1|s, a) at trial t and β > 0, k > 0 and b a 2 R are free parameters that model inverse temperatures, weight on affordance and action selection bias respectively. In total, this model has 9 free parameters (α a and b a for each action type, γ, β and k)

We also tested models incorporating the affordance into the prior mean instead of including the affordance-based selection bias term in the softmax policy. However, these models exhibited an even poorer fit to the data compared to the model described above.

Affordance as a prior model
The prior model uses the affordance-compatibility score Aff(s, a) as the initial action value of an action a to an object s.

Q 0(s, a) = k × Aff(s, a), k > 0

The decision probability and learning rules are identical to the RL model, which results in having one additional free parameter than the RL model (β, α, b pinch, b clench, b poke and k). This model can also be interpreted as implementing a single decision-making system based on affordance and in which the affordance is being relearned based on task experience.

Affordance as a bias model
The bias model is identical to the RL model except for the calculation of the action selection probability. In the affordance as a bias model, action selection is a softmax function of the linear summation of the action values and the affordance-compatibility scores.

P t(a|s) = softmax(β(Q t(s, a) + k × Aff(s, a)) + b a), k > 0

This model has 6 free parameters (β, α, b pinch, b clench, b poke and k).

Fixed arbitration model
In this model, we assumed there are two different decision-making strategies, value-based decision-making and affordance-based decision-making. Value-based decision-making and value learning is modeled using the RL model previously described.

P q,t(a|s) = softmax(βQ t(s, a) + b a)

Action selection by the affordance-based decision-making system is modeled as a softmax function of the affordance-compatibility score.

P aff,t(a|s) = softmax(β(k × Aff(s, a)) + b a)

The action selection policy is calculated as a weighted sum of these two policies

P t(a|s) = w q P q,t(a|s) + (1 − w q)P aff,t(a|s)

where 0 ≤ w q ≤ 1 is the fixed arbitration weight between the two decision-making strategies. This model has 7 free parameters in total (β, α, b pinch, b clench, b poke, k and w q).

Affordance as a cost model
The cost model is an extension of the fixed arbitration model where we assumed that the response-affordance-incompatibility acts as a cost that is contrasted against the reward signal98. Specifically, we modified the delta learning rule to include a cost term which results in a larger reward when an affordance-compatible action is selected so that,

RPE t = (r t + c × k × Aff(s t, a t)) − Q t(s t, a t)

Q t+1(s t, a t) Q t(s t, a t) + α × RPE t

where c > 0 is an additional free parameter that models the effect of affordance in learning as a cost. For example, the additional reward will be smaller when the chosen action has a low affordance-compatibility score compared to when it has a high affordance-compatibility which can be interpreted as there was a cost in selecting affordance-incompatible action.

The other components of the model remain identical to the fixed arbitration model, resulting in a cost model with 8 free parameters (β, α, b pinch, b clench, b poke, k, w q and c).

Conflict-based arbitration model
Like the fixed arbitration model, we assumed that the value-based policy and the affordance-based policy are mixed for the action selection but in this model the arbitration weight dynamically changes trial by trial for each stimulus. The arbitration weight w q,t(s) at trial t is a logistic function of a conflict variable c t(s) which tracks the conflict between the value-based and affordance-based policies given the stimulus s. Thus,

w q,t(s) = 1 / (1 + exp(−η 1 c t(s) + η 0))

where η 1 > 0 and η 0 2 R are free parameters. We assume that the conflict signal between the two policies causes an increase in cognitive control which pushes behavior toward value-based decision-making in this paradigm23,24. The conflict variable c t(s) is calculated as the square root of the Jensen-Shannon Divergence (JSD) between the value-based and affordance-based policies given the stimulus s.

c t(s) = sqrt(JSD(P q,t(s, ·), P aff,t(s, ·)))

The calculation of value-based and affordance-based polices of the model are identical to the fixed arbitration model which makes this model have 8 free parameters (β, α, b pinch, b clench, b poke, k, η 0 and η 1).

We also tested other design choices such as the energy from a Hopfield network25 as the proxy of conflict or using a conflict tracking variable that is updated gradually using trial-by-trial JSD. However, the model described above explained the data better than the other tested options.

Reliability-based arbitration model
This model uses reliabilities21,26,27 of the two strategies as the driving force of arbitration between the value-based and affordance-based decision-making. The reliabilities are calculated using an RL-like updating rule based on unsigned prediction errors26. For example, the reliability of value-based decision-making given a stimulus s is updated using the following delta rule.

χ q,t+1(s t) χ q,t(s t) + α χ ((1 − |RPE t|) − χ q,t(s t)), 0 ≤ α χ ≤ 1

RPE t = r t − Q t(s t, a t)

Therefore, the reliability of value-based decision-making will increase when the unsigned RPE is small. Similarly, we defined the reliability of affordance-based decision-making given a stimulus s is updated based on

χ aff,t+1(s t) χ aff,t(s t) + α χ ((1 − |APE t|) − χ aff,t(s t))

APE t = r t − k × Aff(s t, a t)

where APE is the affordance prediction error and k is between 0 to 1 to ensure the APE is between 0 and 1. The reliabilities were initialized to 0. Then the arbitration weight given a stimulus s is the logistic function of these two reliabilities. Therefore,

w q,t(s) = 1 / (1 + exp(−η q χ q,t(s) + η aff χ aff,t(s) + η 0))

where η q, η aff > 0 and η 0 2 R. Identical to the fixed and conflict-based arbitration model, the value-based and affordance-based policies are softmax functions of action values, or affordance-compatibility scores and the final action selection probability is the reliability-based arbitration-weighted sum of the two policies. As a consequence, this model has 10 free parameters (β, α, b pinch, b clench, b poke, k, α χ, η 0, η q and η aff).

As a simpler alternative, we also tested a value-reliability arbitration model that relies solely on the reliability of the value-based decision-making system. This reduced model is identical to the full reliability-based model, except in the computation of the arbitration weight:

w q,t(s) = 1 / (1 + exp(−η 1 χ q,t(s) + η 0))

This model thus has 9 free parameters.

Performance-based arbitration model
Here, we are proposing a novel arbitration framework that aims to optimize the arbitration weight in terms of maximizing the return. The mathematical derivation of the proposed model is detailed in the Supplementary Methods. Specifically, we defined value, or performance, of a strategy given a stimulus as the expected returns that can be collected by following that particular strategy in response to the stimulus18,22. Then, the arbitration weight on value-based decision-making given a stimulus s is a logistic function of the performances of affordance-based and value-based policies. Thus,

w q,t(s) = 1 / (1 + exp(−η(Perf q,t(s) − Perf aff,t(s)) + η 0)), η > 0, η 0 2 R

The value-based policy’s performance Perf q,t(s) and affordance-based policy’s performance Perf aff,t(s) given the stimulus s are updated trial by trial by the following delta rules which estimate the performances using inverse propensity scoring from the off-policy policy evaluation literature28,29.

Perf q,t+1(s t) Perf q,t(s t) + α p (P q,t(a t|s t) / P t(a t|s t)) (r t − Perf q,t(s t))

Perf aff,t+1(s t) Perf aff,t(s t) + α p (P aff,t(a t|s t) / P t(a t|s t)) (r t − Perf aff,t(s t))

Perf q,0(s) = Perf aff,0(s) = 0

P t(a|s) = w q,t(s)P q,t(a|s) + (1 − w q,t(s))P aff,t(a|s)

where r t denotes the realized outcome at the trial t, P q,t(a|s) is the probability of selecting the chosen action under the value-based policy at the trial t. Similarly, P aff,t(a|s) represents the probability of selecting the chosen action under the affordance-based policy at the trial t, and P t(a|s) denotes the probability of selecting the chosen action. 0 ≤ α p ≤ 1 is a free parameter that denotes the performance updating rate. The performance prediction error is the difference between the realized outcome and the performance of the value-based or affordance-based decision-making respectively. Analogous to the other arbitration models, value-based and affordance-based policies are calculated in the same manner as the fixed arbitration model. The final action selection probability is determined by a performance-based arbitration-weighted sum of the two policies. Notably, we set the performance updating rate α p to be identical to the value learning rate α. As a result, the performance-based arbitration model has a total of 8 free parameters (β, α, b pinch, b clench, b poke, k, η and η 0).

Models with counterfactual value update
We also tested models that update values of unchosen actions when there was no reward. These were implemented using the following learning rule, while keeping the other components the same.

RPE t = r t − Q t(s t, a t)

Q t+1(s t, a t) Q t(s t, a t) + α × RPE t

Q t+1(s t, a0 t) Q t(s t, a0 t) − α 2 × RPE t (only when r t = 0)

where the reward prediction error (RPE) is the difference between the reward r t (0 or 1 depending on the realized outcome) at the trial t and the action value of chosen action a t. a0 t represents the unchosen action at t and 0 ≤ α ≤ 1 is a learning rate parameter. The model comparison and simulation results are shown in Supplementary Fig. 12

Arbitration-weight-decay model
As a simple alternative to active dynamic arbitration models, we implement models with a decaying arbitration weight that gradually places greater weight on the value-based decision-making system. The model implementation is identical to that of the fixed arbitration model, except that the arbitration weight decays over the course of the task. In these models, the arbitration weight is a function of the trial index t. For example, in the case of exponential decay

w q,t = w q * w d t

where 0 < w d ≤ 1; and in the case of hyperbolic decay,

w q,t = w q / (1 + w d × t)

where 0 < w d. These models thus have 8 free parameters in total (β, α, b pinch, b clench, b poke, k, w q, and w d)

Alternative parameterization of affordance compatibility
To address potential concerns regarding the parametric assumptions of affordance compatibility, we tested three additional parameterization methods based on post-task survey data: (1) squared affordance scores, (2) log-transformed affordance scores, and (3) a binarized version where the maximum affordance score of pinch, clench, or poke was set to 1, and all other scores were set to 0.

Consistent results across all parameterizations confirmed that the performance-based arbitration model provided the best fit to both behavioral and fMRI datasets (See Supplementary Fig. 23). Posterior predictive checks were conducted as well using the three performance-based arbitration models with alternative parameterizations. These models successfully captured observed patterns such as initial exploration bias, learning curves, and slopes. Statistical analyses, including mixed-effects GLM regressions of ΔP (performance difference) as a function of the number of rewarded trials, confirmed that the results were robust to the specific parameterization choices (See Supplementary Figs. 24 and 25).

Model recovery and parameter recovery analyses
We conducted a model recovery analysis across the nine models to ensure that the underlying cognitive mechanism that generated data can be reliably identified using the task design. Using the fMRI version of the task, which has half the number of trials compared to the behavior-only version, each of the nine models was simulated 60 times with the estimated parameters from the real fMRI participants’ data. We then fit the models to each simulated behavior using CBM toolkit and did model comparisons based on log evidence which take into account the number of parameters. Each simulated data was labeled with the best-fitting model, and we calculated the probability of the identified model being the true generative model (Supplementary Table 4). It is noteworthy that the probability of the best-fit model being the true generative model for the performance-based and the reliability-based arbitration models were 0.89 and 0.79 each which implies that these models were reliably distinguishable through the model comparison process.

Additionally, we conducted a parameter recovery analysis on the performance-based and the reliability-based arbitration models which were employed for fMRI analyses, to ensure that the model-based variables reliably reflect the underlying cognitive mechanisms. Using the fMRI version of the task which has 480 trials in total, we generated 100 simulated data from each model, with parameters sampled from Gaussian distributions based on sample means and variances of the estimated parameters from the actual fMRI participants data. We estimated the model parameters using CBM toolkit and calculated correlations between the true generative parameters and the estimated parameters. The performance-based arbitration model had a mean Pearson correlation of r = 0.76 between true and estimated parameters, while the reliability-based arbitration model had a mean Pearson correlation of r = 0.53 (See Supplementary Fig. 28 for details). For further analysis on parameter identifiability and potential over parameterization, please refer to Note 3 in the supplementary materials.

Behavior general linear model analyses
We conducted mixed-effect GLM analyses on the RT data from the behavior and fMRI studies to examine the effect of choosing affordance-compatible actions on RTs (Fig. 2a, f). As the RT is influenced by both the affordance-compatibility of the response and the value learning process, we incorporated a set of independent variables in the analyses: indicator variables for the affordance-compatibility of the response, whether the response was correct, whether the trial was in a congruent or incongruent condition, whether the trial was in a high or low condition, a categorical variable representing the movement type of the response (i.e., pinch, clench or poke) and a continuous variable representing the trial index. The dependent variable was RT, which was logarithmically transformed to make it normally distributed. The slopes and intercepts were estimated as random effects for each participant and the fixed-effect coefficients are reported in (Supplementary Table 1)

In addition, we performed mixed-effect GLM analyses to evaluate the impact of affordance-value congruency on the learning curves (Fig. 2d, e, i, j and Fig. 3f, g, l, m). The dependent variable was the difference between congruent and incongruent conditions in terms of frequencies of choosing correct actions, which were extracted for each trial index or the number of rewarded trials so far, for each subject. We included a continuous variable representing the trial index or the number of rewarded trials so far, as regressors in the analyses. The slopes and intercepts were estimated as random effects for each subject and the fixed-effect coefficients are reported in (Supplementary Table 3)

We used the mixedlm function from the statsmodels 0.12.2 package in Python 3.7.7. The package calculated z statistics of each coefficient by dividing the estimates of coefficients with standard errors. Using the z statistics, the p-values were calculated with respect to a standard normal distribution.

fMRI data acquisition
fMRI data were acquired at the Caltech Brain Imaging Center (Pasadena, CA), using a Siemens Prisma 3T scanner with a 32-channel radio-frequency coil. The functional scans were conducted using a multi-band echo-planar imaging (EPI) sequence with 72 slices, -30 degrees slice tilt from AC-PC line, 192 mm × 192 mm field of view, 2 mm isotropic resolution, repetition time (TR) of 1.12 s, echo time (TE) of 30 ms, multi-band acceleration of 4, 54-degree flip angle, in-plane acceleration factor 2, echo spacing of 0.56 ms, and EPI factor of 96. Following each run, both positive and negative polarity EPI-based field maps were collected using similar parameters to the functional sequence, but with a single band, TR of 5.13 s, TE of 41.40 ms, and 90-degree flip angle. T1-weighted and T2-weighted structural images were also acquired for each participant with 0.9 mm isotropic resolution and 230 mm × 230 mm field of view. For the T1-weighted scan, TR of 2.55 s, TE of 1.63 ms, inversion time (TI) of 1.15 s, flip angle of 8 degrees, and in-plane acceleration factor 2 were used. The T2-weighted scan was acquired with TR of 3.2 s, TE of 564 ms, and in-plane acceleration factor of 2.

fMRI data preprocessing
Results included in this manuscript come from preprocessing performed using fMRIPrep 20.2.699,100, RRID:SCR_016216], which is based on Nipype 1.7.0101,102, RRID:SCR_002502].

Anatomical data preprocessing
A total of 1 T1-weighted (T1w) images were found within the input BIDS dataset. The T1-weighted (T1w) image was corrected for intensity non-uniformity (INU) with N4 Bias Field Correction103, distributed with ANTs 2.3.3104, RRID:SCR_004757]. The T1w-reference was then skull-stripped with a Nipype implementation of the antsBrainExtraction.sh workflow (from ANTs), using OASIS30ANTs as target template. Brain tissue segmentation of cerebrospinal fluid (CSF), white-matter (WM) and gray-matter (GM) was performed on the brain-extracted T1w using fast105, FSL 5.0.9, RRID:SCR_002823,]. Brain surfaces were reconstructed using recon-all106, FreeSurfer 6.0.1, RRID:SCR_001847,], and the brain mask estimated previously was refined with a custom variation of the method to reconcile ANTs-derived and FreeSurfer-derived segmentations of the cortical gray-matter of Mindboggle107, RRID:SCR_002438,]. Volume-based spatial normalization to one standard space (MNI152NLin2009cAsym) was performed through nonlinear registration with antsRegistration (ANTs 2.3.3), using brain-extracted versions of both T1w reference and the T1w template. The following template was selected for spatial normalization: ICBM 152 Nonlinear Asymmetrical template version 2009c108, RRID:SCR_008796; TemplateFlow ID: MNI152NLin2009cAsym].

Functional data preprocessing
For each of the 6 BOLD runs per subject (across all tasks and sessions), the following preprocessing was performed. First, a reference volume and its skull-stripped version were generated by aligning and averaging 1 single-band references (SBRefs). A B0-nonuniformity map (or field map) was estimated based on two (or more) echo-planar imaging (EPI) references with opposing phase-encoding directions, with 3dQwarp109 (AFNI 20160207). Based on the estimated susceptibility distortion, a corrected EPI (echo-planar imaging) reference was calculated for a more accurate co-registration with the anatomical reference. The BOLD reference was then co-registered to the T1w reference using bbregister (FreeSurfer) which implements boundary-based registration110. Co-registration was configured with six degrees of freedom. Head-motion parameters with respect to the BOLD reference (transformation matrices, and six corresponding rotation and translation parameters) are estimated before any spatiotemporal filtering using mcflirt111, FSL 5.0.9,]. BOLD runs were slice-time corrected to 0.52s (0.5 of slice acquisition range 0s-1.04s) using 3dTshift from AFNI 20160207109, RRID:SCR_005927]. First, a reference volume and its skull-stripped version were generated using a custom methodology of fMRIPrep. The BOLD time-series (including slice-timing correction when applied) were resampled onto their original, native space by applying a single, composite transform to correct for head-motion and susceptibility distortions. These resampled BOLD time-series will be referred to as preprocessed BOLD in original space, or just preprocessed BOLD. The BOLD time-series were resampled into standard space, generating a preprocessed BOLD run in MNI152NLin2009cAsym space. First, a reference volume and its skull-stripped version were generated using the custom methodology of fMRIPrep. Several confounding time-series were calculated based on the preprocessed BOLD: framewise displacement (FD), DVARS and three region-wise global signals. FD was computed using two formulations following Power (absolute sum of relative motions, 112) and Jenkinson (relative root mean squared displacement between affines, 111). FD and DVARS are calculated for each functional run, both using their implementations in Nipype (following the definitions by ref. 112). The three global signals are extracted within the CSF, the WM, and the whole-brain masks. Additionally, a set of physiological regressors were extracted to allow for component-based noise correction113 CompCor. Principal components are estimated after high-pass filtering the preprocessed BOLD time-series (using a discrete cosine filter with 128s cutoff) for the two CompCor variants: temporal (tCompCor) and anatomical (aCompCor). tCompCor components are then calculated from the top 2% variable voxels within the brain mask. For aCompCor, three probabilistic masks (CSF, WM and combined CSF+WM) are generated in anatomical space. The implementation differs from that of Behzadi et al. in that instead of eroding the masks by 2 pixels in BOLD space, the aCompCor masks subtract a mask of pixels that likely contain a volume fraction of GM. This mask is obtained by dilating a GM mask extracted from FreeSurfer’s aseg segmentation, and it ensures components are not extracted from voxels containing a minimal fraction of GM. Finally, these masks are resampled into BOLD space and binarized by thresholding at 0.99 (as in the original implementation). Components are also calculated separately within the WM and CSF masks. For each CompCor decomposition, the k components with the largest singular values are retained, such that the retained components’ time series are sufficient to explain 50 percent of variance across the nuisance mask (CSF, WM, combined, or temporal). The remaining components are dropped from consideration. Head-motion estimates calculated in the correction step were also placed within the corresponding confounds file. The confound time series derived from head motion estimates and global signals were expanded with the inclusion of temporal derivatives and quadratic terms for each 114. Frames that exceeded a threshold of 0.5 mm FD or 1.5 standardised DVARS were annotated as motion outliers. All resamplings were performed with a single interpolation step by combining all the pertinent transformations (i.e. head-motion transform matrices, susceptibility distortion correction when available, and co-registrations to anatomical and output spaces). Gridded (volumetric) resamplings were performed using antsApplyTransforms (ANTs), configured with Lanczos interpolation to minimize the smoothing effects of other kernels115. Non-gridded (surface) resamplings were performed using mri_vol2surf (FreeSurfer).

Many internal operations of fMRIPrep use Nilearn 0.6.2116, RRID:SCR_001362], mostly within the functional processing workflow. For more details of the pipeline, see the section corresponding to workflows in fMRIPrep’s documentation.

Whole brain general linear model analysis
FSL (FMRIB Software Library) 6.0 software package was used to analyze the fMRI data. Following the preprocessing using fMRIPrep, the fMRI data was convolved with a 3D Gaussian smoothing kernel (8.0 mm FWHM) using FSL SUSAN. Subsequently, BOLD signals for each voxel were scaled to ensure that the mean BOLD signal of each voxel is 100. Before running GLM analyses, first 4 volumes were discarded and a high-pass filter with a cutoff 128s was applied. Framewise displacement, global signal, global signal derivative, 6 head-motion parameters (x, y, z translations and rotations), and temporal CompCor from fMRIPrep were included as nuisance variables in every analysis. Stick regressors that represent onsets of stimulus, fixation cross and outcome were also included in the GLMs. The response, or the hand movement, was modeled using a parametric regressor detailed in the following section. We used the fixed-effect higher-level modeling for individual-level inference that averages across blocks and FSL FLAME1 for group-level inference.

Hand movement parametric regressor
In order to minimize potential confounding effects that may arise from using natural hand gestures as the action in the experiment, we designed a hand movement regressor based on 2-dimensional 21 key points for the right hand extracted from the hand video using Openpose. This regressor was included in all of the analyses.

First, video frames were resized into 162×108 pixels to fit the Openpose net resolution -1×122. By utilizing the BODY_25 model from the Openpose, key points were extracted frame by frame with the output resolution setting -1 × 80. Next, the key points were weighted-moving-averaged with a window of 20 frames (about 500ms) to reduce the noise stemming from Openpose’s estimation error. The confidence scores of Openpose estimations were utilized as the weights in order to mitigate the impact of uncertain estimations. Then, the first principal component of the weighted-moving-averaged 42-dimensional data (21×2) was calculated and used as the parametric regressor that models the hand movement.

In all of our GLM analyses, the hand movement regressor consistently identified the left primary motor cortex, left premotor cortex, left primary somatosensory cortex, and bilateral inferior parietal lobule as the regions associated with the right-hand movement (Supplementary Fig. 17).

Identifying decision-making regions
A GLM analysis was conducted to investigate the regions associated with the affordance-based and value-based decision-making, as well as the action selection which is the integration of those two systems (GLM1). The chosen action’s affordance-compatibility score (Aff(s, a)), chosen action’s value (Q(s, a)), the action selection probability of chosen action (P(a|s)) from the performance-based arbitration model models were included as parametric regressors at the onsets of stimuli. A parametric regressor of the reward prediction error was also included at the onsets of outcomes. Another GLM analysis using model-based parameters from the reliability-based arbitration model was conducted as well (GLM2). The results were cluster corrected with a cluster-defining threshold of z = 3.1.

Identifying arbitration regions
Two GLM analyses were conducted to identify regions related to arbitration between value-based and affordance-based decision-making. One GLM analysis utilized arbitration-related parameters obtained from the performance-based arbitration model (GLM3) and the other one used those from the reliability-based arbitration model (GLM4). GLM3 included performances of the value-based and affordance-based decision-making systems (Perf q,t(s), Perf aff,t(s)), and the difference between those two (Perf aff,t(s) − Perf q,t(s)) as parametric regressors at the onsets of stimuli. Parametric regressors of the performance prediction errors of value-based and affordance-based policies each were also included at the onsets of outcomes in the GLM3.

In GLM4, reliabilities of the value-based and affordance-based decision-making systems (χ q,t(s), χ aff,t(s)), and the difference between those two (χ aff,t(s) − χ q,t(s)) were included as parametric regressors at the onsets of stimuli. A parametric regressor of the affordance prediction error was included as well at the onsets of outcomes in the GLM4. The results were cluster corrected with a cluster-defining threshold z = 3.1.

Neural correlates of affordance-incompatible action execution
To identify neural correlates that are activated more when executing affordance-incompatible actions compared to affordance-compatible actions11,12, a model-free GLM analysis was conducted (GLM5). GLM5 included stick regressors representing the stimuli identities at the stimulus onsets, 2 regressors representing whether the affordance and response were compatible or not at the response onsets, and a parametric regressor of reward at the outcome onsets. While the contrast between the affordance response compatibility regressors have not survived the cluster correction with a cluster-defining threshold z = 3.1, Supplementary Fig. 17g shows the uncorrected results with a threshold set at p < 0.001, corresponding to z ≥ 3.1.

Neural representation strength analysis
To calculate the extent to which the identified regions from GLM analyses can be explained by the corresponding cognitive variables, we computed the mean z-statistics for those specific variables across the voxels within the corresponding group-level regions. The regions were defined using the survived clusters from GLM1, 2, 3 and 4 (For details, see Fig. 4 and Supplementary Fig. 18). We then compared the mean z-statistics for each individual with the frequency of choosing the most rewarding action.

Similarly, the increased BOLD activity for executing affordance-incompatible actions compared to executing affordance-compatible actions was calculated using the contrast between the affordance-response compatibility regressors from GLM5. The z-statistics of the contrast were averaged across the voxels within functionally-defined regions of interest (ROI). The ROIs were defined in the same way as in the previous analyses. We then compared the average z-statistics for each individual with the frequency of choosing the affordance compatible action or the reaction time effect which is the relative increase in reaction time when choosing affordance-incompatible actions compared to affordance-compatible actions (RT effect = RTincomp − RTcomp / RTcomp).

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Yi and O’Doherty present a computational, behavioral, and neuroimaging-based investigation of the role of action affordances in guiding action selection during value-based decision making. They use behavioral and brain responses to arbitrate between three primary computational models. The results suggest that valued-based and affordance-based action selection systems are separate and competitive, with a third meta-controller arbitrating between them on a dynamic basis. I found this to be a remarkably strong paper on multiple fronts. The computational modeling formalizes a clear set of alternative explanations of the psychological phenomenon of interest. The combination of multiple measures – reaction times, choices, and brain responses – provides strong convergent evidence. The experimental method is creative and naturalistic, yet well-controlled. And the overall research question is both important and understudied. Thus, I think that this work will make a very valuable contribution to the literature. I have a few suggestions, below, for how it might be further enhanced:\n1. The paper places a strong emphasis on the visuomotor nature of action affordances, but actions can be construed at multiple levels of abstraction. To take the authors’ own example, a computer keyboard does indeed afford pressing or poking actions. However, it also affords higher-level actions, such as writing, which are not dependent on the execution of a particular motor program (i.e., one can write through typing or via a pen or electronic stylus, or even via dictation). In everyday life, it is often these higher level affordances which guide decision making, because they are more directly tied to the goals that people are attempting to achieve. It might be useful to acknowledge this as outside the scope of the present investigation, yet still worthy of future investigation.\n2. Even basic motor affordances must often be learned. Again returning to the keyboard example, the knowledge that a keyboard affords “pressing” is tied to experience with that object – for example, knowing that the keys are spring-loaded. The same is true for many of the objects investigated in the present study. In future work, it might be helpful to use cross-cultural differences in affordance knowledge to distinguish between more fine-grained computational models. For example, an experienced user of chopsticks likely perceives far more affordances in them than someone who has never used them – and people of both types are likely available for recruitment in the authors’ environs. This will allow future studies to disambiguate whether the affordances under study are habit like or driven by more innate processes.\n3. Many objects offer not just action affordances, but also affective affordances. For example, a can of beer offers not just the motor affordances of grasping or drinking, but also the affective affordance of drunkenness. The effects of such affective affordances – although rarely termed as such – has been extensively investigated in the substance use, addiction, and nutrition literatures within psychology. It would be helpful to make a connection between the present findings and those literatures, because I believe that they could benefit significantly from the conceptual clarity and computational specificity of the current approach (and this paper may receive more – deserving – attention as a result).", "reviewer_2": "The study by Yi and O’Doherty investigates the computational and neural mechanisms underlying the influence of action affordance on value learning. By combining functional neuroimaging, psychophysics, and computational modeling, they aim to understand how action affordance shapes value-based learning in novel environments. Their main finding suggests that action affordance operates as an independent system that concurrently guides action selection alongside value-based decision-making. These two systems engage in a competitive process to determine the best action. Notably, the authors demonstrate that different brain regions contribute to this process: the pre-SMA encodes the performance difference between the action-affordance and value-based decision-making systems, while the posterior parietal cortex (PPC) integrates predictions from both systems to facilitate action selection. Overall, this study provides important and novel insights into how the brain integrates information from multiple sources to optimize action selection. It fits within the scope of Nature Communications, but there are some points that the authors can address to further improve their study.\n1. The authors describe how the brain integrates information about the affordance and the value of action. There are a number of previous studies that explored how the brain integrates information from multiple sources to make decisions. For instance, there are studies by Paul Cisek, Richard Andersen, Paul Glimcher and others (see below), which showed that the brain integrates dynamically information from multiple sources (such as value and effort/cost) when making decisions. Is it likely that the findings of the Yi and O'Doherty study reflects the integration of expected reward and effort cost to perform an action? This will be also consistent with the concept of desirability value, according to which the posterior parietal cortex integrates information from disparate sources, such as expected reward and effort cost. I suggest that the authors describe the previous studies by Cisek, Andersen, Glimcher, Schrater and others and explain how different are their findings from previous studies that have already reported that the brain integrates information about the affordance of action (i.e., cost/effort) and the value associated with that action.\n• Canaveral CA, Lata W, Green AM, Cisek P. Biomechanical costs influence decisions made during ongoing actions. J Neurophysiol. 2024 Jul 11. doi: 10.1152/jn.00090.2024. Epub ahead of print. PMID: 38988286.\n• Cos I, Duque J, Cisek P. Rapid prediction of biomechanical costs during action decisions. J Neurophysiol. 2014 Sep 15;112(6):1256-66. doi: 10.1152/jn.00147.2014. Epub 2014 Jun 3. PMID: 24899673.\n• Christopoulos V, Bonaiuto J, Andersen RA. A biologically plausible computational theory for value integration and action selection in decisions with competing alternatives. PLoS Comput Biol. 2015 Mar 24;11(3):e1004104. doi: 10.1371/journal.pcbi.1004104. PMID: 25803729; PMCID: PMC4372613.\n• Christopoulos V, Schrater PR. Dynamic Integration of Value Information into a Common Probability Currency as a Theory for Flexible Decision Making. PLoS Comput Biol. 2015 Sep 22;11(9):e1004402. doi: 10.1371/journal.pcbi.1004402. PMID: 26394299; PMCID: PMC4578920.\n• Kable J, Glimcher P (2009) The neurobiology of decision: consensus and controversy. Neuron 63: 733–745. pmid:19778504\n2. It will be also interested in assessing if there is any temporal difference between the two systems. Does the brain first assess the affordance of the action before the value or the other way? fMRI has a low temporal resolution, so it might not be possible to look temporal differences between the two systems, but it will be interested in trying to assess it.\n3. Can the author assess whether and how the brain encodes the competition between available actions? For instance, it would be interesting to explore differences in brain activation between quick and slow responses. Previous studies have shown that reaction time reflects the level of competition between actions; that is, the longer it takes to respond, the stronger the competition between the available actions.\nMinor points.\n1. Please add line numbers in the manuscript. It is easier to review it.\n2. The authors need to improve the quality of writing in some parts of the manuscript. For instance, there are some parts that there is no space between the sentences (e.g., last line in page 6, “…experiment.Because…” ).", "reviewer_3": "This study must have been a massively laborious undertaking, combining RL, mode-based fMRI, and automated decoding of gesture-based response. It is thus very unfortunate that I do not find the research question noteworthy. The interaction between action affordability and value-based decision-making is a very interesting topic, but the experimental paradigm in this study is incredibly artificial, at least for two reasons:\n1. It is unclear if this task, in which participants see an object associated with (for instance) a pinching gesture and instead make a clenching gesture to earn a probabilistic reward, captures any real-world learning problems.\n2. The experiment was divided into blocks, in each of which participants learned the most rewarding actions for four items, each corresponding to one of the four conditions (CH, CL, IH, and IL). The descriptions of instructions given to participants are entirely missing in this manuscript, but even if they were not told anything about reward contingencies, it is natural for them to eventually start trying to figure out which items are in the congruent or incongruent condition in each block (it is perhaps one of the most rational strategies they can take).\nIt is unclear the degree to which the reported findings are generalizable beyond this experimental paradigm. The point 2 is particularly crucial for the significance of the modeling results; the \"performance-based arbitration model\" is effectively equivalent to first testing if the most affordable action is the most rewarding one (which is the case 50% of the time) and trying the other two actions if not. The artificiality of the experimental paradigm heavily limits my enthusiasm for this manuscript. It does not help that the authors did not conduct much analysis to demonstrate that their complicated models characterize the behavioral data better than simpler, perhaps heuristic, accounts (such as the one I described above).\nBelow are some of the specific issues I had with the modeling approach:\n3. The models seem overparameterized. Just an example -- even in the simplest RL model, the authors had b_pinch, b_clench, and b_poke as free parameters. They cannot be uniquely estimated. Forgive my frankness, but I am surprised that the authors did not catch this, and such a mistake casts doubts on the entire modeling results. One suggestion would be to make the code publicly available at this point for transparency.\n4. The authors make a lot of specific parametric assumptions in modeling, for example how affordance compatibility is linearly scored based on the post-task survey data. It is unclear to what degree the model comparison results depend on these (mis)specifications.\n5. The authors examined an alternative model, in which learning rates differ between congruent and incongruent conditions. They rejected it solely based on the lack of population-level difference in learning rate estimates, but it would make much more sense to reject it based on its performance (e.g. log evidence).\n6. The authors wrote: \"We didn’t use the hierarchical Bayesian inference function of the toolkit because model simulations using the hierarchically estimated parameters couldn’t reproduce the characteristics of the real data\" (p.24). This sentence is vague (what characteristics?), but I am not convinced that it is a scientifically sound and unbiased reason not to use the hierarchical model, particularly given the common observation in the literature that it tends to help estimate parameters more reliably.\n7. In fMRI analysis, the wrote wrote: \"we found positive correlations between each participant’s propensity to choose the most rewarding actions and the extent to which BOLD activity in the identified arbitration regions can be explained by our model-derived arbitration variables\" (p. 15). Doesn't it simply mean that, for those whose behavioral data was well characterized by a specific model, their MRI data was also well characterized by the same model?"}, "author_response": "We would like to sincerely thank the reviewer for their thoughtful feedback. We appreciate your positive remarks on our work.\n1. The paper places a strong emphasis on the visuomotor nature of action affordances, but actions can be construed at multiple levels of abstraction. To take the authors’ own example, a computer keyboard does indeed afford pressing or poking actions. However, it also affords higher-level actions, such as writing, which are not dependent on the execution of a particular motor program (i.e., one can write through typing or via a pen or electronic stylus, or even via dictation). In everyday life, it is often these higher level affordances which guide decision making, because they are more directly tied to the goals that people are attempting to achieve. It might be useful to acknowledge this as outside the scope of the present investigation, yet still worthy of future investigation.\nWhile this study focuses on the visuomotor nature of action affordances, where object suggest particular motor actions (e.g., pinch, clench, or poke), we acknowledge that actions based on affordance can be constructed at multiple levels of abstraction as pointed out by the reviewer. Based on that we added a paragraph into the discussion section as follows:\n“It is noteworthy that action affordances operate across multiple levels of abstraction, extending beyond the visuomotor focus of our current study (Yi and O’Doherty, 2023). Consider how we interact with a keyboard: while it affords simple physical actions like key pressing, it also enables complex goal-directed behaviors like composing emails, coding software, or creating documents. Such higher-order affordances often guide our decision-making in daily life, as they connect directly to intended outcomes rather than to specific motor patterns. Although the present study focused on basic visuomotor affordances, the relationship between affordance and value-based decision-making investigated here may shed light on how abstract affordances influence choice behavior more broadly. Promising directions for future research include examining how abstract affordances might engage additional neural pathways beyond those revealed here.”\n2. Even basic motor affordances must often be learned. Again returning to the keyboard example, the knowledge that a keyboard affords “pressing” is tied to experience with that object – for example, knowing that the keys are spring-loaded. The same is true for many of the objects investigated in the present study. In future work, it might be helpful to use cross-cultural differences in affordance knowledge to distinguish between more fine-grained computational models. For example, an experienced user of chopsticks likely perceives far more affordances in them than someone who has never used them – and people of both types are likely available for recruitment in the authors’ environs. This will allow future studies to disambiguate whether the affordances under study are habit like or driven by more innate processes.\nWe appreciate this comment about motor affordances being learned behaviors and the extent to which cross-cultural differences might influence affordance knowledge. While we attempted to control for this by selecting objects that predominantly afforded one specific action (pinch, clench, or poke), we recognize that participants' familiarity with objects could affect the underlying neural mechanisms, even if the final action affordances were similar. For instance, our data showed considerable variation in participants' familiarity with 'unfamiliar' objects like car parts or electronic music instruments, reflecting individual life experiences. Those familiar with such objects might process not just basic motor affordances but also complex functional affordances (e.g., car repair or music-making capabilities).\nAlthough the size of our affordance dataset precluded analysis of familiarity-based differences, your suggestion to explore cross-cultural variations in affordance perception presents an excellent avenue for future research. Studying participants with diverse object-use experience could help determine whether affordances are primarily driven by habit, innate processes, or varying by the level of expertise. We have incorporated this perspective into our discussion section as follows:.\n“While our study primarily examined basic motor affordances that were consistent across participants, we recognize that object familiarity may shape affordance perception in important ways. For individuals with specific expertise, objects such as car parts or electronic instruments might suggest sophisticated affordances (e.g., repair capabilities, music production) beyond the simple motor actions (pinch, clench, poke) we studied. Although we controlled for variability by selecting objects with strong statistical preferences for specific actions, the underlying neural mechanisms may differ based on familiarity. The current study was not designed to enable us to study how varying familiarity levels influence affordance perception. However, future research could address this by examining cross-cultural differences in affordance and/or by recruiting participants with diverse object-use experience. Such studies could help determine whether affordances stem primarily from learned habits or innate processes.”\n3. Many objects offer not just action affordances, but also affective affordances. For example, a can of beer offers not just the motor affordances of grasping or drinking, but also the affective affordance of drunkenness. The effects of such affective affordances – although rarely termed as such – has been extensively investigated in the substance use, addiction, and nutrition literatures within psychology. It would be helpful to make a connection between the present findings and those literatures, because I believe that they could benefit significantly from the conceptual clarity and computational specificity of the current approach (and this paper may receive more – deserving – attention as a result).\nThank you very much for your helpful and constructive suggestion. Based on your feedback, we discussed how our computational framework for action-affordance-based and value-based decision-making could be extended to affective affordances, particularly in the context of addiction, substance use, and related treatments. As a result, we have added the following paragraph to the discussion section.\n“The findings of this study on action-affordance-based and value-based decision-making can be extended to the domain of affective affordances, offering valuable insights into substance use and addiction treatment. Relevant questions include how action affordances guide behavior by potentiating specific responses, affective affordances, such as cravings triggered by substances, elicit automatic emotional responses through learned associations. The meta-control mechanism described in the study, involving brain regions like the ACC and pre-SMA, aligns with cognitive control processes essential for regulating cravings and resisting substance-related cues in addiction (Everitt & Robbins, 2016; Goldstein & Volkow, 2011). This suggests that therapies targeting cognitive control, such as cognitive-behavioral therapy (CBT) or mindfulness-based interventions, could strengthen meta-control over affective affordances. Incorporating this dual-system arbitration framework into addiction treatment models may support adaptive and personalized intervention strategies (Kober et al., 2010).”\nReviewer #2 (Remarks to the Author):\nThe study by Yi and O’Doherty investigates the computational and neural mechanisms underlying the influence of action affordance on value learning. By combining functional neuroimaging, psychophysics, and computational modeling, they aim to understand how action affordance shapes value-based learning in novel environments. Their main finding suggests that action affordance operates as an independent system that concurrently guides action selection alongside value-based decision-making. These two systems engage in a competitive process to determine the best action. Notably, the authors demonstrate that different brain regions contribute to this process: the pre-SMA encodes the performance difference between the action-affordance and value-based decision-making systems, while the posterior parietal cortex (PPC) integrates predictions from both systems to facilitate action selection. Overall, this study provides important and novel insights into how the brain integrates information from multiple sources to optimize action selection. It fits within the scope of Nature Communications, but there are some points that the authors can address to further improve their study.\n1. The authors describe how the brain integrates information about the affordance and the value of action. There are a number of previous studies that explored how the brain integrates information from multiple sources to make decisions. For instance, there are studies by Paul Cisek, Richard Andersen, Paul Glimcher and others (see below), which showed that the brain integrates dynamically information from multiple sources (such as value and effort/cost) when making decisions. Is it likely that the findings of the Yi and O'Doherty study reflects the integration of expected reward and effort cost to perform an action?\nThis will be also consistent with the concept of desirability value, according to which the posterior parietal cortex integrates information from disparate sources, such as expected reward and effort cost. I suggest that the authors describe the previous studies by Cisek, Andersen, Glimcher, Schrater and others and explain how different are their findings from previous studies that have already reported that the brain integrates information about the affordance of action (i.e., cost/effort) and the value associated with that action.\n• Canaveral CA, Lata W, Green AM, Cisek P. Biomechanical costs influence decisions made during ongoing actions. J Neurophysiol. 2024 Jul 11. doi: 10.1152/jn.00090.2024. Epub ahead of print. PMID: 38988286.\n• Cos I, Duque J, Cisek P. Rapid prediction of biomechanical costs during action decisions. J Neurophysiol. 2014 Sep 15;112(6):1256-66. doi: 10.1152/jn.00147.2014. Epub 2014 Jun 3. PMID: 24899673.\n• Christopoulos V, Bonaiuto J, Andersen RA. A biologically plausible computational theory for value integration and action selection in decisions with competing alternatives. PLoS Comput Biol. 2015 Mar 24;11(3):e1004104. doi: 10.1371/journal.pcbi.1004104. PMID: 25803729; PMCID: PMC4372613.\n• Christopoulos V, Schrater PR. Dynamic Integration of Value Information into a Common Probability Currency as a Theory for Flexible Decision Making. PLoS Comput Biol. 2015 Sep 22;11(9):e1004402. doi: 10.1371/journal.pcbi.1004402. PMID: 26394299; PMCID: PMC4578920.\n• Kable J, Glimcher P (2009) The neurobiology of decision: consensus and controversy. Neuron 63: 733–745. pmid:19778504\nWe appreciate the comment highlighting the need to contextualize our findings within the broader literature on neural decision-making integration. Key studies by Cisek, Andersen, Glimcher, Schrater, and others have established how the brain integrates multiple decision factors with the posterior parietal cortex (PPC) emerging as a critical hub for processing variables like reward, effort, and biomechanical costs.\nCisek and colleagues, for example, demonstrated rapid computation of biomechanical costs during ongoing movements (Canaveral et al., 2024; Cos et al., 2014). Andersen and Christopoulos developed computational frameworks showing how the PPC concurrently evaluates competing actions' value and effort, enabling real-time decision adjustment (Christopoulos et al., 2015; Christopoulos & Schrater, 2015). Glimcher and Kable revealed how primate brains encode value and choice across different regions, conceptualizing decision-making as a multistage process balancing multiple factors (Kable & Glimcher, 2009).\nOur study extends this foundation by examining a novel aspect: the integration of action affordances with value-based choices during learning. We propose a distinct mechanism where affordance and value-based systems interact through a performance-based arbitration model when learning is involved. Unlike traditional approaches that treat affordance as a desirability factor (e.g., cost or effort), our model frames it as an independent decision-making system balanced against the value system through dynamically computed performance of value- and affordance-based systems. This performance functions as a context-sensitive meta-level controlling signal, with the prefrontal cortex (PFC) acting as the meta-controller to adjust each system's influence dynamically, while the PPC integrates this information for action selection.\nWhile we initially considered modeling affordance as a cost factor, where incompatibility between value-based decisions and affordances would reduce action selection probability, this approach proved inadequate. When simulated using fitted parameters, the cost model predicted an increasing choice accuracy gap—contrary to observed data patterns (See Fig. 2 and Supplementary Fig. 8). This divergence from empirical results led us to develop our current performance-based arbitration model.\nTo further clarify these distinctions, we expanded our discussion with the following paragraphs to include a comparison with these studies:\n“Our findings contribute to the understanding of decision-making by highlighting the dynamic interplay between affordance-based and value-based systems during learning-driven action selection. Previous research has established the PPC as a critical hub for integrating decision-related variables such as reward, effort, and biomechanical costs (Canaveral et al., 2024; Cos et al., 2014; Christopoulos et al., 2015; Christopoulos & Schrater, 2015). However, the present study furthers this understanding by demonstrating that the PPC's role goes beyond simply aggregating desirability factors. Instead, we propose a dynamic performance-based arbitration mechanism, where the PFC functions as a meta-controller, continuously adjusting the influence of affordance and value-based systems based on their predictive accuracy. The PPC then integrates these dynamically weighted inputs to guide action selection, suggesting a more complex and adaptive decision-making process.\nThis performance-based arbitration model helps address limitations observed in traditional cost-based frameworks in explaining our data. Previous models that conceptualized affordances as cost-like constraints, similar to biomechanical effort, appeared less effective when modeling learning-driven decisions. Specifically, simulations based on such models predicted an increasing accuracy gap inconsistent with our empirical data. To reconcile these discrepancies, we developed a model where affordance- and value-based systems operate as parallel processes moderated by performance-based meta-control from the PFC. This framework not only aligns with but also expands Glimcher and Kable's multi-stage decision-making theory (Kable & Glimcher, 2009). emphasizing that the PPC is not merely a passive integrator but an active decision-making arena modulated by adaptive, learning-sensitive signals from the PFC. This context-sensitive mechanism highlights how the brain dynamically balances multiple decision systems to optimize behavior.”\n2. It will be also interested in assessing if there is any temporal difference between the two systems. Does the brain first assess the affordance of the action before the value or the other way? fMRI has a low temporal resolution, so it might not be possible to look temporal differences between the two systems, but it will be interested in trying to assess it.\nIndeed fMRI does not allow us to address this question owing to its poor temporal resolution. However, to answer this question at a behavioral level it could be possible to experimentally manipulate the speed in which a decision needs to be made to see if speeded and non-speeded decisions are differently influenced by affordance and value.\nFor instance, if value influences comes online later than the affordance-based process, then performance in incongruent conditions is likely to be poorer during quick decisions compared to slow decisions, even post-learning. This would suggest that value information takes time to override the dominant affordance-based initial response. Conversely, if affordances comes on line later than value-based mechanisms, the reverse outcome would be anticipated – participants should perform better (choosing the more valued action) during fast decisions and worse during slow decisions. The presence of a central arbitration module would imply no performance difference between fast and slow decisions throughout the learning period. However, since our study did not specifically employ a speeded decision-making paradigm, we instead conducted a supplementary analysis comparing trials with quick versus slow reaction times as an alternative approach. For this we conducted GLM analyses on the incongruent trials (log_rt ~ (choosing_aE + choosing_correct + trial_idx + high_low + C(action)).\n<Behavioral> <fMRI>\nThe coefficients from the GLM analyses showed that choosing a correct action and choosing affordance-compatible actions both were associated with shorter RTs, however, the coefficient of choosing_aE was only significant in the behavior only study not the fMRI study.\nIn summary, performance was better in the fast RT trials but the slow RT trials were not consistently biased to affordance across studies. Thus, we cannot conclude that affordance reliably acts as a bias for value-based decisions, nor that value acts as a bias to affordance-based decisions.\nAs the reviewer pointed out the fMRI data doesn’t have the temporal resolution to study the temporal difference between the two systems. Instead, we conducted connectivity analyses which we detailed in addressing the following comment.\nRelated to this point, we included following paragraphs into line 378 of page 12 of the manuscript.\n“Another question that arises from our findings is the temporal dynamics between affordance-based and value-based processes. We considered behavioral approaches to explore this question. A speeded decision-making paradigm, for instance, could reveal whether one system dominates earlier in the decision process. If value information emerges later than affordance-based processes, then performance in incongruent conditions should be poorer during speeded decisions compared to slower ones, even post-learning. Conversely, if affordance-based processes come online later, participants should perform better in speeded decisions and worse in slower ones in incongruent conditions. Alternatively, if a central arbitration mechanism governs the integration of these processes, no significant performance differences would be expected across speeded and non-speeded decisions in incongruent conditions.\nSince our study did not employ a speeded decision-making paradigm, we instead conducted supplementary analyses to compare trials with quick versus slow reaction times. GLM analyses on incongruent trials revealed that both choosing correct actions and affordance-compatible actions were associated with shorter RTs. Notably, the coefficient for choosing affordance-compatible actions was significant only in the behavior-only study, but not in the fMRI study (Supplementary table 8; β = −0.034, z = −2.47, p ≤ 0.05 for selecting affordance-compatible actions in the behavioral experiment, and β = −0.026, z = −1.57, p = 0.116 in the fMRI experiment). While performance was better in fast RT trials, slow RT trials were not consistently biased toward affordance-based decisions across studies. Therefore, we cannot definitively conclude that affordance acts as a consistent bias for value-based decisions or vice versa.”\n3. Can the author assess whether and how the brain encodes the competition between available actions? For instance, it would be interesting to explore differences in brain activation between quick and slow responses. Previous studies have shown that reaction time reflects the level of competition between actions; that is, the longer it takes to respond, the stronger the competition between the available actions.\nAs noted in our response to comment 2, reaction time in this task reflects a combination of factors, including learned value (learned contingency), affordance-compatibility, task progress, and other influences. Consequently, it is challenging to attribute slower reaction times solely to higher levels of competition between affordance-compatible actions and high-value actions.\nOur GLM analysis demonstrated that the coefficient for affordance-based choices was the only significant predictor of increased reaction times in one of the collected datasets. Therefore, instead of directly using reaction time as a measure of competition, we employed the absolute difference between model-generated parameters of value and affordance for chosen actions as a proxy for competition. This metric was subsequently included in our psychophysiological interaction (PPI) analysis.\nHowever, when using a group-level value region as the seed, we did not observe any regions with selectively stronger connectivity under conditions of higher competition (t(29) = -0.225, p = 0.823 for the value region; t(29) = 0.431, p = 0.669 for the affordance region; t(29) = -0.626, p = 0.536 for the performance difference region; t(29) = -0.578, p = 0.568 for the value-based performance region; t(29) = -0.255, p = 0.801 for the affordance-based performance region).\nWhile these findings do not provide conclusive evidence about how the brain encodes competition between available actions in this task, we acknowledge the importance of this question and plan to address it in future research.\nRelated to this analysis, we added the following sentences in the main manuscript line 485.\n“Additionally, we attempted to identify changes in functional connectivity between the action selection region and other identified regions (chosen value, chosen affordance, performance of value, or performance of affordance) when there is a competition between actions, and presumably weighing more on the value-based choice for the final decision. However, we did not find a statistically significant change in functional connectivity. (For additional details, see Note 4 in the Supplementary Materials.)”\nand a note in the supplementary materials as follows.\n“Note4: Examining functional connectivity changes during action competition\nWe aimed to examine changes in functional connectivity when competing actions arise due to discrepancies between affordance-based and value-based choices. Rather than using RT as a proxy for competition, we employed the absolute difference between model-generated parameters of value and affordance for chosen actions. This decision was based on our GLM analysis, which demonstrated that RT is influenced not only by competition between actions but also by task progress and value. The absolute difference between these model-generated parameters was subsequently incorporated into our psychophysiological interaction (PPI) analysis.\nHowever, when using a group-level value region as the seed, we did not observe any regions with selectively stronger connectivity under conditions of higher competition (t(29) = -0.225, p = 0.823 for the chosen value region; t(29) = 0.431, p = 0.669 for the chosen affordance region; t(29) = -0.626, p = 0.536 for the performance difference region; t(29) = -0.578, p = 0.568 for the value-based performance region; t(29) = -0.255, p = 0.801 for the affordance-based performance region).”\nMinor points.\n1. Please add line numbers in the manuscript. It is easier to review it.\nOk. We added the line numbers in the revised manuscript.\n2. The authors need to improve the quality of writing in some parts of the manuscript. For instance, there are some parts that there is no space between the sentences (e.g., last line in page 6, “…experiment.Because…” ).\nOk. We revised any typos or mistakes in the revised manuscript.\nReviewer #3 (Remarks to the Author):\nThis study must have been a massively laborious undertaking, combining RL, mode-based fMRI, and automated decoding of gesture-based response. It is thus very unfortunate that I do not find the research question noteworthy. The interaction between action affordability and value-based decision-making is a very interesting topic, but the experimental paradigm in this study is incredibly artificial, at least for two reasons:\n1. It is unclear if this task, in which participants see an object associated with (for instance) a pinching gesture and instead make a clenching gesture to earn a probabilistic reward, captures any real-world learning problems.\nThank you for your valuable feedback. We acknowledge your concern regarding the direct real-world applicability of our task. In real-world contexts, there are numerous examples where people must learn unintuitive gestures to use unfamiliar products or tools effectively.\nOne illustrative example is the use of chopsticks. While a stick naturally suggests a simple clench or pinch gesture depending on its size, mastering chopsticks involves learning a specific pinching gesture that is far removed from the basic action affordance for sticks. This adaptation is essential for achieving the rewarding outcome of successfully grasping food.\nAdditionally, this research sheds light on why certain products, like the iPhone or iPad, are remarkably easy to use. These devices are designed with compatible affordances that align with intuitive user interactions, thereby reducing the learning curve. Conversely, products that fail to tap into existing affordances pose greater challenges for users, necessitating more effort and adaptation.\nWe recognize that our task, similar to multi-armed bandit tasks, is a laboratory-based abstraction. Although these may not perfectly mirror real-world scenarios, they enable us to study human cognitive processes with exceptional precision. By abstracting these situations, we gain valuable insights into the mechanisms underlying behavioral adaptation in the presence of action affordances. These insights are crucial for understanding how people interact with technology and adapt to new tools, reflecting a real-world problem that our study addresses. This connection underscores the relevance of our findings to everyday challenges in learning and adaptation, influenced by the affordances present in the user environment.\nWe updated the last paragraph of the introduction as follows.\n“To investigate how affordance influences decision-making, we designed a novel behavioral task that examines conditions in which the most rewarding action aligns with an affordance-compatible action and those in which they are incongruent. The incongruent conditions, in particular, capture a fundamental aspect of real-world action learning: the need to override intuitive action affordances when adapting to novel tools, interfaces, or environments. Many everyday tasks—such as learning to use chopsticks or mastering a musical instrument—require individuals to suppress immediate, affordance-driven responses in favor of newly learned actions that lead to the desired outcome.\nTo systematically explore the mechanisms underlying this process, we employed a computational model-driven approach. We specified a series of computational models to capture different ways in which affordance might influence value learning and tested these models against human behavior. Additionally, we used functional MRI (fMRI) to measure brain activity and identify the neural correlates of these computational processes.\nBy examining how participants adjust their motor responses in response to probabilistic rewards, our study provides valuable insights into the cognitive mechanisms supporting this type of behavioral adaptation. Just as real-world learning involves trial and error, feedback processing, and reinforcement learning, our task allows us to isolate and analyze these processes in a controlled setting. Thus, the overarching goal of this study is to investigate how action affordances interact with value-based action learning across behavioral, computational, and neural levels.”\n2. The experiment was divided into blocks, in each of which participants learned the most rewarding actions for four items, each corresponding to one of the four conditions (CH, CL, IH, and IL). The descriptions of instructions given to participants are entirely missing in this manuscript, but even if they were not told anything about reward contingencies, it is natural for them to eventually start trying to figure out which items are in the congruent or incongruent condition in each block (it is perhaps one of the most rational strategies they can take).\nThe instruction has been added to the supplementary figures 17, 18 and 19 as follows.\nAs shown in the new supplementary figures 17, 18, and 19, participants were not informed that half of the items have congruent actions as the most rewarding actions. Instead, they were simply instructed to identify the best hand gestures for each object, with no mention of the affordances.\nIt can also be observed that participants, unaware of the reward structure, consistently showed an initial exploration bias towards affordance-compatible actions across all blocks. This indicates that participants were unable to decipher the task structure in which half of the items were in congruent conditions and the other half were incongruent. If participants had learned the task structure, there would likely be a gradual increase in the initial exploration bias. However, as can be seen in supplementary figure 2 a and c, the initial exploration bias stayed around 0.5. Furthermore, the mixed linear regression of initial selection bias on block index showed non-significant slopes in both behavioral and fMRI datasets (Fixed-effect coefficients for the slope: β = -0.005, z=-1.115 and p=0.265 in the behavioral data, β=-0.002, z=- 0.232 and p=0.817 in the fMRI data, please check the details in the following table).\n<Behavioral data> <fMRI data>\nWe added the following sentence to the caption of Supplementary Fig.2\n“The mixed linear regression of initial selection bias on block index showed non-significant slopes in both datasets (Fixed-effect coefficients for the slope: 𝛽= -0.005, z=-1.115 and p=0.265 in the behavioral data, 𝛽 =-0.002, z=-0.232 and p=0.817 in the fMRI data, for the intercept: 𝛽 = 0.461, z=9.431 and p<0.001 in the behavioral data, 𝛽 =0.459, z=12.380 and p<0.001 in the fMRI data)”\nIt is unclear the degree to which the reported findings are generalizable beyond this experimental paradigm. The point 2 is particularly crucial for the significance of the modeling results; the \"performance-based arbitration model\" is effectively equivalent to first testing if the most affordable action is the most rewarding one (which is the case 50% of the time) and trying the other two actions if not. The artificiality of the experimental paradigm heavily limits my enthusiasm for this manuscript. It does not help that the authors did not conduct much analysis to demonstrate that their complicated models characterize the behavioral data better than simpler, perhaps heuristic, accounts (such as the one I described above).\nThank you for your observations concerning the complexity of our proposed model. We appreciate the opportunity to clarify why a more complex model is necessary given the nuances of our experimental data.\nVariability in Initial Choices: The experimental results demonstrate that participants did not consistently initiate their choices with affordance-compatible actions, contrary to the expectations of a simpler heuristic model. Specifically, Supplementary Figures 2b and d show that the frequency of initially selecting affordance-compatible actions was notably below 0.5 for half of the participants. This departure from a simple heuristic approach underscores the need for a model that can account for the observed variability in choices.\nImplementation and Evaluation of Simpler Models: Responding to your suggestion for simplicity, we initially tested a win-stay-lose-shift strategy, presuming an initial preference for affordance-compatible actions. However, the model failed to align with the data, indicating that first choices were not always affordance-compatible. To address this, we adapted the model to include a softmax decision rule influenced by affordance compatibility (encoded with 1 or 0 for the simplicity) with an inverse temperature parameter (β), coupled with a win-stay-lose-shift strategy for subsequent choices.\nHowever, the experimental data suggest that participants did not always prefer the affordance-compatible action as their initial choice, contrary to what might be expected under a simple heuristic model. Specifically, as demonstrated in Supplementary Figure 2b and d, the initial selection frequency of affordance-compatible actions was notably below 0.5 for half of the participants. This indicates variability in initial choices that cannot be adequately captured by a straightforward heuristic model.\nDespite these modifications aimed at simplifying the decision rule, the simulations struggled to replicate the learning curves seen in the experiment, though they did accurately represent initial biases (see the above posterior predictive check figures). The log evidence of this simpler model was significantly poorer compared to more comprehensive models (behavioral data: log evidence = -18398; fMRI data: log evidence = -15317; see Figure 3 for log evidences of other models).\n)01\nx\n91=n(\nnoitalumiS\n)01\nx\n03=n(\nnoitalumiS\nCH CL Congruent ΔP\nIH IL Incongruent\nCH CL Congruent ΔP\nIH IL Incongruent\nNecessity of a Complex Model: Consequently, our analyses suggest that the observed behavioral adaptations align more with a model incorporating gradual learning rather than simple heuristic strategies. Therefore, to computationally describe the reviewers’ point on “first test affordance-compatible action and explore afterward”, a gradual adaptation, or learning, is necessary with the affordance-based choice component. To capture this behavior, we employed models that incorporate elements of Bayesian or reinforcement learning alongside strategies that factor in affordance, such as bias, initial values, or distinct decision-making systems.\nThe \"performance-based arbitration\" model, in particular, effectively captures how strategies based on both affordance and value compete and adapt over time, in terms of model fitting metric and simulation. In response to the reviewer's concerns regarding the model's parameter specificity and the integration of affordance scores in decision-making, these aspects will be thoroughly addressed in the respective sections. As can be seen in the corresponding sections, this model has shown effective parameter recovery, and its adaptability to variations—whether through simplification of parameters or alternative methods of incorporating affordance—consistently reinforces the validity of our conclusions.\nIn conclusion, while we acknowledge the concerns regarding model complexity, our analysis supports the necessity of the arbitration mechanism in the model to adequately capture the intricate behaviors observed. Our findings suggest that simpler models, though appealing for their parsimony, fall short in explaining the depth and variability of human decision-making as demonstrated in our study.\nRegarding this point, we updated “Note2: Additional model simulation results section” in the supplementary material as follows.\n“Moreover, we implemented a WSLS-type model to further assess whether a simpler approach could account for the observed “first test affordance-compatible action and explore afterward” pattern in the data. This simplified model incorporated a softmax decision rule influenced by affordance compatibility (encoded as 1 or 0 for simplicity), modulated by an inverse temperature parameter, alongside a win-stay-lose-shift strategy for subsequent choices. While these modifications allowed the model to capture initial biases, it failed to replicate the learning curves observed in the experiment (see Supplementary Fig. 25). Furthermore, the log evidence for this simpler model was substantially lower compared to more comprehensive models, indicating a poorer fit to the data (behavioral data: log evidence = -18398; fMRI data: log evidence = -15317; see Fig. 3 for log evidence of other models).”\nBelow are some of the specific issues I had with the modeling approach:\n3. The models seem overparameterized. Just an example -- even in the simplest RL model, the authors had b_pinch, b_clench, and b_poke as free parameters. They cannot be uniquely estimated. Forgive my frankness, but I am surprised that the authors did not catch this, and such a mistake casts doubts on the entire modeling results. One suggestion would be to make the code publicly available at this point for transparency.\nThank you for the valuable feedback and we acknowledge your concerns regarding potential overparameterization in the model, specifically regarding the parameters b_pinch, b_clench, and b_poke. Below, we address your points and provide evidence to support the robustness of our modeling approach.\nParameter Identifiability: To address this, as detailed in the Methods section, we conducted a parameter recovery analysis to assess identifiability given our task design of 480 trials per participant (960 for the behavioral experiment) by simulating the performance-based arbitration model 100 times with a wide range or free parameters. The analysis yielded an average Pearson correlation of 0.76 across the 8 parameters, indicating strong parameter recovery. The following plot demonstrates recovery results for each parameter, and no evidence of overparameterization was found under this experimental design.\n2\n1\n0\n-1\n-2\n-3\n-3 -2 -1 0 1 2\nTarget\nderevoceR\nAlpha\nr = 0.65, p < 0.001\n6\n4\n2\n0\n-2\n-2 0 2 4 6\nTarget\nderevoceR\nBeta\nr = 0.91, p < 0.001\n8\n6\n4\n2\n0\n-2\n-4\n-4 -2 0 2 4 6 8\nTarget\nderevoceR\nBeta Aff\nr = 0.68, p < 0.001\n4\n3\n2\n1\n0\n-1\n-2\n-2 0 2 4\nTarget\nderevoceR\nBeta Perf\nr = 0.73, p < 0.001\n2\n0\n-2\n-4\n-4 -2 0 2\nTarget\nderevoceR\nBeta0 Perf\nr = 0.58, p < 0.001\n0.5\n0\n-0.5\n-0.5 0 0.5\nTarget\nderevoceR\nB Pinch\nr = 0.74, p < 0.001\n0.5\n0\n-0.5\n-0.5 0 0.5\nTarget\nderevoceR\nB Clench\nr = 0.76, p < 0.001\n0.6\n0.4\n0.2\n0\n-0.2\n-0.4\n-0.6\n-0.5 0 0.5\nTarget\nderevoceR\nB Poke\nr = 0.83, p < 0.001\nAlternative Parameterization: To further address the concern, we explored an alternative parameterization, where the action bias terms were reduced to two free parameters: 𝑏_𝑐𝑙𝑒𝑛𝑐ℎ−𝑏_𝑝𝑖𝑛𝑐ℎ = 𝛽 and 𝑏_𝑝𝑜𝑘𝑒−𝑏_𝑝𝑖𝑛𝑐ℎ = 𝛽 .\n! \"\nDespite this reduced parameterization, the performance-based arbitration model remained the best-fitting model.\n<Behavioral (n=19)> <fMRI (n=30)>\nAdditionally, we compared the fit parameters between the original and updated models, and the correlations (see the plot below) show that the two parameterizations produced highly consistent results.\n<Behavioral (n=19)> <fMRI (n=30)>\n2 1\n0 -1\n-2\n-3\n-4 -4 -2 0 2\nUpdated\nlanigirO\nAlpha\nr = 1.00, p < 0.001\n5 4\n3 2\n1\n0 -1\n0 2 4\nUpdated\nlanigirO\nBeta\nr = 1.00, p < 0.001\n4\n2\n0\n-2\n-2 0 2 4\nUpdated\nlanigirO\nBeta Aff\nr = 1.00, p < 0.001\n3\n2\n1\n0\n0 1 2 3\nUpdated\nlanigirO\nBeta Perf\nr = 1.00, p < 0.001 2\n1\n0\n-1\n-2\n-3\n-2 0 2\nUpdated\nlanigirO\nBeta0 Perf\nr = 1.00, p < 0.001\n0.2 0\n-0.2\n-0.4 -0.6\n-0.8\n-1\n-1 -0.5 0\nUpdated\nlanigirO\nB Clench - B Pinch\nr = 1.00, p < 0.001\n0.5\n0\n-0.5\n-1\n-1 -0.5 0 0.5\nUpdated\nlanigirO\n1\n0 -1\n-2\n-3\n-3 -2 -1 0 1\nUpdated\nB Poke - B Pinch\nr = 1.00, p < 0.001\nlanigirO\nAlpha\nr = 1.00, p < 0.001\n4\n2\n0\n-2\n-2 0 2 4\nUpdated\nlanigirO\nBeta\nr = 1.00, p < 0.001\n6 4\n2\n0\n-2\n-2 0 2 4 6\nUpdated\nlanigirO\nBeta Aff\nr = 1.00, p < 0.001\n4 3\n2\n1 0\n-1\n0 2 4\nUpdated\nlanigirO\nBeta Perf\nr = 1.00, p < 0.001\n2\n0\n-2\n-4\n-4 -2 0 2\nUpdated\nlanigirO\nBeta0 Perf\nr = 1.00, p < 0.001\n1\n0.5\n0\n-0.5\n-0.5 0 0.5 1\nUpdated\nlanigirO\nB Clench - B Pinch\nr = 1.00, p < 0.001\n1.5\n1\n0.5\n0\n-0.5\n-1\n-1 0 1\nUpdated\nlanigirO\nB Poke - B Pinch\nr = 1.00, p < 0.001\nCollectively, the parameter recovery analysis and the results from the alternative parameterization demonstrate that while the original parameterization may theoretically raise concerns about identifiability, the task design and data volume ensured robust parameter estimation. Furthermore, the model-based cognitive variables derived from the reduced-parameter model were identical to those from the original model, reinforcing the robustness of our findings (see the plot below).\n<Behavioral (n=19)> <fMRI (n=30)>\nTo facilitate further evaluation and transparency, we have made our full codebase publicly available at https://github.com/sanghyunyi/AEordance2025. The repository includes all computational models and supporting analyses, which we hope will aid in understanding and verifying the modeling process and results.\nRegarding this further analysis, we included a new Note 3 in the supplementary materials as follows.\n“To evaluate concerns regarding potential overparameterization in our model—specifically related to the parameters b_pinch, b_clench, b_poke—we conducted a thorough parameter recovery analysis and explored an alternative parameterization. We acknowledge that, in theory, these parameters may not be uniquely identifiable in isolation. However, as detailed in the Methods section, we performed a parameter recovery analysis using a simulation-based approach. By generating synthetic data with the performance-based arbitration model across 100 simulations and a broad range of free parameters, we assessed the model’s ability to recover its true parameter values. Given our experimental design of 480 trials per participant (960 for the behavioral experiment), the analysis revealed an average Pearson correlation of 0.76 across the eight parameters, demonstrating strong parameter recovery. The Supplementary Fig. 26 illustrates the recovery results for each parameter, with no evidence of overparameterization under this task design.\nTo further validate the robustness of our modeling approach, we tested an alternative parameterization by reducing the action bias terms to two free parameters 𝑏_𝑐𝑙𝑒𝑛𝑐ℎ−𝑏_𝑝𝑖𝑛𝑐ℎ = 𝛽 and 𝑏_𝑝𝑜𝑘𝑒−𝑏_𝑝𝑖𝑛𝑐ℎ = 𝛽 . Despite this simplification, the performance-based arbitration model continued to outperform other models, confirming its superior fit to the data in both behavioral and fMRI experiments (See Supplementary Fig. 27).\nMoreover, comparisons between the fit parameters and model-derived cognitive variables from the original and reduced models demonstrated perfect correlations (Pearson r=1.0), indicating that both parameterizations produced consistent results (See Supplementary Fig.28).”\n4. The authors make a lot of specific parametric assumptions in modeling, for example how affordance compatibility is linearly scored based on the post-task survey data. It is unclear to what degree the model comparison results depend on these (mis)specifications.\nWe appreciate the reviewer’s concern regarding the parametric assumptions made in modeling, specifically how affordance compatibility is linearly scored based on the post-task survey data. To address this, we tested three additional parameterization methods for affordance compatibility:\n1. Squared affordance scores,\n2. Log-transformed affordance scores,\n3. A binarized version, where only the maximum affordance score of pinch, clench, and poke is parameterized as 1, while the scores for other actions are set to 0.\nAdditionally, in response to the reviewer’s third point, we incorporated two free parameters to model action biases in these new sets of models.\nOur results remain consistent across all three alternative parameterization settings. For instance, we still observed that the performance-based arbitration model achieved the best model fitting performance across both behavioral and fMRI datasets (see the following graphs).\nFurthermore, posterior predictive checks using the three types of performance-based arbitration models successfully captured the observed patterns of initial exploration bias, learning curves, and slopes. The results are consistent with the statistical tests reported in the manuscript, where we regressed the performance difference between congruent and incongruent conditions (ΔP) on the number of trials or the number of rewarded trials using the same mixed-effect GLM (see the attached plots with beta coefficients, z-statistics, and p-values).\n<Behavioral simulation>\nSimulation (n=19 x 10)\nderauqS\ncimhtiragoL\ndeziraniB\nCH CL\nIH IL\nCongruent\nIncongruent\nΔP\n<fMRI simulation>\nSimulation (n=30x 10)\nBased on these findings, we can confidently conclude that our primary conclusion that the performance-based arbitration model is the best is robust and not sensitive to the specific parameterization of the affordance score. We included the following subsection in the methods section.\n“Alternative parameterization of aHordance compatibility\nderauqS\ncimhtiragoL\ndeziraniB\nCongruent\nIncongruent\nΔP\nCH CL\nIH IL\nTo address potential concerns regarding the parametric assumptions of affordance compatibility, we tested three additional parameterization methods based on post-task survey data: (1) squared affordance scores, (2) log-transformed affordance scores, and (3) a binarized version where the maximum affordance score of pinch, clench, or poke was set to 1, and all other scores were set to 0.\nConsistent results across all parameterizations confirmed that the performance-based arbitration model provided the best fit to both behavioral and fMRI datasets (See Supplementary figure 20). Posterior predictive checks were conducted as well using the three performance-based arbitration models with alternative parameterizations. These models successfully captured observed patterns such as initial exploration bias, learning curves, and slopes. Statistical analyses, including mixed-effects GLM regressions of Δ𝑃(performance difference) on trial progression and rewarded trials, further supported the robustness of the model's predictive accuracy and its insensitivity to specific parameterization choices (See Supplementary figures 21 and 22).\"\n5. The authors examined an alternative model, in which learning rates differ between congruent and incongruent conditions. They rejected it solely based on the lack of population-level difference in learning rate estimates, but it would make much more sense to reject it based on its performance (e.g. log evidence).\nThank you for pointing this out. Indeed, we conducted additional analyses to compare the models based on their performance. Specifically, the two learning rate models (2lr) exhibited worse log evidence compared to the single learning rate model, as shown in the provided log evidence graph, which includes the two learning rate fixed arbitration model.\nFurthermore, the posterior predictive checks for the two learning rate models revealed their inability to capture the decreasing performance gap between congruent and incongruent conditions.\nBased on both model evidence (log evidence) and posterior predictive checks, we found sufficient grounds to reject the two learning rate model and added the following sentence to the manuscript.\n“Additionally, the model with two distinct learning rates exhibited lower log evidence compared to the single learning rate model and was unable to replicate the decreasing performance gap observed between congruent and incongruent conditions (see Supplementary Fig. 23).”\n6. The authors wrote: \"We didn’t use the hierarchical Bayesian inference function of the toolkit because model simulations using the hierarchically estimated parameters couldn’t reproduce the characteristics of the real data\" (p.24). This sentence is vague (what characteristics?), but I am not convinced that it is a scientifically sound and unbiased reason not to use the hierarchical model, particularly given the common observation in the literature that it tends to help estimate parameters more reliably.\n)01\nx\n91=n(\nnoitalumiS\n)01\nx\n03=n(\nnoitalumiS\nThank you for your comment regarding the use of hierarchical Bayesian inference and for pointing out the need to provide further details. First of all we would like to highlight that in our experiment, each participant completed either 480 or 960 trials—approximately 5 or 10 times more trials per participant than the data sets commonly found in studies demonstrating the effectiveness of hierarchical Bayesian inference (Ahn et al., 2013; Piray et al., 2019). While hierarchical inference is highly valuable in situations where participant-level data are sparse, its utility diminishes when there are sufficient data points for reliable individual parameter estimation. In our case, we observed cases when the hierarchical approach introduced additional noise from other participants rather than improving parameter estimates. For example, in parameter recovery analysis using 100 times simulated data, the average correlation between recovered parameters of Performance-based arbitration model using the hierarchical Bayesian inference was 0.73 but without the hierarchical inference 0.76. This resulted in failures to reproduce the actual behavioral patterns in the posterior predictive checks (See the following figures. Statistical test results with a red underscore indicate failed cases).\nFor further context on the trade-offs of hierarchical Bayesian inference as a function of trial count, please refer to Figure 5 in Piray et al. (2019).\n)01\nx 91=n(\nnoitalumiS\n)01\nx 03=n(\nnoitalumiS\nPerformance-based\n)01\nx 91=n(\nnoitalumiS\n)01\nx 03=n(\nnoitalumiS\nReliability-based\n)01\nx 91=n(\nnoitalumiS\n)01\nx\n03=n(\nnoitalumiS\nConflict-based\n)01\nx 91=n(\nnoitalumiS\n)01\nx\n03=n(\nnoitalumiS\nFixed\nWe hope this additional explanation clarifies our rationale for not using hierarchical Bayesian inference in this study and demonstrates that our decision was justified and tailored to the specific characteristics of our dataset. Accordingly, we updated the corresponding sentence into as follows.\n“We did not utilize hierarchical parameter estimation in this study and relied instead on estimating individuals subject-level parameters. This was done because while hierarchical Bayesian inference for parameter estimation is especially useful in situations where participant-level data are sparse, its utility diminishes when there are sufficient data points for reliable individual parameter estimation (Ahn et al., 2013; Piray et al., 2019). In our case, each subject completed 480 or 960 trials, allowing for reliable estimation of individual subject-level parameters. In additional analyses we found that the hierarchical approach introduced additional noise from other participants into the individual subject-level estimates resulting in less accurate rather than more accurate parameter estimates. For example, in parameter recovery analysis using 100 times simulated data, the average correlation between recovered parameters of Performance-based arbitration model using the hierarchical Bayesian inference was 0.73 but without the hierarchical inference 0.76. This resulted in failures to reproduce the actual behavioral patterns in the posterior predictive checks (See Supplementary Fig. 24)”\n7. In fMRI analysis, the wrote wrote: \"we found positive correlations between each participant’s propensity to choose the most rewarding actions and the extent to which BOLD activity in the identified arbitration regions can be explained by our model-derived arbitration variables\" (p. 15). Doesn't it simply mean that, for those whose behavioral data was well characterized by a specific model, their MRI data was also well characterized by the same model?\nThank you for your observation regarding our interpretation of the fMRI findings. To clarify, our analysis revealed a correlation between participants' task performance and the extent to which their neural data could be explained by our model-derived arbitration variables. However, we did not find direct evidence to support the interpretation that participants whose behavioral data were well characterized by the performance-based model also exhibited neural data that closely aligned with the same model.\nTo further test this, we examined the relationship between the log evidence of the performance-based arbitration model and participants’ behaivoral task performance. For this analysis we indeed found a strong positive correlation, indicating that participants who were better explained by the performance-based arbitration model also performed better on the task.\nHowever, when we turned to the fMRI data we did not observe a significant correlation between the log evidence of the performance-based arbitration model and the extent to which BOLD activity was explained by the same model-derived variables.\nThus, we hesitate to conclude that behavioral data well-characterized by the performance-based arbitration model directly implies that fMRI data will also be well-characterized by the same model. Instead, our findings suggest that arbitration between value-based and affordance-based action selection is crucial for achieving high task performance.\nFurthermore, our neural data support this idea, showing that individuals who utilized arbitration-related brain regions more effectively demonstrated better task performance.\nThus, we’d like to reserve the interpretation that when the behavioral data well explained by the performance-based arbitration model, their fMRI data as well. Instead, we’d like to keep that arbitration between value-based and affordance-based action selection is necessary for achieving the high performance and the neural data further support the idea by showing that the people who are utilizing arbitration region for the arbitration showed better performance."}, {"round": 2, "reviewer_comments": {"reviewer_2": "The authors have thoroughly addressed all the key points raised in my review, and I appreciate their thoughtful revisions. I have no further concerns and recommend the manuscript for publication.", "reviewer_4": "In this paper the authors explore how reward information and action affordances co-shape value-based learning.\nThis is an interesting research question, which also has applied implications.\nThe reported experiments are well designed and reported; and the same holds for most analyses in the manuscript.\nMy main concern is whether the authors have sufficient evidence to back up their main computational claim. Indeed, the behavioural data rule-out computational models in which the action affordance bias is fixed over time. The authors proceed and propose a series of models in which a dynamic bias emerges from cognitive control mechanisms. However, can the authors rule out the possibility that the affordance bias has its own (passive/ decaying) temporal dynamics? Incorporating a time-varying bias could potentially \"patch-up\" the simpler models the authors tried out. Where is the evidence that there is an active control mechanism causing this time-varying bias? Perhaps there are certain qualitative aspects of the behavioural data that would lend support to the arbitration models over a time-varying bias. In some, there seems to be another class of models in-between the fixed and arbitration ones that the authors need to assess.\nOther comments:\n-- The modelling endeavour is commendable, however the exposition of the modelling results can be better streamlined. For instance, comparing a Bayesian and RL model early on but then sticking with an RL formulation for the arbitration models is not well motivated/ explained.\n-- It is not clear to me that the Affordance Prediction Error is a meaningful quantity. It combines incommensurable quantities: affordance compatibility and reward value. I also think that the notion of reliability is relatively fixed for affordance reliability. Reward reliability does increase as a function of learning, so I was wondering whether the authors have tried a simpler reliability model which resorts to the affordance bias only when value-based reliability is low (at the early trials).\n-- The authors state that the RT behavioural effects are indistinguishable in early and late trials. However, the choice effects manifest differently at early trials (larger affordance bias) relative to late trials. It is not clear whether these two statements are at odds with each other.\n-- At the end of section 2.3 the authors acknowledge that their experimental paradigm does not allow them to delineate the temporal dynamics of processing value information and action affordances. I agree with their statements. Across these lines, I find that the slow vs. fast RT analyses is not informative. This analysis could be informative if we were sure that all slow responses are attributed to a higher decision boundary (with reference to an evidence accumulation framework). However, slow responses can also be attributed to other factors (low arousal levels or drift rate, longer non-decision time etc) which limits the inferential potential of a median split analysis in this paradigm.", "reviewer_5": "This is an impressive multi-modal approach to investigating the interaction of action affordability and value-based decision making in a methodologically and theoretically strong paper. The authors admirably integrated the helpful comments by the reviewers in this revision, which has clarified several points about the strength of the modeling approach, the significance of the question, clarification on the methods and generalization of the paradigm/results. Helpful additions include supplementary figures 17-19, testing of new models (e.g., win-stay-lose-shift strategy) and addressing concerns about overparameterization. The authors also made their codebase publicly available to enhance transparency. Overall, this revised paper will make a significant contribution to the literature and is well-suited for publication in Nature Communication."}, "author_response": "We sincerely thank the reviewer for their thoughtful feedback throughout the review process and for their positive evaluation of our revisions. We greatly appreciate the time and effort they invested in improving the quality of the manuscript.\nResponse to reviewer #4\nThank you for highlighting this important theoretical alternative. To directly address this point, we implemented and tested additional models incorporating a passively decaying affordance bias. Two forms of decay, exponential and hyperbolic over trials, were examined. However, when compared to our arbitration models, these passive decay models performed significantly worse in terms of model evidence and Bayesian model comparison. As shown in Supplementary Figure 29, models with passive decay consistently underperformed relative to those implementing arbitration via active control mechanisms.\n<Supplementary Figure 29>\nFurthermore, the passive decay models failed to capture key behavioral signatures that were successfully reproduced by the best-performing arbitration model, most notably, the decreasing gap between congruent and incongruent conditions in the frequency of correct action selection as a function of rewarded trials. Simulated results from both decay variants did not show a significant reduction in the gap between learning slopes for the congruent and incongruent conditions, as presented in Supplementary Figure 30.\n<Supplementary Figure 30>\nThese findings suggest that models assuming a passive transition from affordance-driven behavior to goal-directed decision-making, irrespective of task context, are less likely to account for the observed faster adaptation in incongruent conditions compared to congruent ones for the same number of rewards.\nWe have clarified this in the revised Results section and added the following paragraph.\n“To further rule out alternative explanations, we additionally implemented and tested a series of simpler models. These included two passive-decay models in which the affordance bias decreased over time according to either an exponential or hyperbolic decay function, as well as a restricted arbitration model in which the arbitration weight was determined solely by the reliability of the value-based decision system. All of these models were significantly outperformed by the full arbitration models that incorporate dynamic performance or reliability tracking across both systems (see Supplementary Fig. 29). Moreover, simulations from these simplified models failed to capture a key behavioral feature: the decreasing gap between congruent and incongruent conditions in the frequency of correct action selection as a function of the number of rewarded trials. In both the decay-based and the value-reliability-only models, this gap failed to decrease significantly (see Supplementary Fig. 30). These findings suggest that neither a passive transition away from affordance-driven behavior nor an arbitration process based only on value reliability is sufficient to explain the observed data. Rather, a dynamic, bidirectional arbitration mechanism—tracking the performance or reliability of both systems—is necessary to reproduce the behavioral effects.”\nDetails about the model implementation are also included in the Methods section, as follows:\n“Arbitration-weight-decay model.\nAs a simple alternative to active dynamic arbitration models, we implement models with a decaying arbitration weight that gradually places greater weight on the value-based decision-making system. The model implementation is identical to that of the fixed arbitration model, except that the arbitration weight decays over the course of the task. In these models, the arbitration weight is a function of the trial index $t$. For example, in the case of exponential decay\n𝑤 = 𝑤 ∗𝑤𝑡\n𝑞,𝑡 𝑞 𝑑\nwhere 0 < 𝑤 ≤ 1; and in the case of hyperbolic decay,\nd\n1\n𝑤 = 𝑤\n𝑞,𝑡 𝑞1+𝑤 ×𝑡\nd\nwhere 0<𝑤 . These models thus have 8 free parameters in total (𝛽,𝛼,𝑏 ,𝑏 ,𝑏 ,𝑘,𝑤 ,\nd 𝑝𝑖𝑛𝑐ℎ 𝑐𝑙𝑒𝑛𝑐ℎ 𝑝𝑜𝑘𝑒 𝑞\nand 𝑤 )”\nd\nThese new models have also been added to the public codebase available at\nhttps://github.com/sanghyunyi/Affordance2025\nOther comments:\n-- The modelling endeavour is commendable, however the exposition of the modelling results can be better streamlined. For instance, comparing a Bayesian and RL model early on but then sticking with an RL formulation for the arbitration models is not well motivated/ explained.\nThank you for the thoughtful feedback. We agree that while we initially compared both Bayesian and RL frameworks, our transition to focusing exclusively on RL-based arbitration models in the later stages of model development may not have been sufficiently justified in the manuscript.\nThe Bayesian model offers a normative approach to decision-making by optimally integrating new learning signals. We explored several ways to incorporate affordance effects within this framework. One approach was to model the affordance effect as a prior, consistent with Bayesian principles. However, Bayesian models are generally less flexible when it comes to capturing suboptimal or non-normative learning behavior, particularly when participants do not update beliefs in an optimal manner. As a result, this approach placed natural limits on the model’s ability to account for the behavioral data.\nA second approach was to incorporate affordance as a bias and model adaptation as belief updates about the likelihood that a given action would lead to reward. Nonetheless, this class of models consistently yielded poorer fits to the behavioral data compared to even relatively simple RL models, which offered greater expressive power.\nGiven the flexibility of RL and its strong empirical grounding in modeling value-based learning, we ultimately chose to explore a broader set of RL-based model variants for arbitration.\nWe made these points clearer by adding the following sentences into the main section 2.3.\n“We also tested the possibility that value learning could be supported by Bayesian inference rather than RL (Bayes model). Although Bayesian learning offers a normative framework for optimal belief updating, we found it less well-suited for capturing the flexible and sometimes suboptimal behavioral patterns observed in our task. We implemented several Bayesian variants, including models where affordance was incorporated either as a prior or as a bias influencing belief updates. However, these Bayesian models consistently showed poorer fits to the data compared to RL models (See Fig. 3) which allowed for more expressive adaptation to feedback. Given the greater flexibility of RL and its empirical success in modeling value-based choice behavior, we focused our subsequent modeling efforts on RL-based models.”\n-- It is not clear to me that the Affordance Prediction Error is a meaningful quantity. It combines incommensurable quantities: affordance compatibility and reward value. I also think that the notion of reliability is relatively fixed for affordance reliability. Reward reliability does increase as a function of learning, so I was wondering whether the authors have tried a simpler reliability model which resorts to the affordance bias only when value-based reliability is low (at the early trials).\nWe agree that the Affordance Prediction Error (APE) is a computational construct designed to capture deviations from expected affordance-based choices, and we acknowledge that it is not directly comparable to the Reward Prediction Error (RPE) in terms of scale or psychological interpretation. Our intention in introducing the APE was not to suggest a neural or conceptual equivalence with RPE, but rather to provide a tractable mechanism for monitoring the predictive performance, or reliability, of the affordance-based decision system within the broader arbitration framework. In this context, APE serves as a heuristic for estimating how effectively the affordance-based system guides behavior over time. Specifically, the system’s reliability dynamically decreases when affordance-based choices fail to yield rewards and increases when affordance-compatible actions lead to successful outcomes.\nWe recognize that the concept of “reliability of affordance” can also be interpreted as the inherent strength of stimulus–action associations, which we captured using static affordance scores in our models. However, in response to the reviewer’s comment, we additionally tested a simplified variant in which arbitration weights were determined solely by the reliability of the value-based system. This value-only reliability model performed worse than the full reliability-based arbitration model that tracks both systems (see Supplementary Figure 29). Moreover, this model failed to reproduce the key behavioral signature: a decreasing gap between congruent and incongruent conditions in correct action choices as a function of the number of rewarded trials (see Supplementary Figure 30).\nWe have clarified this in the revised section 2.3 and added the following sentences.\n“To further rule out alternative explanations, we additionally implemented and tested a series of simpler models. These included two passive-decay models in which the affordance bias decreased over time according to either an exponential or hyperbolic decay function, as well as a restricted arbitration model in which the arbitration weight was determined solely by the reliability of the value-based decision system. All of these models were significantly outperformed by the full arbitration models that incorporate dynamic performance or reliability tracking across both systems (see Supplementary Fig. 29). Moreover, simulations from these simplified models failed to capture a key behavioral feature: the decreasing gap between congruent and incongruent conditions in the frequency of correct action selection as a function of the number of rewarded trials. In both the decay-based and the value-reliability-only models, this gap failed to decrease significantly (see Supplementary Fig. 30). These findings suggest that neither a passive transition away from affordance-driven behavior nor an arbitration process based only on value reliability is sufficient to explain the observed data. Rather, a dynamic, bidirectional arbitration mechanism—tracking the performance or reliability of both systems—is necessary to reproduce the behavioral effects.”\nDetails about the model implementation are also included in the Methods section, specifically within the reliability arbitration model subsection, as follows:\n“As a simpler alternative, we also tested a value-reliability arbitration model that relies solely on the reliability of the value-based decision-making system. This reduced model is identical to the full reliability-based model, except in the computation of the arbitration weight:\n1\n𝑤 (𝑠) = \n𝑞,𝑡 1+𝑒𝑥𝑝(−𝜂 𝜒 (𝑠)+𝜂 )\n𝑞 𝑞,𝑡 0\nThis model thus has 9 free parameters.”\nThis new model has also been added to the public codebase available at\nhttps://github.com/sanghyunyi/Affordance2025\n-- The authors state that the RT behavioural effects are indistinguishable in early and late trials. However, the choice effects manifest differently at early trials (larger affordance bias) relative to late trials. It is not clear whether these two statements are at odds with each other.\nWe appreciate the reviewer’s perspective and fully acknowledge that, in many decision-making frameworks such as the drift diffusion model (DDM), choice and RT are closely coupled, with longer RTs generally reflecting slower evidence accumulation due to weaker or more ambiguous information, or a greater distance between the starting point of the diffusion process and the decision boundary.\nHowever, in the current paradigm, RT reflects a complex mixture of factors beyond decision uncertainty alone. These include trial progression, experimental condition, action type, reward status of the chosen action, and affordance compatibility. To disentangle these influences, we conducted mixed-effects regression analyses controlling for these variables, and found that the RT difference between affordance-compatible and incompatible actions remains robust across both early and late trials (see Supplementary Table 1).\nImportantly, the persistence of this RT difference across learning suggests that it is not simply a function of changing decision certainty or learning-related dynamics. Instead, we interpret this RT effect as a motoric or perceptual facilitation consistent with stimulus–response compatibility, which exerts a stable influence irrespective of evolving reward contingencies. This is supported by the lack of significant change in the RT difference over time.\nIn contrast, the choice effect, reflected in a selection bias toward affordance-compatible actions, is dynamic and diminishes as participants learn which actions are more rewarding. This divergence between RT and choice effects suggests that the two measures are driven by distinct sources: RTs are shaped by stable sensorimotor mappings (i.e., affordance-based motor priming), while choices reflect the evolving arbitration between value-based and affordance-based decision systems.\nThus, while DDM frameworks predict a coupling between RT and choice under a unified evidence accumulation process, our findings suggest that, in this context, RT can be influenced by automatic motor biases that are largely independent of value-based decision-making.\nTherefore, the apparent dissociation between RT and choice effects in our task does not reflect a contradiction, but instead highlights the multifaceted nature of affordance influences on behavior, some of which are dynamic and decision-related, and others that are stable and rooted in sensorimotor priming.\nTo make these points clear, we added the following sentences to the discussion.\n“Our findings also offer an important qualification to decision-making frameworks that assume a tight coupling between RT and choice, such as drift diffusion models. While these models predict that longer RTs reflect slower or more uncertain decisions along a unified evidence accumulation process, our results reveal a partial dissociation: the RT difference between affordance-compatible and incompatible actions remained robust across learning, whereas the choice bias diminished over time. This suggests that RTs in our paradigm are shaped by stable sensorimotor facilitation effects (e.g., stimulus–response compatibility), while choices reflect the dynamic arbitration between value-based and affordance-based systems. In this way, affordance exerts both automatic and adaptive influences on behavior, operating at distinct levels of the decision hierarchy.”\n-- At the end of section 2.3 the authors acknowledge that their experimental paradigm does not allow them to delineate the temporal dynamics of processing value information and action affordances. I agree with their statements. Across these lines, I find that the slow vs. fast RT analyses is not informative. This analysis could be informative if we were sure that all slow responses are attributed to a higher decision boundary (with reference to an evidence accumulation framework). However, slow responses can also be attributed to other factors (low arousal levels or drift rate, longer non-decision time etc) which limits the inferential potential of a median split analysis in this paradigm.\nWe appreciate the reviewer’s thoughtful observation and agree that our current paradigm is limited in its ability to precisely delineate the temporal dynamics of affordance and value-based processing. As correctly pointed out, interpreting slow versus fast RTs within an evidence accumulation framework would require strong assumptions about the sources of RT variability, such as drift rate, boundary separation, or non-decision time, that our task was not specifically designed to isolate or estimate. Therefore, we acknowledge that attributing slower RTs solely to affordance-stimulus compatibility is overly speculative, as alternative explanations like attentional fluctuations, or motor preparation differences cannot be ruled out.\nConsidering this, we recognize that the RT analysis offers limited inferential power and should be interpreted with caution. In the revised manuscript, we added the following paragraph to clarify that this analysis is exploratory in nature and does not provide conclusive insight into decision dynamics. We have also noted the importance of future work employing paradigms explicitly tailored to capture temporal aspects of decision-making, such as time-pressure manipulations or modeling approaches grounded in sequential sampling theory. We thank the reviewer for highlighting this important limitation and helping us more accurately frame our interpretations.\n“Building on this, our findings highlight the importance of separating automatic motor influences from deliberative decision processes when interpreting RT patterns in value-based tasks. While RT is often used as a proxy for decision difficulty or uncertainty, our data suggest that this assumption may not hold in settings where stimulus–response compatibility introduces a persistent, low-level facilitation effect. To more clearly dissociate these processes, future work will benefit from paradigms that are explicitly designed to capture decision dynamics, such as those using time-pressure manipulations, sequential sampling models, or temporally resolved neural recordings. These approaches could help clarify how affordance-related signals interact with value-based evidence accumulation over time, and whether they modulate the decision process directly or operate through parallel, independent channels.”\nThank you for your positive feedback. We appreciate your careful review and are glad that our revisions have addressed all of your concerns."}, {"round": 3, "reviewer_comments": {"reviewer_4": "The authors engaged in-depth with my comments and addressed all my concerns. I have no further concerns."}, "author_response": ""}]

Decision letter