Author response: Humans use forward thinking to exploit social controllability
2021-05-24

Article Figures and data Abstract Introduction Results Discussion Materials and methods Appendix 1 Data availability References Decision letter Author response Article and author information Metrics Abstract The controllability of our social environment has a profound impact on our behavior and mental health. Nevertheless, neurocomputational mechanisms underlying social controllability remain elusive. Here, 48 participants performed a task where their current choices either did (Controllable), or did not (Uncontrollable), influence partners’ future proposals. Computational modeling revealed that people engaged a mental model of forward thinking (FT; i.e., calculating the downstream effects of current actions) to estimate social controllability in both Controllable and Uncontrollable conditions. A large-scale online replication study (n=1342) supported this finding. Using functional magnetic resonance imaging (n=48), we further demonstrated that the ventromedial prefrontal cortex (vmPFC) computed the projected total values of current actions during forward planning, supporting the neural realization of the forward-thinking model. These findings demonstrate that humans use vmPFC-dependent FT to estimate and exploit social controllability, expanding the role of this neurocomputational mechanism beyond spatial and cognitive contexts. Introduction Humans do not always have influence over the environments which they occupy. A lack of controllability has a profound impact on mental health, as has been demonstrated by decades of research on uncontrollable stress, pain, and learned helplessness (Maier and Seligman, 1976; Maier and Watkins, 2005; Overmier, 1968; Weiss, 1968). Conversely, high levels of controllability have been associated with better mental health outcomes such as higher subjective well-being (Lachman and Weaver, 1998) and less negative affect (Maier and Seligman, 2016; Southwick and Southwick, 2018). For humans, one of the most important types of controllability we need to track concerns our social environment. Doing this could be one of the roles of the various neural systems whose involvement in social cognition is supported by mounting evidence (Atzil et al., 2018; Dunbar and Shultz, 2007). Nevertheless, despite the importance, the neurocomputational mechanisms underlying social controllability have not been systematically investigated. Based on previous work demonstrating the computational mechanisms of controllability in non-social environments, here we hypothesize that people use mental models to exploit social controllability, for instance via forward simulation. In non-social contexts, it has been proposed that controllability quantifies the extent to which the acquisition of outcomes, and particularly desired outcomes, can be influenced by the choice of actions (Huys and Dayan, 2009; Dorfman and Gershman, 2019; Ligneul, 2021). In these non-social settings, agents need to learn the association between actions and state (event) transitions and potential outcomes in order to simulate future possibilities (Pezzulo et al., 2013; Szpunar et al., 2014) and make decisions (Daw et al., 2011; Dolan and Dayan, 2013; Doll et al., 2015; Gläscher et al., 2010). It has also been hypothesized that both under- and over-estimation of controllability could be detrimental to behavior (Huys and Dayan, 2009) depending on the complexity of the environment. Yet, it remains unknown whether this is true for social controllability. Studies on strategic decision-making (Camerer, 2011) have provided initial insight into the possible mechanisms underlying social controllability and influence. For example, Hampton et al., 2008 showed that people can learn the influence of their own actions on others during an iterative inspection game; and that the medial prefrontal cortex (mPFC) tracked expected reward given the degree of expected influence (Hampton et al., 2008). In other types of strategic games such as bargaining, it has been suggested that individuals differ drastically in their ability to manage their social images and exert influence on others, a behavioral phenomenon subserved by underlying neural differences in prefrontal regions (Bhatt et al., 2010). Furthermore, through the application of an interactive partially observable Markov decision process model, Hula et al., 2015 found that humans are able to use forward planning and mentally simulate future interactions in an iterative trust game (Hula et al., 2015). All of these studies suggest that learning the structure of the social environment is crucial for exerting influence, yet none have systematically examined the computational underpinnings of social controllability in a group setting where an agent plays with multiple other players that constitute a more social-like environment. Neurally, along with recent findings about its role in providing a representational substrate for cognitive tasks (Behrens et al., 2018; Niv, 2019; Schuck et al., 2016), the ventromedial prefrontal cortex (vmPFC) has been shown to signal expected values across a wide range of settings (Boorman et al., 2009; Kable and Glimcher, 2007; FitzGerald et al., 2009; Behrens et al., 2008; Bartra et al., 2013; Venkatraman et al., 2009). The majority of studies have focused on the role of the vmPFC in encoding the subjective values of non-social choices (Boorman et al., 2009; FitzGerald et al., 2009; Kable and Glimcher, 2007; Venkatraman et al., 2009). Nevertheless, accumulating evidence also pinpoints to a central role of the vmPFC in computing the value of social choices (Behrens et al., 2008; Hampton et al., 2008; Hiser and Koenigs, 2018), such as expected values computed based on learned influence (Hampton et al., 2008). A recent meta-analysis suggests that both social and non-social subjective values reliably activate the vmPFC (Bartra et al., 2013). Thus, we expect that the vmPFC will also play an important role in social controllability where the value of future events should be simulated and computed. In the current study, we hypothesize that humans exploit social controllability by implementing forward thinking (FT) and mentally simulating future interactions. In particular, we consider the long-lasting effect that one’s current interaction with one other person can have on future interactions with many others who constitute the social environment, for instance by developing a reputation. We predict that social agents will use forward planning to take into account not only decision variables related to the present interaction with a current partner, but also those related to future interactions with other partners from the same milieu. Finally, we hypothesize that the choice values integrating the planned paths would be signaled in the vmPFC. We used computational modeling and functional magnetic resonance imaging (fMRI; n=48), in the context of a social exchange paradigm (see Figure 1 and Materials and methods), to test the hypothesis that FT serves as a mechanism for social controllability. Furthermore, we replicated our computational findings in a large-scale online study involving more geographically diverse participants (n=1342). Both in-person and online participants completed an economic exchange task where they did (Controllable) or did not (Uncontrollable) influence their partners’ proposals of monetary offers in the future (see Figure 1a and b, and Materials and methods for details). Participants were told that they were playing members coming from two different teams, one each for the two controllability conditions (in a counterbalanced order across subjects); in fact, they played with a computer algorithm in both cases. Supplementary file 2 provides the task instruction provided to participants. To directly compare the impact of social versus non-social contexts on individuals’ decision strategies, we further administered a matched controllability experiment where participants were explicitly told that they were playing against a computer algorithm (Figure 2—figure supplement 1 and Supplementary file 1a). Figure 1 with 1 supplement see all Download asset Open asset Experimental paradigm. (a) Participants played a social exchange task based on the ultimatum game. There were two blocks: one ‘Controllable’ condition and one ‘Uncontrollable’ condition. Order of the conditions was counterbalanced across participants. Each block had 40 (fMRI sample) or 30 (online sample) trials. In each trial, participants needed to decide whether to accept or reject the split of 20 from their partners and asked to decide whether to accept or reject the offer. Unbeknownst to participants, the actual offer was randomly drawn from a normal distribution (rounded and restricted to be between 8 (inclusive) for the fMRI sample and between 9 (inclusive) for the online sample; the first offer was always 1 and 0, 2 (probability of 1⁄3 each, subject to the constraints) if the participant rejected the present offer, and decreased the next offers by 1, or 5. At the end of the task, after all the trials were completed, we asked participants to rate how much control they believed they had over their partners’ offers in each condition using a 0–100 scale to measure their perceived action-offer contingency (‘self-reported/perceived controllability’ hereafter). In the fMRI study, on 60% of the trials, participants were also asked about their emotional state (How do you feel?) on a scale of 0 (unhappy) to 100 (happy) after they made a choice (i.e., 24 ratings per condition; see Figure 1—figure supplement 1). Note that participants were not instructed about the statistics of the task environment nor the nature of the condition they were playing, although the instruction about the existence of two separate teams was provided to encourage participants to learn contingent rules and norms within each condition (Supplementary file 2). If participants were able to detect social controllability correctly within each condition, they would show strategic decisions that exert appropriate levels of control over others’ subsequent choices. Results Participants distinguished between controllable and uncontrollable environments We first examined whether participants’ choices were sensitive to the difference in controllability between the two social environments, noting that there was no explicit instruction about this difference. Our primary measures here were the offer sizes participants received in each condition, their rejection behavior, and their self-reported controllability. If individuals learned the action-offer contingency of the controllable environment, we should observe that (1) offers received under the Controllable condition would be pushed up to a higher level than those under the Uncontrollable condition; (2) people would need to reject more offers to obtain larger future offers under the Controllable than the Uncontrollable condition; and (3) people would report higher self-reported controllability for the Controllable than for the Uncontrollable condition. First, we found that despite the same starting offer of 1–3): meanC=77%, meanU=87%, t(22)=–1.35, p=0.19; middle ( 7–9): meanC=28%, meanU=8%, t(72.50)=4.00, p<0.001). Each offer bin for the Controllable in (b2) represents 23, 48, and 41 participants who were proposed the corresponding offers at least once, whereas each bin for the Uncontrollable represents all 48 participants. The t-test for each bin was conducted for those who had the corresponding offers for both conditions. (c) The self-reported controllability ratings were higher for the Controllable than Uncontrollable condition (meanC=65.9, meanU=43.7, t(74.55)=4.10, p<0.001; eight participants were excluded due to missing data). (d) Response times were longer for the Controllable than the Uncontrollable condition (meanC=1.75±0.38, meanU=1.53±0.38; paired t-test t(47)=4.34, p<0.001), suggesting that participants were likely to engage more deliberation during decision-making in the Controllable condition. A paired t-test was used for the rejection rates for low and middle offers and the self-reported controllability ratings. The t-statistics for the mean offer size, overall rejection rate, rejection rate for high offers, and self-reported controllability are from two-sample t-tests assuming unequal variance using Satterthwaite’s approximation according to the results of the F-tests for equal variance. Error bars and shades represent SEM; ***p<0.001; n.s. indicates not significant. For (a2, b1, c, d), each line represents a participant and each bold line represents the mean. Next, we examined the rejection patterns from the two conditions. On average, rejection rates in the two conditions were comparable (meanC=50.8%, meanU=49.1%, t(67.87)=0.43, p=0.67; Figure 2b1). By separating the trials each individual experienced into three levels of offer sizes (low: 4–6, and high: 4–9) offers in the Controllable condition, while they showed comparable rejection rates for the low offers ( 1–3): meanC=77%, meanU=87%, t(22)=–1.35, p=0.19; middle ( 7–9): meanC=28%, meanU=8%, t(72.50)=4.00, p<0.001; Figure 2b2; see Figure 2—figure supplement 2 for rejection rates by each offer size). These results suggest that participants behaved in a strategic way to utilize their influence over the partners. One possible confound is that individuals may have experienced different affective states in the two conditions and changed their choice behaviors. However, this seemed unlikely because there was no significant difference in emotional rating between the Controllable and the Uncontrollable conditions (Figure 1—figure supplement 1). As additional evidence that participants distinguished the controllability between conditions, we compared self-reported beliefs about controllability between the two conditions. Indeed, participants reported higher self-reported controllability for the Controllable than the Uncontrollable condition (meanC=65.9, meanU=43.7, t(74.55)=4.10, p<0.001; Figure 2c). Besides the clear indication of individuals’ recognition of the difference in controllability between conditions, the mean level of self-reported controllability for the Uncontrollable condition was 43.7%, which was still substantially higher than their actual level of controllability on future offers made by the partners (0%). This result might suggest that participants could develop an illusory sense of control when they had no actual influence over their partners’ offers. In addition, we examined response times as an exploratory analysis and found that participants took longer time to make their decisions in the Controllable condition than the Uncontrollable condition. These results again suggest that participants differentiated the controllability between conditions (meanC=1.75±0.38, meanU=1.53±0.38; paired t-test t(47)=4.34, p<0.001; Figure 2d). Taken together, these findings demonstrate that participants were able to exploit and perceive their influence in a social environment when they had influence, although they have developed an illusion of control, at least to some degree, even when controllability did not exist. We delineate the computational mechanisms underlying these behaviors in the next sections. Participants used forward thinking to exploit social controllability We constructed computational models of participants’ choices and sought to investigate what cognitive processes might underlie people’s ability to exploit social controllability. Previous studies on value-based decision-making have shown that people can use future-oriented thinking and mentally simulate future scenarios when their current actions have an impact on the future (Daw et al., 2011; Gläscher et al., 2010; Lee et al., 2014; Moran et al., 2019). Relying on this framework, we hypothesized that individuals use FT to estimate the impact of their behavior on future social interactions. To test this hypothesis, we constructed a set of FT models which assume that an agent computes the values of action (here, accepting or rejecting) by summing up the current value (CV) and the future value (FV) based on her estimation of the amount of controllability she has over the social interactions. These models also incorporate social norm adaptation (Gu et al., 2015) to characterize how individuals’ aversion thresholds to unfairness is adjusted by observing the counterpart teams’ proposals (Fehr and Schmidt, 1999) (see Materials and methods for details). The key individual-level parameter-of-interest in this model is the ‘expected influence,’ δ, representing the amount of the offer changes that participants thought they would induce by rejecting the current offer (see Materials and methods). We constrained the range of δ using a sigmoid function to − 2, in order to match with the range participants observed in the Controllable condition ( 2 to $0). Moreover, we considered the number of steps one calculates into the future (i.e., planning horizon; Figure 3a). We compared models that considered from one to four steps further in the future in addition to standalone social learning (‘0-step;’ also see Figure 3—figure supplement 5 for comparison with a model-free [MF] learning). The 0-step model only considers the utility at the current state. All other components including the utility function of the immediate rewards, and the variable initial norm and norm learning incorporated in the utility function are shared across all the candidate models. In model fitting, we excluded the first 5 out of 40 trials for the fMRI sample (30 trials for the online sample) to exclude initial exploratory behaviors and to focus on stable estimation of controllability. We also excluded the last five trials because subjects might adopt a different strategy toward the end of the interaction (e.g., ‘cashing out’ instead of trying to raise the offers higher). Figure 3 with 5 supplements see all Download asset Open asset Computational modeling of social controllability. (a) The figure depicts how individuals’ simulated value of the offers evolves contingent upon the choices along the future steps under the Controllable condition. Future simulation was assumed to be deterministic (only one path is simulated instead of all paths being visited in a probabilistic manner). The solid and thicker arrows represent an example of a simulated path. To examine how many steps along the temporal horizon participants might simulate to exert control, we tested the candidate models considering from zero to four steps of the future horizon. (b) For both the Controllable and Uncontrollable conditions, the forward thinking (FT) models better explained participants’ behavior than the 0-step model. The 2-step FT model was selected for further analyses, because the improvement in the DIC score (Draper’s Information Criteria; Draper, 1995) was marginal for the models including further simulations (paired t-test comparing 2-step FT model with (i) 0-step Controllable: t(47)=–4.45, p<0.0001, Uncontrollable: t(47)=–4.21, p<0.001; (ii) 1-step Controllable: t(47)=–4.41, p<0.0001, Uncontrollable: t(47)=–3.01, p<0.001; (iii) Controllable: Uncontrollable: Controllable: Uncontrollable: (c) The choices by the 2-step FT model were matched with individuals’ actual choices with an rate of for the Controllable and for the Each bold line represents mean (d) The levels of expected influence drawn from the 2-step FT model were higher for the Controllable than the Uncontrollable Each line represents a participant and each bold line represents the mean. The expected influence was between the Controllable and the Uncontrollable conditions The self-reported controllability was not between the conditions Under the Controllable condition, expected influence with mean offers Each represents a Error bars and shades represent SEM; ***p<0.001; The results showed that for both conditions all FT models better explained participants’ choices than the standalone norm learning model FT (Gu et al., as by Information 1995) across individuals (paired t-test comparing 2-step FT model with 0-step model Controllable: t(47)=–4.45, Uncontrollable: t(47)=–4.21, p<0.001; Figure In addition, not all were in analysis using the 0-step model (e.g., to norm Controllable: Uncontrollable: whereas all the from the FT models were (see Figure 3—figure supplement for of the 2-step These results suggest that participants engaged in future-oriented thinking and how their current choice might affect subsequent social of the actual level of controllability of the environment. The FT models with longer planning horizon to show DIC (i.e., better model but the improvement marginal after two steps (paired t-test comparing 2-step FT model with (i) 1-step Controllable: t(47)=–4.41, p<0.0001, Uncontrollable: t(47)=–3.01, p<0.001; (ii) Controllable: Uncontrollable: (iii) Controllable: Uncontrollable: Figure The 2-step FT model participants’ choices with an rate of for the Controllable and for the Uncontrollable condition (Figure which was higher than the 1-step model for the Controllable condition p<0.001), Uncontrollable and comparable with the models with longer planning horizon Controllable Uncontrollable Controllable Uncontrollable the of our expected influence δ, was better and in general for the 2-step model Uncontrollable compared to the other models Controllable Uncontrollable Controllable Uncontrollable Controllable Uncontrollable We used from the 2-step FT model for subsequent (see 1 for a of from this 1 from the 2-step forward thinking (FT) model. to norm It might that participants engaged a 2-step FT model to estimate the future impact of their current choices under the Uncontrollable condition. However, as in most where the controllability of our social interactions is unknown or participants were not explicitly told about the of the environment. Indeed, they that they could exert at least some control (Figure 2c). Thus, we that individuals to make strategic decisions with that they have some controllability over the social environment of the actual controllability. that participants were in offers in the Controllable condition (Figure we that the expected influence δ would differ between the two conditions. Indeed, we found that the expected influence drawn from the 2-step FT model were higher for the Controllable than for the Uncontrollable condition Figure indicating that participants simulated levels of controllability when environments were in controllable than when they were despite the systematic difference between the two conditions, the expected influence was still between the conditions Figure suggesting a of the This is in with the self-reported about controllability, which was not between the conditions Figure between expected influence and self-reported controllability is in Figure supplement Furthermore, we observed a association between expected influence and task during the Controllable condition Figure This result suggests that those who simulated a level of controllability were able to raise the offers indicating the effect of with a non-social controllability task To investigate whether our results are to the social we a non-social of the task in which participants played the same game with the instruction of with instead of with virtual Using the same computational we found that not only participants choice patterns (Figure 2—figure supplement but also the 2-step FT model was still in the non-social task (Figure 2—figure supplement and that was still higher for the Controllable than the Uncontrollable condition (Figure 2—figure supplement a of subjective data revealed two differences in the non-social task compared to the social First, participants’ subjective report of controllability did not between conditions in the non-social task (Figure 2—figure supplement which suggests