Damage to Ventromedial Prefrontal Cortex
There has been a long history of analyzing people with brain lesions in cognitive neuroscience (Damasio and Damasio, 1989 ) . We focused on a particular study conducted by Bechara et al. ( 1994 ) for the purpose of this article and predicted that reducing the description length of RR-RL 2 should correspond to the behavior of brain-lesioned patients.

Experimental Design: To probe decision-making in clinical populations, Bechara et al. ( 1994 ) introduced an experimental paradigm called the Iowa Gambling Task (IGT). The IGT involves 100 choices in a single four-armed bandit problem. Two of the arms are high-risk arms, while the other two are low-risk arms. High-risk arms result in a constant positive reward of 100 but have a chance to yield a noisy penalty with an expected value of 125 . Low-risk arms result in a constant positive reward of 50 but have a chance to yield a noisy penalty with an expected value of 25 . A complete list of trials is printed in Table D1 . Bechara et al. ( 1994 ) used the IGT to compare the decision-making of healthy participants to that of participants with ventromedial prefrontal cortex (vmPFC) damage.

Analysis: The focus of our analysis was the proportion of selected low- and high-risk arms across the entire experiment. High-risk arms cause an average loss of 25 points per trial, while low-risk arms provide an average payoff of 25 points per trial. We should therefore expect an agent to select the superior low-risk arms with higher frequency. Healthy participants are indeed able to learn about the structure of the task and will after a while start to sample the superior low-risk arms. Participants with vmPFC damage, however, continue to sample to the inferior high-risk as illustrated in Figure 3 (a). This pattern is striking because performance in these subjects remains worse than chance regardless of how often they interact with the task.

Results: RR-RL 2 requires a distribution over bandit problems for meta-learning, but participants in the IGT only encountered a single bandit task. Therefore, we cannot directly use the task of the original study for meta-learning as we have done in the previous example. We instead constructed a distribution over bandit problems that maintains the key characteristics of the IGT:

The positive reward component was independently sampled for each arm from a uniform distribution with a minimum value of 0 and a maximum value of 150 .

The mean across all trials of the negative reward component was also sampled from a uniform distribution with a minimum value of 0 and a maximum value of 150 .

The negative reward component had an occurrence probability sampled randomly from a uniform distribution with a minimum value of 0.05 and a maximum value of 0.95 .

We furthermore added additive noise sampled from a zero-mean normal distribution with a standard deviation of 10 to the negative reward component in each time-step.

We trained RR-RL 2 with a targeted description length of \{100,200,\ldots,10000\} nats on the previously described distribution. When tested on the IGT, we find that RR-RL 2 replicates the pattern reported by Bechara et al. ( 1994 ) . Models with a high description length successfully solve the task by selecting low-risk arms in the majority of time-steps. If description length is however sufficiently reduced, RR-RL 2 predominately samples high-risk arms. We illustrate this behavior for two example models in Figure 3 (b). Figure D2 provides a more detailed picture of how description length mediates choice behavior.

In summary, our analysis sheds light on why brain-lesioned patients display below-chance performance in the IGT. Intuitively, any resource-limited agent must primarily devote its computational resources to things that are easy to estimate. In the IGT, the deterministic positive reward component is easier to estimate than the noisy negative component. An agent with significantly restricted resources will thus focus on the positive component while ignoring the negative. In turn, the agent will assign higher estimated payoffs to the inferior high-risk arms and therefore select them more frequently. We found that RR-RL 2 implements this behavior and that reducing its description length captured participants with lesioned vmPFC.