Peer review of: Unconscious reinforcement learning of hidden brain states supported by confidence (2020)

Reviewers' comments: 

Reviewer #1 (Remarks to the Author): 

This manuscript reports the results of an fMRI study in which a real-time neurofeedback loop is used 
to define states that are subsequently used to guide the reinforcement of actions. Essentially the state 
that a decoder retrieves from either VC or PFC about the direction of a random dot stimulus defines 
which action leads to reward with high probability (and in session 1 and 2 the decoded state is not 
explicitly made known to the participant). Participants learn to perform the task across two sessions, 
with performance better than chance by session 2, albeit not significantly different from session 1. The 
authors take this data and then use it to test a hypothesis about the role of metacognition in 
facilitating learning about the use of “latent and “high dimensional brain states”. The authors recorded 
confidence ratings – and found that confidence ratings increased over the course of the experiment, 
which they also found was correlated with performance, while correct discrimination accuracy per se 
was correlated more weakly with performance. Levaraging neuroimaging correlations with RPE across 
the three sessions, the authors conclude that metacognition boosts RL performance via a frontostriatal 
mechanism by which the brain untangles the course of dimensionality. 

While the finding that people are able to learn to perform a task such as this is an interesting result 
and a solid technical achievement, the main claims of the paper about the boosting relationship of 
meta-cognition on learning of RL task representations, is not well supported by the experimental 
design, or the findings. Essentially what the paper does report is a correlation between metacognitive 
judgement and learning capacity both across subjects, and within subjects across sessions, but it is 
unclear in what direction does that correlation go. 

1/ The basic issue with this whole manuscript is that to address a question about the relationship 
between meta-cognitive judgments and reinforcement learning over latent states, there are many 
experimental designs that could be selected for doing so, but using a neural decoding approach is not 
one of them. A much simpler and more powerful design would involve giving people tasks with hidden 
latent states without any neural decoding loop, and then tracking the degree to which they are able to 
solve the task successfully by making inferences over the hidden states. Experimental manipulations 
could be used to increase or decrease confidence and increase or decrease discrimination accuracy – 
indeed with a clever design they could be independently manipulated. The relationship between those 
inferences and confidence judgements could be tracked and analyzed. There is absolutely no need to 
have the neurofeedback component, and the design is problematical because there is no explicit 
experimental manipulation of any of the relevant decision variables. As a result, we can’t conclude 
whether confidence has any causal role in driving performance of the latent state inference, or not. 
Though these two variables are correlated across the experiment we don’t know what direction is the 
causality – only that the two are related (performance on the task and confidence in that 
performance). 

2/ The argument that participants are performing a search through a massively high dimensional 
space to solve the task does not seem credible. First of all, the same subjects completed a motion dot 
discrimination task a week before that was used to train the classifier – in which the discrimination 
was left vs right movement. In the task itself the participants are asked to indicate whether 
movement was left or right. So, the participants are clearly clued into the key states of the task that 
are likely to be relevant for learning: left vs right moving dots. They are not faced with an 
unconstrained multidimensional problem. Instead, it should be obvious (whether consciously or not) 
given the way the task is structured that they need to focus on movement of the dots being either 
leftward or rightward. 

3/ None of the computational modeling results help to make the case stronger about a role for 
metacognition in boosting learning about latent state spaces. The computational model analyses do 
help to validate the RL model by showing that a model which can utilize the relevant latent spaces will 
account better for participants behavior than one which does not, as participants do learn the task by 
session 2. The modeling also provides further support of a relationship between confidence 
judgements and learning accuracy, as indexed by a model-derived variable of unsigned prediction 
errors. So overall, the modeling is a nice validation of the role of RL in describing performance on this 
task, and on the relationship between conscious ratings of confidence and performance. However, 
these modeling results don’t make the case any stronger about whether or how metacognition is 
causally relevant for learning. 

4/ The neuroimaging results unfortunately are not informative and don’t provide specific insight into 
the claims made by the authors. The authors descriptively report differences 
between session 1 and 2 and session 3 in correlations with RPE signals, concluding that there is 
initially a “global search” involving multiple brain regions that converges to “reward” and “belief” 
processing hubs in PFC and basal ganglia by session 3. But session 3 is a different task to session 1 
and 2 (in session 3 the latent states are revealed to the participant) so it cannot be compared in kind 
to task 1 and 2, and in any event there is no control for changes in brain activity that would simply 
happen as a function of time on task / habituation etc. Furthermore, we have no idea whether 
changes in resting state reported before vs after the task are anything to do with task performance, as 
opposed to reflecting changes that just happen after being in the scanner for a certain period of time. 

5/ The sample size is very low (n=18) – this is especially the case because the decoder was trained on 
two different brain areas VC vs PFC in half of the subjects. The authors report no difference between 
the subjects in terms of where decoding was performed but with such a small sample size, this is not 
an interpretable result. The sample size does affect the strength of some of the claims – for instance 
the lack of a significant correlation between task performance and discrimination accuracy could be 
entirely due to the small sample size. 

Reviewer #2 (Remarks to the Author): 

I have reviewed this manuscript previously. I thought that it was novel and interesting then. It has 
now been further revised and improved. I have no further comments. I have included my previous 
comments below. 

------ 

This is a rather beautiful study that deserves to be seen by a wide audience. The findings will be of 
broad interest to visual and cognitive neuroscientists and to researchers building neuroimaging 
analysis tools and brain-inspired neural networks. The manuscript presents evidence for a very 
interesting conceptual advance that although intuitive is novel – that metacognition can be employed 
to distinguish one’s own brain states. The argument is based on both careful behavioural analysis and 
careful interrogation of neural activity measurements. The methods and analyses employed are 
advanced and compelling. I have noted a number of points below but they are all minor in nature. 

1 P6 The difference between the state free and the state dependent learning models initially seemed 
to me to have been explained clearly in lines 147-153 and in the legend for figure 3e on lines 187-
195. However, it did not understand why the state-free model predicts learning that “would just follow 

any average bias of the multivoxel patterns to one state - that is, the prevalence of one latent state 
over the other, if any (figure 3A-B)”? Nevertheless, this seems to be an important idea that underlies 
several of the analyses illustrated in figure 3. Is it because if learning is dependent on the action the 
subject makes given the brain state, Q(s,a), rather than dependent on just the state, Q(s), or just the 
action, Q(a), then neither one state or the other is likely to predominate in the brain because it is not 
directly associated with reward? Instead what is associated with reward is the action given the brain 
state. Why, however, should there be any drift in the underlying brain state towards one value in the 
state free Q(a) model? 

2 Figure 3. What do the orange and blue dot colours indicate? At first I wondered if these indicate 
participants from the PFC or the V1 recording experiments. The legend does not seem to suggest this. 
It does mention “Coloured dots (light/dark green) represent single subjects” but I could not easily see 
any green dots. 

3 Line 298 “Resting state functional connectivity is believed to be modified by recent co-activation of 
two brain areas”. Direct evidence that this is the case is provided by Johnen et al., (eLife, 2015) who 
directly activated pairs of human brain areas and then recorded connectivity with fMRI. 

4 Lines 308-312 These lines refer to the changes in connectivity with voxels in the basal ganglia that 
are illustrated in figure 6b. They emphasize connectivity with “putamen, thalamus and cerebellum 
subregions” but these changes are hard to discern on panel 6b. Maybe they could be labelled? Instead 
what is more striking is the increased connectivity with parts of prefrontal cortex and anterior 
temporal lobe and the decreases in connectivity with more posterior cortical regions (bottom right). 
However, this seems unmentioned in the text. 

5 Line 312 group-specific changes in functional connectivity are described as being shown in figure S6 
but I think that the relevant figure is S5. I think that the key analyses already performed are 
interesting and I understand why they have been seeded in the manner they have. Nevertheless, it 
might be possible to obtain stronger striatal connectivity group by seeding from a posterior striatal 
location in the posterior striatum which is more closely linked to representations of rewarding visual 
stimuli (eg work from Hikosaka’s laboratory such as Kunimatsu et al., J. Neuroscience, 2019; Amita et 
al., EJN, 2019). I do not, however, think that this is a particularly critical analysis. 

6 Figure 7c The target distributions are described as gold coloured in the legend but they do not look 
gold coloured on my pdf (more grey coloured). 

Reviewer #3 (Remarks to the Author): 

Cortese and colleagues investigated whether humans can be trained to make strategic use of latent 
representations in their own brains. To this aim, they scanned subjects performing a RL task, where 
the mapping between action and reward was determined by “decoded mental states” about random 
dot motion (the classifier being based on a previous scanning session). They further propose that 
stated confidence mediates dimensionality reduction across the otherwise vast search space of latent 
brain states. The topic is interesting, and the authors should be praised for developing and combining 
a lot of sophisticated analyses. However, I think the paper presents several major issues, that I am 
not sure can be addressed in a revision without a replication experiment. 

1. First, I really can’t see how the results and the analysis support the claim that “meta-cognition 

facilitates the search in latent task representations”. The authors’ interpretation strongly relies on a 
causal link that goes from confidence reporting to latent space search, but as all their evidence is 
correlational, it is, at least, equally likely to suppose a parsimonious alternative model where stated 
confidence merely reflects (or reads out) a successful search. All claims about confidence “helping” 
searching in the latent space should be qualified as speculative, as it may be that successful search is 
actually “helping” confidence expression. 

2. Second, and more worrying, virtually most of the main results of the paper are significant at a very 
permissive threshold (uncorrected p~0.05 or p~0.01), which poses a serious multiple comparison 
issue, considering the considerable number of statistical tests performed. The same statistical 
concerns apply to imaging results which are reported at p < 0.005 uncorrected. This is even more 
problematic, as the study is clearly exploratory and relies on non-standard analytical procedures. Just 
to be clear, I am not suggesting that exploratory and innovative research should not be performed (or 
highlighted by important journals), but I feel that the current amount statistical evidence is not 
convincing enough. Fortunately, there is a straightforward solution for this: run the same experiment 
in a replication sample and see whether the results hold. 

3. Third, also concerning the statistical analyses, in several instances I noted a “difference in 
significance does not imply significant difference” issue (for example p(opt action) significant in 
session 2, but not in session 1 as a proof of learning). Please, test directly the difference and refrain 
from interpreting difference in significance. In addition, as several crucial measures require test 
against “chance level”, full linear models, where the intercept can be taken as difference from chance, 
would be better. 

4. Please make clearer what “perceptual discrimination” in session 1 and 2 means. Since random dot 
motion was random and the ‘state’ was determined the decoded from neural activity, it is not 
immediately clear to see what is an accurate response here. In general the paper could benefit from 
clearer definitions of the dependent variables. 

5. The fact that there was a significant trend for a correlation between accuracy in the two tasks, is 
also problematic (especially because the difference between the two tasks is only significant at 
p<0.028…). Discrimination accuracy should be entered explicity as a control for p(opt action). I fear 
that if the authors regressed out the shared variance, the residual will show no sign of significant 
conditioning. This is also true for the analysis of accuracy as a function of the level of perceptual 
confidence, where the discrimination accuracy follows a very similar pattern. 

6. To link the learning effect to meta-cognition, the authors start by showing a correlation between 
meta-d’ and the sum of the rewards (to note, also marginal p<0.041...). Furthermore, it is very 
unclear why here the metric “sum of the rewards” has been chosen as a proxy for performance in the 
RL task. The analysis should be replaced with p(opt action), which has been the main dependent 
variable all along. Also it is unclear whether this correlation holds for both the PFC and the VC. In any 
case counting a subject twice does not make any sense. 

7. The authors regress RPE against brain activity (model-based fMRI), but see Lebretone et al. (Nature 
Human Behavior; 2019) for issues related to this technique. In particular the authors should specify 
whether the RPE regressors have been z-scored, across subjects and/or session, justify their approach 
and specify whether the results are affected by their analytical choices. 

Minors 
The authors equate reward contingencies to RL states. I see why this is the case in their design, but 

different states may have the same reward contingencies and vice versa. Please consider avoid this 
terminology confusion. 
Oddly, the result session starts with a statement about the interpretation of the results, before 
providing any relevant quantitative and statistical evidence. Please consider replacing this sentence 
with more factual information. 
“implicit metacognition” is mentioned in the abstract, but not well defined. The definition given in the 
abstract is not clear for a scientist not expert in meta-cognition. 
In the abstract there is mention of possible applications of these findings in artificial intelligence. This 
is a pretty wild speculation (the models proposed are very simple, computers do not have brain and 
the role of meta-cognition in AI is very much debated), so please remove it from the abstract. 
The term “brain states” should be clarified at its first occurence (in the instruction). Different 
neuroscientists will have very different interpretations of what a brain state is. 
The term “discrimination confidence” should be clarified at its first occurence (in the instruction). It is 
unclear for a scientist outside meta-cognition field.