Detecting, avoiding &amp; eliminating confounds in MVPA/decoding studies
Creators: name: Goergen, Kai
name: Hebart, Martin N
name: Allefeld, Carsten
name: John-Dylan Haynes
Publisher: name: F1000Research
Year: 2016

description: Classical design principles (e.g. randomization) and control analyses (e.g. on error, reaction time, age) are routinely applied in neuroimaging studies. We investigated the validity of this practice for studies using new analysis methods such as multi-variate pattern analysis (MVPA) and found that this can lead to the exact opposite of what it should achieve: Classical design principles can <i>induce</i> confounds instead of controlling them, and standard control analyses can misleadingly increase the certainty that confounds have been successfully controlled – even if they were not.<br> <br>Investigating this problem we found a simple remedy that allows to detect, avoid, and eliminate a big class of possible confounds, which we provide at the end of this text.<br> <br>We exemplify the problem by looking at cross-validated decoding, one of the most commonly applied MVPA procedures in neuroimaging. We demonstrate the effects and provide illustrative examples from both simulated and empirical data, among them an explanation on how applying standard experimental design principles can systematically cause so-far mysterious <i>below-chance</i> decoding accuracies.<br> <br>In short, we argue that:Standard classical experimental design theory does not ensure control of potential confounds in experiments that apply new (mostly cross-validated) multivariate analysis methods such as decoding.Instead, it can cause systematic positive or negative biases (such as significant below-chance decoding accuracies), potentially yielding false positive or – equally bad – false negative results.The common practice of applying different statistics for control analyses and the analyses of neuroimaging data, e.g. t- or F-tests for control data and cross-validated decoding for neuroimaging, is not valid, because different statistical methods quantify, and are thus sensitive to, different structural features.Consequently, the control analyses might overlook the influence that a hidden confounding factor (e.g. difficulty or age) exerts on both imaging and control data, even if the confound is present in both datasets in exactly the same way! Conversely, the control analysis might point out effects as confounds to which only the control analysis is sensitive to, while MVPA is not. <br>We propose a simple solution to these problems: <i>Applying the same type of analysis</i> to (i) imaging data, (ii) design variables, and (iii) control data. This allows to detect potentially confounding features in designs and to conduct appropriate control analyses. For example, if cross-validated decoding is used to distinguish trials of different experimental conditions using imaging data, then cross-validated decoding should also be carried out on the reaction times from these trials. Because the same analysis method is sensitive to the same features of the data, this procedure is able to detect a wide range of possible confounding effects that so far had been tested individually (or have not been known at all), such as structural problems, sequence effects, problems induced by different number of trials and many more. We demonstrate how this can be done, supply example code for easy implementation, and provide information on how not only detect, but also avoid and eliminate confounds in actual experiments.