# Commentary: Seeing the conflict: an attentional account of reasoning errors

Darren Frey, Bence Bago, Wim De Neys

► To cite this version:

Darren Frey, Bence Bago, Wim De Neys. Commentary: Seeing the conflict: an attentional account of reasoning errors. Frontiers in Psychology, 2017, 8, pp.1284. 10.3389/fpsyg.2017.01284. hal-04058557

HAL Id: hal-04058557

https://hal.science/hal-04058557

Submitted on 30 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_1.png" /></figure>

HAL
open science

# Commentary: Seeing the conflict: an attentional account of reasoning errors

Darren Frey, Bence Bago, Wim De Neys

► To cite this version:

Darren Frey, Bence Bago, Wim De Neys. Commentary: Seeing the conflict: an attentional account of reasoning errors. Frontiers in Psychology, 2017, 8, pp.1284. 10.3389/fpsyg.2017.01284. hal-04058557

HAL Id: hal-04058557

https://hal.science/hal-04058557v1

Submitted on 30 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

Check for
updates

# Commentary: Seeing the conflict: an attentional account of reasoning errors

Darren P. Frey<fnref n="1" />,²,³*, Bence Bago<fnref n="1" />,²,³ and Wim De Neys<fnref n="1" />,²,³

**Keywords:** intuition, reasoning, decision making, conflict detection, attention

## A commentary on

### Seeing the conflict: an attentional account of reasoning errors

by Mata, A., Ferreira, M. B., Voss, A., and Kollei, T. (2017). *Psychon. Bull. Rev.* 24, 1–7.
doi: 10.3758/s13423-017-1234-7

In a recent study, Mata et al. (2017) test a two-stage account of reasoning that differentiates between the initial interpretation and selection of information within a problem, and the subsequent operation on this information. Crucially, on this account, reasoning errors can result from mistakes at either stage. In previous work, the authors found indirect experimental evidence in support of this account (e.g., Mata et al., 2014), and their most recent experiments address it by analyzing reasoners' on-line attentional resources in situations of conflict. To do so, the authors record eye-tracking measurements as participants evaluate both classical reasoning tasks that induce conflict (variants of the bat-and-ball item from the Cognitive Reflection Test, Frederick, 2005), and comparably structured no-conflict items. Their two-stage account makes at least two strong predictions. All things being equal, correct responders should: (1) attend to conflict problems more than no-conflict items; (2) and they should attend to conflict items more than incorrect responders.

Mata et al. present two studies, the second of which is a replication of the first that avoided a potential confound present in Study 1. In that study, eye-tracking recordings ended when participants indicated that they were ready to respond, raising the concern that crucial aspects of the reasoning process might have been neglected (Ball et al., 2006). By and large, the results of their studies support the claim that correct responders attend more to conflict than no-conflict items, and they allocate more attentional resources to the critical components of the tricky conflict items. Using a technique that is fairly novel in this context (e.g., see Ball et al., 2006, for eye-tracking studies with a different reasoning task; see Ball, 2013 for a review), the authors have generated interesting insights into how it is that reasoners may be led to err, suggesting a largely overlooked attentional path to such mistakes.

However, the authors further interpret their results to indicate that incorrect responders are generally insensitive to conflicts. It is one thing to assert that correct responders do a better job at attending to conflicts than incorrect responders, but it is quite another to claim that incorrect responders are entirely insensitive to such conflicts. In line with Mata et al.'s findings, there is previous experimental literature that suggests that correct responders are more sensitive to conflicts than incorrect responders (e.g., Pennycook et al., 2015). Among incorrect responders, there are prominent individual differences. Within samples of biased responders there are subgroups of participants across various tasks who do, indeed, fail to detect reasoning conflicts. This group tends to range from 15 to 40% of incorrect responders (Mevel et al., 2015; Pennycook et al., 2015; Frey et al., 2017). Yet, the majority of even biased individuals detect conflict across a number of tasks that make use of different measures, suggesting that there is no good empirical reason to universally

**OPEN ACCESS**

**Edited by:**

Ulrich Hoffrage,
University of Lausanne, Switzerland

**Reviewed by:**

Edward J. N. Stupple,
University of Derby, United Kingdom
Gordon Pennycook,
Yale University, United States

*Correspondence:**

Darren P. Frey
darren.frey@gmail.com

**Specialty section:**

This article was submitted to
Cognition,
a section of the journal
Frontiers in Psychology

**Received:** 17 May 2017

**Accepted:** 13 July 2017

**Published:** 25 July 2017

**Citation:**

Frey DP, Bago B and De Neys W (2017) *Commentary: Seeing the conflict: an attentional account of reasoning errors*. *Front. Psychol.* 8:1284.
doi: 10.3389/fpsyg.2017.01284

<footnote n="1">¹Université Paris Descartes, Paris, France, ²University of Caen Normandy, Caen, France, ³UMR8240 Laboratoire de Psychologie du Développement et de l'Éducation de L'enfant, Paris, France</footnote>
<figure><img src="image_3.png" /></figure>

Check for
updates

# Commentary: Seeing the conflict: an attentional account of reasoning errors

Darren P. Frey¹,²,³*, Bence Bago¹,²,³ and Wim De Neys¹,²,³

¹Université Paris Descartes, Paris, France, ²University of Caen Normandy, Caen, France, ³UMR8240 Laboratoire de Psychologie du Développement et de l'Éducation de L'enfant, Paris, France

**Keywords:** intuition, reasoning, decision making, conflict detection, attention

## A commentary on

### Seeing the conflict: an attentional account of reasoning errors

by Mata, A., Ferreira, M. B., Voss, A., and Kollei, T. (2017). *Psychon. Bull. Rev.* 24, 1–7.
doi: 10.3758/s13423-017-1234-7

In a recent study, Mata et al. (2017) test a two-stage account of reasoning that differentiates between the initial interpretation and selection of information within a problem, and the subsequent operation on this information. Crucially, on this account, reasoning errors can result from mistakes at either stage. In previous work, the authors found indirect experimental evidence in support of this account (e.g., Mata et al., 2014), and their most recent experiments address it by analyzing reasoners' on-line attentional resources in situations of conflict. To do so, the authors record eye-tracking measurements as participants evaluate both classical reasoning tasks that induce conflict (variants of the bat-and-ball item from the Cognitive Reflection Test, Frederick, 2005), and comparably structured no-conflict items. Their two-stage account makes at least two strong predictions. All things being equal, correct responders should: (1) attend to conflict problems more than no-conflict items; (2) and they should attend to conflict items more than incorrect responders.

Mata et al. present two studies, the second of which is a replication of the first that avoided a potential confound present in Study 1. In that study, eye-tracking recordings ended when participants indicated that they were ready to respond, raising the concern that crucial aspects of the reasoning process might have been neglected (Ball et al., 2006). By and large, the results of their studies support the claim that correct responders attend more to conflict than no-conflict items, and they allocate more attentional resources to the critical components of the tricky conflict items. Using a technique that is fairly novel in this context (e.g., see Ball et al., 2006, for eye-tracking studies with a different reasoning task; see Ball, 2013 for a review), the authors have generated interesting insights into how it is that reasoners may be led to err, suggesting a largely overlooked attentional path to such mistakes.

However, the authors further interpret their results to indicate that incorrect responders are generally insensitive to conflicts. It is one thing to assert that correct responders do a better job at attending to conflicts than incorrect responders, but it is quite another to claim that incorrect responders are entirely insensitive to such conflicts. In line with Mata et al.'s findings, there is previous experimental literature that suggests that correct responders are more sensitive to conflicts than incorrect responders (e.g., Pennycook et al., 2015). Among incorrect responders, there are prominent individual differences. Within samples of biased responders there are subgroups of participants across various tasks who do, indeed, fail to detect reasoning conflicts. This group tends to range from 15 to 40% of incorrect responders (Mevel et al., 2015; Pennycook et al., 2015; Frey et al., 2017). Yet, the majority of even biased individuals detect conflict across a number of tasks that make use of different measures, suggesting that there is no good empirical reason to universally

**OPEN ACCESS**

**Edited by:**

Ulrich Hoffrage,
University of Lausanne, Switzerland

**Reviewed by:**

Edward J. N. Stupple,
University of Derby, United Kingdom
Gordon Pennycook,
Yale University, United States

*Correspondence:**

Darren P. Frey
darren.frey@gmail.com

**Specialty section:**

This article was submitted to
Cognition,
a section of the journal
Frontiers in Psychology

Received: 17 May 2017

Accepted: 13 July 2017

Published: 25 July 2017

**Citation:**

Frey DP, Bago B and De Neys W (2017) Commentary: Seeing the conflict: an attentional account of reasoning errors.
Front. Psychol. 8:1284.
doi: 10.3389/fpsyg.2017.01284
equate incorrect responding with insensitivity to conflict. A great deal of evidence suggests that incorrect responders are often at least minimally sensitive to conflicts across a wide range of tasks. Surveying this literature, De Neys (2012, 2014) cites evidence of conflict sensitivity on a plurality of diverse tasks (base rate neglect, conjunction fallacy, ratio bias, syllogistic reasoning). More importantly, using the very same task as Mata et al. (2017)—the bat-and-ball problem—different teams have found evidence that incorrect responders detect conflicts on such items (De Neys et al., 2013; Gangemi et al., 2015; Johnson et al., 2016). Since the latter findings rely on different measures (reaction times and confidence/error ratings) than those employed by Mata et al. (2017), one might argue that these contradictory results are a function of the distinct techniques the experiments used. Perhaps the subtler, less invasive eye-tracking results are particularly well positioned to uncover misrepresentations of the conflict premises attributable to early attentional divergences. Moreover, there are dissenting accounts using different techniques. For instance, Travers et al. (2016) find no evidence of conflict detection among incorrect responders using mouse-movement tracking methods.

Because this is somewhat controversial territory, perhaps it is worth considering the actual data Mata et al. (2017) use to draw their conclusion. The authors base their claim for the insensitivity of incorrect responders to reasoning conflict on null results drawn from frequentist statistics in a small sample (n = ~30 in Study 1; and n = 27 in Study 2). The authors briefly note the possibility that their sample of incorrect responders might have been underpowered and therefore incapable of discerning differences in this group. To test this, we ran a Bayesian analysis on their data using Bayes factors. This approach allows us to quantify the degree of support for the null hypothesis (e.g., Wagenmakers, 2007; Masson, 2011; Morey et al., 2014).

Given the potential confound resulting from the non-continuous nature of the eye-tracking in Study 1, we focused on Study 2, and the authors were kind enough to share their subject-averaged results with us. Using the JASP package (JASP Team, 2016), we ran a Bayesian paired samples t-test with default priors (Cauchy prior width = 0.707). We entered the average fixation time and number of revisits on the critical problem premises (sentence 3) on conflict and no-conflict items for the group of incorrect responders in the analyses. Results

showed that the Bayes factor in favor of the null hypothesis (no effect of conflict) over the alternative hypothesis (effect of conflict) was 1.34 for fixation time (non-directional; directional alternative hypothesis conflict > no-conflict, Bayes factor = 0.71) and 2.84 for the number of revisits (non-directional; directional alternative hypothesis conflict > no-conflict, Bayes factor = 1.67). These Bayes factor values indicate that there is only weak support in favor of the null hypothesis. Based on the classification of Lee and Wagenmakers (2013), the evidence for the hypothesis that incorrect responders do not distinguish between the conflict and no-conflict versions of the problem can be classified as merely anecdotal. Indeed, the only clear conclusion that we can draw based on these data is that we need more data to support a convincing conclusion in either direction.

In sum, although there is much to like about the Mata et al. (2017) paper, our key point is that the study does not present substantial evidence for the claim that incorrect responders cannot discriminate between conflict and no-conflict problems. Given the prior contradictory findings on the exact same task, we propose that caution is needed when drawing strong conclusions about incorrect responders' insensitivity to conflict on the basis of this study.

## AUTHOR CONTRIBUTIONS

DF wrote the article. BB and WD analyzed the data. WD revised the article.

## FUNDING

This research was supported by a research grant (DIAGNOR, ANR-16-CE28-0010-01) from the Agence National de la Recherche. Additionally, DF is supported by the Sorbonne Paris Cité International Grant (INSPIRE), and BB is supported by the École de Neurosciences Paris (ENP).

## ACKNOWLEDGMENTS

We are grateful to the authors of the original article for sharing their data with us.

## REFERENCES

Ball, L. (2013). "Eye-tracking and reasoning: what your eyes tell about your inferences," in *New Approaches in Reasoning Research*, eds W. De Neys and M. Osman (Hove: Psych Press), 51–69.

Ball, L. J., Phillips, P., Wade, C. N., and Quayle, J. D. (2006). Effects of belief and logic on syllogistic reasoning: eye-movement evidence for selective processing models. *Exp. Psychol.* 53, 77–86. doi: 10.1027/1618-3169.53.1.77

De Neys, W. (2012). Bias and conflict: a case for logical intuitions. *Perspect. Psychol. Sci.* 7, 28–38. doi: 10.1177/1745691611429354

De Neys, W. (2014). Conflict detection, dual processes, and logical intuitions: some clarifications. *Think. Reason.* 20, 169–187. doi: 10.1080/13546783.2013.854725

De Neys, W., Rossi, S., and Houdé, O. (2013). Bats, balls, and substitution sensitivity: cognitive misers are no happy fools. *Psychon. Bull. Rev.* 20, 269–273. doi: 10.3758/s13423-013-0384-5

Frederick, S. (2005). Cognitive reflection and decision making. *J. Econ. Perspect.* 19, 25–42. doi: 10.1257/089533005775196732

Frey, D., Johnson, E. D., and De Neys, W. (2017). Individual differences in conflict detection during reasoning. *Q. J. Exp. Psychol.* 1–52.

Gangemi, A., Bourgeois-Gironde, S., and Mancini, F. (2015). Feelings of error in reasoning—in search of a phenomenon. *Think. Reason.* 21, 383–396. doi: 10.1080/13546783.2014.980755

JASP Team (2016). JASP (Version 0.8) [Computer software].

Johnson, E. D., Tubau, E., and De Neys, W. (2016). The Doubting System 1: evidence for automatic substitution sensitivity. *Acta Psychol.* 164, 56–64. doi: 10.1016/j.actpsy.2015.12.008

Lee, M. D., and Wagenmakers, E. J. (2013). Bayesian Data Analysis for Cognitive Science: A Practical Course. New York, NY: Cambridge University Press.

Masson, M. E. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis significance testing. *Behav. Res. Methods* 43, 679–690. doi: 10.3758/s13428-010-0049-5
equate incorrect responding with insensitivity to conflict. A great deal of evidence suggests that incorrect responders are often at least minimally sensitive to conflicts across a wide range of tasks. Surveying this literature, De Neys (2012, 2014) cites evidence of conflict sensitivity on a plurality of diverse tasks (base rate neglect, conjunction fallacy, ratio bias, syllogistic reasoning). More importantly, using the very same task as Mata et al. (2017)—the bat-and-ball problem—different teams have found evidence that incorrect responders detect conflicts on such items (De Neys et al., 2013; Gangemi et al., 2015; Johnson et al., 2016). Since the latter findings rely on different measures (reaction times and confidence/error ratings) than those employed by Mata et al. (2017), one might argue that these contradictory results are a function of the distinct techniques the experiments used. Perhaps the subtler, less invasive eye-tracking results are particularly well positioned to uncover misrepresentations of the conflict premises attributable to early attentional divergences. Moreover, there are dissenting accounts using different techniques. For instance, Travers et al. (2016) find no evidence of conflict detection among incorrect responders using mouse-movement tracking methods.

Because this is somewhat controversial territory, perhaps it is worth considering the actual data Mata et al. (2017) use to draw their conclusion. The authors base their claim for the insensitivity of incorrect responders to reasoning conflict on null results drawn from frequentist statistics in a small sample (n = ~30 in Study 1; and n = 27 in Study 2). The authors briefly note the possibility that their sample of incorrect responders might have been underpowered and therefore incapable of discerning differences in this group. To test this, we ran a Bayesian analysis on their data using Bayes factors. This approach allows us to quantify the degree of support for the null hypothesis (e.g., Wagenmakers, 2007; Masson, 2011; Morey et al., 2014).

Given the potential confound resulting from the non-continuous nature of the eye-tracking in Study 1, we focused on Study 2, and the authors were kind enough to share their subject-averaged results with us. Using the JASP package (JASP Team, 2016), we ran a Bayesian paired samples t-test with default priors (Cauchy prior width = 0.707). We entered the average fixation time and number of revisits on the critical problem premises (sentence 3) on conflict and no-conflict items for the group of incorrect responders in the analyses. Results

showed that the Bayes factor in favor of the null hypothesis (no effect of conflict) over the alternative hypothesis (effect of conflict) was 1.34 for fixation time (non-directional; directional alternative hypothesis conflict > no-conflict, Bayes factor = 0.71) and 2.84 for the number of revisits (non-directional; directional alternative hypothesis conflict > no-conflict, Bayes factor = 1.67). These Bayes factor values indicate that there is only weak support in favor of the null hypothesis. Based on the classification of Lee and Wagenmakers (2013), the evidence for the hypothesis that incorrect responders do not distinguish between the conflict and no-conflict versions of the problem can be classified as merely anecdotal. Indeed, the only clear conclusion that we can draw based on these data is that we need more data to support a convincing conclusion in either direction.

In sum, although there is much to like about the Mata et al. (2017) paper, our key point is that the study does not present substantial evidence for the claim that incorrect responders cannot discriminate between conflict and no-conflict problems. Given the prior contradictory findings on the exact same task, we propose that caution is needed when drawing strong conclusions about incorrect responders' insensitivity to conflict on the basis of this study.

## AUTHOR CONTRIBUTIONS

DF wrote the article. BB and WD analyzed the data. WD revised the article.

## FUNDING

This research was supported by a research grant (DIAGNOR, ANR-16-CE28-0010-01) from the Agence National de la Recherche. Additionally, DF is supported by the Sorbonne Paris Cité International Grant (INSPIRE), and BB is supported by the École de Neurosciences Paris (ENP).

## ACKNOWLEDGMENTS

We are grateful to the authors of the original article for sharing their data with us.

## REFERENCES

Ball, L. (2013). "Eye-tracking and reasoning: what your eyes tell about your inferences," in *New Approaches in Reasoning Research*, eds W. De Neys and M. Osman (Hove: Psych Press), 51–69.

Ball, L. J., Phillips, P., Wade, C. N., and Quayle, J. D. (2006). Effects of belief and logic on syllogistic reasoning: eye-movement evidence for selective processing models. *Exp. Psychol.* 53, 77–86. doi: 10.1027/1618-3169.53.1.77

De Neys, W. (2012). Bias and conflict: a case for logical intuitions. *Perspect. Psychol. Sci.* 7, 28–38. doi: 10.1177/1745691611429354

De Neys, W. (2014). Conflict detection, dual processes, and logical intuitions: some clarifications. *Think. Reason.* 20, 169–187. doi: 10.1080/13546783.2013.854725

De Neys, W., Rossi, S., and Houdé, O. (2013). Bats, balls, and substitution sensitivity: cognitive misers are no happy fools. *Psychon. Bull. Rev.* 20, 269–273. doi: 10.3758/s13423-013-0384-5

Frederick, S. (2005). Cognitive reflection and decision making. *J. Econ. Perspect.* 19, 25–42. doi: 10.1257/089533005775196732

Frey, D., Johnson, E. D., and De Neys, W. (2017). Individual differences in conflict detection during reasoning. *Q. J. Exp. Psychol.* 1–52.

Gangemi, A., Bourgeois-Gironde, S., and Mancini, F. (2015). Feelings of error in reasoning—in search of a phenomenon. *Think. Reason.* 21, 383–396. doi: 10.1080/13546783.2014.980755

JASP Team (2016). JASP (Version 0.8) [Computer software].

Johnson, E. D., Tubau, E., and De Neys, W. (2016). The Doubting System 1: evidence for automatic substitution sensitivity. *Acta Psychol.* 164, 56–64. doi: 10.1016/j.actpsy.2015.12.008

Lee, M. D., and Wagenmakers, E. J. (2013). Bayesian Data Analysis for Cognitive Science: A Practical Course. New York, NY: Cambridge University Press.

Masson, M. E. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis significance testing. *Behav. Res. Methods* 43, 679–690. doi: 10.3758/s13428-010-0049-5
Mata, A., Ferreira, M. B., Voss, A., and Kollei, T. (2017). Seeing the conflict: an attentional account of reasoning errors. *Psychon. Bul. Rev.* 24, 1–7. doi: 10.3758/s13423-017-1234-7

Mata, A., Schubert, A.-L., and B., Ferreira, M. (2014). The role of language comprehension in reasoning: How “good-enough” representations induce biases. *Cognition* 133, 457–463. doi: 10.1016/j.cognition.2014.07.011

Mevel, K., Poirel, N., Rossi, S., Cassotti, M., Simon, G., Houdé, O., et al. (2015). Bias detection: response confidence evidence for conflict sensitivity in the ratio bias task. *J. Cogn. Psychol.* 27, 227–237. doi: 10.1080/20445911.2014.986487

Morey, R. D., Rouder, J. N., Verhagen, J., and Wagenmakers, E.-J. (2014). Why hypothesis tests are essential for psychological science a comment on Cumming (2014). *Psychol. Sci.* 25, 1289–1290. doi: 10.1177/0956797614525969

Pennycook, G., Fugelsang, J. A., and Koehler, D. J. (2015). What makes us think? A three-stage dual-process model of analytic engagement. *Cogn. Psychol.* 80, 34–72. doi: 10.1016/j.cogpsych.2015.05.001

Travers, E., Rolison, J. J., and Feeney, A. (2016). The time course of conflict on the Cognitive Reflection Test. *Cognition* 150, 109–118. doi: 10.1016/j.cognition.2016.01.015

Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. *Psychon. Bull. Rev.* 14, 779–804. doi: 10.3758/BF03194105

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Frey, Bago and De Neys. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Mata, A., Ferreira, M. B., Voss, A., and Kollei, T. (2017). Seeing the conflict: an attentional account of reasoning errors. *Psychon. Bul. Rev.* 24, 1-7. doi: 10.3758/s13423-017-1234-7

Mata, A., Schubert, A.-L., and B., Ferreira, M. (2014). The role of language comprehension in reasoning: How “good-enough” representations induce biases. *Cognition* 133, 457–463. doi: 10.1016/j.cognition.2014.07.011

Mevel, K., Poirel, N., Rossi, S., Cassotti, M., Simon, G., Houdé, O., et al. (2015). Bias detection: response confidence evidence for conflict sensitivity in the ratio bias task. *J. Cogn. Psychol.* 27, 227–237. doi: 10.1080/20445911.2014.986487

Morey, R. D., Rouder, J. N., Verhagen, J., and Wagenmakers, E.-J. (2014). Why hypothesis tests are essential for psychological science a comment on Cumming (2014). *Psychol. Sci.* 25, 1289–1290. doi: 10.1177/0956797614525969

Pennycook, G., Fugelsang, J. A., and Koehler, D. J. (2015). What makes us think? A three-stage dual-process model of analytic engagement. *Cogn. Psychol.* 80, 34–72. doi: 10.1016/j.cogpsych.2015.05.001

Travers, E., Rolison, J. J., and Feeney, A. (2016). The time course of conflict on the Cognitive Reflection Test. *Cognition* 150, 109–118. doi: 10.1016/j.cognition.2016.01.015

Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. *Psychon. Bull. Rev.* 14, 779–804. doi: 10.3758/BF03194105

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Frey, Bago and De Neys. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.