Untitled

A recent paper by Chatterjee, Rose, and Sinha (2013) reported impressively large “money priming” effects: incidental exposure to concepts relating to cash or credit cards made participants much less generous with their time and money (after cash primes) or much more generous (after credit card primes ). Primes also altered participants’ choices in a word-stem completion task. To explore these effects, we carried out re-analyses of the raw data. A number of strange oddities were brought to light, including a dramatic similarity of the filler word-stem completion responses produced by the 20 subjects who contributed most to the priming effects. We suggest that these oddities undermine the credibility of the paper and require further investigation. Chatterjee, Rose, and Sinha (2013) presented results from three experiments investigating social priming— specifically, priming effects induced by incidental exposure to concepts relating to cash or credit cards. They reported that exposing people to cash concepts made them less generous with their time and money, whereas exposing them to credit card concepts made them more generous. The article by Chatterjee et al. (2013) was brought to the attention of the present authors by an investigator working on social priming. Struck by the large effect sizes in the first two studies, and hoping to better understand how such large effects had emerged, we requested the raw data. The authors provided us with three Microsoft Excel files containing summary data for each experiment, as well as the materials used in the studies. Later, they also provided an Excel file with additional data for the third study: the subjects’ word-stem completion responses. Examining these data brought to light a number of oddities. These peculiarities—described in detail next— included extraordinary similarity of word-stem completion responses attributed to the subset of subjects who contributed the most to the large effects reported in the original article. As we show, this high rate of reduplication appears in a number of distinct aspects of the data, and the likelihood that it would have occurred by chance are found to be infinitesimally small. Furthermore, as pointed out to us by a reviewer of a previous version of our paper, there were puzzling inconsistencies between the materials used in data collection and the subjects’ reported responses. We argue here that these findings make it extremely unlikely that the data were produced by the process described in the Method section of the original article. Prior to the preparation of the current article, we brought the concerns described here to the attention of the original authors via extensive e-mail correspondence. Over a long period, these authors argued that our findings did not undermine the credibility of their data, and in a review of an earlier version of this paper and in a lengthy e-mail correspondence, they sought to rebut our concerns. We discuss some of their reactions next. Overview of the experiments In all three studies reported by Chatterjee et al. (2013), the priming manipulation involved asking subjects to solve sentence descrambling tasks. Here, subjects see five words and try to construct a valid sentence using four of the words. Subjects were randomly assigned to descramble sentences related to cash (cash condition) or credit cards (credit condition) or (in some studies) neither one (neutral condition). For example, in the credit condition a subject might be asked to create a sentence with four of the five words in “TV shall watch we Visa,” for which a solution might be “we shall watch TV. ” The authors contended that their data demonstrated that “priming cash concepts reduces CONTACT Hal Pashler hpashler@ucsd.edu Department of Psychology, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093. Color versions of one or more of the figures in the article can be found online at www.tandfonline. com/hbas. © 2016 Taylor & Francis D ow nl oa de d by [C ol um bi a U ni ve rs ity ] a t 1 8: 26 0 6 Fe br ua ry 2 01 6 willingness to help others, while activating credit card concepts reverses these effects” (p. 109). The choice of dependent variables varied slightly across studies. In Experiment 1, subjects were given four 25-cent coins (“quarters”) in return for completing the study. Then, as each subject left the lab, the experimenter mentioned that the lab was accepting donations for the “University Student Fund” and that, if they wished, subjects could put some of the quarters in a box to support this charitable effort. The article reported that, on average, subjects in the credit condition donated more money (Mà 73 cents) than did subjects in the neutral condition (41 cents), who in turn donated more money than did subjects in the credit condition (27 cents). Experiment 2 was similar to Experiment 1, except that subjects were invited to volunteer time rather than money for a charitable cause. Experiment 3 focused on cash and credit priming effects upon subjects’ completions of word stems (although donation choices were again reported). The specific procedures of Experiment 3 are discussed in more detail further below. Unusually large effect sizes Although Chatterjee et al. (2013) did not report effect sizes, before requesting the raw data we calculated effect sizes in the form of Cohen’s d (d represents the difference between the conditions divided by the pooled standard deviation). In Experiment 1, for the comparison between credit and cash conditions, the effect size for amount of money donated was dà 2.19. In Experiment 2, subjects indicated how many hours per month they would be willing to volunteer to help an organization described as “University Student Welfare. ” The effect sizes for the differences between the control condition and the cash and credit conditions were each about dà1. 6 in magnitude, and the difference between the latter two conditions measured dà 2.98. These are all very large effects for any behavioral science experiment— particularly surprising, it seemed to us, to find in the context of a study with such a subtle and indirect manipulation. To place these effect sizes in context, it may be illuminating to compare them with reasonable priors for effect sizes in general. Large-scale surveys of the social-psychological literature show that published effects average dà . 45 (Richard, Bond, & StokesZoota, 2003; Westfall, 2015), but of course these are undoubtedly inflated due to publication bias. To provide a more accurate estimate of typical effect sizes, Simmons, Nelson, and Simonsohn (2013) studied a large sample (697 participants) and deliberately chose a number of glaringly “obvious” effects to measure, reporting all of their measures. For example, they tested the hypothesis that men weigh more than women and the hypothesis that people who like eggs tend to eat more egg salad than do people who do not like eggs. The effect sizes for these two effects were dà 0.59 and 1.07, respectively. As another point of comparison involving an experimental manipulation, Lobbestael, Arntz, and Wiers (2008) compared four different methods used by emotion researchers to elicit anger in subjects. The manipulations were far from subtle, for example, repeatedly verbally harassing subjects produced effects. Yet the observed effects on subjects’ self-reported anger averaged in the range from dà 0.64 to dà 0.74. As Simmons et al. pointed out, a d of 1.0 tends to occur only with effects so potent that they are easily detected through casual observation. Thus, to return to the Chatterjee et al. (2013) article, finding that subtle priming manipulations yield effects in the range of 1.5–3 seemed to us to be quite extraordinary. Of course, the very large effect sizes reported by Chatterjee et al. (2013) could have been inflated by sampling error. Even if that were the case, however, the occurrence of multiple unusually large effects within a single article seemed odd to us. Seeking to explore the data set in greater detail, we requested the raw data to see if the data sets had other unusual features. The authors were kind enough to provide their data (which can be downloaded at http://laplab.ucsd.edu/Chattdata/). Our examination of these data is the main focus of the remainder of this article. Reduplication of nontarget responses in