DOI: 10.5061/dryad.23j8n
title: Data from: It is premature to regard the ego-depletion effect as ‘too incredible’

description: The ‘strength’ model conceptualizes self-control as a limited resource
 (Baumeister et al., 1998). Individuals are able to exert self-control, but
 only for a limited period after which capacity declines leading to reduced
 self-control capacity; a state known as ego-depletion. The model has
 generated a sizable literature confirming the ego-depletion effect in
 multiple spheres. Our meta-analysis of published ego-depletion studies
 computed a medium-sized effect (d=0.62) across 198 tests (Hagger et al.,
 2010). Carter and McCullough (2013) recently applied analyses aimed at
 testing for publication bias to our data including Schimmack’s (2012)
 incredibility index and two regression techniques (Egger et al., 1997;
 Moreno et al., 2009). Regression analyses indicated that the ego-depletion
 effect was substantially smaller than reported in our analysis and may
 even be zero, and the incredibility index indicated low statistical power
 and the chances of finding so many significant effects improbable. They
 concluded that the ego-depletion effect is subject to considerable
 publication bias and questioned whether it is a ‘real’ effect at all. We
 replicated these analyses and found similar results. We have made our
 analyses available to download from the open-access Dryad Digital
 Repository [doi:10.5061/dryad.23j8n]. We thank Carter and McCullough
 (2013) raising the issue of bias. We take this opportunity to present some
 alternative conclusions to the ones they presented. We agree that journal
 editors should be more judicious in demanding bias tests in meta-analyses,
 but believe that that recommendation does not resolve the problem of
 interpreting the bias. An important addendum to the regression analyses is
 that the bias detected by a significant regression line cannot be
 definitively attributed to publication bias. Sterne and coworkers (2000;
 2001) suggest that such bias could be attributed to a number of possible
 sources. Instead, they use the term ‘small study’ effect; the tendency for
 smaller studies to report larger effect sizes. One possible reason would
 be due to publication bias: journals tending to favor the publication of
 small studies with statistically significant results and
 disproportionately large effect sizes. However, the findings may also be
 due to methodological inadequacies or true heterogeneity in the effect. A
 definitive response to resolving the nature of bias detected by these
 methods (i.e. whether it is publication bias or other source of bias that
 causes a ‘small study effect’) would be to demand authors conducting
 meta-analyses be diligent in the pursuit of ‘fugitive literature’:
 unpublished studies with null findings, or findings that conflict with the
 commonly-accepted paradigm, that Rosenthal (1994) eloquently predicted
 would reside in the ‘file drawers’ of researchers who could not get them
 published. In the case of ego-depletion, a unique contribution would be to
 identify unpublished studies including those with null or negative
 effects, as well as studies that have since been published, and
 recalculate the meta-analytic effect size. Such an undertaking would not
 only yield a more robust effect size ostensibly independent of publication
 bias but also be informative as to whether the ‘small study effect’
 detected in the analyses was due to publication bias, other forms of bias,
 or true heterogeneity. We encourage researchers to make their replications
 of ego-depletion studies freely available to aid future meta-analyses. We
 would also like to express concerns regarding Carter and McCullough’s
 prediction, based on their regression analyses, that the ego-depletion
 effect may be zero. This prediction was based on the intercept of the
 regression of the ego-depletion effect size on precision. However, if the
 true ego-depletion effect size is zero or close to it, one would expect
 the effect sizes in the literature to be randomly distributed in both
 positive and negative directions about zero. If this is the case, then
 where are those negative findings? There are scant few ego-depletion
 experiments that have found opposite effects i.e. an improvement in
 second-task performance after engaging in an initial self-control task,
 let alone null effects. Given the intensiveness of research in this field,
 would it not be reasonable to expect to have seen the negative findings
 published? The absence of these effects creates a problem for the claim
 that the true effect is zero and the credibility of the analysis. It could
 be argued that such negative effects might not have been published because
 their interpretation might contradict commonly-accepted theory and may lie
 in the file drawers of the researchers who found them. However, we think
 that such findings would likely have seen the light of day in journals
 because they contradict the strength model and support alternative
 hypotheses consistent with other theories such as adaptation or learned
 industriousness (Converse & DeShon, 2009). For example, one could
 pose an alternative hypothesis that improvement in self-control
 performance in ego-depletion experiments could be due to learning the
 capacity to self-regulate which was transferable. In such cases one would
 expect statistically-significant improvements in performance on a
 subsequent self-control task after engaging in an initial task that taxes
 self-control. Of course, we would have to assume that researchers were
 sufficiently virtuous in not turning their null results into supportive
 evidence using selective reporting (Francis, in press). We contend that if
 the predicted effect size for ego-depletion is zero, then negative effects
 should be present in this literature and we would expect such effects to
 be published given their pivotal role in testing alternative hypotheses
 based on other theories. As a final point, while we thank Carter and
 McCullough for raising a notable question regarding the existence of bias
 in ego-depletion meta-analysis, their analysis tells us little about its
 source and does not acknowledge that other effects in published
 meta-analyses are subject to similar bias. We think it is important to
 view and interpret the bias found for ego-depletion using these methods in
 context. For example, how does the small study effect found for ego
 depletion match up to the relative to the incidence of bias in the
 discipline of social psychology as whole? A useful future endeavor would
 be to systematically identify meta-analyses published in social psychology
 over a substantive period, subject each to the bias-identification
 analyses, and comment on the extent of the bias within the discipline.
description: Ego-depletion
 meta-analysis raw dataRaw data from our
 meta-analysis of the ego depletion effect. Columns are: (1) Ego-depletion
 effect size for each study in Cohen's d (standardized mean difference
 between experimental and control group); (2) Experimental group sample
 size; (3) Control group sample size; (4) Depleting task domain code, CA =
 controlling attention, CI = controlling impulses, CV = choice and
 volition, CE = controlling emotions, CT = controlling thoughts, SP =
 social processing, CP = cognitive processing; (5) Study authors, year and
 (if applicable) experiment/sample number within study. For details see:
 Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. D. (2010).
 Ego depletion and the strength model of self-control: A meta-analysis.
 Psychological Bulletin, 136(4), 495-525. doi:
 10.1037/a0019486ego_depletion_raw_data.txtFunnel plots of ego-depletion dataFunnel plots for ego-depletion effect sizes from our meta-analysis (see Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. D. (2010). Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin, 136(4), 495-525. doi: 10.1037/a0019486). There are two separate plots: Standard - this is the effect size (standardized mean difference, Cohen's d) against the standard error of each study; Precision - this is the effect size plotted against the study precision (1/standard error). For description see: Duval, S., & Tweedie, R. L. (2000). Trim and fill: A simple funnel plot based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. Files are MS word format.Funnel_plots.zipSmall-study effect and bias reportsWord files from CMA summarising bias statistics for the ego-depletion data from our meta-analysis (Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. D. (2010). Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin, 136(4), 495-525. doi: 10.1037/a0019486). There are separate files for the overall effect and each separate set of data for depleting task domain (see our meta-analysis or main ego-depletion data file in this data set for description of depleting task domains). Each file contains a description of the funnel plot, the Classic fail-safe N, Begg and Mazumdar's Rank Correlation Test, Egger's Test of the Intercept, Duval and Tweedie's Trim and Fill, and an important caveat about the interpretation of bias, which we also outline in our commentary: Hagger, M. S. (2014). Avoiding the ‘déjà-variable’ phenomenon: Social psychology needs more guides to constructs. Frontiers in Psychology, 5, 52. doi: 10.3389/fpsyg.2014.00052. It is also important to note that almost all of these data files indicate substantial heterogeniety, so may question the validity of the use of these statistics to make definitive decisions on small-study or publication bias.Bias_reports.zipEgo depletion meta-analysis full raw data filesFull raw data sets for our meta-analysis (Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. D. (2010). Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin, 136(4), 495-525. doi: 10.1037/a0019486) including data for all moderator variables, additional dependent variable, and tests of additional 'Strength Model' hypotheses. See accompanying readme for details.ego_depletion_meta.zip