A Multiverse Analysis of the Effects of Ego Depletion on Selfish Behaviour
Creators: name: Seroka, Julia
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: Social Statistics
subject: Life Sciences
subject: Social and Behavioral Sciences
subject: Psychology
subject: FOS: Psychology
subject: Research Methods in Life Sciences
subject: Multiverse Analysis

description: The present study aims to address issues at the heart of the current replication crisis across sciences with a focus on social psychology through an investigation of whether findings "under scrutiny" in a chosen publication are "either fragile or sturdy" (Wagenmakers et al., 2021), i.e, establishing their robustness. This is aimed to be achieved through examining the extent to which the effects of interest vary over a range of alternative, sensible data pre-processing and analytic choices, also known as "researcher degrees of freedom" (Simmons et al., 2011). The article in question, J. M. Osgood's "Ego-depletion increases selfish decision making, but may also increase self-conflict and regret about those decisions" published in The Journal of Social Psychology in 2019 has been selected, and the current study will primarily focus on two effects from Osgood's Study 1 - the increased selfish behaviour and decreased prosocial behaviour in participants subjected to an ego depletion task, expressed as an increased number of fish harvested in a social dilemma game, and a decreased number of fish returned to the pond, respectively (data available at https://osf.io/45mxw). The inclusion criteria for selecting a paper included interesting main findings that may appear too perfect, nearing non-significance, shaky, perhaps in conflict with existing literature, or questionable for other justifiable reasons such as insufficient sample sizes, suspicion of the study being underpowered, etc. A number of potential papers was rejected, and the aforementioned publication accepted following a review and discussion with supervisor. Exclusion criteria included overly complex statistical modelling that would require time and resources beyond the scope of the degree project to analyse, as well as publications with crucial data not openly available. The paper in question was selected on the basis of the effects' p-values nearing non-significance (.039, .048, one-tailed), as well as some observed conceptual and methodological inconsistencies and issues that will be discussed and argued to be common hindrances to the ultimate value of psychology as a science, the preservation of which should be the collective priority of all researchers within the field. A Multiverse of alternative data pre-processing and analytic decisions that could have been made by the original researcher has been identified and represented visually in a "garden of forking paths" (Gelman &amp; Loken 2014), with 48 paths for each reported effect, i.e. 96 total paths, each leading to an individual analysis. The original authors followed one of these paths, however as previously mentioned, the choices along this path are arbitrary "researcher degrees of freedom" and could justifiably be altered, and some of these possible alterations are demonstrated in the "garden of forking paths". Each of the 96 paths in this study involves different, equally justifiable data modifications and statistical analysis choices, such as alternative outlier removal, data subsets, variable classifications and statistical tests. The specific modifications represented as branches opposite those with the researchers' original choice on each level of the garden of forking paths diagram are 1) excluding participants with extreme task completion times, 2) including originally excluded participants, 3) running the analyses on males and females separately, and 4) controlling for age. The rationale behind each will be discussed. Executing each path will generate an individual p-value and associated effect size which will then be compared and contrasted with others, and appropriate conclusions will be drawn. The purpose of these analyses is to examine the "vibration of effects" (Patel et al., 2015) across the p-values and effect sizes of interest, and hence their robustness. All data modifications were conducted in the statistical software SPSS and all analyses were planned, and will be run in SPSS Statistics, JASP and R Studio. 24 frequentist statistical analyses for both effects will be conducted in SPSS by running an independent samples t-test or an ANCOVA with all of the modifications' combinations. The following 24 paths are Bayesian equivalents of the frequentist tests in question that will be executed in JASP. No participant recruitment or testing will be taking place. The research has been approved provisionally by the School of Psychological Science Research Ethics Committee, approval code: 10070. The dataset was randomised for the purpose of the Multiverse Analysis plan to avoid any a-priori bias, and the actual analyses to be reported will be conducted on the original dataset. References: Gelman, A., Loken E., (2014). The statistical crisis in science: data-dependent analysis--a "garden of forking paths"--explains why many statistically significant comparisons don't hold up. American Scientist, 102(6). Sigma Xi. The Scientific Research Society. Osgood J. M. (2019). Ego-depletion increases selfish decision making, but may also increase self-conflict and regret about those decisions. The Journal of social psychology, 159(4). 417–430. https://doi.org/10.1080/00224545.2018.1505706 Patel, C. J., Burford, B., &amp; Ioannidis, J. P. (2015). Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. Journal of clinical epidemiology, 68(9). 1046–1058. https://doi.org/10.1016/j.jclinepi.2015.05.029 Simmons, J. P., Nelson, L. D., &amp; Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11). 1359–1366. https://doi.org/10.1177/0956797611417632 Wagenmakers, EJ., Sarafoglou, A., Aarts, S. et al. Seven steps toward more transparency in statistical practice. Nat Hum Behav 5, 1473–1480 (2021). https://doi.org/10.1038/s41562-021-01211-8