Motivated Reasoning in Scientific Evidence Collection
Creators: name: Drummond Otten, Caitlin
name: Anglin, Stephanie
name: Broomell, Stephen
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: Social and Behavioral Sciences

description: Title: Motivated Reasoning in Scientific Evidence Collection Investigators: Stephanie Anglin, Stephen Broomell, Caitlin Drummond Study Design: • After providing informed consent, participants answer a screener task consisting of 3 questions about their beliefs regarding Covid-19. The target question asks whether they believe that hydroxychloroquine is effective at preventing and treating Covid-19. Participants have the option of answering yes, no, or don’t know. We will only invite those who say yes or no to continue with the study, and we will use a quota to recruit equal numbers of participants who say yes and no. • Participants are next told that they will be reading about real research testing the effectiveness of a drug for preventing [treating; wording varies by study condition] Covid-19. They are told that they will be given a link to the research at the end of the study. • Participants are randomly assigned to study condition: they read about a study that found that hydroxychloroquine was not effective at preventing Covid-19, or about a study that found that hydroxychloroquine was effective at treating Covid-19. • Participants are randomly assigned to drug condition: hydroxychloroquine is referred to either as hydroxychloroquine or “Drug A” with a promise that Drug A will be revealed at the end of the study. DVs: • Main DV: After reading about the study, participants are asked the main DV, “Testing drugs takes time and resources. Medical researchers may conduct multiple studies of the same drug, if they think they need more evidence regarding the drug's effects. But, devoting time and resources to testing one drug means that there will be less time and resources available to test other promising drugs. In your opinion, based on the study you just read, should medical researchers prioritize conducting another study testing hydroxychloroquine [Drug A]?” Responses are on a 7-item Likert scale from strongly agree to strongly disagree. Respondents will also be able to respond to an open-ended question asking why they chose their answer. • Study Findings DV: Participants are next asked what they think would happen if medical researchers conducted another study testing hydroxychloroquine [Drug A]: they would find that hydroxychloroquine [Drug A] is effective, that hydroxychloroquine [Drug A] is ineffective, the results would be inconclusive, or don’t know. • Usage DV: Participants are next asked, “Based on the study that you just read, do you think that hydroxychloroquine should be used to prevent [treat] COVID-19?” They have the option of answering yes or no or I don’t know. • Quality DV: Participants are next asked three questions about their evaluation of the quality of the evidence in the study they read, drawn from Drummond &amp; Fischhoff (2019), and elicited on Likert scales: The study’s evidence was convincing; the way the study was conducted made sense; the study was conducted carefully. Following previous research, we will average these together to form an index variable for study quality. • Participants will complete the 11-item Scientific Reasoning Scale (SRS). • Finally, participants are asked to provide demographic information including age, gender, education, high school science education, college science education, political party, political ideology, who they voted for in 2016, and whether English is their native language. Analyses and Hypotheses: • Descriptive results: we will report descriptive statistics (proportions, distributions, means and SDs, where appropriate) for the four dependent variables expressed above, overall and aggregated by study condition, drug condition, and prior belief. • Participants were randomly assigned to read about one of two real studies that tested hydroxychloroquine. These studies occurred in different settings, tested different hypotheses, and differed in other ways. Because of the differences across these two studies, we will analyze our data within study, meaning that we will analyze the data of those who saw the study that found hydroxychloroquine effective separately from those who saw the study the found hydroxychloroquine ineffective. • Within each study, we will conduct a two-way ANOVA of prior belief (hydroxychloroquine is effective/ hydroxychloroquine is ineffective) by drug condition (hydroxychloroquine/Drug A). • Hypothesis for Our Main Dependent Variable: We predict to find an interaction between prior belief and drug condition such that the proportion of respondents indicating another study should be conducted does not vary by prior belief when presented with Drug A, but does vary by prior belief when presented with hydroxychloroquine, such that those for whom the initial study’s findings is inconsistent with prior beliefs are more likely to indicate another study should be done. • We will conduct exploratory analyses of our secondary DVs: • Study Findings DV: We predict to find an interaction between prior belief and drug condition such that the proportion of respondents indicating another study would find Drug A effective does not vary by prior belief when presented with Drug A, but does vary by prior belief when presented with hydroxychloroquine, such that those who believe hydroxychloroquine is effective are more likely to indicate another study would find hydroxychloroquine effective. • Usage DV: We predict to find an interaction between prior belief and drug condition such that the proportion of respondents indicating Drug A should be used to prevent Covid-19 does not vary by prior belief when presented with Drug A, but does vary by prior belief when presented with hydroxychloroquine, such that those who believe hydroxychloroquine is effective are more likely to indicate hydroxychloroquine should be used. • Quality DV: We predict to find an interaction between prior belief and drug condition such that perceived study quality does not vary by prior belief when presented with Drug A, but does vary by prior belief when presented with hydroxychloroquine, such that those for whom the initial study’s findings is inconsistent provide lower quality ratings. • We will conduct exploratory analyses assessing the robustness of these results after controlling for SRS and demographics, via linear and logistic regressions. Power Analysis, Participant Recruitment, and Inclusion/Exclusion Criteria: • We will use the following exclusion criteria. There is a three-item screener task at the start of the survey, including a target question about whether they believe that hydroxychloroquine is effective at preventing and treating Covid-19. Those who indicate they “don’t know” whether hydroxychloroquine is effective will not be asked to take the full study. Additionally, we will use a quota to recruit roughly equal numbers of participants who say “yes” and “no” to the hydroxychloroquine question; those who are over quota will not be asked to take the full study. Within the study, at the end of the Scientific Reasoning Scale, there is an instructed-choice attention check question. Participants who do not answer this question correctly will be excluded from the analyses. • Using an adjusted alpha of 0.025 (0.05/2), adjusted to reflect that we are testing our main hypothesis twice (because of the two study conditions), we would need to recruit 155 participants per study condition to have 80% power to detect an interaction between prior belief and drug condition, using a two-way ANOVA, with a medium-sized effect size of Cohen’s f = 0.25. Our target N is 310 participants. • We will recruit a nationally representative sample of participants using Lucid. Based on prior research, we expect 15% of the sample to fail the attention check question. Thus, we plan to recruit 357 participants.