Pre-registration for Study 2: Influence of true/false news ratio on discernment
Creators: name: Modirrousta-Galian, Ariana
name: Altay, Sacha
name: Lyons, Benjamin
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: Social and Behavioral Sciences

description: Pre-registration for Study 2: Influence of true/false news ratio on discernment 1. Data collection &amp; Exclusion No data have been collected for this study yet. All participants will be recruited via Prolific Academic, and will have to be located in the U.S. They will be required to be fluent English speakers. We will use Prolific’s “balance by sex” option. Participants will be paid $1 for the est. 6 minute survey (est. 10$/hr). We will exclude participants who fail the pre-treatment attention check by not answering “I pay attention” when prompted. 2. Design We pre-tested 109 headlines, and kept 42 true headlines rated above 2.5 and 42 false news rated below 2 (on a 4pts scale). Participants who took part in the pre-test will not be eligible for the final study. The true headlines were selected from US &amp; UK mainstream news outlets. In the pre-test, half of these news outlets were of high prominence and the other half of low prominence (see:https://www.pewresearch.org/wp-content/uploads/sites/8/2014/10/Political-Polarization-and-Media-Habits-FINAL-REPORT-7-27-15.pdf) The false headlines were taken from fact-checking websites, and were rated as false. Some false headlines were also taken from a previously tested stimuli set (Chen et al, 2022) https://osf.io/q6gwy/ (Study 3). We followed this procedure to create the headlines on Facebook: https://online.ucpress.edu/collabra/article/7/1/25293/117809/A-Practical-Guide-to-Doing-Behavioral-Research-on We will use a four-point Likert scale to measure accuracy (from not at all accurate to very accurate) as our primary headline accuracy measure. Discernment is computed by taking the difference in the mean perceived accuracy between true and false news headlines (i.e., mean perceived mainstream news accuracy − mean perceived false news accuracy). Potential moderators and background measures Before rating the headlines, participants will first respond to the following: Four questions from the CRT-2 (Thompson &amp; Oppenheimer, 2016). We will then create an additive scale based on the number of correct responses (ranging from 0-4). Interest in news question How interested, if at all, would you say you are in news? [Not at all interested to Extremely interested] 7pts Political orientation question Some people talk about 'left', 'right' and 'centre' to describe parties and politicians. (Generally, socialist parties would be considered ‘left wing’ whilst conservative parties would be considered ‘right wing’). With this in mind, where would you place yourself on the following scale? [Very left-wing to Very right-wing] 7pts Secondary outcome measures After rating the headlines, participants will first respond to the following: Confidence in performance question How do you think you compare to other Americans in how well you performed in this study today at recognizing news that is made up? Please respond using the scale below, where 1 means you’re at the very bottom (worse than 99% of people) and 100 means you’re at the very top (better than 99% of people). We will also create a measure of overconfidence. First, using our discernment measure detailed above, we will order respondents and calculate their percentile. We will then combine these variables to compute the overconfidence measure as the difference between people’s self-reported ability and their actual performance. The result is a scale that can range from −100 to 100. Trust in news question To what extent do you agree with the following statement? “I think you can trust most news most of the time” [Strongly disagree to Strongly agree] 5pts Participants will be assigned in one of the following three conditions: 3. Procedure True prime The first 6 headlines are treated as ‘primes’ (6 true). The last 12 (6 true &amp; 6 false) are the target headlines. 50/50 mix The first 6 headlines are treated as ‘primes’ (3 true &amp; 3 false). The last 12 (6 true &amp; 6 false) are the target headlines. False prime The first 6 headlines are treated as ‘primes’ (6 false). The last 12 (6 true &amp; 6 false) are the target headlines. In each condition participants will rate the accuracy of 18 headlines. The target headlines will not be randomly selected from the set of 42 true and 42 false headlines. Instead, we created 7 sets of target headlines (84/12). The presentation of the sets will be randomized. The prime headlines will be randomly selected from the set of 42 true and 42 false headlines (minus the headlines presented as target headlines; i.e. a participant will not see the same headlines as target and prime). In the 50/50 mix participants will either first see 3 false headlines or 3 true headlines (randomized). 3. Hypotheses &amp; statistical analyses H1. Response bias: The higher the proportion of false news in the mix, the more participants will rate true and false news as inaccurate. And the higher the proportion of true news in the mix, the more participants will rate true and false news as accurate. The mix is coded as a continuous variable 1-2-3. Participants and headlines are added as random effects. lmer(accuracy_true_news ~ mix + (1|Participants) +(1|Set_Prime_Headlines)). lmer(accuracy_false_news ~ mix + (1|Participants(1|Set_Prime_Headlines)). RQ1. How will the proportion of false news affect discernment? Discernment = true - false The mix is coded as a continuous variable 1-2-3. Participants and headlines are added as random effects. lm(discernment ~ mix). Note that for both H1 and RQ1 if the effect of condition (the mix) is not linear we will also code it as a categorical variable. RQ2a. How does the proportion of false news affect confidence? lm(confidence ~ mix). RQ2b. How does the proportion of false news affect overconfidence? overconfidence = confidence - actual performance compared to other participants lm(overconfidence ~ mix). RQ3. How does the proportion of false news affect trust in news? lm(trust ~ mix). H1 and RQ1 will also be tested with Signal Detection Theory analyses Receiver operating characteristic (ROC) analysis will be used to analyse the data. This will involve calculating the area under the curve (AUC) using the trapezoidal rule for each participant. A linear mixed model with mix as the independent variable and participant and headline as random variables will be used to analyse the AUCs (RQ1): lmer(AUC ~ mix + (1|Participants) +(1Set_Prime_Headlines)). To analyse response bias, B"D will be calculated for each participant. A linear mixed model with mix as the independent variable and participant and headline as random variables will be used to analyse the B"Ds (H1): lmer(B"D ~ mix + (1|Participants) +(1Set_Prime_Headlines)). Exploratory analyses We will also look at the effect of the CRT and demographics on treatment effects. 4. Sample size We will recruit 1000 US participants on Prolific Academic.