STUDENTS’ CIVIC ONLINE REASONING

1

Students’ Civic Online Reasoning: A National Portrait

Joel Breakstone1, Mark Smith1, Sam Wineburg1,
Amie Rapaport2, Jill Carle2, Marshall Garland2, and Anna Saavedra3
1

Graduate School of Education, Stanford University
2

3

Gibson Consulting Group, Austin, Texas

Dornesife Center for Economic and Social Research, University of Southern California

Author Note
This is a prepublication version of an article that has been accepted for publication in
Educational Researcher.
Correspondence concerning this article should be addressed to Joel Breakstone, Graduate
School of Education, Stanford University, 485 Lasuen Mall, Stanford, CA 94305. Email:breakstone@stanford.edu

STUDENTS’ CIVIC ONLINE REASONING

2

Abstract
Are today’s students able to discern quality information from sham online? In the largest
investigation of its kind, we administered an assessment to 3,446 high school students. Equippedwith a live Internet connection, students responded to six constructed-response tasks. Students
struggled on all of them. Asked to investigate a site claiming to “disseminate factual reports” onclimate science, 96% never learned about the organization’s ties to the fossil fuel industry. Twothirds were unable to distinguish news stories from ads on a popular website’s homepage. Over
half believed that an anonymously posted Facebook video, shot in Russia, provided “strong
evidence” of U.S. voter fraud. Instead of investigating the organization or group behind a site,
students were often duped by weak signs of credibility: a website’s “look”, its top-level domain,
the content on its About page, and the sheer quantity of information it provided. The study’s
sample reflected the demographic profile of high school students in the United States, and amultilevel regression model explored whether scores varied by student characteristics. Findings
revealed differences in student abilities by grade level, self-reported grades, locality,
socioeconomic status, race, maternal education, and free/reduced price lunch status. Taken
together, these findings reveal an urgent need to prepare students to thrive in a world in which
information flows ceaselessly across their screens.
Keywords: assessment, digital literacy, civics

STUDENTS’ CIVIC ONLINE REASONING

3

Students’ Civic Online Reasoning: A National Portrait
Teenagers swim in a sea of digital information. Outside of school and homework, they
spend more than seven hours a day online (Rideout & Robb, 2019). Three quarters say they getnews from social media “often” or “sometimes,” and nearly 70% do so through Internet videos(Dautrich, 2018). Given this reality, teenagers need to be able to sort fact from fiction online tounderstand issues that affect themselves, their communities, and their country. Are they equipped
to do so? Evidence suggests that young people, like many Americans, struggle to evaluate online
sources (Gasser et al., 2012; Hargittai et al., 2010; Lurie & Mustafaraj, 2018; McGrew et al.,2018; Pan et al., 2007). However, prior research has rarely asked students to contend with actual
digital content and has never yielded a nation-wide perspective on students’ abilities. This study
is the first to offer a national portrait across demographic groups of how students evaluatedonline sources.
Background
The health of a democracy depends on people’s ability to access reliable information(Hobbs, 2010; Mihailidis & Thevenin, 2013). If youth consume information without the ability
to assess its credibility—unable to tell who is behind a cause and what their motives might be—they are easy marks for rogues of all stripes. On the Internet, traditional gatekeepers andhallmarks of authority are largely absent. It is thus imperative to democratic functioning thatstudents know how to assess the quality of information on which they base their decisions(Lynch, 2017; Metzger et al., 2010).
Unfortunately, research has revealed shortcomings in students’ approaches to evaluating
digital content. Students who said they would base evaluations on source information rarely did
so when observed in real time (Hargittai et al., 2010). Students frequently ignored source

STUDENTS’ CIVIC ONLINE REASONING

4

information (Bartlett & Miller, 2011; Barzilai & Zohar, 2012), focusing instead on the relevanceof the information provided (Walraven et al., 2009) and basing their conclusions on a website’ssurface-level features (Coiro et al., 2015; McGrew et al., 2018). These findings may reflect
deficiencies in how students are taught to judge the credibility of Internet sources. Many of themost widely used website evaluation materials—including those appearing on prestigious
university websites—feature outdated strategies that can lead students dangerously astray(Breakstone, et al., 2018; Caulfield, 2016; Sullivan, 2019; Wineburg et al., 2020; Wineburg &Ziv, 2020).
Despite research pointing to students’ lack of preparation, few tools exist to measure
students’ competency in the digital realm. Extant assessments rarely ask students to evaluate realsources. Instead, they fill out Likert-scale items (Ashley et al., 2013) or respond to scaffolded
hypothetical scenarios (Leu et al., 2012). From such proxy measures, it is hard to know what
students actually do online.
Limited assessment options spurred us to develop new measures. To do so, we firstsought to identify expertise in the evaluation of online sources. We asked three groups to
examine a series of websites: Stanford University students, university professors from fivedifferent institutions, and professional fact checkers from leading news outlets (Wineburg &McGrew, 2017). Students and academics often approached sites by “reading vertically,” startingat the top of their screen and proceeding to the bottom, pausing to comment on the site’s prose
and layout, the presence of references and contact information, and the site’s top-level domain(dot-orgs good, dot-coms bad). Fact checkers took a radically different tack. Landing on an
unfamiliar website, they clicked away from it and engaged in lateral reading (Wineburg &McGrew, 2019). They opened up new tabs across the horizontal axis of their browser andSTUDENTS’ CIVIC ONLINE REASONING

5

searched for information about the organization or individual behind it. Only after surveying
what other sites had to say did they return to their original starting place. Using this approach,fact checkers quickly sized up sites that masked their intent and backers. Students andacademics, on the other hand, often dwelled on the original site, resulting in confusion about its
real agenda or sponsor. Lateral reading allowed fact checkers to answer a crucial question:
Who’s behind the information? Their evaluations were guided by two others: (1) What’s theevidence? (2) What do other sources say? Taken together, these strategies allowed fact checkers
to better evaluate digital content than either academics or college students.
We refer to fact checkers’ approach to searching for and evaluating social and politicalinformation as Civic Online Reasoning. Our purview is more targeted than the expansive fieldsof media and digital literacy, which can embrace topics ranging from cyberbullying to identity
theft. It focuses squarely on how to sort fact from fiction online, a prerequisite for responsible
civic engagement in the twenty-first century (Kahne et al., 2012; Mihailidis & Thevenin, 2013).
In 2015, our research team launched a two-year project to develop assessments thatmeasured Civic Online Reasoning. We developed a range of tasks through an iterative process ofprototyping, expert review, and piloting with students in classrooms across the country. We also
used think-aloud protocols to ensure the assessments tapped the intended skills. These tasks
asked students to evaluate real digital sources, ranging from Facebook videos to websites ofadvocacy groups pretending to be nonpartisan think tanks. Students from middle school tocollege struggled to perform basic evaluations of digital content (McGrew et al., 2018). Middle
school students confused sponsored content with news stories. High school students trusted a
photograph posted anonymously to a social media site. College students deemed a website aboutminimum wage policy as credible even though it was run by a public relations firm.

STUDENTS’ CIVIC ONLINE REASONING

6

Despite widespread concern about digital misinformation, no prior research has gauged
students’ ability to evaluate online sources on a national scale. In this study, students completed
a holistic assessment in which they went online to evaluate a range of sources. We analyzed datafrom 3,446 students, a sample that reflected the demographic profile of high school students in
the United States. This approach yielded evidence of students’ ability to evaluate online
information and evidence of how student performance on the assessment varied by region,
locality, grade level, race/ethnicity, gender, home language, grades in school, maternal
education, and free/reduced lunch status.
Methods
Participating Districts
Using the 2016-2017 Common Core of Data from the National Center for Education
Statistics (NCES), we compiled a list of all traditional public and charter schools in the UnitedStates that enrolled students through grade 12 (National Center for Education Statistics, n.d.-a).
We categorized districts by region (Northeast, Midwest, South, West) and grouped them by
locale within regions (urban, suburban, and rural/town), resulting in twelve separate categories.
We recruited at least one district in each of the twelve categories. We prioritized districts
with racial and ethnic diversity, and we aimed to maximize the number of states represented.1
The final sample included 16 districts in 14 states. Participating districts varied considerably inenrollment, racial and ethnic diversity, percentage of students receiving special education
services, and proportion of schools eligible for Title I under the Elementary and Secondary
Education Act (see Table 1).

1

We calculated the multi-group racial/ethnic diversity for each school and district with Simpson’s Diversity Index
formula. We used Reardon and Townsend’s (2018) Stata -seg- command. The measure reflects the probability that
two students selected from a random draw in a given entity (e.g., a school or district) are members of the same
racial/ethnic group.

7

STUDENTS’ CIVIC ONLINE REASONING
Table 1
Characteristics of participating districts
Black/
AfricanAmericanAsianNativeAmerican
0.2%

Special
Education
21.9%

LimitedEnglish
Prof
4.0%

52.9%

3.6%

Region
Northeast

LocaleUrban

Enrollment
22,359Title I
Schools
89.7%

Northeast

Suburban

15,878

0.0%

15.1%

73.1%

5.2%

4.0%

0.1%

17.2%

1.3%

Northeast

Rural

2,089

0.0%

1.8%

91.0%

1.6%

5.3%

0.3%

14.4%

1.0%

2.5%

1.3%

0.1%

17.7%

0.7%
12.3%

Hispanic
3.1%

White
32.7%

Northeast

Rural

2,034

83.3%

3.7%

88.9%

MidwestUrban

45,455

92.3%

14.6%

2.5%

80.7%

1.7%

0.2%

17.0%

MidwestSuburban

20,322

0.0%

3.1%

76.8%

4.0%

11.7%

0.2%

11.4%

2.2%

MidwestSuburban

21,029100.0%

8.7%

44.4%

23.7%

16.2%

0.3%

14.2%

10.5%

Midwest

Rural

5,122

100.0%

3.0%

89.3%

1.3%

1.6%

0.6%

15.5%

1.2%

Midwest

Rural

5,100

38.5%

4.5%

85.6%

1.8%

4.6%

1.2%

14.0%

2.1%

South

Urban

12,115

10.0%

15.7%

51.5%

11.2%

15.0%

0.3%

10.2%

10.4%

South

Suburban

15,942

60.0%

13.9%

58.5%

6.1%

3.1%

4.6%

17.2%

4.3%

South

Rural

427

100.0%

15.2%

71.9%

6.6%

0.9%

0.2%

10.5%

0.7%

South

Rural

2,486

100.0%

30.0%

66.3%

1.0%

0.6%

0.6%

9.5%

11.1%

West

Urban

29,682

9.6%

17.8%

73.8%

1.1%

3.2%

0.5%

*

5.1%

West
West

Suburban
Rural

4,074
6,579

0.0%
68.2%

13.7%
19.3%

76.1%
65.6%

1.2%
6.5%

9.5%
14.5%

1.7%
4.6%

0.6%
0.9%

1.3%
1.2%

Teacher Recruitment and Registration
We randomly selected three high schools (grades 9-12) to participate within each district;
if a district had three or fewer high schools, all were included.2 We invited all social studies
teachers to participate at each of the selected schools. For each teacher who volunteered, we
randomly selected two sections from their schedule and asked them to administer assessments in
those sections. As a token of appreciation, we provided $50 gift cards for each of the selectedclass sections that participated. Teachers could choose to administer the assessment in othersections as well.
Administration

2

There were two exceptions. A Northeastern district had a large proportion of magnet schools. To ensure that weselected schools that represented the district, we stratified by magnet/traditional school type and then randomly
selected two schools from each type. In a Midwestern district, some teachers from the three schools we selected also
administered the assessment to students in a fourth school that we had not selected because those teachers taught in
multiple schools.

STUDENTS’ CIVIC ONLINE REASONING8

The assessment was administered with an online survey platform. Participation was
voluntary. Students completed the survey using Internet-connected computers during theirregularly scheduled social studies class, and they were allowed to use the Internet at any time toanswer the questions. The tasks appeared in the same order for all students. Students were notallowed to return to items after they had submitted their responses; however, no time limits were
imposed. Teachers were allowed to provide technical assistance to students, but they could not
help students evaluate sources, interpret questions, or compose responses.
Analytic Sample
We collected completed assessments from 6,445 students in 79 classrooms at 31 schools
across 16 districts. To ensure our sample reflected the U.S. population of students by region andlocale, we used the 2016-17 Common Core of Data from NCES to calculate the percentages ofpublic school students in each geographical region of the U.S. (Midwest, Northeast, South, andWest) and by locale within each region (rural, suburban, and urban). Next, we generated arandom sample of 2,937 students that was proportionate to the distribution of students in eachregion and locale.
We examined the racial and ethnic composition of this sample and found that the
proportions of Hispanic and Black/African American students were lower than the national
population of public school students.3 To improve the representativeness of our sample, we
returned to the set of completed assessments and added the remaining 509 students who
identified themselves as Hispanic or Black/African American to the analytic sample regardless of3

We created a composite variable for race/ethnicity based on students’ responses to two questions: 1) Are you
Hispanic or Latino? (yes/no), and 2) What is your race? (American Indian or Alaska Native, Asian, Black or African
American, Native Hawaiian or other Pacific Islander, White, Other—please specify, Prefer not to say). In
accordance with National Center for Education Statistics (n.d.-b) guidelines, all students who said “yes” to the firstquestion were counted as Hispanic regardless their answer to the second question. Students who either selected “no”
or did not respond to the first question were grouped by their response to the second question.

9

STUDENTS’ CIVIC ONLINE REASONING
their district. Adding these students resulted in a final analytic sample of 3,446 students that
better matched the racial/ethnic makeup of American public schools (see Table 2).
Table 2Analytic Sample by Region, Locale, Race/Ethnicity, Gender, Grade Level, Free/Reduced Lunch
Status, and Mother’s Education% United
StatesEnrollment a

Analytic
Sample N

Analytic
SampleDifference
BetweenAnalytic
Sample and% United
StatesEnrollment

16%
21%
37%
25%

732
852
1,084
778

21%
24%
31%
22%

5%*
3%*
-6%*
-3%*

-

31%
42%
27%

967
1,659
820

28%
48%
24%

-3%*
6%*
-3%*

-

5%

112

3%

-2%*

4%

1%
15%
26%
3%
49%
-

29
306
698
42
224
1,373
28
634

1%
9%
20%
1%
7%
40%
1%
18%

0%
-6%*
-6%*
4%*
-9%*
-

1%
11%
25%
2%
8%
49%
-

52%
48%

1,283
1,443

37%
42%

-15%*
-6%*

43%
49%

-

230
490

7%
14%

-

8%
-

27%
26%
24%
24%

17
820
491
867
761

0%
24%
14%
25%
22%

-3%*
-12%*
1%
-2%*

1%
28%
17%
29%
26%

Analytic
Samplewithout
Missing

Region
Northeast
MidwestSouth
West
LocaleUrban
Suburban
Rural
Race/Ethnicity
Asian or Pacific Islander
American Indian/Alaska
NativeBlack or African American
Hispanic
Other
Two or more races
White
Prefer not to say
Missing
GenderMale
FemaleNon-binary/selfdescribe/prefer not to say
Missing
Grade Level
8th
9th
10th
11th
12th

10

STUDENTS’ CIVIC ONLINE REASONING
Missing-% United
StatesEnrollment

490

Analytic
Sample N

14%

Analytic
SampleDifference
BetweenAnalytic
Sample and% United
StatesEnrollment-

Analytic
Samplewithout
MissingNational School Lunch
Program Status
Eligible/Participant
52%
850
25%
-27%*
29%
Not eligible/Non-Participant
48%
1,475
43%
-5%*
50%
I don’t know
533
15%
18%
Prefer not to say
84
2%
3%
Missing504
15%Mother’s Education level bLess than high school
222
6%
8%
*p<.05. Statistical significance of the difference between the analytic sample and the United States enrollment target
for each student subpopulation was obtained from a one-sample difference in proportion tests.
aU.S. enrollment data is from the National Center for Education Statistics Common Core of Data (2016-17).
bCommon Core of Data does not include data on mother’s education for U.S. public school students.

Adding the Black/African American and Hispanic students changed the composition of
the sample by region/locality, but the impact was small. Table 2 shows the geographic
distribution of the final analytic sample. Students from suburban schools in the Northeast andMidwest were slightly overrepresented and students from the South were slightly
underrepresented, but the overall sample reflected the geographic distribution of students in the
United States. We also considered whether the sample reflected the self-reported grade level,
gender, and free/reduced lunch eligibility (see Table 2).
Assessment Instrument
The assessment included six constructed-response tasks previously developed by ourteam (cf. McGrew et al., 2018). The tasks sampled a range of Civic Online Reasoning skills,
such as lateral reading and evaluating evidence on social media. (Appendix A provides
descriptions of the tasks; Appendix B includes the instrument.) For example, the Website
Evaluation task (task 3 in Appendix B) assessed whether students could investigate who was

STUDENTS’ CIVIC ONLINE REASONING

11

behind a website. Students were provided a link to the homepage of co2science.org, anorganization whose About page states that they seek to “disseminate factual reports and soundcommentary” on the environmental effects of carbon dioxide. The group is actually funded byfossil fuel companies, including ExxonMobil. Students were asked to judge whether the site wasreliable, and a screen prompt reminded them that they could search online to answer.
A second task, Evaluating Evidence (task 1 in Appendix B), gauged students’ ability toevaluate the trustworthiness of a social media post. The post presented students with a Facebook
video from a user named “I on Flicks” and asked students if it constituted “strong evidence” ofvoter fraud during the 2016 Democratic primaries. The video included four silent clips of poll
workers surreptitiously stuffing ballots into bins. Captions told viewers that the clips depicted
Democratic primary elections in Illinois, Pennsylvania, and Arizona. The post accompanying thevideo read: “Have you ever noticed that the ONLY people caught committing voter fraud are
Democrats?” None of this was true. The clips actually showed voter fraud in Russia, somethingeasily ascertained by a brief web search.
After students completed the six tasks, they self-reported a range of demographic
characteristics, including: age, grade, gender, race, ethnicity, primary language, typical grades inschool, mother’s education, and free/reduced lunch participation.
Rubric Design and Scoring Procedures
Rubric design was informed by prior research and development of Civic Online
Reasoning assessments (McGrew et al., 2018; Wineburg et al., 2016). In Mastery responses,
students evaluated online content by investigating the source of information, by interrogating theevidence presented, or by seeking out information from other reliable sources. Emerging
responses were partially incorrect or did not fully articulate sound reasoning. Beginning

STUDENTS’ CIVIC ONLINE REASONING

12

responses included incorrect or irrelevant strategies for evaluating online information. (SeeAppendix C for a sample rubric.)
Two independent raters scored student responses using a three-level rubric (Beginning 0; Emerging - 1; Mastery - 2).4 Estimates of interrater reliability were high, with Kappa
coefficients between .93 and .97 for each task (Task 1 = .95; Task 2 = .97; Task 3 = .94; Task 4 =.96; Task 5 = .94; Task 6 = .93).
Multivariate Regression Analysis
We used multivariate regression to explore the relationship between student
characteristics and performance on the assessment. We first created an overall composite score
for students (n = 3,402) across the six tasks. We dropped forty-four students in the analytic
sample from the composite score analysis because none of their completed tasks could be scored(e.g., they indicated that they encountered technical problems on the tasks they completed). We
then fit a multilevel regression model to explore differences in composite scores by a variety ofstudent characteristics. Below, we outline how we constructed the composite score and specify
the model used for the analysis.
Composite Score
The composite score was an average of students’ scores on the items they completed.5 So,
if a student completed all six items, then their composite score was the mean of their six scores;
if they completed three items, their score was the mean of those three scores, and so on. The

4

Scores from the first rater were used for the regression analysis.
To ensure that combining the items into a single score was psychometrically sound, we conducted an exploratory
factor analysis and found that the correlation patterns between items supported constructing a single scale. The
eigenvalue for a single factor scale was 3.04, and a single factor model accounted for nearly 60% of the total
variance in scores. We also estimated internal consistency and found that Cronbach’s alpha was .58. Tasks 4 and 5
share the same stimulus, so we checked to ensure that scores for these two items were sufficiently independent totreat as separate items. Scores for these two items were only weakly correlated (r = .19), and crosstabs showed that
there was little overlap in scores across the two questions.
5

STUDENTS’ CIVIC ONLINE REASONING

13

average composite score ranged between 0 (a student received a score of 0 on all completed
items) to 2 (a student received a score of 2 on all completed items).
Multivariate Analysis
We fit a single two-level hierarchical linear model (HLM), with random intercepts at theschool level.6 We included students’ self-reported grades as a measure of prior academic
achievement because higher achieving students might score systematically higher. The “student
grades” ranged from 1 (student reported they earned “mostly D’s”) to 7 (student reported“mostly A’s”). We also anticipated that students in upper grades might perform better thanstudents in lower grades, so we included grade level in the model.
Additionally, we included students’ geographic region and locale to adjust for any
systematic differences by student location. Finally, we included demographic variables that raw
scores suggested might be relevant to student performance on the tasks, including race/ethnicity,
gender, free/reduced lunch status, maternal education, and first language spoken at home.7
Appendix D describes these variables and how they were measured; Appendix E provides thepercentage of students scoring Emerging or Mastery by task for each of the variables.
Results
Students struggled on all of the tasks. At least two thirds of student responses were scoredas Beginning for each of the six tasks. On four of the six tasks, over 90% of students’ responses
were scored Beginning. Out of all the student responses to the six tasks, fewer than 3% received

6

The intraclass correlation coefficient was .04, which indicated that most of the variability in the regressionadjusted composite scores was within schools and not between them. This may have been a byproduct of our study
design, which prioritized schools with heterogeneous student populations. Diverse schools may have more withinschool variability than schools with homogenous populations.
7
For each demographic subpopulation, we included dummy variables, with the largest group serving as thereference category. For instance, for student race/ethnicity, we included dummy variables for students classified as
Asian, other, Hispanic, two or more races, and Black, with White respondents serving as the reference population.
Table 4 provides coefficient estimates for each demographic subgroup included in the model.

14

STUDENTS’ CIVIC ONLINE REASONINGa Mastery score. Claims on Social Media 1 (task 4 in Appendix B) had the lowest proportion ofMastery responses, with less than 1% of students demonstrating a strong understanding of theonline reasoning competencies measured. Evaluating Evidence had the highest proportion, with
8.7% earning a Mastery score. (See Table 3 for the distribution of scores by task.)
The mean composite score across all students was 0.17 out of a possible score of 2. (SeeAppendix F for the distribution of composite scores.) Fifty-eight percent of students had an
average composite score of 0, indicating that they scored at the Beginning level on all of thetasks they completed. Of the 3,402 students in the analytic sample with a composite score, only
13 scored at the Mastery level on all six tasks (0.38%).
Table 3
Percent of Responses at Each Rubric Level by Task
Evaluating
Webpage
WebsiteEvidence
Comparison Evaluation
Beginning
77.4%
92.1%
96.8%
Emerging
13.9%
6.7%
1.4%
Mastery
8.7%
1.2%
1.8%
Total (N)
3,119
2,678
2,994

SocialMedia 1
92.8%
6.9%
0.3%
2,431

SocialMedia 2
95.4%
2.7%
1.9%
2,431

Home Page
Analysis
68.7%
26.6%
4.7%
2,969

Note. Some students in the analytic sample did not complete all tasks or did not provide
responses that could be scored. Consequently, the total number of responses varied by task.

Website Evaluation Task
The Website Evaluation task, which presented students with a climate change website
funded by fossil fuel companies, had the highest proportion of Beginning scores, with 96.8% of
students receiving the lowest score. Instead of leaving the site, the vast majority of students
remained glued to the site itself, drawn in by its top-level domain (.org), the recency of its
updates, and the sheer quantity of information it included. A student from a suburban district in
the Northeast wrote: “This page is a reliable source to obtain information from. You see in the
URL that it ends in .org as opposed to .com.” This student was seemingly unaware that, unlike

STUDENTS’ CIVIC ONLINE REASONING

15

dot-gov or dot-edu sites, dot-org is an “open” domain; any individual or group can obtain one
without passing a character test or attesting to benevolent intent (Wineburg & Ziv, 2019).
Students also gave undue weight to the site’s About page, not recognizing it as a curated portraitof how a group wished to be perceived. A student from the urban South wrote:
The “about us” tab does show that the organization is a non-profit dedicated to simply
providing the research and stating what it may represent. In their position paper on CO2,
they provide evidence for both sides and state matters in a scientific manner. Therefore, I
would say they are an unbiased and reliable source.
In contrast, strong student responses used other online sources to discover thatco2science.org undermines climate change science on behalf of corporate funders. Establishing
this was not hard but did require that students leave the site to investigate. Entering the group’sname in their browsers and reading laterally called up a host of results that shed light on thegroup’s agenda. Less than two percent of students did this. One student who did wrote:co2science.org is not a reliable source because it has ties to large companies that want topurposefully mislead people when it comes to climate change. According to USA
TODAY, Exxon has sponsored this nonprofit to pump out misleading information on
climate change. According to the Seattle Post-Intelligencer, many of their scientists also
have ties with energy lobbyists.
Evaluating Evidence Task
Over half of the students (52%) were taken in by the video purportedly showing ballot
stuffing in the United States. They concluded that the video provided “strong evidence” of
Democratic voter fraud in primary elections. A student from a rural district in the Midwest took
the video at face value: “Yes, it shows video evidence of fraud in multiple different states at

STUDENTS’ CIVIC ONLINE REASONING

16

multiple different times.” Another student from an urban district in the Northeast wrote, “Yes,because the video showed people entering fake votes into boxes.”
A quarter of the students rejected the video but did not provide a coherent rationale fordoing so. Students focused on irrelevant features like the lack of audio, the video’s grainy
quality, or insufficient narration from the post’s author. Others accepted the veracity of the video
but ended up rejecting it because there were too few examples. A student from the South wrote,“The video only shows a few specific instances, and is not enough to blame the whole
Democratic party of voter fraud.” Still others struggled to interpret what was depicted in theclips, ignoring the source and origin of the video altogether. A student explained:
I have no idea what is going on exactly and I see people put papers in the box, but there
are cases where the votes are sneaked in. I am not sure this is a strong case of faking
votes for a person, but it is a case for vote fraud.
Only 8.7% of students received a Mastery score by rejecting the video and providing a relevant
explanation. These students understood they had no way of knowing if “I on Flicks” was areliable source or whether the video depicted American voting fraud. A student from an urban
district in the West questioned the video’s origin:
It does not provide enough evidence because we don't know who the people are in thevideo, and who the people are that made the video. There is very little context andinformation given, besides the place, and a little bit of commentary making claims toconvince you.
Of the 3,119 responses on this task, only three students actually tracked down the source of thevideo. Again, doing so required little more than choosing relevant search terms (e.g., “video” and“Democratic ballot stuffing”) and scanning the search engine results page for sites like Snopes or

STUDENTS’ CIVIC ONLINE REASONING

17

the BBC. A student from a suburban district in the West showed what could be learned byleaving the video and reading laterally:
Doing some research on the Internet, I found the exact same video as shown above.
However, the video which I found, which was presented by BBC News, showed that the
cam footage had nothing to do with the 2016 Democratic primary elections. Instead, BBC
presented the cam footage as that of the Russian 2016 election. According to theannouncer of BBC News, the video showed that the people in Russia—not America—
were committing voter fraud.
Such responses were few and far between.
Multivariate Results
Our regression analysis revealed significant differences in students’ composite scores fora variety of demographic factors. Table 4 shows the regression coefficients and standard errors
for each of the variables in our model. Adjusting for all other included variables, students whoreported earning higher grades performed significantly better than students who reported earning
lower grades. A one-step increase in self-reported grades (e.g., from “mostly B’s” to “mostly A’s
and B’s”) was associated with a .03 improvement in average composite score (a noticeable
improvement in a scale with a maximum of 2). Similarly, grade level was a significant predictorof student performance. Each increase in grade level (e.g., from sophomore to junior) was
associated with a .03 increase in students’ composite scores for students who were similar for allother reported characteristics.

18

STUDENTS’ CIVIC ONLINE REASONING
Table 4
Results from Mixed Effects Regression Model
Student grades
National School Lunch Program eligible
Mother's education
Grade level
Region (compared with Midwest)
Northeast
South
West
Locality (compared with Rural)
Suburban
Urban
Gender (compared with Female)Male
Non-binary/Prefer to self-describeHome Language (compared with English)
Other
SpanishRace/Ethnicity (compared with White)
AsianBlack or African-American
Other
Two or more races
Hispanic_cons
Variance: School
Intra-class correlation (r)Schools: n
Wald X^2

Composite Score
0.029 (.00)***-0.044 (.02)**0.014 (0.01)*
0.029 (.07)**-0.036 (.04)
0.009 (.03)
0.001 (.04)
0.043 (.03)
0.107 (.03)***0.017 (.01)-0.020 (.03)-0.036 (.03)-0.012 (.02)
0.019 (.03)-0.120 (.03)***-0.083 (.04)*-0.039 (.02)-0.014 (.18)-0.157 (.05)**.003 (.001)*.044
31: 1,690
87.59

Note: *p < .05, **p <.01, *** p<.001; Standard errors in parentheses.

We also observed significant results for proxy measures of socioeconomic status.
Maternal education was a significant predictor of average composite score. For every one-stepincrease in a mother’s reported education level (e.g., from less than high school to high school,or from some college to a bachelor’s degree), performance improved by about .01 points on
average. Free/reduced price lunch status was also a significant predictor of performance, with

STUDENTS’ CIVIC ONLINE REASONING

19

students eligible to receive free/reduced lunch performing .04 points lower, on average, thanstudents not eligible, adjusting for all other variables in the model.
Our model also revealed significant differences by other characteristics. Urban students
significantly outperformed their rural and suburban peers. Table 5 includes contrasts between
each locale pair not included in the initial model. Adjusting for all other variables, urban students
scored .11 points higher than rural students and .06 points higher than suburban students.
Race/ethnicity was also a significant factor in our model. After adjusting for all other variables,
Black/African American students in our sample had lower composite scores than students who
were Asian/Pacific Islander (-.14 points), Hispanic (-.11 points), multiracial (-.08 points), orWhite (-.12 points). Table 5 includes contrasts for each race/ethnicity comparison not included in
the initial model. Other variables were not significant predictors of student scores. Gender,
geographic region, and home language were not significant predictors when adjusting for allother variables.
Table 5Contrasts of Marginal Effects for Race/Ethnicity and Locale
Contrast
Race/Ethnicity
Black vs. Asian-.139 (.04)**
Black vs. White-.120 (.03) ***Black vs. Other-.037 (.04)
Black vs. Two or more races-.081 (.03)**Black vs. Hispanic-.105 (.03)**
Locality
Urban vs. Rural
0.107*(0.03)
Urban vs. Suburban
0.064*(0.03)
Note: *p < .05, **p <.01, *** p<.001; Standard errors in parentheses.
Limitations
This study’s sample was not randomly generated; as a result, it was not perfectly
representative on all observable characteristics. Assembling a truly random, representativeSTUDENTS’ CIVIC ONLINE REASONING

20

sample of students in the United States for this type of assessment would have presented
prohibitive administrative and financial barriers. In contrast to the National Assessment of
Educational Progress (NAEP), we had no legislative mandate for administration and far fewer
resources for data collection. In contrast to NAEP, we relied only on the goodwill of school
districts to participate, negotiated research access with districts individually, and worked withtechnology administrators to ensure students’ access to the open Internet during assessment
administration. Additionally, it was a low-stakes assessment. Students were not required toparticipate, and their performance did not affect course grades. However, the same holds true for
NAEP exams, which are held up as the “gold standard” of large-scale testing (National Center
for Education Statistics, n.d.-c). Despite these limitations, this is the largest study of students’
online reasoning skills to date in the United States.
Measuring students’ Civic Online Reasoning at scale presented a variety of challenges. In
the interest of ecological validity, we asked students to evaluate real online sources about socialand political issues using the Internet. This openness stands in stark contrast to the tightly
controlled environments typical of large-scale assessment. We could not control the sources thatstudents encountered while searching online. Nor could we directly observe how students arrived
at their answers. Giving students free rein online also meant greater heterogeneity in the
responses collected, which made scoring more difficult. Going forward, researchers will need toweigh the benefits of asking students to engage with authentic sources against the measurement
challenges that the accompanying freedom creates.
There are important dimensions of evaluating online sources that this study did not
address. In particular, it did not account for students’ political beliefs. Although the assessment
deliberately included sources from across the political spectrum, students’ beliefs may have

STUDENTS’ CIVIC ONLINE REASONING

21

influenced their evaluations of sources about contentious social and political issues. Theinteraction between students’ political orientations and their evaluations of digital sources
obviously merits further study.
Discussion
The results of this study should give pause to anyone who cares about the future of
democracy. Over half of students believed that an anonymously posted Facebook video, actually
shot in Russia, provided strong evidence of U.S. voter fraud. Students displayed a troublingtendency to remain on a website and take it at face value. Over 96% never learned about ties
between a climate change website and the fossil fuel industry. Instead of investigating the group
behind the site, students were duped by weak signs of credibility: the website’s “look”, its toplevel domain, or the content on its About page. Each of these features is ludicrously easy togame.
All students would benefit from carefully designed, thoughtfully delivered, andrigorously evaluated curriculum. At the same time, our findings suggest that the students mostunderserved by our nation’s schools are similarly underserved in the realm of access to quality
materials for evaluating digital sources. Socioeconomic status affects educational outcomes in
the United States (cf. Mandara et al., 2009; Reardon et al., 2015) and indicators ofsocioeconomic status were predictors of scores here, too. After controlling for other variables,
eligibility for the National School Lunch Program was negatively associated with student scores;
higher maternal educational attainment was associated with higher scores. A pernicious racial
opportunity gap persists in American schools (cf. Howard, 2019; Reardon et al., 2015), and our
findings suggest it was at play as well.

STUDENTS’ CIVIC ONLINE REASONING

22

It would be a mistake to see our results as an indictment of students. Presented withsomething to read, they mostly did what they’ve been taught. They read. Unless students
possessed deep subject matter knowledge, a site like co2science.org—studded with citations toscientific articles, “major reports”, and lists of awards—looked, well, pretty good. The few
students who learned that the website was backed by corporate interests did so not because they
stayed on the site to engage in “close reading” (Common Core State Standards Initiative, n.d.).
They succeeded because they didn’t.
When the Internet is how we become informed citizens, traditional ways of reading are
not merely ineffective (Gigerenzer & Gassmaier, 2011; Hertwig & Engel, 2016; Kozyreva et al.,2020). They’re dangerous (Caulfield, 2018; Warzel, 2021). As far back as 1971, Herbert Simon
presciently noted that information overload results in a scarcity of attention. Today, whenadvertisers, corporations, lobbyists, clickbait sites, conspiracy theorists, hate groups, and foreign
governments work overtime to hijack attention, often the wisest thing to do is not to read, but topreserve attention by practicing strategic ignoring (Wineburg & McGrew, 2019). Underconditions of limited attention, the most crucial decision to make is where to allocate it. Above
all else, lateral reading is an act of attention conservation. One uses the awesome powers of theInternet to quickly decide if an unfamiliar group, organization, or claim merits sustained
attention. The few students who left a website to navigate the open Internet found answers
waiting for them at the ready.
We have emphasized the need to leave a site to get an accurate read on it. Students would
obviously benefit by mastering a host of other skills: choosing effective search terms, conducting
reverse image searches, checking archived versions of webpages in the Internet Archive, and soon. Nor have we addressed the thorny issue of motivated reasoning, which renders the most

STUDENTS’ CIVIC ONLINE REASONING

23

sophisticated searches moot if the searcher’s mind is made up from the beginning. But we needto start somewhere. Unlike learning how to apply Boolean operators or decode a video’s
metadata, the basics of lateral reading are straightforward. Students open new tabs on theirbrowsers and enter search terms, something they already know how to do. But they won’t do sounless they understand, in deep and compelling ways, that the Web plays by a different set ofrules from the vetted texts provided to them in school.
A much greater investment in research and development is needed to prepare students tonavigate treacherous digital terrain. Small-scale but promising studies suggest that it is possible
to improve students’ digital savvy (Breakstone et al., 2021; Brodsky et al., 2019; Kohnen et al.,2020; McGrew, 2020; McGrew et al., 2019). Given the threat to civic and public health posed bydigital illiteracy, these preliminary interventions need to be taken up at scale to identify
approaches that can be widely adopted. Investigating how teachers across subject areas and gradelevels integrate these skills into existing instruction will be critical. It’s doubtful that students
will become skilled consumers of digital content if instruction is restricted to a single course or
one-off workshop.
Conclusion
Across regions and demographic groups, students in this study fell short. Their credulity
makes them—along with the rest of us—easy prey for groups that seek to deceive, mislead, andmanipulate. The toxic effects of disinformation have chipped away at the foundations ofdemocracy the world over (Diamond, 2020; Levitsky & Ziblatt, 2018).
In October 1957, a whirling orbital ball known as Sputnik roused Americans from their

STUDENTS’ CIVIC ONLINE REASONING

24

slumber. It led to the passage of the National Defense Education Act, a massive federal
investment in American education. Today, the threat posed to national security by a digitally
credulous citizenry warrants a similarly urgent response.
Anything less guarantees the continued erosion of democratic society.
Acknowledgements
This research was supported by the William and Flora Hewlett Foundation (Grant No.
2017-5939, Sam Wineburg, Principal Investigator). The content is solely the responsibility of theauthors and does not necessarily represent the views of the William and Flora Hewlett
Foundation.
Sarah McGrew and Teresa Ortega were integral to the research and development of Civic
Online Reasoning assessments. This study would not have happened without Marc Chun’s
encouragement and support. Rex Long and Sarah Narvaiz provided valuable research assistance.

STUDENTS’ CIVIC ONLINE REASONING

25

References
Ashley, S., Maksl, A., & Craft, S. (2013). Developing a news media literacy scale. Journalism &Mass Communication Educator, 68(1), 7–21. https://doi.org/10.1177/1077695812469802
Bartlett, J., & Miller, C. (2011). Truth, lies and the Internet. Demos.
https://www.demos.co.uk/files/Truth_-_web.pdf
Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online
sources. Cognition and Instruction, 30(1), 39–85.
https://doi.org/10.1080/07370008.2011.636495
Breakstone, J., McGrew, S., Smith, M., Ortega, T., & Wineburg, S. (2018). Why we need a new
approach to teaching digital literacy. Phi Delta Kappan, 99(6), 27-32.
https://doi.org/10.1177/0031721718762419
Breakstone, J., Smith, M., Connors, P., Ortega, T., Kerr, D., & Wineburg, S. (2021). Lateralreading: College students learn to critically evaluate internet sources in an online course.
Harvard Kennedy School Misinformation Review, 2(1), 1-17.
https://doi.org/10.37016/mr-2020-56
Brodsky, J., Brooks, P. J., Scimeca, D., Todorova, R., Galati, P., Batson, M., Grosso, R.,
Matthews, M., Miller, V., Tachiera, T., & Caulfield, M. (2019, October 3-5). Teachingcollege students the four moves of expert fact-checkers [Paper presentation]. Technology,
Mind, & Society, an American Psychological Association Conference, Washington, DC,
United States.
Caulfield, M. (2016, December 19). Yes, digital literacy. But which one? Hapgood.
https://hapgood.us/2016/12/19/yes-digital-literacy-but-which-one/

STUDENTS’ CIVIC ONLINE REASONING

26

Caulfield, M. (2018, December 19). Recalibrating our approach to misinformation. EdSurge.
https://www.edsurge.com/news/2018-12-19-recalibrating-our-approach-tomisinformation
Coiro, J., Coscarelli, C., Maykel, C., & Forzani, E. (2015). Investigating criteria that seventh
graders use to evaluate the quality of online information. Journal of Adolescent and Adult
Literacy, 59(3), 287–297. https://doi.org/10.1002/jaal.448
Common Core State Standards Initiative. (n.d.). English language arts standards.
http://www.corestandards.org/ELA-Literacy/CCRA/R/
Dautrich, K. (2018). The future of the First Amendment: 2018 survey of high school students andteachers. John S. and James L. Knight Foundation.
https://knightfoundation.org/reports/the-future-of-the-first-amendment-2018/
Diamond, L. (2020). Ill winds: Saving democracy from Russian rage, Chinese ambition, andAmerican complacency. Penguin.
Gasser, U., Cortesi, S., Malik, M., & Lee, A. (2012). Youth and digital media: From credibility
to information quality. Berkman Klein Center for Internet & Society.
Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of
Psychology, 65, 451-482. https://doi.org/10.1146/annurev-psych-120709-145346
Hargittai, E., Fullerton, L., Menchen-Trevino, E., & Thomas, K. Y. (2010). Trust online: Young
adults’ evaluation of Web content. International Journal of Communication, 4, 468–494.
Hertwig, R., & Engel, C. (2016). Homo ignorans: Deliberately choosing not to know. Perspectives on
Psychological Science, 11(3), 359–372. https://doi.org/10.1177/1745691616635594
Hobbs, R. (2010). Digital and media literacy: A plan of action. The Aspen Institute.
https://assets.aspeninstitute.org/content/uploads/2010/11/Digital_and_Media_Literacy.pdf

STUDENTS’ CIVIC ONLINE REASONING

27

Howard, T. C. (2019). Why race and culture matter in schools: Closing the achievement gap in
America's classrooms. Teachers College Press.
Kahne, J., Lee, N., & Feezell, J. T. (2012). Digital media literacy education and online civic andpolitical participation. International Journal of Communication, 6, 1–24.
Kohnen, A. M., Mertens, G. E., & Boehm, S. M. (2020). Can middle schoolers learn to read the
web like experts? Possibilities and limits of a strategy-based intervention. Journal ofMedia Literacy Education, 12(2), 64-79. https://doi.org/10.23860/JMLE-2020-12-2-6
Kozyreva, A., Lewandowsky, S., & Hertwig, R. (2020). Citizens versus the Internet: Confronting
digital challenges with cognitive tools. Psychological Science in the Public Interest,21(3), 103-156. https://doi.org/10.1177/1529100620946707
Leu, D. J., Coiro, J., Kulikowich, J. M., & Cui, W. (2012, November 28-December 1). Using thepsychometric characteristics of multiple-choice, open Internet, and closed (simulated)
Internet formats to refine the development of online research and comprehension
assessments in science: Year three of the ORCA project [Paper presentation]. Literacy
Research Association 62nd Annual Conference, San Diego, CA, United States.
Levitsky, S., & Ziblatt, D. (2018). How democracies die. Penguin.
Lurie, E., & Mustafaraj, E. (2018). Investigating the effects of Google’s search engine result
page in evaluating the credibility of online news sources. Proceedings of the 10th ACM
Conference on Web Science. ACM Digital Library.
https://doi.org/10.1145/3201064.3201095
Lynch, M. P. (2017). An Internet of us: Knowing more and understanding less in an age of big
data. W.W. Norton.

STUDENTS’ CIVIC ONLINE REASONING

28

Mandara, J., Varner, F., Greene, N., & Richman, S. (2009). Intergenerational family predictorsof the Black–White achievement gap. Journal of Educational Psychology, 101(4), 867–
878. https://doi.org/10.1037/a0016644
McGrew, S. (2020). Learning to evaluate: An intervention in civic online reasoning. Computers& Education, 145, 1-13. https://doi.org/10.1016/j.compedu.2019.103711
McGrew, S., Breakstone, J., Ortega, T., Smith, M., & Wineburg, S. (2018). Can students
evaluate online sources? Learning from assessments of civic online reasoning. Theory
and Research in Social Education, 46(2), 165–193.
https://doi.org/10.1080/00933104.2017.1416320
McGrew, S., Smith, M., Breakstone, J., Ortega, T., & Wineburg, S. (2019). Improving students’web savvy: An intervention study. British Journal of Educational Psychology, 89(3),485-500. https://doi.org/10.1111/bjep.12279
Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010). Social and heuristic approaches tocredibility evaluation online. Journal of Communication, 60(3), 413–439.
https://doi.org/10.1111/j.1460-2466.2010.01488.x
Mihailidis, P., & Thevenin, B. (2013). Media literacy as a core competency for engaged
citizenship in participatory democracy. American Behavioral Scientist, 57(11), 1611–
1622. https://doi.org/10.1177/0002764213489015
National Center for Education Statistics. (n.d.-a). Information on the Common Core of Data.
https://nces.ed.gov/ccd/files.asp
National Center for Education Statistics. (n.d.-b). Collecting race and ethnicity data fromstudents and staff using the new categories. https://nces.ed.gov/ipeds/report-yourdata/race-ethnicity-collecting-data-for-reporting-purposes

STUDENTS’ CIVIC ONLINE REASONING

29

National Center for Education Statistics. (n.d.-c). NAEP is the leader in large-scale assessment.
https://nces.ed.gov/nationsreportcard/about/leader_assessments.aspx
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007). In Google wetrust: Users’ decisions on rank, position, and relevance. Journal of Computer-MediatedCommunication, 12(3), 801–823. https://doi.org/10.1111/j.1083-6101.2007.00351.x
Reardon, S. F., Robinson-Cimpian, J. P., & Weathers, E. S. (2015). Patterns and trends in
racial/ethnic and socioeconomic academic achievement gaps. In H. F. Ladd & M. E.
Goertz (Eds.), Handbook of research in education finance and policy (2nd ed., pp. 491509). Routledge.
Reardon, S. F., & Townsend, J. B. (2018). SEG: Stata module to compute multiple-groupdiversity and segregation indices [Statistical software module]. EconPapers.
https://econpapers.repec.org/software/bocbocode/s375001.htm
Rideout, V., & Robb, M. B. (2019). The Common Sense census: Media use by tweens and teens,2019. Common Sense Media. https://www.commonsensemedia.org/research/thecommon-sense-census-media-use-by-tweens-and-teens-2019
Simon, H. A. (1971). Designing organizations for an information-rich world. In M. Greenberger(Ed.), Computers, communications, and the public interest (pp. 38-72). The Johns
Hopkins Press.
https://digitalcollections.library.cmu.edu/awweb/awarchive?type=file&item=33748
Sullivan, M. C. (2019). Why librarians can’t fight fake news. Journal of Librarianship andInformation Science, 51(4), 1146-1156. https://doi.org/10.1177/0961000618764258

STUDENTS’ CIVIC ONLINE REASONING

30

Walraven, A., Brand-Gruwel, S., & Boshuizen, H. P. A. (2009). How students evaluate
information and sources when searching the World Wide Web for information.
Computers & Education, 52(1), 234–246. https://doi.org/10.1016/j.compedu.2008.08.003
Warzel, C. (2021, February 18). Don’t go down the rabbit hole. The New York Times.
https://www.nytimes.com/2021/02/18/opinion/fake-news-media-attention.html
Wineburg, S., Breakstone, J., Ziv, N., & Smith, M. (2020). Educating for misunderstanding:
How approaches to teaching digital literacy make students susceptible to scammers,rogues, bad actors, and hate mongers (Working Paper A-21322), Stanford History
Education Group. https://purl.stanford.edu/mf412bt5333
Wineburg, S., & McGrew, S. (2017). Lateral reading: Reading less and learning more whenevaluating digital information (Working Paper No. 2017-A1), Stanford History
Education Group. http://dx.doi.org/10.2139/ssrn.3048994
Wineburg, S., & McGrew, S. (2019). Lateral reading and the nature of expertise: Reading lessand learning more when evaluating digital information. Teachers College Record,
121(11), 1-40.
Wineburg, S., McGrew, S., Breakstone, J., & Ortega, T. (2016). Evaluating information: The
cornerstone of Civic Online Reasoning (Working Paper No. 2016-A1), Stanford History
Education Group. http://purl.stanford.edu/fv751yt5934
Wineburg, S., & Ziv, N. (2019, December 5). The meaninglessness of the .org domain. The New
York Times. https://www.nytimes.com/2019/12/05/opinion/dot-org-domain.html
Wineburg, S., & Ziv, N. (2020, November 6). Why can’t a generation that grew up online spot
the misinformation in front of them? Los Angeles Times.

STUDENTS’ CIVIC ONLINE REASONING
https://www.latimes.com/opinion/story/2020-11-06/colleges-students-recognizemisinformation

31

STUDENTS’ CIVIC ONLINE REASONING
Appendix A
Description of Tasks on COR Assessment
Item
COR Question(s) Addressed
Description
1. Evaluating
What’s the evidence?
Evaluate whether a video posted on
Video Evidence
Who’s behind the information? Facebook is good evidence of voter
fraud.
2. Webpage
Comparison

Who’s behind the information? Explain which of two webpages is abetter source of information on gun
control.

3. Article
Evaluation

What do other sources say?
Using any online sources, explain
Who’s behind the information? whether a website is a reliable
source of information about global
warming.

4. Claims on
Social Media 1

What’s the evidence?
Explain why a social media post is a
Who’s behind the information? useful source of information aboutbackground checks for firearms.

5. Claims on
Social Media 2

What’s the evidence?
Explain how a social media post
Who’s behind the information? about background checks might not
be a useful source of information.

6. Home Page
Analysis

Who’s behind the information? Explain whether tiles on the
homepage of a website areadvertisements or news stories.

32

STUDENTS’ CIVIC ONLINE REASONING

33

Appendix B

Students’ Online Reasoning Study
You are invited to participate in a research project that looks at how people evaluate online
sources. Findings from this project will help us create resources to teach students how tonavigate and evaluate information on the Internet.
Completing the tasks will take about 30-40 minutes. Your answers are completely anonymous.
Your answers will not affect your grade in this course in any way.
On the following pages you will be presented with five tasks. Please keep the following in mind
as you complete each one:
• You may open a new tab and complete an Internet search to help you answer any of thequestions.
• Once you advance, you will not be able to return to previous pages. Your browser's backbutton won't work either.
Thank you for participating!

STUDENTS’ CIVIC ONLINE REASONING

34

Task 1 of 5
Below is a screenshot of a Facebook post from October 2016. Please watch the video
posted below the screenshot.

You can watch the video from the post here: (Note, there is no audio.)

Question: Does the Facebook post provide strong evidence of voter fraud during the2016 Democratic primary election? (The primary was the election to decide who would be
the Democratic candidate for president.)
• Yes
• No
Explain your reasoning.
Were you able to view the video above? (Note: The video does not have sound.)
• Yes
• No

STUDENTS’ CIVIC ONLINE REASONING

35

Task 2 of 5
Imagine you are doing research on gun control and you find the following webpages.
Because webpages can change frequently, these links will take you to archived versionsof the articles. Please do not consider that in your evaluations.
• http://web.archive.org/web/20161205141632/http://people.duke.edu/~gnsmith/articles/myths.htm• http://web.archive.org/web/20170222133426/https://en.wikipedia.org/wiki/Gun_politics_in_the_
United_States
Question: Which of these two webpages do you think is a better place to start yourresearch?
• http://people.duke.edu/~gnsmith/articles/myths.htm• https://en.wikipedia.org/wiki/Gun_politics_in_the_United_States
Explain your choice. Refer to both webpages in your answer.

Were you able to access both links about gun control?
• Yes
• No

STUDENTS’ CIVIC ONLINE REASONING

Task 3 of 5
Visit the following webpage: http://www.co2science.org/
Question: Is this a reliable source of information about global warming? You may use
any information on this website, or you can open a new tab and do an Internet search if
you want.
• Yes
• No
Explain your answer, citing evidence from the webpages you used. Be sure to provide
the URLs to the webpages you cite.

Were you able to access the above link about climate science?
• Yes
• No

36

STUDENTS’ CIVIC ONLINE REASONING

37

Task 4 of 5
The following tweet appears in your Twitter feed (please click on the link to view):
https://twitter.com/MoveOn/status/666772893846675456?lang=en
Question: Why might this tweet be a useful source about NRA members' opinions on
background checks? List any sources you used to make your decision.

Question: Why might this tweet not be a useful source about NRA members' opinions on
background checks? List any sources you used to make your decision.

Were you able to view the tweet above?
• Yes
• No

STUDENTS’ CIVIC ONLINE REASONING

Task 5 of 5
Here is the home page of Slate.com. Some of the things that appear on Slate.com arenews stories, and others are advertisements.

Question: Is #1 an advertisement?
• Yes
• No
Explain your reasoning.
Question: Is #2 an advertisement?
• Yes
• No
Explain your reasoning.
Question: Is #3 an advertisement?
• Yes
• No
Explain your reasoning.

38

39

STUDENTS’ CIVIC ONLINE REASONING

Additional Questions
On a scale of 0-10, how seriously did you take these tasks? "0" indicates that you did nottake the tasks seriously at all. "5" indicates that you took the tasks fairly seriously. "10"indicates that you took the tasks very seriously.

1

2

3

4

5

6

7

8

9

10

Thank you for completing this exercise! Please take a few minutes to answer the questions below.
Your answers to these questions help us learn about the students who completed the exercises.
All of your responses are still anonymous.
Please select your school district.
Please select your school.

In which class you are completing
this survey?
• US/American History I• US/American History II• World History• AP US History• AP US Government & Politics• Other (please enter class nameIs this class an Honors or
IB class?
• Yes, Honors• Yes, IB• No

How old are you?

What is your gender?

•
•
•
•
•

12 or
younger
13
14
15
16

•
•
•
•

17
18
19
20 or older

•
•
•
•
•

What is your race? Mark one of moreboxes
• American Indian or Alaska Native• Asian• Black or African American• Native Hawaiian or Other PacificIslander• WhiteFemaleMale
Non-binary
Prefer to self-describe
Prefer not to say

What grade are you in?
•
•
•
•
•

8th grade or below
9th
10th
11th
12th

Are you Hispanic or
Latino?
• Yes• No• Prefer not to say

What is the highest degree or level ofschool that your mother completed?
• Less than high school• High school• Some college• Bachelor’s degree• Advanced degree (e.g., MA, JD, MD,
PhD).

40

STUDENTS’ CIVIC ONLINE REASONING
•
•

Other
Prefer not to say••

I don’t know
Prefer not to say

What is the language(s) most used in your home? Drag and drop up to three languages
from the left column to the right box. Then rank them in order of how much they're used inyour home.
• English• Arabic• Burmese• Cambodian• Cantonese• Cape Verdean Creole• French• Greek• Hmong• Haitian-Creole• Italian• Korean• Mandarin• Portuguese• Russian• Somali• Spanish• Toishanese• Vietnamese• Other (Please specify):• Prefer not to say

What is the zip code where you live? (If you live in multiple zip codes, choose the one inwhich you spend more time.)

Are you eligible for free or reduced-price lunch at school?
• Yes
• No
• I don’t know.
• Prefer not to say
Which best describes the grades you usually earn on your report card?
• Mostly A’s

STUDENTS’ CIVIC ONLINE REASONING
•
•
•
•
•
•
•

Mostly A’s and B’s
Mostly B’s
Mostly B’s and C’s
Mostly C’s
Mostly C’s and D’s
Mostly D’s
Prefer not to say

41

STUDENTS’ CIVIC ONLINE REASONING

42

Appendix C
Task 1: Evaluating Evidence Rubric
This assessment asks students to determine whether a video posted on Facebook provides strong
evidence of voter fraud in the 2016 election. Successful students will conclude that the video is
not strong evidence and provide a clear explanation why it isn’t. Successful explanations canfocus on a variety factors, including: (1) there is no evidence that the video was actually shot
where the poster claims it was; (2) nothing is known about the identity of the person who posted
the video (I on Flicks) and they are likely not a reliable source of information; (3) after doing
research on the video, it is clear that the video is not what the poster claims it is; or (4) after
researching the identity of the person who posted the video, it is clear that “I on Flicks” may not
be a reliable source of information.

(2) MASTERY

Student rejects the video as strong evidence and does at least one of thefollowing:• Questions who posted the video• Questions the source/location of the video• Investigates who posted the video and concludes they are unreliable• Investigates the video and determines the recordings are not from the2016 Democratic primaryStudent questions the content of the video rather than the source,
implying or stating that an unedited video would be automaticallyreliableOR(1) EMERGING

Student questions the source of the post/video, but also does one of thefollowing:• Waffles/straddles the fence, both accepting the video as goodevidence but also raising relevant concerns• States incorrect reasoning for why the post/video is unreliable
Student rejects the video but uses completely incorrect or incoherent
reasoning.

(0) BEGINNING

OR
Student concludes the post does provide strong evidence.

STUDENTS’ CIVIC ONLINE REASONING

43

Appendix D
Regression Model Demographic Variables
Race/ethnicity: We included the composite race/ethnicity variable in the multivariate models.
Students who did not answer the question or who answered “prefer not to say” were notincluded.
Gender: We included a categorical measure of gender with three categories: female, male, and athird category that combined nonbinary and “prefer to self-describe” response options.
Individuals who did not answer the question or answered “prefer not to say” were not included in
the multivariate models.
Free/reduced lunch status: FRL status was a binary indicator (eligible/participating versus not
eligible/not-participating). Students who did not answer the question or who responded “I don’tknow” or “prefer not to say” were not included in the multivariate models.
Mother’s education: Student reports on their mother’s highest level of education were coded 1(less than a high school education) to 5 (advanced degree). Students who did not answer thequestion or who responded “I don’t know” or "prefer not to say” were not included in themultivariate models.
Home language: The assessments were in English, so students for whom English was not a first
language may have had more difficulty interpreting the sources and conveying their thinking in
English. As a result, we included a categorical measure of home language (English versus not
English) based on the students’ response about the language(s) spoken at home. Students who
did not answer the question were not included in the multivariate models.

44

STUDENTS’ CIVIC ONLINE REASONING
Appendix E
Descriptive Results Tables
Table E1
Percent Scoring Emerging or Mastery by Locale for Tasks 1 to 5
LocaleN

Evaluating
Evidence %Rural
Suburban
Urban
Total

820
1,659
967
3,446

20.9%
20.9%
27.3%
3,119

Webpage
Comparison%

WebsiteEvaluation
%

4.2%
7.9%
11.2%
2,678

2.6%
2.7%
4.6%
2,994

Claims on
Social Media
1
%
5.0%
6.8%
10.6%
2,431

Claims on
SocialMedia 2
%
2.3%
3.7%
9.3%
2,431

Home Page
Analysis
%
23.1%
31.0%
39.4%
2,969

Table E2
Percent Scoring Emerging or Mastery by Region for Tasks 1 to 5
RegionN

Evaluating
Evidence %Webpage
Comparison %WebsiteEvaluation %Midwest
Northeast
South
West
Total

852
732
1,084
778
3,446

16.4%
15.3%
27.6%
29.2%
3,119

4.5%
5.2%
10.4%
10.1%
2,678

1.7%
2.3%
4.2%
3.9%
2,994

Claims on
SocialMedia 1
%
4.0%
4.2%
10.0%
5.4%
2,431

Claims on
Social Media
2
%
2.5%
1.1%
7.2%
3.7%
2,431

Home Page
Analysis %
26.9%
22.4%
37.1%
36.4%
2,969

Table E3
Percent Scoring Emerging or Mastery by Grade for Tasks 1 to 5

Grade
8th Grade
9th Grade10th Grade
11th Grade
12th Grade
Missing
Total (N)

N
17
820
491
867
761
490
3,446

Evaluating
Evidence %Webpage
Comparison %WebsiteEvaluation %

14.3%
15.1%
19.8%
26.7%
29.2%
20.0%
3,119

0.0%
5.2%
5.8%
9.6%
11.8%
4.9%
2,678

0.0%
1.3%
1.5%
3.6%
5.6%
3.8%
2,994

Claims on
SocialMedia 1
%
0.0%
4.2%
6.0%
7.2%
12.1%
5.5%
2,431

Claims on
SocialMedia 2
%
0.0%
1.5%
4.3%
5.5%
8.0%
1.1%
2,431

Home Page
Analysis %
13.3%
24.4%
25.7%
36.1%
38.8%
10.9%
2,969

45

STUDENTS’ CIVIC ONLINE REASONING

Table E4
Percent Scoring Emerging or Mastery by Race/Ethnicity and Task

Race/Ethnicity

N

Eval
Evidence%

Asian/Pacific Islander
American Indian/Alaska
NativeBlack or African
AmericanHispanic
Other
Two or more races
White
Prefer not to say
Missing
Total

112

33.6%

22.8%

5.4%

Claims on
SocialMedia 1
%
16.7%

29

17.9%

3.8%

0.0%

11.5%

3.8%

31.0%

306
698
42
224
1,373
28
634
3,446

11.0%
17.6%
13.5%
29.5%
27.8%
19.2%
18.0%
3,119

2.8%
3.9%
6.7%
9.0%
11.5%
10.0%
3.6%
2,678

0.0%
0.9%
2.7%
3.2%
5.0%
7.4%
2.2%
2,994

2.2%
3.9%
6.5%
6.5%
9.7%
8.7%
3.6%
2,431

1.3%
3.2%
9.4%
6.0%
6.1%
4.3%
0.5%
2,431

17.6%
24.7%
31.0%
32.6%
38.3%
37.0%
17.0%
2,969

Webpage
Comparison %WebsiteEvaluation%

Claims on
SocialMedia 2
%
7.2%

Home
Page
Analysis%
43.8%

Table E5
Percent Scoring Emerging or Mastery by Average Grades for Tasks 1 to 5
GradesMostly As
Mostly As and Bs
Mostly Bs
Mostly Bs and Cs
Mostly Cs
Mostly Cs and Ds
Mostly Ds
Prefer not to say
Missing
Total

N

Evaluating
Evidence%

Webpage
Comparison%

WebsiteEvaluation%

Claims on
SocialMedia 1 %

794
1,116
268
445
108
77
33
101
504
3,446

38.9%
21.4%
16.4%
11.1%
12.9%
10.8%
7.7%
7.8%
19.5%
3,119

14.7%
6.6%
6.9%
3.7%
3.5%
10.2%
0.0%
3.6%
5.0%
2,678

6.5%
2.4%
2.7%
1.0%
0.0%
0.0%
0.0%
1.1%
3.5%
2,994

13.8%
6.4%
4.4%
2.8%
2.8%
1.8%
0.0%
1.4%
4.9%
2,431

Claims on
SocialMedia 2
%
10.0%
3.5%
3.8%
1.1%
1.4%
1.8%
0.0%
0.0%
1.0%
2,431

Home Page
Analysis %
43.1%
32.1%
24.8%
21.1%
24.5%
23.4%
26.7%
20.0%
10.7%
2,969

46

STUDENTS’ CIVIC ONLINE REASONING
Table E6
Percent Scoring Emerging or Mastery by Gender for Tasks 1 to 5
GenderFemaleMale
Non-binary
Prefer to selfdescribe
Prefer not tosayMissing
Total

N

Evaluating
Evidence %Webpage
Comparison%

WebsiteEvaluation%

Claims on
Social Media 1%

Claims on
Social Media 2%Home Page
Analysis %

1,443
1,283
28

24.5%
22.2%
20.0%

7.1%
10.2%
5.0%

2.7%
4.0%
9.1%

7.1%
7.8%
0.0%

4.3%
5.7%
5.3%

31.8%
32.0%
46.2%

144

15.8%

1.8%

0.0%

5.3%

2.2%

25.4%

58
490
3,446

15.8%
20.0%
3,119

1.8%
4.9%
2,678

0.0%
3.8%
2,994

5.3%
5.5%
2,431

2.2%
1.1%
2,431

25.4%
10.9%
2,969

Table E7
Percent Scoring Emerging or Mastery by National School Lunch Program Eligibility for Tasks 1 to 5
Claims on
Evaluating
Webpage
WebsiteClaims on
Eligible for School
SocialN
Evidence
Comparison Evaluation
SocialLunch Program
Media 1%%%
Media 2 %
%
Yes
850
16.0%
4.4%
0.5%
3.7%
0.9%
No
1,475
29.7%
11.5%
5.3%
10.8%
8.6%
I don’t know
533
16.8%
6.1%
1.6%
4.2%
1.2%
Prefer not to say
84
13.9%
2.7%
0.0%
3.3%
0.0%
Missing
504
19.5%
5.0%
3.5%
4.9%
1.0%
Total
3,446
3,119
2,678
2,994
2,431
2,431

Home Page
Analysis
%
21.1%
39.9%
28.9%
16.0%
10.7%
2,969

Table E8
Percent Scoring Emerging or Mastery by Mother's Education for Tasks 1 to 5
Highest degree/level of
education mothercompleted?
Less than high school
High schoolSome college
Bachelors degree
Advanced degree
I don’t know
Prefer not to say
Missing
Total

N

Evaluating
Evidence%

Webpage
Comparison%

WebsiteEvaluation%

Claims on
SocialMedia 1 %

222
506
460
736
473
411
148
490
3,446

20.3%
14.5%
22.8%
28.1%
34.6%
17.4%
8.1%
20.0%
3,119

5.4%
3.7%
9.2%
11.4%
13.4%
5.1%
3.2%
4.9%
2,678

1.4%
0.4%
2.3%
4.5%
7.7%
1.3%
0.7%
3.8%
2,994

5.9%
3.6%
4.9%
9.5%
15.0%
3.6%
1.9%
5.5%
2,431

Claims on
SocialMedia 2
%
3.5%
1.3%
3.3%
7.2%
11.7%
0.3%
0.0%
1.1%
2,431

Home Page
Analysis
%
25.6%
21.9%
29.5%
39.5%
43.6%
25.7%
21.5%
10.9%
2,969

47

STUDENTS’ CIVIC ONLINE REASONING
Table E9
Percent Scoring Emerging or Mastery by Home Language for Tasks 1 to 5
Home
Language
EnglishSpanishOther
Missing
Total

N

Evaluating
Evidence %Webpage
Comparison%

WebsiteEvaluation%

Claims on
SocialMedia 1 %

2,425
302
219
500
3,446

24.8%
13.1%
17.8%
19.5%
3,119

8.9%
2.6%
9.1%
4.7%
2,678

3.6%
0.3%
2.0%
3.6%
2,994

8.0%
2.1%
6.3%
5.2%
2,431

Claims on
SocialMedia 2
%
5.4%
0.4%
3.2%
1.1%
2,431

Home Page
Analysis %
33.7%
21.4%
24.0%
11.1%
2,969

STUDENTS’ CIVIC ONLINE REASONING

48

Appendix F
Distribution of the Composite Scores
Composite Score

N

%

0.0
1,974
58.0%
0.17 to 0.49
966
28.4%
0.5 to 0.99
351
10.3%
1.0 to 1.49
88
2.6%
1.5 to 1.99
10
0.29%
2.0
13
0.38%
a
100%
Total
3,402
a
Forty-four students in the analytic sample dropped out of the composite score analysis because the students indicatedtechnical problems or provided nonsensical answers.