1

Lateral Reading and the Nature of Expertise:
Reading Less and Learning More When Evaluating Digital Information
Sam Wineburg & Sarah McGrewin press, Teachers College Record
July 28, 2018

Sam Wineburg
wineburg@stanford.edu
485 Lasuen Mall, Stanford CA 94305
Sam Wineburg is the Margaret Jacks Professor of Education and (by courtesy) History at
Stanford University. His latest book is Why Learn History (When It’s Already on Your Phone)published by the University of Chicago Press.
Sarah McGrew
smcgrew@stanford.edu
485 Lasuen Mall, Stanford CA 94305
Sarah McGrew is a doctoral candidate at the Stanford Graduate School of Education. She studies
how young people reason about online information and how teachers can support students tothoughtfully engage with digital content. Her recent publications include “Can Students Evaluate
Online Sources? Learning from Assessments of Civic Online Reasoning” in Theory & Researchin Social Education.
1 or 2 sentence description of article: This study compares how professional fact checkers,
historians, and first year college students evaluated online information and presents the strategies
fact checkers used to efficiently and effectively find trustworthy information.

READING LESS AND LEARNING MORE
Structured Abstract [200 to 400 words]
Background/Context: The Internet has democratized access to information but in so doing has
opened the floodgates to misinformation, fake news, and rank propaganda masquerading as
dispassionate analysis. Despite mounting attention to the problem of online misinformation andgrowing agreement that digital literacy efforts are important, prior research offers few concrete
ideas about what skilled evaluations look like.
Purpose/Objective/Research Question/Focus of Study: Our purpose in this study was to seekout those who are skilled in online evaluations in order to understand how their strategies andapproaches to evaluating digital content might inform educational efforts. We sampled 45
experienced users of the Internet: 10 Ph.D. historians, 10 professional fact checkers, and 25
Stanford University undergraduates. Analysis focused on the strategies participants used toevaluate online information and arrive at judgments of credibility.
Research Design: In this expert/novice study, participants thought aloud as they evaluated live
websites and searched for information on social and political issues such as bullying, minimum
wage, and teacher tenure. We analyze and present findings from three of the tasks participants
completed.
Findings/Results: Historians and students often fell victim to easily manipulated features of
websites, such as official-looking logos and domain names. They read vertically, staying withina website to evaluate its reliability. In contrast, fact checkers read laterally, leaving a site after aquick scan and opening up new browser tabs in order to judge the credibility of the original site.
Compared to the other groups, fact checkers arrived at more warranted conclusions in a fractionof the time.
Conclusions/Recommendations: We draw on insights gleaned from the fact checkers’
practices to examine current curricular approaches to teaching web credibility as well as tosuggest alternatives.

2

READING LESS AND LEARNING MORE

3

Executive Summary
BACKGROUND
The Internet has democratized access to information but in so doing has opened the floodgates tomisinformation, fake news, and rank propaganda masquerading as dispassionate analysis.
Despite mounting attention to the problem of online misinformation and growing agreement that
digital literacy efforts are important, prior research offers few concrete ideas about what skilled
evaluations look like.
FOCUS
Our work focused on civic online reasoning, a subset of the broader fields of digital and media
literacy, whose broad goals include everything from how to protect one’s digital footprint tolearning how to write code. We define civic online reasoning as the ability to search for,evaluate, and verify online information related to social and political issues; proficiency inonline reasoning is necessary to become informed about the social and political choices citizens
face.
RESEARCH QUESTIONS
Two questions guided our inquiry: How do experienced users of the Internet arrive at judgmentsof credibility as they evaluate unfamiliar sites and investigate questions of social and politicalimport? And, what strategies and heuristics do they use to locate online information andestablish its trustworthiness?
SAMPLE AND METHODS
We sampled 45 experienced users of the Internet: 10 Ph.D. historians, 10 professional fact
checkers, and 25 Stanford University undergraduates. Historians were selected because priorresearch has documented their ability to interrogate sources to establish reliability andtrustworthiness. Because much historical work still takes place with print sources, we also
sampled professional fact checkers, whose work is done sitting before a computer screen.
Finally, we recruited a group of talented undergraduates, who attend a university where
technology startups sprout within campus labs and where computer science is the mostpopular major on campus. In live Internet sessions, participants completed a series of webbased tasks about issues such as minimum wage policy, school-based bullying, and teacher
tenure. QuickTime Player was used for screen captures, recording all mouse movements,
clicks, and websites visited. Using think-aloud methodology, participants verbalized theirthoughts as they engaged in web searching.
RESULTS    READING LESS AND LEARNING MORE

4

Historians and students often fell victim to easily manipulated features of websites, such as
official-looking logos and domain names. Fact checkers, on the other hand, engaged in threepractices—taking bearing, lateral reading and click restraint—that allowed them to read less but
learn more about the topics they investigated. To take bearings, searchers obeyed the following
imperative: before diving deeply into unfamiliar content, chart a plan for moving forward.
Rather than staying on a single website to evaluate its reliability, lateral reading involved
leaving a site after a quick scan, opening up new browser tabs along the screen’s horizontal axis
to judge the credibility of the original site. Click restraint stands in contrast to whimsicalclicking; before clicking on any one result, users evaluated the list of search results to understandthe digital terrain in which they have landed. Only then did they initiate a search.
IMPLICATIONS
Civic online reasoning demands reading practices that differ from those developed to read
traditional print sources. Staying on a single webpage and carefully parsing its content, before
one has even established whether a site is what it says it is, is a woefully inefficient way tonavigate treacherous digital territory. We need to teach students a set of flexible strategies thatwill eliminate some of their most common errors. Educators can begin this effort in a separate
course on digital literacy, but the work cannot stop there. An age when information comes to us
via a computer screen demands a fundamental reconsideration of how we teach all of the coreschool subjects.

    READING LESS AND LEARNING MORE
The 2016 election cycle made it clear that more people are relying on the Internet for
information—often with little idea about how to judge what they find there. Among the false
stories that swirled during election season was “Pizzagate,” which alleged that Hillary Clinton
was involved in a child sex trafficking ring that operated from a pizza shop in Washington, DC
Rumors spread on Facebook and Twitter; 4chan and Reddit users investigated purported“evidence” about the scandal; and websites like InfoWars and Breitbart ran lurid stories. The
combination of purposeful and haphazard spread of mis- and disinformation resulted in realconsequences: On December 4, 2016, Edgar Maddison Welch entered Comet Ping Pong
restaurant to investigate and rescue the sex trafficking victims he expected to find. Policearrested him after he fired three rifle shots (Robb, 2017).
Educators and policy makers have responded to growing fears about online
misinformation with a potential solution: media literacy. Four states have already passed laws
mandating media literacy education, and other states are on their way (Foley, 2017). But even if
states mandate media literacy, we still face a thorny problem— what, exactly, to teach.
There is no shortage of resources for instruction in how to evaluate online information.
Most share something in common: they focus students’ attention on a website’s internalfeatures. Was the site recently updated? Are there banner ads? Are the author’s credentials
provided? Is the language free of emotion? Yet, any organization with a competent web
designer and a modicum of digital savvy can design a site that aces these questions, whether
the organization is a trusted purveyor of information or not. Web development and design
have outpaced our methods for evaluation (Wineburg, Breakstone, McGrew, & Ortega, 2016;
Wineburg & McGrew, 2016). Despite mounting agreement that media literacy is important,
  
  

5

READING LESS AND LEARNING MORE
we have few concrete ideas about what skilled evaluations look like. Our purpose in this study
was to seek out those who are skilled in online evaluations in order to understand how theirapproaches to evaluating digital content might inform educational efforts.
Students’ Online Evaluations
Students often rely on the Internet for information about the world, but they struggle toevaluate the information they find there (Bennett, 2012; Gasser, Cortesi, Malik, & Lee, 2012).
In one of the most extensive think-aloud studies to date, Hargittai, Fullerton, MenchenTrevino, and Thomas (2010) observed 102 college students as they searched online. Students
overwhelmingly ceded to Google the responsibility for determining the credibility of
information—the higher up a site ranked in Google’s results, the more reliable students
considered it to be. Another study found that undergraduates ignored the valuable information
contained in Google’s snippets (the few sentences accompanying each search result) andclicked instead on websites in higher positions even when those websites were less relevant(Pan et al., 2007).
Students do no better when they turn to evaluating websites. A study of nearly 8,000
responses to exercises about web credibility revealed widespread problems among middle
school, high school, and college students. Students could not distinguish between traditional
news and sponsored content (advertisements), they rarely evaluated sources (sponsoringorganizations or authors) and were instead swayed by content that appeared to present strong
evidence (in the form of photographs, data displays, etc.), and they judged websites based onsuperficial features such as their graphic design or how authoritative their logo or references
made them seem (McGrew, Breakstone, Ortega, Smith, & Wineburg, 2018; Wineburg,
  
  

6

READING LESS AND LEARNING MORE

7

Breakstone, McGrew, & Ortega, 2016).
Wiley et al. (2009) found that college students rarely considered where information camefrom when evaluating reliability, a finding replicated across a range of studies and students ofdifferent ages and in different countries (e.g., Barzilai & Zohar, 2012; List, Grossnickle, &Alexander, 2016; Walraven, Brand-Gruwel & Boshuizen, 2009). Instead, young people judge a
website based on its relevance to their searching needs (Iding, Crosby, Auernheimer, & Klemm,
2009; Julien & Barker, 2009; Walraven et al., 2009) and its appearance (Agosto, 2002; Barzilai& Zohar, 2012).
Extant research suggests that young people struggle with many aspects of finding
information online—from selecting search results to judging whether a site is trustworthy. What,then, might a more sophisticated approach look like?
Frameworks of Credibility
Since the early days of the web, scholars have created models for how people search forand evaluate information. Two early frameworks posited that prominence of information (howlikely information was to be noticed) played a primary role in users’ evaluations. Information
foraging theory (Pirolli & Card, 1995) argued that people who search for information are likeanimals searching for their next source of food. Information’s prominence (i.e., its placement andaccessibility) and its relevance (how closely digital content matches what the user is searchingfor) drive decisions about whether to use it. Fogg’s (2003) prominence-interpretation theory
extended the importance of prominence to judgments of websites themselves: Fogg argued that
evaluations could be explained based on whether a user noticed an element of a website (due toits prominence) and what judgment the user made based on that element (interpretation).
    READING LESS AND LEARNING MORE

8

Subsequent frameworks argued that users base evaluations on surface features, such aslength, references, and writing style (Lucassen & Schraagen, 2011) or genre and level offamiliarity (Hilligoss & Rieh, 2008). Users also judge the actual contents of the information;
they leverage their knowledge of the topic to check whether it is accurate and relevant to theirneeds. Finally, users judge the credibility of the source itself, deciding whether the source has
expertise and is trustworthy (Hilligoss & Rieh, 2008; Lucassen & Schraagen, 2011; Wathen &Burkell, 2002). Whether the provider of information is deemed reliable is not necessarily more
important than other considerations in these models.
Conceptualizing Expertise in Civic Online Reasoning
These models of credibility judgments share one major shortcoming: They describe what
average users do instead of investigating expert practice. Studies of what skilled users do are lesscommon. Lucassen and Schraagen (2011) studied people active on a car enthusiasts’ forum as aproxy for expert knowledge about car engines. Unsurprisingly, people who knew more aboutcars were better able to detect errors in Wikipedia than those who knew less. Similarly, a groupof Dutch researchers compared psychology students and psychology faculty as they selected
online sources on psychological topics; faculty spent more time scanning search results whilestudents made more superficial evaluations (Brand-Gruwel, Kammerer, van Meeuwen, & vanGog, 2017). In another study, researchers designated a group of graduate students in educational
technology as “experts” and compared their online research processes with those of university
freshmen (“novices”) (Brand-Gruwel, Wopereis, & Vermetten, 2005). But the authors provided
few clues about how experts went about selecting and evaluating information.

    READING LESS AND LEARNING MORE

9

We set out to purposefully sample skilled Internet users as they evaluated digital sources
about social and political issues. The Internet is reshaping participatory politics, changing how
we learn about politics, communicate with our representatives in government, and organize
political protests. In response, civic education should, among other goals, prepare students to“analyze and evaluate information in order to learn about and investigate pressing civic andpolitical issues” (Kahne, Hodgin, & Eidman-Aadahl, 2016, p. 9). Civic online reasoning(McGrew, Breakstone, Ortega, Smith, & Wineburg, 2018) encompasses the ability toeffectively search for, evaluate, and verify social and political information online; proficiency inonline reasoning is necessary for citizens to use the Internet to inform their social and politicalchoices. Civic online reasoning can be considered a subset of the wider fields of digital andmedia literacy, whose sweeping goals can include everything from how to protect one’s online
identity to learning to code or produce digital videos (e.g., Common Sense Media, n.d.; Mozilla,
n.d., National Association for Media Literacy Education, 2007).
The present work set out to understand in greater detail what experts do when judging
social and political information online. Before we could tackle this issue, we needed to figureout who qualified as an expert.
We turned to a group of professionals who evaluate sources for a living: historians.
Ample research has established how historians source documents, interrogating a document’s
author and the circumstances of its creation as keys to determining its trustworthiness (Leinhardt& Young, 1996; Shanahan & Shanahan, 2008; Wineburg, 1991, 1998). Shanahan, Shanahan, andMisischia (2011) found wide variations in sourcing among academics from different fields.
While mathematicians explicitly ignored the author of a paper, as it “would only be a distraction    READING LESS AND LEARNING MORE

10

and could help in no way with the process of making sense of the text,” historians engaged in“extensive sourcing,” speculating about “who the author was and what he or she represented”(2011, pp. 408-409).
Despite the growth of digital history, the majority of historians still conduct their researchin archives of print documents (Dougherty & Nawrotzki, 2013). We thus set out to study asecond group whose work is largely done on a computer screen: fact checkers, whose job it is toascertain truth in digital form. These professionals are charged with evaluating claims andevidence and spend much of their time vetting digital information.
Finally, we recruited a third group: undergraduates at Stanford University. In 2016,
Stanford rejected 95% of its applicants, making it the most competitive university in the
United States (Anderson, 2016). Nearly all admitted students were in the top 10% of their highschool classes and scored above the 90th percentile on the SAT (Stanford University, 2015).
These young people attended a university in the heart of Silicon Valley, where technology
startups sprout within campus labs and where computer science is the most popular major(Stanford University, 2017). These students are earmarked—at least according to Stanford
University brochures—to lead the digital future.
MethodThe purpose of this study was to investigate how experienced Internet users arrive at
judgments of trustworthiness. We asked two main research questions: How do experienced usersof the Internet arrive at judgments of credibility as they evaluate unfamiliar sites and investigate
questions of social and political import? What strategies or heuristics do they use to efficiently
find reliable information? Based on other expert/novice research (Ericsson, Chraness, Feltovich,    READING LESS AND LEARNING MORE

11

& Hoffman, 2006), our hope in studying expertise was to distill a set of practices that can informthe development of curricula and assessment.
ParticipantsHistorians. Ten historians were recruited; all held a Ph.D. in history and were facultyat five different four-year colleges and universities in California and Washington state. Six
were male; four were female. Their ages ranged from 39 to 69 (M = 47). [insert Footnote 1
here]
Fact Checkers. The fact checkers were all employed at well-regarded news andpolitical fact-checking organizations. Eight were located in New York City or Washington,
DC; two were based on the West Coast. As with the historians, six were male and four female.
Ages ranged from 23 to 60 (M = 34). Two participants held master’s degrees while one held aPh.D.; the rest had bachelor’s degrees.
College Students. Students were recruited using fliers posted on the Stanford campus.
Each received a $25 Amazon gift card for participating. All students were enrolled in thesecond or third quarter of their first year and were between the ages of 18 and 19; 11
identified as male, 13 as female, and one as non-binary. Every student reported spending atleast four hours online each day.
ProtocolWe developed a set of six online tasks that took approximately 45 minutes tocomplete. Our focus was on evaluating digital sources that addressed social and politicalissues. Space limitations require that we narrow our discussion here to three of the main tasks
participants completed (see Table 1). [insert Footnote 2]    READING LESS AND LEARNING MORE

  
  

12

READING LESS AND LEARNING MORE

13

Table 1Main Web Evaluations
Topic
Bullying in schools
URLs:
https://www.acpeds.org/th
e-college-speaks/positionstatements/societalissues/bullying-at-schoolnever-acceptable

Processes
Elicited

Participants could

Evaluations internal andexternal to a site;
comparing sites

Scroll, click on links, andleave the site to accessany information online
Time Limit: 10 minutes

Minimum wage policy
URL:https://www.minimu
mwage.com/2014/10/denmarks-dollar-forty-onemenu/

Evaluations internal andexternal to a site

Scroll, click on links, andleave the site to accessany information online
Time Limit: 5 minutes

Teacher tenure: Funding
for plaintiffs in Vergara
v. California

Open web search to findout who paid for the$1.2 million legal fees

Access any information
onlineTime Limit: 5 minutes

https://www.aap.org/enus/about-the-aap/aappress-room/pages/StigmaAt-the-Root-of-Ostracismand-Bullying.aspx

    READING LESS AND LEARNING MORE
Procedure
Sessions with historians and fact checkers were conducted by the authors; sessions
with students were conducted by one of the authors and other members of the research team.
Participants were asked to complete a series of web-based tasks on a 13-inch MacBook Air.
Websites were live, and participants were able to search the Internet as they normally do—clicking on links, opening new tabs, and leaving a site to search elsewhere. We employed athink-aloud methodology (Ericsson & Simon, 1993; Pressley & Afflerbach, 1995) that has
been used to understand the nature of expertise in a wide variety of fields, including thethought processes of literary critics (Warren, 2011), mathematicians (Shanahan & Shanahan,2008), historians (Wineburg, 1991), and physicians making clinical judgements (Elstein,
Kagan, Shulman, Jason, & Loupe, 1972). At the beginning of the interview, the think-aloudprocedure was explained: participants were told to say everything that came to mind as they
completed the tasks. After introducing each task, we refrained from speaking unless theparticipant fell completely silent. In that case, questions like, “What are you thinking?” wereused to encourage participants to verbalize their thoughts. Meta-analyses of think-aloudprotocols show that they do not cause extensive disruption in the kinds of thought processes inwhich participants engage (Taylor & Dionne, 2000).
QuickTime Player version 10 was used to record audio and to capture video of thecomputer screen. QuickTime video-recorded all action on the screen, from scrolling
movements to clicks, while simultaneously recording sound. We also used an iPhone 6 as a
backup source of audio in case parts of the QuickTime sound files were muffled.
Participants were encouraged to do what they normally would when evaluating
  
  

14

READING LESS AND LEARNING MORE
information and determining its trustworthiness. We used a variety of prompts to encourage
natural behavior, including: “You can open up new tabs—do whatever you normally would tolearn about a site” and “We’re interested in your take. You can stay on the page or go out toanother website, anything you would normally do.” We repeated these instructions at the
beginning of each task. We also noted the time limit for each task and gave participants a oneminute warning before time was up.
Data Analysis
We developed rubrics to rate the quality of participants’ conclusions for each task.
These rubrics were developed after extensive pilot testing with Ph.D. graduate students anduniversity professors (we describe these rubrics in greater detail in subsequent sections that
describe each task).
In addition to assigning rubric scores, we developed codes specific to each task
(discussed in greater detail in Findings). We independently analyzed 10% of the data corpus,
met to discuss coding and resolve inconsistencies, and updated the coding scheme. One rater(the second author) then coded the remainder of the protocols. A second coder who did notparticipate in the creation of the rubrics or coding scheme analyzed a random sample of 10
protocols (22% of the data corpus); interrater agreement was 92% across the three tasks(Cohen’s Kappa = 0.90).
Additional analyses varied by task. These included tracking the time participants took
to settle on a conclusion; whether they stayed on or left a site, and, if they left, which othersites they visited; and whether they took steps to find out more about the individuals or groups
behind the sites they consulted.
  
  

15

READING LESS AND LEARNING MORE

16

Results
Task 1: Bullying
Participants evaluated articles about bullying on the websites of the American Academyof Pediatrics (“the Academy”) and the American College of Pediatricians (“the College”).
Despite the similarity in names, the two organizations could not be more different.
The Academy. Established in 1932, the Academy is the largest professional organizationof pediatricians in the world, with 66,000 members and a paid staff of 450. The Academy
publishes Pediatrics, the field’s flagship journal, and offers continuing education on everythingfrom Sudden Infant Death Syndrome to the importance of wearing bicycle helmets duringadolescence.
The website of the Academy bears a logo and trademarked motto. Resources andprofessional education opportunities for members are featured, including details on
membership, the group’s history since its founding in 1930, and opportunities to browse books
and journals that the Academy publishes. Participants viewed an article on the Academy
website entitled “Stigma: At the Root of Ostracism and Bullying.” The article describes a
symposium in which six papers were presented, including “Discrimination and Stigmatizationof Non-heterosexual Children and Youth.” Additional presentations focused on factors thatmight place youth at risk for bullying, such as weight, sexual orientation, race, and income(American Academy of Pediatrics, 2014).
The College. By comparison, the College is a splinter group that in 2002 broke from theAcademy over the issue of adoption by same-sex couples. It is estimated to have between 200500 members, one full-time employee, and publishes no journal (Throckmorton, 2011). The
group has come under withering criticism for its virulently anti-gay stance, its advocacy of    READING LESS AND LEARNING MORE

17

“reparative therapy” (currently outlawed for minors in thirteen U.S. states), and incendiary posts
such as one that advocates adding P for pedophile to the acronym LGBT, because pedophilia,
they claim, is “intrinsically woven into [the LGBT] agenda” (American College of Pediatricians,2015). The Southern Poverty Law Center has labeled the College a hate group that is“deceptively named” and acts to “vilify gay people” (Lenz, 2012; Southern Poverty Law Center,2016). The College’s portrayal of research findings on LGBT youth has provoked the ire of thenation’s leading scientists, including Francis Collins, MD, the former director of National
Institutes of Health, who wrote that “the American College of Pediatricians pulled language outof context from a book I wrote . . . to support an ideology that can cause unnecessary anguishand encourage prejudice. The information they present is misleading and incorrect” (as cited in
Bradshaw, Weight, & Packard, March 3, 2011). [insert Footnote 3]
At the same time, a quick glance at the College’s site might lead one to conclude that it
is a politically neutral medical organization (Turban, 2017). The website bears an officiallooking logo and sports the motto “Best for Children.” An anodyne “About Us” page informs
the reader that the College “produce[s] sound policy, based upon the best available research,
to assist parents and to influence society in the endeavor of childrearing.” However, a closer
look shows that the College does not mask its positions. The “Mission of the College” states:“We recognize the basic father-mother family unit, within the context of marriage, to be theoptimal setting for childhood development.” The College’s “Position Statements” aretransparent on issues ranging from abortion (prematurely and unnecessarily ending a humanlife) to corporal punishment (effective under certain circumstances).
Participants began by evaluating an article on the College website entitled “Bullying at
School: Never Acceptable,” where a section labeled “Prevention” advises schools to refrain
    READING LESS AND LEARNING MORE

18

from recognizing any students as particularly at risk of being bullied:
By focusing a program upon the special characteristic or activity of one student orgroup, the school opens the floodgates for other programs promoted by its advocates,
i.e. over issues involving religion, ethnicity, stature, intelligence, race, or even athletic
abilities. By focusing anti-bullying programs, instead, on the topic of general
respectfulness, the school…avoids the pitfalls of calling undue attention to a particulargroup or perhaps venturing into controversial teachings. (Trumbull, 2013)
Many studies have shown that students who identify as LGBT are more likely to be bullied thantheir heterosexual peers—over 80% of LGBT students were “verbally harassed” and over 40%
were “physically harassed at school…because of their sexual orientation,” according to a study
cited in the White House Conference on Bullying (Espelage, 2011, p. 65). Yet, the College
implies that programs to reduce bullying against LGBT students amount to “special treatment,”
and that these programs may “validat[e] individuals displaying temporary behaviors ororientations” (Trumbull, 2013).
Participants were given up to five minutes per site to evaluate the trustworthiness of eachas a source of information about bullying. If they did not explicitly compare the two sites before
the ten minutes were up, we asked: “If you had to say which website was more reliable andwhich was less reliable, what would you say?”
We developed a rubric to characterize the quality of the conclusions participants reached:
we awarded two points for specific, correct, and warranted descriptions of the sites. Suchanswers concluded that the Academy is the main professional organization and the College is asmall splinter group, which was formed because of ideological disagreements with its parent
    READING LESS AND LEARNING MORE
organization. We awarded one point for vague or indecisive evaluations, and zero points whenparticipants reached wrong conclusions (such as equating both organizations in terms oftrustworthiness).
For the College website, a Kruskal-Wallis nonparametric analysis of varianceindicated significant differences in the conclusions reached by participants on the College
website: fact checkers had a perfect mean score of 2 (SD = 0); historians, 0.7 (SD = 0.95); andstudents, .16 (SD = 0.37) (H (2) corrected for ties = 27.5, p < .001). Follow-up Mann-Whitney
U tests showed significant differences between fact checkers and historians (p = .003) and fact
checkers and students (p < .001).
There were also significant differences in the quality of conclusion scores for the
Academy site (H (2) corrected for ties = 25.2; p < .001). Fact checkers again had a perfectscore (M = 2, SD = 0), historians a 1.2 (SD = 0.79), and students a 0.4 (SD = 0.58). Follow-up
Mann-Whitney U tests yielded significant differences between fact checkers and historians (p= .01), fact checkers and students (p < .001), and historians and students (p = .007).
There were striking differences in which site participants judged the most reliable.
Every fact checker unreservedly viewed the Academy’s site as the more reliable; historians
often equivocated, expressing the belief that both sites were reliable; and students
overwhelmingly judged the College’s site the more reliable (see Figure 1).

  
  

19

READING LESS AND LEARNING MORE

Figure 1. Percentage of participants in each group selecting the College or the Academy
as more reliable.

  
  

20

READING LESS AND LEARNING MORE

21

Taking Bearings. Fact checkers’ success was closely tied to what we think of as
taking bearings, a concept borrowed from the world of navigation. Exploring an unfamiliar
forest, experienced hikers know how easy it is to lose their way. Only foolhardy hikers trust
their instincts and go traipsing off. Instead they rotate their compass’s bezel to determine
bearings—the angle, measured in degrees, between North and their desired destination.
Obviously, taking bearings on the web is not as precise as measuring an angle in degrees. It
begins, however, with a similar premise: When navigating unfamiliar terrain, first gain a senseof direction.
Checker C’s approach exemplified the advantages of taking bearings. He spent a mere
eight seconds on the College’s landing page before going elsewhere. “The first thing I would do
is see if I can find anything on the organization,” he said as he typed the organization’s nameinto Google. He clicked on Wikipedia’s entry about the College and read that it is a “sociallyconservative association of pediatricians…founded in 2002…as a protest against the [American
Academy’s] support for adoption by gay couples.” Wikipedia’s entry linked to sources including
a Boston Globe story (“Beliefs drive research agenda of new think tanks,” Kranish, 2005), areport from the Southern Poverty Law Center (“American College of Pediatricians Defames
Gays and Lesbians in the Name of Protecting Children,” Lenz, 2012), and a brief from theAmerican Civil Liberties Union (“Misinformation from Doctors . . . Out to Hurt Students?,”
Coleman, 2010).
It was a full minute and twenty seconds before Checker C returned to the College’s
article on bullying. Reading the abstract that he had glanced at in the task’s opening seconds(see Figure 2), he paused at the phrase “no group should be singled out,” and remarked that    READING LESS AND LEARNING MORE
this is “often code for, you know, kids who are more likely to be bullied—students of color or
gay or queer children,” adding, “That’s the kind of thing that I never would have known if I
had just looked at [the article on bullying].”

  
  

22

READING LESS AND LEARNING MORE

23

Figure 2. Abstract of “Bullying at School: Never Acceptable” (emphasis added).

Rendered in under two minutes, Checker C’s conclusion was not only an accurate
evaluation of the bullying article but also of the rest of the College’s website, which presents ananti-gay stance throughout. [insert Footnote 4] Overall, fact checkers left the landing page of theCollege in about half a minute (M = 32 s, SD = 29 s). In contrast, historians took almost threetimes as long (M = 88 s, SD = 103 s) (eight of the 10 left the landing page; two did not). The 16
students who left the landing page (nine never did) took an average of 100 seconds (SD = 52 s).
Fact checkers’ comments as they left the landing page (see Table 2) showed an
immediate impulse to take bearings. They understood the web as a maze filled with trap doors
and blind alleys, where things are not always what they seem. Their stance toward the    READING LESS AND LEARNING MORE
unfamiliar was cautious: while things may be as they seem, in the words of Checker D, “I
always want to make sure.”

  
  

24

READING LESS AND LEARNING MORE

25

Table 2
Examples of Fact Checkers’ Comments Upon Leaving the Landing Page
Checker

Comment

A“I immediately want to know more about [the College]. So I’m going to go to
About Us.”D

“My first move to figure out whether something is reliable is to click on the
About Us page. . . . At face, the American College of Pediatricians sounds
pretty formal, but I always want to make sure.”E“I want to learn a lot more about the American College of Pediatricians.”

H“It’s kind of hard to tell how mainstream this organization is, so I might open
another tab just to read a little bit more about, if this is the main American
pediatricians’ professional organization or if this is a splinter group forsome reason.”    READING LESS AND LEARNING MORE
Historians’ Reading. Two of ten historians resembled fact checkers in how they took
bearings. Leaving the landing page after a 20-second glance, Historian H opened the site’s“Resources” tab and clicked on the link to focusonthefamily.com to confirm that it was in fact
the organization founded by evangelist Dr. James Dobson. He returned to the College’s“Resources” page with a hypothesis: “They probably have an agenda to quote, cure, unquotehomosexuality, which is another fundamentalist point of view.” Historian S also left theCollege’s site in less than half a minute. Googling the organization’s name, he clicked on a
Breitbart headline, “American College of Pediatricians on Same-Sex Marriage Ruling: A
Tragic Day for America’s Children.” He concluded that the College is “a heavily ideologicalsite.”
Historians H and S were the exceptions. Asked whether the website of the splintergroup or the 64,000-member Academy was the more trustworthy site, five of their colleagues
equivocated. Seven of the historians never took bearings; one did so only after analyzing thebullying article for four minutes. After ten minutes of review, most scholars had learned
virtually nothing about the respective agendas of the two pediatrics organizations.
Historians were often taken in by the College’s name and logo; its .org domain; its
layout and aesthetics; and its “scientific” appearance, complete with abstract, references, andarticles authored by medical doctors. Reading the “Bullying at School” article, Historian M
commented on the presence of a scientific abstract and references, compared the site to
WebMD, and noted that the article was signed by a doctor (true, but it was not something sheverified, since she never left the landing page). She concluded:
I think I would probably find this pretty reliable on the basis that it’s written by an
  
  

26

READING LESS AND LEARNING MORE
expert, it’s citing expert opinions, it’s been reviewed by at least some people from theCollege of Pediatricians, so it agrees with an expert opinion. But it is still nonethelessstill an opinion piece, it’s just an opinion piece that I agree with, and…reflects theopinion of a group that I want to know the opinion of.
There was no basis for Historian M’s far-reaching conclusions other than the surface featuresof the site, its presentation of information, and the M.D. listed after the author’s name.
One feature played a key role in shaping historians’ judgment: the presence ofreferences  at the bottom of the College’s entry. Seven of 10 historians explicitly commented
on them (see Table 3), viewing citations to Pediatrics and the Journal of Criminology, among
others, as conferring legitimacy on the article’s content.

  
  

27

READING LESS AND LEARNING MORE

28

Table 3
Historians’ Comments About References
HistorianaComment

A“It has references to kind of standard scientific literature, of backing up some ofits claims so it has a kind of authoritative tone to it.”B

“I would look at the references and see who the [author] is citing.”

E“These are all references to professional journals so that definitely reinforces
my sense that it’s a genuine site and that the information found here can betrusted.”I“I am looking at some of the footnotes and they all seem like perfectly
credible sources. . . . I can trust this site.”

K“Who are they actually citing? So Pediatrics, okay, so they’re citing realjournals so I trust them a little bit more. . . . So the citations suggest that it
has some reputable characteristics.”

L“I like to look at the sources to see where they are getting things. These are allacademic journals as opposed to random Google News, which you neverknow about.”N“I am looking at the references now and to what extent they’re linked up tojournals that strike me as peer-reviewed journals and have some kind of
credibility. So, they all seem to come from something that strikes me—I
don’t know, Pediatrics—but I assume it seems to be in some kind of
academic form.”

Not all references were to scientific articles. Among the 10 references, one was to Free Dictionary, two to Yahoo
News blogs, one to Alliance Defense Fund, and the rest to refereed journal articles.

    READING LESS AND LEARNING MORE
Students’ Reading. By the end of ten minutes, only three of the 25 students had
distinguished between the stances of the College and the Academy. Fully 60% of students
chose the College as the more reliable site. Even the five who favored the Academy learned
little about the vast differences between the two organizations.
Students rarely took bearings when landing on an unfamiliar site. Nine of the 25 neverleft the original site; those who did tended to click on links that spoke to a personal interest
rather than a search designed to find out more about the organization behind the website.
Student 19, who planned to major in either ancient Greek or bioengineering, based her
evaluation almost exclusively on features like the organization’s name (“sounds pretty
legitimate”); the site’s layout, which included bullet points (“nice to understand quickly”) andsection headings (“that’s really smart”); and the absence of banner ads (“makes you focus on
the article”). On the basis of graphic design, she concluded that the College’s page was themore reliable of the two: “What struck me was how [the College’s site] was laid out.” Student
19’s approach was representative of how the majority of students conducted their evaluations(see Table 4).

  
  

29

READING LESS AND LEARNING MORE

30

Table 4Students’ Comments About Why They Trusted the College’s Webpage

    Reason for
conferring
trustworthiness

Examples of Reasoning

Scientific
Presentation:
abstract,
references,authored by a
medical doctor“This seems like it’ll be pretty promising. There’s an abstract, so I feellike this is like a research thing.” (Student 12)“So now I see an abstract, which makes me think that this is a very
research-based paper. . . . This seems like a very scientific article,
because everything is in list form and very specific. The diction andthe language is pretty scientific in general. I like that they are citing
their sources with links and stuff.” (Student 15)“It’s written by a doctor. . . . There’re references. Seems like a legitimate
article.” (Student 20)

Usefulness:amount of
information,clarity andaccessibility ofarticle“It has a very clear title on what its view of bullying is. . . . I really like
how it’s laid out with the little headings to easily find what you need,and bullet points are always easier to look through also. And thereferences are really useful if I were to be doing research project,because then I could just look at these references afterwards. Yeah, I
think this would be a useful site. It does seem like they have a lot of
information.” (Student 13)“If I were writing a paper…then I would choose [the College] over [theAcademy] simply because this just provides more information
relevant to the topic.” (Student 6)
Answering which is more reliable after looking at both sites: “The[College article] because that actually gave me more information
about bullying.” (Student 11)

Graphic design:pleasant layout,
color scheme,lack ofadvertisements“They seemed equally reliable to me. I enjoyed the interface of the[College website] better. But they seemed equally reliable. They’reboth from academies or institutions that deal with this stuff everyday.” (Student 5)“Nice how there’s not really any advertisements on this site. Makes it
seem much more legitimate.” (Student 19)

Organization’s
Apparent
Authority:
name, logo,
URL“I can automatically see this source and trust it just because of how
official it looks—American College of Pediatricians, even the font
and the way the logo looks makes me think this is a mind hive thatcompiled this.” (Student 7)
First statement on arriving at the site: “American College of
Pediatricians. Seems like a credible website, run by pediatricians.”

READING LESS AND LEARNING MORE
(Student 16)
First statement on arriving at the site: “.org. So this looks like it might
have been subsidized by a government agency.” (Student 18)

  
  

31

READING LESS AND LEARNING MORE
Three of the 25 students selected the Academy as more trustworthy because theylearned something about, and rejected, the College’s ideological stance. Two of the threestumbled upon information that provided insight into the College’s views, but did notdeliberately seek it out. Only one in 25 students took bearings in a way that could becompared to the fact checkers’ approach. Even then, the student spent nearly four minutes
reading “Bullying at School: Never Acceptable” before leaving the site.
Task 2: Minimum Wage
Participants evaluated an article entitled “Denmark’s Dollar Forty-One Menu” on thewebsite minimumwage.com (see Figure 3). The article argues that if the U.S. followed theexample of Denmark and raised wages, it would face higher food prices and diminished job
opportunities. The article links to stories in the New York Times and Columbia Journalism
Review, while the website includes tabs for research reports and news stories. Its “About”
page says it is a project of the Employment Policies Institute (EPI), a group described as a
“nonprofit research organization . . . . [that] sponsors nonpartisan research which is conductedby independent economists at major universities.”

  
  

32

READING LESS AND LEARNING MORE

Figure 3. “Denmark’s Dollar Forty-One Menu” on minimumwage.com.

  
  

33

READING LESS AND LEARNING MORE
Despite their nonpartisan declarations, minimumwage.com and the Employment
Policies Institute are cloaked websites (Daniels, 2009). They are the handiwork of Berman andCompany, a Washington, DC-based public relations firm that lobbies on behalf of therestaurant and hotel industries. Berman’s specialty, in the words of the New York Times, is tocreate “official-sounding nonprofit groups to disseminate information on behalf of corporateclients” (Lipton, 2014). None of this information, however, is disclosed onminimumwage.com or the Employment Policies Institute website. A 2013 Salon article
characterized the tactics of Berman and Company with the headline, “Industry P.R. Firm
Poses as Think Tank” (Graves, 2013).
Participants were given up to five minutes to evaluate minimumwage.com. They could
use any Internet resources (including leaving the site) to help them; we repeated the
instructions to do what “they would normally do” when landing on an unfamiliar site.
Participants who had not reached the Employment Policies Institute website after five minutes
were given this prompt: “Minimumwage.com is paid for by another person or organization.
Spend up to three minutes to figure out who is behind this site.”
We used the following rubric to rate participants’ responses:
Score
0
1
2
3
4
  
  

Description
Evaluates minimumwage.com based on surface features; does not identify
connection to the Employment Policies Institute.
Determines that the Employment Policies Institute sponsors minimumwage.com,
but learns nothing about the Employment Policies Institute.
Determines that the Employment Policies Institute sponsors minimumwage.com;
describes the Employment Policies Institute as a non-profit and non-partisan
think tank or research organization.
Determines that the Employment Policies Institute sponsors minimumwage.com;
describes the Employment Policies Institute as an advocacy organization orraises substantial questions/concerns about its trustworthiness.
Determines that the Employment Policies Institute sponsors minimumwage.com

34

READING LESS AND LEARNING MORE
and is a front site created by Berman and Company, a public relations firm.
There were dramatic differences in what fact checkers, historians, and students learned
during the task’s eight minutes. Before prompting, fact checkers’ conclusions averaged 3.3(SD = .82) out of 4, versus historians’ average of 1.3 (SD = 1.4) and students’ .52 (SD = 1.16).
A Kruskal-Wallis test showed significance (H (2) corrected for ties = 21.4, p < .001); followup Mann-Whitney U tests showed differences among fact checkers and historians (p = .003)
and fact checkers and students (p < .001).
Without prompting, and in less than a minute, the fact checkers learned that EPI was
minimumwage.com’s parent (See Figure 4; M = 51 s, SD = 43 s). Historians took nearly fourtimes as long (M = 3 min, 40 s, SD = 2 min). Six of the 10 needed to be prompted to find EPI.
Among the three groups, students took the longest to get to EPI: an average of 5 minutes and
18 seconds (SD = 1 min, 24 s); the overwhelming majority of students (four-fifths) needed
prompting.
Every fact checker concluded that Richard Berman (or Berman and Company)
sponsored EPI and minimumwage.com. Only six historians did so, and those who did tooknearly twice the time as checkers (Mcheckers = 3 m, 25 s, SD = 1 min, 42 s; Mhistorians = 6 m, SD =2 min, 35 s). Only forty percent of students made it to Berman and Company; those that did
took an average of nearly seven minutes (M = 6 min, 59 s, SD = 1 min, 51 s).

  
  

35

READING LESS AND LEARNING MORE

Figure 4. Average time for participants to determine Employment Policies Institute’s
sponsorship of minimuwage.com; average time and percentage of each participant group to
determine Richard Berman or Berman and Company’s sponsorship of both websites.

  
  

36

READING LESS AND LEARNING MORE
Reading Laterally. Fact checkers learned more about minimumwage.com and did so
in less time than the others. They employed a powerful heuristic for taking bearings: lateralreading. Fact checkers almost immediately opened up a series of new tabs on the horizontal
axis of their browsers before fully reading the article.
Checker A glanced at “Denmark’s Dollar Forty-One Menu” for six seconds before
clicking on the page’s “About” tab, where she learned that the site was “a project of the
Employment Policies Institute.” She used keyboard shortcuts (pressing the command key
while clicking) to open the link to the Employment Policies Institute site in a new tabalongside minimumwage.com (see Figure 5). After just three seconds on EPI’s home page,
she went to their “About Us,” scanned the bland description (“Founded in 1991, theEmployment Policies Institute is a non-profit research organization dedicated to studying
public policy issues”), and quipped, “This is profoundly not helpful.” In just over a half
minute, she opened a new tab and searched for Employment Policies Institute.

  
  

37

READING LESS AND LEARNING MORE

Figure 5. Checker A’s lateral reading.

  
  

38

READING LESS AND LEARNING MORE
Scanning Google’s snippets, Checker A skipped the first four results and selected
SourceWatch’s entry on EPI: “So this says it’s one of several front groups created by a PR
firm.” She scrolled until she hit a linked quotation from a New York Times reporter who“detailed his visit to the EPI, saying, ‘I didn’t see any evidence at all that there was an
Employment Policies Institute office.’” One minute and twenty-seven seconds into the task,
she clicked on SourceWatch’s citation for this quote, which led to a National Public Radio
story, “A Closer Look at How Corporations Influence Congress.” Rather than reading it,
Checker A used Command-F to search for EPI and corroborate the claims made by
SourceWatch. A little over two minutes into the task, she had EPI sized up:
Obviously this isn’t a legitimate organization, based on the reporting of this New YorkTimes reporter. He talks about actually going there, he doesn’t see any evidence at allthat they actually had an office, there are no employees, all the staff there actuallywork for the PR firm.
Only then did she return to her original starting place, minimumwage.com, declaring, “[TheNew York Times reporter] is right. It’s a very legitimate looking website, but clearly, this isalso advancing an agenda.”
With breakneck speed, Checker A deftly traversed a digital morass, ignoring massive
amounts of material (she barely read the original article) to conclude that minimumwage.com
and EPI were not what they seemed. Though slightly less efficient, the other checkers
mirrored Checker A’s lateral approach. The average time they took to leave the starting page
was just over half a minute (M = 37 s, SD = 41 s). None accepted EPI’s description at facevalue; instead, they read laterally, visiting an average of six pages before concluding that
  
  

39

READING LESS AND LEARNING MORE
minimumwage.com and EPI were cloaked sites that represented corporate interests.
Historians’ Reading. Historians took longer, on average, to go fromminimumwage.com to EPI than fact checkers took to conclude that both sites were theproducts of Berman and Company. Before prompting, only four of 10 historians connected
minimumwage.com to the Employment Policies Institute. As in the previous task, Historians
H and S were outliers. They left the landing page four times as fast as the others, averaging 26
seconds; their eight colleagues averaged 2 minutes, 5 seconds. Both were efficient lateralreaders, wasting little time before opening additional tabs. Three of their colleagues, on theother hand, remained stuck on minimumwage.com for the entire task.
Even when some of the historians sought to read laterally—opening new tabs toresearch minimumwage.com or the Employment Policies Institute—they lacked essential
searching skills. For example, a minute into the task, Historian K tried to learn more aboutminimumwage.com by opening a new tab to search for the name of the organization. But
instead of putting the name of the organization in quotation marks and adding keywords like“funding” or “who is behind,” she typed [minimum wage.com] into the search bar, separating“minimum” from “wage” and adding no additional terms. The outcome was an entire page ofresults issued by the very organization she was trying to investigate. Sensing a dead end, she
added [conservative?] to the search bar, which produced yet another page of fruitless results(see Figure 6).

  
  

40

READING LESS AND LEARNING MORE

Figure 6. Historian K’s search results for [minimum wage.com conservative?]

  
  

41

READING LESS AND LEARNING MORE
Stymied, the historian abandoned lateral reading and returned to the original“Denmark’s Dollar Forty-One Menu” page, no wiser than before. She clicked the page’s“Research” tab to engage in a more familiar task: “Let me see how I can interpret thelegitimacy of their research.” Historian K was not alone: her colleagues fumbled such basic
moves as putting terms in quotation marks so that Google could search for contiguous terms.
Each of these historians was an astute reader, but reading skills alone weren’t enough to pullback the curtain from a cloaked website.
Students’ Reading. Students struggled to get to the bottom of minimumwage.com.
They either spent too much time reading vertically, staying on the page and reading it as theymight a print document, or they engaged in fluttering, aimlessly moving across the screen,“touching or not touching pieces of information … unconscious to its value and without aplan” (Kirschner & Von Merriënboer, 2013, p. 171). When five minutes were up and before
being prompted, 80% of students had devoted no time to investigating who was behindminimumwage.com.
Although some students left the landing page quickly, their exit was a far cry from thestrategy of taking bearings. Instead, they meandered to different parts of the site, making
decisions about where to click based on aspects that struck their fancy. A prospective
chemical engineering major quickly glanced at “Denmark’s Dollar Forty-One Menu” before
scrolling to the bottom of the page and clicking on “In Your State,” an interactive map whereusers could click on different states and compare minimum wage rates and unemployment
statistics. He spent two minutes playing with it, longer than he spent reading the initial article.
Other students engaged in similar kinds of fluttering, clicking on features that piqued their
  
  

42

READING LESS AND LEARNING MORE
curiosity rather than those that would justifiably inform their judgment about the site’strustworthiness (see Table 5).

  
  

43

READING LESS AND LEARNING MORE

44

Table 5Students’ Fluttering on Minimumwage.com
Links
ClickedStudent’s Comment while Clicking

Clicking Sequence

https://www.
minimumwa
ge.com/media/“It’s interesting how the Media page is kept
very minimalistic, and then you click onother things [clicking on ‘News Reports,’
which leads to an EPI page] and it brings
you to different pages [clicks back to‘Media’ page]. But I think it’s actuallysmart to keep that elsewhere just toorganize it.” (Student 19)

Visited “Media” page after
visiting the “Home,”“Myths,” “Research,”and “In Your State”
pages

https://www.
minimumwa
ge.com/research/“I don’t really want to read their blog, andI’m not interested right now in what’s mystate’s minimum wage and teen
unemployment. . . . And videos andgraphics are too time consuming.”(Student 3)

Explaining her reasoning for
clicking on the“Research” page insteadof the “Blog,” “In Your
State,” or “Video andGraphics” pages

https://www.
minimumwa
ge.com/news/“I like the layout of the blog, I think it’s also
just very clear and everything’s very
cleanly laid out in a single column. Samewith this [‘Research’] page. . . . Oh, andthen here’s a description of the website.
Um, this is a pretty cool page too.”(Student 12)

Clicked through several
pages of the website,
including “Home,” “InYour State,” “Blog,”“Research,” “About,”
and “Myths.” On eachpage, she focusedcomments on
appearance andorganization

https://www.
minimumwa
ge.com/media/“Maybe this is an impartial website. Is there Clicked to “Media” andany such thing [clicks to ‘Videos and“Videos and Graphics”
Graphics’ page] as an impartial website? I
pages after viewing the
don’t know. [reading advertisements“Home” and “In Your
posted on site] ‘Unhappy New Year,’ ‘If 7
State” pages
out of 10 doctors said you were sick, you
would listen.’” (Student 1)

    READING LESS AND LEARNING MORE
Task 3: Vergara v. California
In May 2012, lawyers in California filed a lawsuit on behalf of nine public school
students, including Beatriz Vergara. They argued that the system of teacher tenure in
California violated the state constitution by denying equal protection to students withineffective teachers. In June 2014, a California Superior Court ruled in favor of the students.
The case cost more than a million dollars to prosecute, a sum that typically exceeds the
spending money of nine adolescents. In fact, the legal team was hired and financed by David
Welch, a Silicon Valley entrepreneur who founded the organization Students Matter.
The press, however, often omitted this detail. What made for good copy was a Davidversus-Goliath tale of adolescents taking on a powerful teachers’ union: nine students, mostly
students of color, courageously confronting a rotten bureaucracy and demanding better
teachers. A news item on the website of KABC, the Los Angeles ABC affiliate, reported that
“The verdict is a win for nine students who sued the state saying that tenure policies havemade it impossible for bad teachers to be fired” (“California Teacher Tenure,” 2014). It madeno mention of Students Matter, David Welch, or any of the big money that backed the suit.
Unlike the two previous tasks, this one began with a paper stimulus: the 379-word
article from KABC. We gave participants time to read the article before telling them that thenine students had a million-dollar legal bill. We then asked them to spend five minutes
searching for who paid the tab. Participants needed to, as it were, “follow the money” bylocating information that named Students Matter, and ultimately David Welch, as the mainbacker of the lawsuit.
Vergara was a politically charged case with far-reaching implications. Students Matter
  
  

45

READING LESS AND LEARNING MORE
argued that the case was about getting rid of laws that were “handcuffing schools from doing
what’s best for kids when it comes to teachers” (“Vergara v. California,” n.d.); the California
Teachers Association painted it as a “lawsuit brought by wealthy corporate special interests
looking to eradicate educators’ professional and due process rights” (“Vergara v. State ofCalifornia,” n.d.). Given these conflicting claims and the number of bona fide news sources
and partisan sites that wrote about the case, site selection and verification were essential. If
participants could verify that Welch was the source of the plaintiffs’ funding across bona fidesources, they could be more certain that they had successfully navigated politically muddy
waters to arrive at the correct answer.
The 25 Stanford students were the fastest in identifying Welch as the source of funding(M = 1 minute, 42 seconds, SD = 86 s). Fact checkers and historians were slower. Historians
took 2 minutes, 1 second (SD = 56 s), and checkers averaged 2 minutes, 8 seconds (SD = 93s). Although they were the slowest to reach their conclusions, fact checkers were the mostselective when it came to the sites they visited and took the most time to verify their answers.
We rated the quality of participants’ conclusions using a 5-point scale. Participants
were given a 0 if they never identified Welch; a 1 if they identified Welch but did so only
through a questionable source; a 2 if they identified and verified Welch’s role based on two or
more questionable sources; a 3 if they identified Welch using a bona fide source; and a 4 ifthey identified and verified Welch’s role through at least one bona fide source and one
additional source. We defined bona fide sources as those that have systems in place to ensure
the quality and accuracy of the information they provide (e.g., they have a retraction or
correction policy, employ an ombudsman, separate their editorial and news divisions, and are
  
  

46

READING LESS AND LEARNING MORE
explicit about the methods with which they verify news) (Bounegru, Gray, Venturini, &Mauri, 2017; Bulger & Davison, 2018).
Using our rubric, the fact checkers’ conclusions merited a 3.6 (SD = 0.70), versus
historians’ 2.4 (SD = 1.3) and students’ 2.3 (SD = 1.5). Fifteen students scored a 0, 1, or 2,
while all but one of the fact checkers’ responses scored a 3 or 4. A Kruskal-Wallis test showedsignificance (H (2) corrected for ties = 27.5, p < .001); follow-up Mann-Whitney U tests
showed differences between fact checkers and students (p = .016).
The differences between the students’ and the fact checkers’ approaches can be seen
by comparing Checker D with Student 17, a mathematical and computational science major.
Both identified Welch in under a minute (34 seconds for the student, 50 seconds for the
checker). The student spent just a few seconds on the results yielded by searching for [vergara
v california]. He clicked on the first result (the Students Matter page), but quickly returned tothe search results, reminding himself, “I’m looking for the ‘who paid.’” He selectedvergaratrial.com, a partisan site created by the California Federation of Teachers, where helocated Welch’s name. He neither commented on the website’s political stance nor whether hefound it trustworthy; he simply located Welch’s name and accepted it as fact.
Checker D initially searched for [vergara v california] before quickly adjusting it to[vergara v. california court records]. As she scrolled down the results, she said, “I’m coming
up with a lot of different information. I’d rather click on some press reports.” She skipped thefirst three results, all of which were affiliated with Students Matter, along withvergaratrial.com and cacs.org (an organization she did not recognize), and instead opened
articles from three news organizations and Wikipedia. Exhibiting what we call click restraint,
  
  

47

READING LESS AND LEARNING MORE
she spent nearly 20 seconds scanning the search engine results page and reading the snippets
before clicking on any link. Although she opened four additional webpages in new tabs (seeFigure 7), her use of keyboard shortcuts meant that her eyes and focus never wavered from theresults page. Checker D went first to Wikipedia, where she skipped over most of the entry byusing the “Contents” menu to navigate to “Litigants.” There, she read that “funding for theplaintiff school students was provided by David Welch, a Silicon Valley entrepreneur.” Shethen clicked on the Washington Post article she had opened in a different tab. She used thecommand-F shortcut to search for Welch’s name and confirmed his role in the case.
Checker D took 16 seconds longer than Student 17 to find Welch’s name. However,
she was more purposeful in the sites she opened, more discerning in the information she
considered trustworthy, and more thorough in ascertaining that David Welch was indeed themoney behind Vergara v. California.

  
  

48

READING LESS AND LEARNING MORE
Figure 7. Checker D’s search results showing the sites she opened.

  
  

49

READING LESS AND LEARNING MORE

Historians. Historians were only slightly better than students in the quality of theirconclusions (Mhistorians = 2.4 versus Mstudents = 2.3). Although several historians excelled, quickly
locating Welch’s name and verifying his role on trusted sites, two of them relied exclusively
on partisan or questionable sources and made no attempt to verify their conclusions.
A third, Historian N, never made it to Welch. He searched for [Vergara v. California]
and started with Wikipedia. Rather than using Wikipedia to quickly locate Welch’s name,Historian N went directly to the references to find “a link to the case itself.” For nearly threeminutes, he examined the original court brief (number BC484642), scrolling up and down the
PDF document, pausing at “Procedural History” and learning that the plaintiffs argued that the
California Educational Code violated the equal protection clause of the state constitution.
After searching in vain for the plaintiffs’ backers, he abandoned Wikipedia and initiated a new
search, adding “plaintiffs” and “attorneys” to his original query.
He clicked on the first result (studentsmatter.org, Welch’s organization) and went to“Our Team,” where he recognized the name of the lead attorney (“someone I know … the
Solicitor General under Bush”). By the end of the task the only thing he could say was that theplaintiffs were represented by a “team with deep legal pockets.” An accurate statement, butthen again, this was the starting point for the task—participants were told at the task’s startthat legal fees in this case were “over a million dollars” and that their goal was to find out who
paid the bill. By the task’s end, this historian was no closer to answering the question thanwhen he started. How come?
The simplest answer was that Historian N did what historians are trained to do: search
  
  

50

READING LESS AND LEARNING MORE

51

for primary sources. Had the task been to write a history of the Vergara case, initiating the
research process with the court briefing might’ve made sense. However, when the goal was toquickly learn who backed the teenagers, a close reading of a labyrinthine legal document—
which never mentioned Welch—took precious time and sapped limited energy.
Limitations
The purpose of this exploratory study was to better understand the nature of expertise in
the evaluation of online information. We recognize, however, that any task that involves
researchers peering over participants’ shoulders creates an artificial environment that can distort
what people ordinarily do. Despite repeated imperatives to “do what you normally do,” it has tobe odd to be shown sites not of one’s choosing and given one-minute warnings to stop searching.
Studies are needed that observe people evaluating websites in more natural settings. At the sametime, we reasoned that tasks without time limits threaten ecological validity—just-in-timesearches are generally matters of minutes or seconds, not hours (Liu, White, & Dumais, 2010;
Nielsen, 2011). It’s also possible that a different sample of sites might have yielded different
results. We chose sites that covered a range of topics and perspectives and that varied in how
forthcoming they were about their agendas. But even within the categories we selected, there are
innumerable options, each with unknown content effects. More extensive research is needed toknow if the strategies we identified are generalizable across topics, sites, and searches.
Additionally, it may have been the case that participants didn’t put forth their best efforts,
although we find that prospect unlikely. Our sample was comprised of people with high levels of
self-regard and intellectual confidence. Looking foolish or appearing cavalier, especially whenrendering judgments about issues of social and political moment, threatens that self-regard.
    READING LESS AND LEARNING MORE

52

We are also aware that professional fact checkers were not the only possible group of
experts we could have sampled. Others, such as Wikipedia editors who have earned the highest
badges, specialists in cyber security, and professional librarians and information scientists, arealso worthy of study. Teacher educators, especially those responsible for preparing new teachers
to use information technologies, would be a fruitful sample, too. In their approach to websites,
two of the ten historians resembled the fact checkers more than their fellow historians. Small
sample sizes exaggerate differences (Jussim, 2012): we can’t rule out the possibility that
doubling or tripling sample size would have produced different results. Studies that require
intensive protocol analysis always present a trade-off between sample size and available
resources. That said, a sample of 45 nearly hour-long protocols is on the higher end for this genreof research.
Discussion
The participants in this study were all capable readers. Historians had esteemed
publications to their credit and held coveted positions in a field where tenured faculty lines are
increasingly rare. The fact checkers worked for prestigious publications and rubbed shoulders
with famous authors who depended on them to get things right. Our college students gained
admission to the most competitive university in the U.S. in 2015 (Stanford University, 2015).
Yet, despite our participants’ talents, there were unmistakable differences in how they navigated
the web.
Only two of the ten historians adroitly evaluated digital information. Others were often
indistinguishable from college students. Both groups fell victim to the same digital ruses.
Considering our participants’ experience and accomplishments, we are left to ask: What is it    READING LESS AND LEARNING MORE

53

about the Internet that bedevils intelligent people? What did fact checkers do that allowed themto quickly and accurately discern the trustworthiness of information?
The answer, we suggest, lies with two concepts we introduced earlier: taking bearings
and lateral reading. In order to take bearings, searchers obey the following imperative: before
diving deeply into unfamiliar content, chart a plan for moving forward. Taking bearings is what
sailors, aviators, and hikers do to plot their course toward a desired destination. Landing in
unfamiliar territory, fact checkers set out for their destination—making a judgment ofcredibility— only after gaining a sense of where they had landed. To take bearings, web
searchers obviously don’t use a physical compass. But they need metaphorical compasses just as
much as hikers need real ones.
In an Internet teeming with cloaked sites and astroturfers (see Daniels, 2009;
SourceWatch, n.d.), taking bearings often assumes the form of lateral reading. When reading
laterally, one leaves a website and opens new tabs along the browser’s horizontal axis, drawing
on the resources of the Internet to learn more about a site and its claims. When reading laterally,fact checkers paid little attention to features of a website like its appearance or contents. Instead,they quickly leapt off the landing page to open new tabs. Fact checkers, in short, learned mostabout a site by leaving it.
Lateral reading differs from one of the main approaches to literacy advocated by the
Common Core State Standards. As defined by the Common Core’s framers, close reading
encourages students to “read and reread deliberately” in order to “reflect on the meanings ofindividual words and sentences; the order in which sentences unfold; and the development ofideas over the course of the text, which ultimately leads students to arrive at an understanding of    READING LESS AND LEARNING MORE

54

the text as a whole” (PARCC Model Content Frameworks, 2011, p. 7). Close reading trains
students to parse grammatical structures, to ponder the metaphors and similes an author employs,
and to consider how minor shifts in word choice influence meaning. Close reading is essential to
any thoughtful curriculum (Shanahan, 2012; Wolf, 2007). However, the close reading of a digitalsource when one doesn’t yet know if the source can be trusted is neither efficient nor effective.
Just as close reading focuses students’ attention on carefully analyzing text, so a popular
approach to teaching web evaluation advocates parsing the features and contents of an individual
webpage. Often presented as checklists, these guidelines focus attention on elements internal to a
website: Is its URL a .org or a .com? Is an author listed? Are there ads on the page? Are
hyperlinks functional? The questions that appear on these lists range from ten to as many as 30(see Common Sense Media, 2012; Media Education Lab, n.d., News Literacy Project, n.d.),nearly all of which focus on easily manipulated features. Similarly, college library websites often
advise students to use “Five Criteria for Web Evaluation.” These criteria (“Authority, Accuracy,
Objectivity, Currency, and Coverage,”)—or variations on the theme (including the CRAAP test:“Currency, Relevance, Authority, Accuracy, and Purpose”)— focus, like checklists, on closely
reading and analyzing the contents of a single webpage. [insert Footnote 5]
Even if we set aside the concern that students (and the rest of us) lack the patience tospend fifteen minutes answering lists of questions about a single site, a bigger problem looms:
identifying a .org URL, locating an author, and making sure a site is free of typos hardly confers
credibility. The Employment Policies Institute not only carried a .org domain but was labeled a
501(c)(3) “charitable organization.” At a time when the Internet is characterized by polished web
design, search engine optimization, and organizations vying to appear trustworthy, such    READING LESS AND LEARNING MORE

55

guidelines create a false sense of security. Students and historians often stumbled because theyclosely read and followed the advice dispensed by checklists. Fact checkers succeeded because
they didn’t.
Instead of closely reading or ticking off elements on a list, checkers ignored massive
amounts of irrelevant (or less crucial) text in order to make informed judgments about thetrustworthiness of digital information. In short, fact checkers read less but learned more.
This approach contrasted with students’ and historians’ tendency to closely read andanalyze webpages. There were, however, moments when fact checkers slowed down. College
students were faster at finding the name of the financial backer in the Vergara case. But speed
came at the expense of quality. Students arrived at David Welch’s name by promiscuous
clicking, often without regard to a source’s reliability. Fact checkers took longer not because of
faulty search strategies or unhelpful keywords, but because they slowed down and carefully
reviewed search results before clicking on any one of them. They displayed click restraint:
Before clicking on any result, they mined Google’s snippets for the wealth of information they
contain. They examined each URL, considered the source of the information, and scanned thebrief but fecund sentence fragments before alighting on a link to click. They stood back from theresults page and viewed it as a whole, gaining a sense of the “information neighborhood”(Klurfeld & Schneider, 2014) into which they had landed. A searcher’s first click can mean
destiny, either putting them on a path toward warranted conclusions or sending them into thewilderness of infinite regress. Click restraint tips the balance toward the former.
The strategies and heuristics that fact checkers deployed relied on three sources ofbackground knowledge: knowledge of digital sources, knowledge of how the Internet and    READING LESS AND LEARNING MORE

56

searches are structured, and knowledge of strategies to make searching and navigating effective
and efficient. Fact checkers recognized and distinguished among an array of online venues,
including where sites spread out across the political spectrum (Daily Kos is liberal, Daily Caller
conservative). They understood the characteristics that make a source reliable versus those thatact as fallible proxies for reliability, such as how an unfamiliar organization describes itself on its“About” page. For example, the Employment Policies Institute describes itself as “a non-profit
research organization dedicated to studying public policy issues.” Checker A’s reaction was curt:“This is profoundly not helpful.” She knew that a nonprofit status does not stamp an organizationas unquestioningly altruistic.
Yet knowledge of sources is necessary but not sufficient. Fact checkers also possessedknowledge of online structures, particularly how search results are organized and presented.
They knew that the first result was not necessarily the most authoritative, and they spent time
scrolling through results, often scanning the entire first page (and sometimes the second andthird) before clicking on a link. They understood how search engine optimizers use sophisticatedkeywords and other techniques to game results, pushing some sites to the top and moreauthoritative sites to the bottom. Students and sometimes historians often clicked on the first few
search results, rarely articulating a rationale for why they selected a particular link (a finding
documented by others; e.g., Hargittai et al., 2010; Pan et al., 2007). Unlike fact checkers, theyrarely took time to scan the entire search engine results page to gain an overall picture of thedigital territory into which they had landed. They seldom clicked to the second or third page ofsearch results, a finding common to over seventy percent of Internet searches (Schwartz, 2014).

    READING LESS AND LEARNING MORE

57

Finally, lateral reading relies on canny strategies and heuristics for navigating theInternet. Some of these are easy to teach, like knowing that Google searches for contiguous
words only when they are placed in quotation marks, or that right-clicking opens up new
windows along the screen’s horizontal axis (instead of piling window upon window). Other
heuristics, like lateral reading and click restraint, are more involved and are undergirded by anunderstanding of the Internet and how it differs from print sources.
Teaching strategies like lateral reading and click restraint—as well as the knowledge thatenables their effective use—will not solve all our information woes. However, these strategies
may go part of the way in eliminating some of the most common errors in judging digitalsources. Facing an onslaught of digital information, all of us need efficient strategies forseparating truth from falsehood, good arguments from bad. As an example, consider the daunting
challenge faced by California voters trying to sift through seventeen separate initiatives on the2016 ballot: plans to increase the tobacco tax, ban plastic bags, limit the sale of ammunition,legalize recreational marijuana, require porn stars to wear condoms while filming, approve abond to build new schools, repeal the death penalty or make it easier to mete out, and so on. If
the average citizen spent ten minutes researching each initiative, we would consider this an act of
responsible citizenship. The educational challenge we face is this: What can we teach to ensurethat those ten minutes count?
Addressing this challenge in an age where we encounter the world through a screen can
begin with a course on media literacy. However, it cannot stop there. The digital revolution
demands a fundamental reconsideration of how we teach all of the core school subjects. Whatshould history teaching look like when students can go online and find “evidence” for the canard
    READING LESS AND LEARNING MORE

58

that “thousands” of black men put on grey uniforms to take up arms for the Confederacy? Or
science teaching when anti-vaxxer sites maintain a “proven” link between autism and measles
shots, despite a retraction by the journal publishing the claim and the fact that “no respectable
body of opinion” supports the linkage (Walker- Smith v. General Medical Council, 2012)? Or
language arts class when ad hominem arguments and “alternative facts” overwhelm civil
discourse?
Technology can do many things, but it can’t teach discernment. In Thomas Jefferson’sday, an earlier technological revolution, the advent of moveable type, drastically lowered the costof print. Jefferson watched with dismay as untrustworthy pamphlets and broadsides of dubious
quality littered the streets. Jefferson understood that a dark side accompanies the expansion of
expression. We would do well to take his solution to heart, as well as to realize it won’t comecheaply: “If we think [the people] not enlightened enough to exercise their control with awholesome discretion, the remedy is not to take it from them, but to inform their discretion byeducation.”    READING LESS AND LEARNING MORE

59

References
Agosto, D. E. (2002). A model of young people’s decision-making in using the Web. Library &Information Science Research, 24, 311–341. doi:10.1016/S0740-8188(02)00131-7
American Academy of Pediatrics. (2014). Stigma: At the root of ostracism and bullying.
Retrieved from https://www.aap.org/en-us/about-the-aap/aap-press-room/pages/StigmaAt-the-Root-of-Ostracism-and-Bullying.aspx
American College of Pediatricians. (2015). “P” for pedophile. Retrieved fromhttps://www.acpeds.org/p-for-pedophile
Anderson, N. (2016, April 1). Applied to Stanford or Harvard? You probably didn’t get in.
Admit rates drop, again. Washington Post. Retrieved fromhttps://www.washingtonpost.com/news/grade-point/wp/2016/04/01/applied-to-stanfordor-harvard-you-probably-didnt-get-in-admit-rates-drop-again/?noredirect=on
Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online
sources. Cognition and Instruction, 30, 39-85. doi:10.1080/07370008.2011.636495
Bennett, S. (2012). Digital natives. In Z. Yan (Ed.), Encyclopedia of cyber behavior: Volume 1(pp. 212-219). Hershey, PA: IGI Global.
Bounegru. L., Gray, J., Venturini, T., & Mauri, M. (2017). A field guide to “fake news” andother information disorders. Amsterdam, NL: Public Data Lab. Retrieved fromhttps://fakenews.publicdatalab.org/
Bradshaw, W., Weight, D. G., & Packard, T. (2011, Mary 3). Same sex attraction not a matter ofchoice. The Salt Lake Tribune. Retrieved fromhttp://archive.sltrib.com/printfriendly.php?id=51356807&itype=cmsid
    READING LESS AND LEARNING MORE

60

Brand-Gruwel, S., Kammerer, Y., van Meeuwen, L., & van Gog, T. (2017). Source evaluationof domain experts and novices during Web search. Journal of Computer Assisted
Learning, 33, 234-251. doi:10.1111/jcal.12162
Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by
experts and novices: Analysis of a complex cognitive skill. Computers in Human
Behavior, 21, 487-508. doi:10.1016/j.chb.2004.10.005
Bulger, M., & Davison, P. (2018). The promises, challenges, and futures of media literacy. New
York, NY: Data & Society. Retrieved from https://datasociety.net/output/the-promiseschallenges-and-futures-of-media-literacy/
California teacher tenure law unconstitutional, judge says. (2014, June 10). Retrieved fromhttp://abc7.com/education/california-teacher-tenure-law-unconstitutional/106228/
Coleman, T. (2010). Misinformation from doctors . . . out to hurt students? Retrieved fromhttps://www.aclu.org/blog/speakeasy/misinformation-doctorsout-hurt-students
Common Sense Media. (2012). Identifying high-quality sites. Retrieved fromhttps://www.commonsensemedia.org/educators/lesson/identifying-high-quality-sites-6-8
Common Sense Media. (n.d.). Scope and sequence: Common Sense K-12 digital citizenshipcurriculum. Retrieved from https://www.commonsense.org/education/scope-andsequence
Daniels, J. (2009). Cloaked websites: Propaganda, cyber-racism and epistemology in the digitalera. New Media and Society, 11, 659-683. doi:10.1177/1461444809105345
Dougherty, J. & Nawrotzki, K. (Eds.). (2013). Writing history in the digital age. Ann Arbor, MI:
University of Michigan Press.
    READING LESS AND LEARNING MORE

61

Elstein, A. S., Kagan, N., Shulman, L. S., Jason, H., & Loupe, M. J. (1972). Methods and theory
in the study of medical inquiry. Journal of Medical Education, 47, 85-92.
Ericsson, K. A., Charness, N., Feltovich, P, & Hoffman, R. (2006). Cambridge handbook ofexpertise and expert performance. New York, NY: Cambridge.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.).
Cambridge, MA: MIT Press.
Espelage, D. (2011). Bullying and the lesbian, gay, bisexual, transgender, questioning (LGBTQ)
community. Retrieved from https://www.stopbullying.gov/atrisk/groups/lgbt/white_house_conference_materials.pdf
Fogg, B. J. (2003, April). Prominence-interpretation theory: Explaining how people assess
credibility online. Paper presented at the  ACM Conference on Human Factors in
Computing Systems, Ft. Lauderdale, FL.
Foley, B. (2017, December 31). Spread of fake news prompts literacy efforts in schools. PBS
New Hour. Retrieved from https://www.pbs.org/newshour/education/spread-of-fakenews-prompts-literacy-efforts-in-schools
Gasser, U., Cortesi, S., Malik, M., & Lee, A. (2012). Youth and digital media: From credibility
to information quality. Cambridge, MA: Berkman Center for Internet and Society.
Graves, L. (2013, November 13). Corporate America’s new scam: Industry P.R. firm poses asthink tank! Salon. Retrieved fromhttp://www.salon.com/2013/11/13/corporate_americas_new_scam_industry_p_r_firm_po
ses_as_think_tank/    READING LESS AND LEARNING MORE

62

Hargittai, E., Fullerton, L., Menchen-Trevino, E., & Thomas, K. Y. (2010). Trust online: Young
adults’ evaluation of web content. International Journal of Communication, 4, 468-494.
doi:1932–8036/20100468
Herold, B. (2018, February 28). Media literacy. Education Week, 37(22), 4.
Hilligoss, B., & Rieh, S. Y. (2008). Developing a unifying framework of credibility assessment:
Construct, heuristics, and interaction in context. Information Process and Management,
44, 1467-1484. doi:10.1016/j.ipm.2007.10.001
Iding, M. K., Crosby, M. E., Auernheimer, B., & Klemm, B. (2009). Web site credibility: Why
do people believe what they believe? Instructional Science, 37, 43-63.
doi:10.1007/s11251008-9080-7
Julien, H., & Barker, S. (2009). How high school students find and evaluate scientificinformation: A basis for information literacy skills development. Library & Information
Science Research, 31, 12-17. doi:10.1016/j.lisr.2008.10.008
Jussim, L. (2012). Social perception and social reality. New York, NY: Oxford University Press.
Kahne, J., Hodgin, E., & Eidman-Aadahl, E. (2016). Redesigning civic education for the digitalage: Participatory politics and the pursuit of democratic engagement. Theory & Researchin Social Education, 44, 1-35. doi:10.1080/00933104.2015.1132646
Klurfeld, J., & Schneider, H. (2014). News literacy: Teaching the Internet generation to makereliable information choices. Washington, DC: Brookings Institution. Retrieved fromhttps://www.brookings.edu/wp-content/uploads/2016/07/Klurfeld-Schneider_NewsLiteracy_June-2014.pdf
Kranish, M. (2005, July 31). Beliefs drive research agenda of new think tanks: Study on gay
    READING LESS AND LEARNING MORE

63

adoption disputed by specialists. Boston Globe. Retrieved fromhttp://archive.boston.com/news/nation/washington/articles/2005/07/31/beliefs_drive_rese
arch_agenda_of_new_think_tanks/
Leinhardt, G., & Young, K. M. (1996). Two texts, three readers: Distance and expertise inreading history. Cognition and Instruction, 14, 441-486. doi:10.1207/s1532690xci1404_2
Lenz, R. (2012). American College of Pediatricians defames gays and lesbians in thename of protecting children. Retrieved from https://www.splcenter.org
Lipton, E. (2014, February 9). Fight over minimum wage illustrates web of industry ties.
The New York Times. Retrieved fromhttp://www.nytimes.com/2014/02/10/us/politics/fight-overminimum-wage-illustratesweb-of-industry-ties.html
List, A., Grossnickle, E. M., & Alexander, P. A. (2016). Undergraduate students’ justifications
for source selection in a digital academic context. Journal of Educational Computing
Research, 54, 22-61. doi:10.1177/0735633115606659
Liu, C., White, R. W., & Dumais, S. (2010). Understanding web browsing behaviors throughWeibull analysis of dwell time. Proceedings of the 33rd International ACM SIGIR
Conference on Research and Development in Information Retrieval. Geneva,
Switzerland.
Lucassen, T., & Schraagen, J. M. (2011). Factual accuracy and trust in information: The role ofexpertise. Journal of the American Society for Information Science and Technology, 62,
1232-1242. doi:10.1002/asi.21545
McGrew, S., Breakstone, J., Ortega, T., Smith, M., & Wineburg, S. (2018). Can students
    READING LESS AND LEARNING MORE

64

evaluate online sources? Learning from assessments of civic online reasoning. Theory& Research in Social Education, 46, 165-193. doi:10.1080/00933104.2017.1416320
Media Education Lab. (n.d.) Who do you trust? Retrieved fromhttp://mediaeducationlab.com/secondary-school-unit-2-who-do-you-trust
Mozilla. (n.d.). Web literacy. Retrieved from https://learning.mozilla.org/en-US/web-literacy
National Association for Media Literacy Education. (2007). Core principles of media literacy
education in the United States. Retrieved from https://namle.net/publications/core
principles/
News Literacy Project (n.d.). Ten questions for fake news detection. Retrieved fromwww.thenewsliteracyproject.org/sites/default/files/GOTenQuestionsForFakeNewsFINAL.pdf
Nielsen, J. (2011). How long do users stay on web pages? Retrieved fromhttps://www.nngroup.com
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007). In Google wetrust: Users’ decisions on rank, position, and relevance. Journal of Computer-MediatedCommunication, 12, 801-823. doi:10.1111/j.1083-6101.2007.00351.x
PARCC Model Content Frameworks. (2011). Retrieved from https://parccassessment.org/content/uploads/2017/11/PARCCMCFELALiteracyAugust2012_FINAL.pd
f
Pirolli, P., & Card, S., (1995, May). Information foraging in information access environments.
Paper presented at the CHI Conference on Human Factors in Computing Systems,
Denver, CO.
    READING LESS AND LEARNING MORE

65

Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively
responsive reading. Hillsdale, NJ: Erlbaum.
Robb, A. (2017, November 16). Anatomy of a fake news scandal. Rolling Stone. Retrieved fromhttps://www.rollingstone.com/politics/news/pizzagate-anatomy-of-a-fake-news-scandalw511904
Schwartz, B. (2014). A new click rate study for Google organic results. Retrieved fromhttps://marketingland.com/new-click-rate-study-google-organic-results-102149
Shanahan, C., Shanahan, T., & Misischia, C. (2011). Analysis of expert readers in three
disciplines: History, mathematics, and chemistry. Journal of Literacy Research, 43, 393429. doi:10.1177/1086296X11424071
Shanahan, T. (2012, June 18). What is close reading? [Blog post]. Retrieved fromhttp://shanahanonliteracy.com/blog/what-isclosereading#sthash.mxxi0paG.rtIrn0KW.dpbs
Shanahan, T., & Shanahan, C. (2008). Teaching disciplinary literacy to adolescents: Rethinking
content-area literacy. Harvard Educational Review, 78, 40–59.
SourceWatch (n.d.). Portal: Front groups. Retrieved fromhttps://www.sourcewatch.org/index.php/Portal:Front_groups
Southern Poverty Law Center. (2017). Active hate groups 2016. Retrieved fromhttps://www.splcenter.org/fighting-hate/intelligence-report/2017/active-hate-groups-2016
Stanford University. (2017). Facts 2017: Academics. Retrieved fromhttp://facts.stanford.edu/academics/undergraduate-facts.

    READING LESS AND LEARNING MORE

66

Stanford University. (2015). Our selection process: Applicant profile. Retrieved fromhttp://admission.stanford.edu/basics/selection/profile15.html
Taylor, K. L., & Dionne, J. P. (2000). Accessing problem-solving strategy knowledge: The
complementary use of concurrent verbal protocols and retrospective debriefing. Journalof Educational Psychology, 92, 413-425. doi:10.1037/0022-0663.92.3.413
Throckmorton, W. (2011, October 6). The American College of Pediatricians versus theAmerican College of Pediatrics: Who leads and who follows? [Blog post]. Retrieved
from http://www.patheos.com/blogs/warrenthrockmorton/2011/10/06/the-americancollege-of-pediatricians-versus-the-american-academy-of-pediatrics-who-leads-and-whofollows/
Trumbull, D. (2013). Bullying at school: Never acceptable. Retrieved fromhttp://www.acpeds.org/the-college-speaks/position-statements/societal-issues/bullying-atschool-never-acceptable.
Turban, J. (2017, May 8). The American College of Pediatricians is an anti-LGBT group [Blogpost]. Retrieved from https://www.psychologytoday.com
Vergara v. California. (n.d.). Retrieved from http://studentsmatter.org/case/vergara/
Vergara v. State of California. (n.d.). Retrieved from http://www.cta.org/Vergara
Walker- Smith v. General Medical Council. (2012). EWHC (UK).
Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2009). How students evaluate informationand sources when searching the World Wide Web for information. Computers &Education, 52, 234-246. doi:10.1016/j.compedu.2008.08.003

    READING LESS AND LEARNING MORE

67

Warren, J. E. (2011). “Generic” and “specific” expertise in English: An expert/expert study inpoetry interpretation and academic argument. Cognition and Instruction, 29, 349-374.
doi:10.1080/07370008.2011.607929
Wathen, C. N., & Burkell, J., (2002). Believe it or not: Factors influencing credibility on theweb. Journal of the American Society for Information Science and Technology, 53, 134144. doi:10.1002/asi.10016
Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A.
(2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks.
American Educational Research Journal, 46, 1060-1106.
doi:10.3102/0002831209333183
Wineburg, S. (1991). Historical problem solving: A study of the cognitive processes used in theevaluation of documentary and pictorial evidence. Journal of Educational Psychology,
83, 73–87. doi:10.1037/0022-0663.83.1.7
Wineburg, S. (1998). Reading Abraham Lincoln: An expert/expert study in the interpretation ofhistorical texts. Cognitive Science, 22, 319–346. doi:10.1016/S0364-0213(99)80043-3
Wineburg, S., Breakstone, J., McGrew, S., & Ortega, T. (2016). Evaluating information: The
cornerstone of civic online reasoning. Stanford, CA: Stanford History Education Group.
Wineburg, S., & McGrew, S. (2016, November 1). Why students can’t Google their way to thetruth. Education Week, 36(11), 22. Retrieved fromhttps://www.edweek.org/ew/articles/2016/11/02/whystudents-cant-google-their-way
to.html
Wolf, M. (2007). Proust and the squid: The story and science of the reading brain. New York,    READING LESS AND LEARNING MORE
NY: HarperCollins.

  
  

68

READING LESS AND LEARNING MORE69

ACKNOWLEDGEMENTS
This research was generously supported by the Robert R. McCormick Foundation and the
Spencer Foundation (Grant 201600012, Sam Wineburg, Principal Investigator). The authors
alone are responsible for its contents. We gratefully acknowledge the indispensable feedback ofMike Caulfield, Joel Breakstone, Teresa Ortega, Mark Smith, and two anonymous reviewers.

    READING LESS AND LEARNING MORE

70

Footnotes
Footnote 1: We sampled historians and fact checkers on the basis of professional affiliation andfull-time employment as historians at four-year colleges and universities. We did not ask theseparticipants for information about their race or ethnicity, languages spoken, or country of origin.
Footnote 2: In addition to the tasks presented here, the full protocol included 1) brief evaluationsof four static sites, 2) an open web search on a historical question with contemporary
ramifications, and 3) locating the registrant of a website. The findings from those tasks arebroadly consistent with what we present here. A description of the full protocol is available fromthe authors.
Footnote 3: The statement from Collins, which was posted on the National Institutes of Healthwebsite, is also available via the Web Archive:
http://web.archive.org/web/20110727115017/http://www.nih.gov/about/director/04152010_state
ment_ACP.htm.
Footnote 4: The stance is prominent in other parts of the website, such as a “Position Statement”
entitled “On the Promotion of Homosexuality in the Schools,” which states that “the homosexual
lifestyle carries grave health risks”; that “validating a student’s same-sex attraction during theadolescent years is premature and may be harmful;” and “sexual reorientation therapy can beeffective.” Retrieved from https://www.acpeds.org/wordpress/wp-content/uploads/On-thePromotion-of...pdf.
Footnote 5: The University of Alaska/Fairbanks guide is located at https://library.uaf.edu/ls101evaluation, while Illinois State University’s ishttps://guides.library.illinoisstate.edu/evaluating/craap.