21 OCTOBER 2020EDUCATING FORMISUNDERSTANDING:HOW APPROACHES TO TEACHING DIGITAL LITERACY MAKE
STUDENTS SUSCEPTIBLE TO SCAMMERS, ROGUES, BAD ACTORS,AND HATE MONGERS
WORKING PAPER A-21322
SAM WINEBURG, JOEL BREAKSTONE, NADAV ZIV, AND MARK SMITH | STANFORD UNIVERSITYThe Stanford History Education Group: sheg@gse.stanford.edu

3

EXECUTIVE SUMMARY4INTRODUCTION

4THE STUDY

7

STRATEGIES THAT FAIL

14

IMPLICATIONS

18

RECOMMENDATIONS

20ENDNOTES

How to cite this report: Sam Wineburg, Joel Breakstone, Nadav Ziv, & Mark Smith,"Educating for Misunderstanding: How Approaches to Teaching Digital Literacy
Make Students Susceptible to Scammers, Rogues, Bad Actors, and Hate Mongers"(Working Paper A-21322, Stanford History Education Group, Stanford University,
Stanford, CA, 2020). https://purl.stanford.edu/mf412bt5333.

Executive Summary
Russian troll farms sow disinformation. Fake news runs amok onsocial media. Bots impersonate real people. Real people assume false
identities. How do we know what’s true? The only thing certain in our
digital age is our uncertainty.
This confusion impairs our ability to make wise,fact-based decisions that shape our nation’s
future. What has been the educational responseto this predicament? The most common
approaches—“media literacy,” “news literacy,”“digital literacy,” and even that catch-all, “criticalthinking"—share a commitment to teachingpeople how to tell truth from fiction, recognize
hoaxes, and practice caution before passing
along dubious content to family and friends. Are
these approaches effective in helping today’s
college students make thoughtful choices about
what to believe?
The Study
To address this question, we surveyed 263
college sophomores, juniors, and seniors at alarge state university on the East Coast. On one
task, students evaluated the trustworthiness of a“news story” that came from a satirical website.
On a second task, students evaluated the website
of a group claiming to sponsor “nonpartisan
research.” In fact the site was created by aWashington, D.C., public relations firm run by
a former corporate lobbyist. For both tasks,
students had a live internet connection and were
instructed to “use any online resources” to maketheir evaluations.
The Results
Students struggled. They employed inefficient
strategies that made them vulnerable to forces,
whether satirical or malevolent, that threaten
informed citizenship.
STANFORD HISTORY EDUCATION GROUP• Over two-thirds never identified the “newsstory” as satirical.
• Ninety-five percent never located the PR firm
behind the supposedly “nonpartisan” website.
Often students:• Focused exclusively on the website or prompt,
rarely consulting the broader web• Trusted how a site presented itself on its
About page
• Applied out-of-date and in some cases incorrect
strategies (such as accepting or rejecting a site
because of its top-level domain)• Attributed undue weight to easily manipulated
signals of credibility—such as an organization’s
non-profit status, its links to authoritative
sources, or “look”
Students Learned What We Taught Them
Alarmingly, students’ approach was consistent withguidelines that can be found on many college anduniversity websites. Sometimes these materials are
just plain wrong. Sometimes they are incomplete.
Sometimes they are so inconsistent that they offer
scant guidance for navigating the treacherous
terrain of today’s internet.
Educational institutions must do a better job
helping students become discerning consumers ofdigital information. Our society and its democratic
institutions depend on it.

3INTRODUCTION
Educators have addressed the information
crisis in a variety of ways. Common to each is
a commitment to teaching young people how
to tell truth from fiction, recognize hoaxes,
and practice caution before passing along
dubious content to family and friends. Do theseapproaches help today’s college students makeinformed choices about what to believe?
The answer to this question has implications
that extend beyond students’ college years.
Recognizing the duty to prepare informed
citizens, dozens of colleges are weighing
requirements in “civic literacy,” with some
proposing that students pass a test in order tograduate.1 Ensuring that students understand thechecks and balances of American government or
the freedoms guaranteed by the Bill of Rights isimportant. But a civics test worthy of our digitalage must assess students’ ability to distinguishbetween reliable and misleading information.
A free society that can’t tell the difference
jeopardizes its freedom.
For nearly every issue facing today’s young
people—from climate change and income
inequality to student debt and the legalizationof marijuana—the place to find up-to-date
information is the internet. However, bothreliable information and deceitful sources flood
students’ screens. To thrive as members of ademocratic society, students must be able to sort
through this digital thicket.
THE STUDY
The internet is an indispensable fixture of college
life. Across the curriculum, students are asked to
go online to do research and complete course
assignments. College and university websites
provide guidelines about evaluating a range of

EDUCATING FOR MISUNDERSTANDING

sources. As misinformation and fake news havebecome features of modern life, these websites
commonly provide advice aimed at helping
students evaluate digital content.
To understand how today’s students evaluate
digital sources, we surveyed sophomores,
juniors, and seniors at an East Coast university
enrolling over 16,000 undergraduate students.
According to the 2019 rankings of U.S. News& World Report, this university is considered“selective.”
We sent out a description of the study toprofessors across a range of departments. Twelve
professors agreed to give up thirty minutes ofscheduled class time to participate. Students
were drawn from courses in biology, architecture,public health, classics, history, social work,
education, and anthropology. In all, 263 students
participated: 89% were juniors or seniors and
11% sophomores. We excluded freshmen
because we wanted to study students who had
already completed introductory college courses.
In each classroom, a member of the researchteam described the study and explained thatparticipation was voluntary. Students who
volunteered logged on to the university’s networkwith their own laptops and typed their answers
into Qualtrics, an online survey platform. Abouthalf of the students completed a task about “fakenews”; the other half a task about a website that
hid its true backers. As a token of appreciation,
students were provided with a link to redeem a$5 Starbucks gift card.
Task 1: Fake News
This task presented students with an image of amessaging app that showed what appeared to
be a news story from a publication called The
Seattle Tribune. The story carried the headline“Government Considers Further Discussion

4Surrounding Religion in School.” Students wereasked to evaluate the trustworthiness of thearticle and could use “any online resources”
to make their evaluation. A timer that counted
down from seven minutes appeared in the rightcorner of their screens.
Students who typed “Seattle Tribune” into theirbrowsers were able to answer the prompt inseconds (Figure 1). They quickly found that The
Seattle Tribune’s website contained a disclaimer
explaining that it was a satirical site and that all its
news articles were “fictional.” A junior sports andrecreation major wrote that the site “appears to bemuch like The Onion . . . It should be taken with agrain of salt.” A junior architecture major quoted
directly from The Seattle Tribune's disclaimer(Figure 2): “‘The Seattle Tribune is a news andentertainment satire web publication. . . . All news
articles contained within The Seattle Tribune are
fictional and presumably satirical.’”Figure 1
The Seattle Tribune Task

Look at the message below and answer thequestion that follows.
You may use any online resources to help
you. You have up to seven minutes tocomplete the task.

Responses like these, however, were decidedly
in the minority. About two-thirds of the students(83/125) failed to recognize The Seattle Tribune
as a satirical site (Figure 3). They either judged
the article as trustworthy (25/125) or rejected it
based on an irrelevant feature, such as its top-leveldomain (58/125). In both cases, students’ attention
remained glued to the article itself, hardly everleaving the prompt to search the open web.
Among students who judged the article as credible,
some provided responses that showed little effort.
Others diligently read the item and justified their
responses by noting that the story came from“local news,” that it “looks professional,” that it
addressed a “timely subject,” or that it matched“what I think.” These students placed their faith inweak signals such as the appearance of the article
or used their personal beliefs as barometers for
credibility. A senior majoring in anthropology based
her assessment on the fact that the story contained“statistical facts” that “can be checked if needed.”
STANFORD HISTORY EDUCATION GROUP

How trustworthy is the article in thescreenshot that Toni shared?
Explain your answer.

5

Figure 2
The Seattle Tribune Disclaimer

Task 2: Cloaked Website
On the cloaked website task, students judged
the credibility of an article that appeared onminimumwage.com, a site that claims to provide
nonpartisan information about minimum wage
policy. Its About page sends readers to the site’ssponsor: the Employment Policies Institute (EPI;
www.epionline.org), which describes itself as“a non-profit research organization dedicatedto studying public policy issues surrounding
employment growth.” Students were given twelve
minutes to assess the trustworthiness of theinformation provided by minimumwage.com.
Minimumwage.com and its parent organization,
the Employment Policies Institute, embody
how many public policy campaigns are waged
on today’s internet. Sites proclaiming to offer
nonpartisan research often turn out to be thework of special interests. Minimumwage.com
and the Employment Policies Institute are
run out of the offices of the Washington, D.C.,
public relations firm of Berman and Company.
The firm, headed by former lobbyist Richard
Berman, receives funding from the restaurant
lobby, an interest group dedicated to keeping theminimum wage low.2
EDUCATING FOR MISUNDERSTANDING

Figure 3
Detecting Fake News/Satire

34%
CORRECT

Identified site
as satire

66%

INCORRECTDid not identify
site as satiren = 125

Establishing the connection between the
Employment Policies Institute and Berman’s
PR firm takes less than a minute by searchingfor “Employment Policies Institute” (Figure 4).
Students who did so easily found informationthat cast doubt on the group’s “nonpartisan”
stance. A senior history and anthropology major
6wrote that “a quick internet search shows thatthe ‘institute’ is run and owned by a restaurant
industry lobbyist who has a vested interest in
not having the minimum wage increased.” A
senior English major looked past the site’s neutral
language and searched the open web: “While the[website’s] language seems professional, and thearticle mentioned newspapers of record . . . closer
research into the validity of the website reveals
that it is a pseudo think-tank website that isfunded by groups with corporate interests against
labor unions, workers rights and labor rights.”
Answers like these two were exceedingly rare.
Eighty-five percent of students never turnedto the broader internet to investigate thetrustworthiness of minimumwage.com. Instead,they relied on the information provided by thesite itself. Nearly forty percent (54/138) judged
minimumwage.com trustworthy. Forty-five
percent (63/138) rejected the site but based their
judgment on irrelevant features, like the site’s
top-level domain. Ten percent (14/138) remained
unsure of the site’s quality. Only seven students
out of 138 connected minimumwage.com to the
public relations firm of Berman and Company(Figure 5).

Figure 4
Search Sequence for Identifying Group Behindminimumwage.com
STEP 1Minimumwage.com indicates it is a project of
the Employment Policies InstituteDenmark’s Dollar Forty-One Menu

https://www.minimumwage.com/2014/10/denmarks-dollar-forty-one-menu/

STEP 2
Employment Policies Website
Employment Policies Institute - Public ...

https://www.epionline.org

STRATEGIES THAT FAIL
Students approached these tasks with an arrayof strategies. Too often, however, their strategies
were ineffective or, worse, led them astray.
Why did so many of these college students
make simple mistakes? Alarmingly, many of thestrategies they used were ones recommended bycollege and university websites.

STEP 3
A search for Employment Policies Institute returns
sites that cast doubt on the Institute's trustworthiness
Employment Policies Institute - Google Sea

https://www.google.com/search?q=employment+policies+institute&oq=employment+policies+institute&aqs=chrome.0.69i59.8301j0j4&sour

The most ubiquitous tool for teaching web
credibility at the college level is known as the
CRAAP test, a set of guidelines corresponding to
the categories of Currency, Relevance, Authority,
Accuracy, and Purpose (hence, CRAAP).3 A Google
STANFORD HISTORY EDUCATION GROUP

7

search brings up more than 100,000 results for the
CRAAP test, which can be found on the websitesof elite research universities, regional colleges,
and scores of institutions in between (Figure6). The CRAAP test prompts students to ask
questions (sometimes as many as 30) to assessa site’s credibility.4 While the kinds and numberof questions vary, most versions of the CRAAP
test direct students’ attention to a site’s top-leveldomain, the information on its About page, theauthority of its links, the presence or absence ofbanner ads, the listing of contact information, andthe currency and frequency of updates. The basic
assumptions of the CRAAP test are rooted in an
analog age: Websites are like print texts. The best
way to evaluate them is to read them carefully. But
websites are not variations of print documents.
The internet operates by wholly different rules.

Figure 5Students' Search Strategies for minimumwage.com

10%

Left site but
remained unsure

5%

Identified
PR firm ofBerman and Co.

85%

Stayed on Siten = 138

Figure 6
Examples of CRAAP Test from College and University Websites

California State University, Chico
Indiana University Southeast

Evaluating Information – Applying the CRAAP Test
Meriam Library California State University, Chico

When you search for information, you're going to find lots of it . . . but is it good information? You will haveto determine that for yourself, and the CRAAP Test can help. The CRAAP Test is a list of questions to help
you evaluate the information you find. Different criteria will be more or less important depending on yoursituation or need.
Key:  indicates criteria is for Web

Evaluation Criteria

Pikes Peak Community College

Currency: The timeliness of the information.

 When was the information published or posted?
 Has the information been revised or updated?
 Does your topic require current information, or will older sources work as well?
Are the links functional?

Relevance: The importance of the information for your needs.

 Does the information relate to your topic or answer your question?
 Who is the intended audience?
 Is the information at an appropriate level (i.e. not too elementary or advanced for your needs)?
 Have you looked at a variety of sources before determining this is one you will use?
 Would you be comfortable citing this source in your research paper?

Authority: The source of the information.

 Who is the author/publisher/source/sponsor?
 What are the author's credentials or organizational affiliations?
 Is the author qualified to write on the topic?
 Is there contact information, such as a publisher or email address?
Does the URL reveal anything about the author or source?
examples: .com .edu .gov .org .net

Accuracy: The reliability, truthfulness and correctness of the content.

 Where does the information come from?
 Is the information supported by evidence?
 Has the information been reviewed or refereed?
 Can you verify any of the information in another source or from personal knowledge?
 Does the language or tone seem unbiased and free of emotion?
 Are there spelling, grammar or typographical errors?

UC/San Diego

Purpose: The reason the information exists.

 What is the purpose of the information? Is it to inform, teach, sell, entertain or persuade?
 Do the authors/sponsors make their intentions or purpose clear?

 Is the information fact, opinion or propaganda?
Miami Dade College
 Does the point of view appear objective and impartial?

 Are there political, ideological, cultural, religious, institutional or personal biases?
9/17/10EDUCATING FOR MISUNDERSTANDING

8

Deceived by Domains
Instructional materials like the CRAAP test train
students’ attention on a site’s top-level domain.
They warn that the commercial nature of dotcoms makes them less trustworthy than dot-orgsor dot-edus. “Anyone can get [a dot-com],” alerts
one college guide. “Be cautious in using these.
Confirm the information somewhere else.”5
Evaluating a site by its top-level domain courts
error. Some students rejected The Seattle Tribune
not because they identified it as “fake news.”
Rather, their evaluations began—and abruptly
ended—at the URL bearing the letters com. A
junior education major wrote, “You see a .com anda random URL name. This is not a reliable source.”
A senior parks and recreation management major
concluded, “It is a .com so I would not consider
it very trustworthy.” A junior interior architecture
major noted, “[The Seattle Tribune] is not thattrustworthy because the website ends in .com.” In
total, more than 10% (13/125) of students referredto the site’s top-level domain in their evaluations.
A similar pattern emerged on the second task.
More than 15% (21/138) included the site’s URL
in their explanations of the trustworthiness ofminimumwage.com. A junior classics major
reasoned, “The dot-com is a giveaway that it
received revenue based on advertisements andclicks.” A junior special education major seconded:“What is throwing me off is the .com at the end of
the website. I was always taught during research
assignments not to trust those.”
The problem with judging a site by its top-leveldomain—in these cases, flatly rejecting dotcoms—is that practically every bona fide news
source, from The New York Times to The Wall Street
Journal to The Washington Post, is registered asa dot-com. The wholesale rejection of dot-coms
leaves students little recourse for learning about
the political and social world.
STANFORD HISTORY EDUCATION GROUP

If dot-coms ignited students’ suspicion, whatboosted their confidence? A website that ended
in dot-org. On the cloaked website task, asophomore social work major elaborated: “I donot think this is a reliable source because themost reliable sources have .org at the end of theirURL.” A senior majoring in English and education
added, “Any time I am completing any sort ofresearch, I prefer to use .org and .edu sites.” On
the fake news task, a junior education major
maintained, “In order for it to be trustworthy
it should have a .edu, .org, or a .gov.” Anothereducation major answered similarly, “The article
is not very trustworthy because the website is notfrom a .gov or .org website.”
Students’ belief about the benevolence of dot-org
sites is widely shared. A 2013 survey showed that
half of Americans (and greater percentages inFrance, India, and Brazil) believed an organization
had to pass an approval process and “meetcriteria” to register as a dot-org.6 Such beliefs
are not only wrong. They’re dangerous. Dotorg is a “legacy” domain, one of the top-leveldomains created by the U.S. government in the
mid-1980s. The dot-org domain was created asa “catch-all,” a category to slot sites that didn’t
fit neatly in domains such as .edu (education),.gov (government), or .mil (military).7 Unlike
these three domains, each of which maintained
a vetting process from the start, dot-org was
created—and has remained— “open” (in thatsense, just like dot-com). Anyone can buy a dotorg domain without filing special paperwork,without proving benevolent intent, or even
without showing nonprofit status.8
There are over ten million registered dot-orgs.9
Many are legitimate groups that have 501(c)(3) tax-exempt status from the United States
Internal Revenue Service (IRS). Many others,
however, don’t. In fact, there’s nothing to preventa commercial site from obtaining a dot-org
9

Table 1:
College and University Advice about Dot-Org & Nonprofit Status12
Institution

Advice*"Is the Web document linked to a federal agency (.gov),
a non-profit site (.org), an educational institution(.edu), or a business (.com)?"

Harvard University".org: typically used by non-profit entities that are noteducational or commercial entities."
Northwestern University".org for nonprofit organizations"

Boston College

Pennsylvannia
State University"Who is supplying the information? Is it an educational
institution (.edu extension)? A government agency(.gov)? A commercial supplier (.com)? A non-profit
organization (.org)?"".org - Non-Profit Organization"

University of Washington*emphasis ours

domain. Craigslist, one of the largest classified
advertising sites in the world, is registered as
craigslist.org. Religious cults, tobacco industry(and now vaping industry) trade groups, andgroups associated with “dark money,” suchas the Koch brothers’ Americans for Prosperity
Foundation, all bear the dot-org domain. Moremenacingly, a study of the groups listed on theSouthern Poverty Law Center’s 2019 “Hate Map”
showed that among those sites that maintain a
web presence, forty-nine percent carried the dotorg domain.10
College and university guidelines often give dotorgs a free pass. Some guidelines equate dot-orgs
EDUCATING FOR MISUNDERSTANDING

with nonprofits and say nothing more—a stancethat fuels the misconception that the dot-org
brand alone makes a site credible.
Web evaluation guidelines often link dot-org with
tax-exempt status, a designation issued by theUnited States Internal Revenue Service (Table
1). On the cloaked website task, ten students
cited the Employment Policies Institute’s nonprofit status when justifying the trustworthinessof minimumwage.com. A senior English major
reasoned: “[Employment Policies Institute]is a not-for-profit organization that has been
operating since 1991. I don't see any bias, andit appears that they are primarily research and10data driven.” A senior geography major approvedof the site by explaining that “the information iscited from a NON-profit research organization.”
Yet, like a dot-org domain, tax-exempt status
does not ensure that a group acts on behalf ofthe public good. Hundreds of trade associations
and lobby groups carry the designation. Eachyear the IRS is overwhelmed by applications for
tax-exempt status. In 2015 alone, it reviewed
101,962 applications. Of these, 94%, or 95,372,
were approved. Research by the Stanford Centeron Philanthropy and Civil Society concluded thatobtaining non-profit status “is an embarrassingly
easy thing to do. It is hardly an exaggeration
to say that when it comes to oversight of theapplication process to become a public charity,
nearly anything goes.”11
About Page
A website’s About page figures prominently on
many college and university guides. Northwesternadvises students to consider whether the site has
an “about” section or “introductory/backgroundinformation that you can review to help determine
a mission, point of view, or agenda,” andBrandeis tells students to “read the ‘About’ pagecarefully.” When guiding students how to assess

an organization’s credibility, Penn State points
students to “look for an ‘About Us’ link on the
homepage.”13
Many students accorded a special status to
About pages. They failed to realize that for manygroups the About page could just as easily becalled the spin page. For cloaked websites, deceit
reaches its apex on the About page. Althoughstudents may understand that a friend’s social
media profile is a curated depiction of how theywant to appear, they seemed to think that anorganization’s About page provided an official—perhaps even vetted—description of its intent.
This pattern finds support in what students aretaught. Savannah Technical College tells students
to “go to the ‘About’ page” of a site and “look forthe credentials of the person or organization anda contact address.”14 But without corroborating
this information on other sites, how can one trust
an organization's self-portrayal? Putting stock ina site's About page is advice that's widely shared.
National Public Radio, in an article entitled “Fakeor Real? How to Self-Check the News and Get
the Facts,” tells readers that the language on
an About page should be “straightforward” andadvises them to be skeptical if a site’s About page

Figure 8
NPR's "How to Self-Check the News"STANFORD HISTORY EDUCATION GROUP

11

language is “melodramatic”—implying, perhaps
unwittingly, that if language is neutral anddispassionate, a site might be more trustworthy.15

anyone with a modest budget. Functional links
are no longer a signal that differentiates between
trustworthy and dubious sites.

The objective-sounding language of the
About pages from minimumwage.com and the
Employment Policies Institute achieved its
intended effect. Fourteen percent (20/138) ofstudents relied upon information from the Aboutpages to make their evaluations. A senior publichealth major wrote that minimumwage.com
was reliable because its About page said that
“Minimum Wage is a project of the Employment
Policies Institute (EPI). Founded in 1991, EPI isa non-profit research organization dedicated to
studying public policy issues”—a direct quotefrom the website. Other students followed suit,
such as a junior social work major: “It is a project
coming from an organization known as the
Employment Policies Institute founded in 1991.
The research is being conducted by independent
economists from major universities. https://www.
minimumwage.com/about/.”

Universities also advise students to weighthe prestige and authority of a site’s links.
More prestigious links, the thinking goes,indicate a more reliable site. Cornell University
recommends that students ask whether “thelinks on [a] site lead to other reputable sites”(Table 2). Rare were colleges that warned
students that links to prestigious sources canthemselves be tools of deception.

Sites like the Employment Policies Institute hide
their real sponsors. The most important thing
to know about such groups—and others whose
intent is to hide their backers—is that they are
incredibly good at what they do. They know how
to create compelling websites. They know how tomanipulate signals of credibility. And, above all,they know how to create About pages that projectauthority and objectivity.16
Links
Students used a website’s links as proxies oftrustworthiness. This feature of web evaluation
also finds support in university guidelines, wherestudents are instructed to check if a site’s links
are in working order. In the early days of theinternet, broken links signaled unprofessionalismand decreased a site’s credibility. But today,
making sure that links work is a trivial task for
EDUCATING FOR MISUNDERSTANDING

On the cloaked website task, ten percent
of students (15/138) interpreted links torecognizable sources as markers of reliability.
Minimumwage.com linked to both the New YorkTimes and the Columbia Journalism Review. Seeing
these links boosted many students’ confidence inthe site’s trustworthiness (Table 3).
When a site like minimumwage.com links to
The New York Times or the Columbia Journalism
Review, the hope is that the reputation of the link
will carry the day—just as it did for these college
students. Minimumwage.com bets that typicalusers will hover over a link just long enoughto identify it as legitimate or, if they click on it,
will be satisfied with a cursory glance.17 Amongthe 138 respondents, only one, a sophomore
biology major, read enough of the linked New YorkTimes article to realize that it failed to support
minimumwage.com's claims: “The link within
the page that connects to The New York Times iscontradicting what the minimumwage.com has tosay. The New York Times explains that Denmark is
actually thriving when it comes to higher minimum
wage. The Minimum Wage website is suggesting
the opposite.”Today’s hyperlink is the grandchild of thescholarly reference: the footnote. But there are12

Table 2:
College and University Advice About Links18
Institution

Advice*"Do links on this site lead to other reputable sites?"

Cornell University"Is there a bibliography or a source list? Are there
footnotes or links to reputable cited sources?"
Binghamton University"Do the links still work?"

Boston College"Are links or references to other sources up to date?"

University ofNotre Dame"Are there links to other sources to back up the facts?"

Rutgers: The State Universityof New Jersey*emphasis ours

big differences between footnotes appearing
in scholarly journals and links on unregulated
websites. For established scholarly journals,
the practice of citation is held in check byoverlapping safeguards that range from peer
reviewers acquainted with the scientific literature
to the reputation of the authors to the selfcorrecting nature of science. None of thesesafeguards apply to unregulated websites.
Minimumwage.com demonstrates the danger ofusing links as proxies of credibility. Had students
read the Columbia Journalism Review link
(Figure 9), they would have found a damning

EDUCATING FOR MISUNDERSTANDING

indictment of minimumwage.com’s sponsor. The
article’s penultimate paragraph dismisses the
Employment Policies Institute as “Rick Berman’s
restaurant-lobby front,” labeling the so-called
think tank an “anti-labor industry shill”—hardly
a ringing endorsement for an organization thatclaims nonpartisanship.19 More than anything,
this example shows how a website can enhance
its reputation by linking to authoritative sources—even if those sources undermine its credibility.

13

Table 3:
Student Evaluations Based on Links
Year

Major

Response

Senior

Biology“minimumwage.com is reliable because it cites sources like
https://mobile.nytimes.com/2014/10/28/business/international/living-wages-served-in-denmark-fast-food-restaurants.html”

Senior

Education“The article gives you a link to an article by The New York Times
which is believed to be a credible source of information. Also
the Columbia Journalism Review is used and it seems to bea credible source, but maybe it is less credible than The New
York Times.”

Sophomore

Social Work“The links on the site work, some of the sites it cited are sites I
have heard of.”

Aesthetics
Looks matter. Judging a site by its “look”
receives support from a range of universities.
Among guidelines offered by the University of
Arizona, students are told to favor sites that look
“professional” as judged by their “colors/fonts”(see Figure 10).20 Brigham Young University tells
students to consider “the look and feel of thewebsite,” adding that, “reliable websites usually
have a more professional look and feel thanpersonal Web sites.”21 The University of Illinois
recommends that students ask, “Does the page
look professional and function well?”22
Students' reliance on such surface featuresechoed loudly. A senior biology major noted thatminimumwage.com “looks reliable and it seems
like it provides factual information.” The site’sgraphics similarly impressed a senior history
major: “The interactive map [on] the page offers
a visual while looking at other states’ minimum
wage. The blog feature allows people to give theirown personal opinions and media helps give avisual to show what is being portrayed.” In an
age of slick website templates and inexpensive
hosting, relying on a site’s aesthetics plays into

EDUCATING FOR MISUNDERSTANDING

Figure 9
Excerpt from Columbia Journalism Review

the hands of rogues seeking to manufacture an
impression of respectability.
IMPLICATIONS
If the future of our democracy depends on
young people’s ability to distinguish trustworthy
from spurious information, we’ve got a lot ofwork ahead of us. College students embraced

14

ways of navigating the internet that made themvulnerable to the very forces that threaten
informed citizenship. But young people are not toblame. The roots of their strategies can often be
found in the materials educators have designed.

Figure 11
The Seattle Tribune Task & Student Response

Look at the message below and answer thequestion that follows.

Common sense seems to dictate that we should
examine a website carefully in order to judge
its credibility. That’s the advice many colleges
recommend: consider a site’s domain, examine
its About page, search for the telltale signs thatsomething’s off (like flashing banner ads or
clickbait), check to see that the site provides
contact information, and verify that its links are inworking order. This approach, however, does morethan mislead. Spending precious minutes scouring
a site before first determining whether it’s worth
the effort is a colossal waste of time.
Misplaced effort was especially apparent in thisarchitecture major’s lengthy response to the
Fake News task (Figure 11). On its face, thisstudent provides an intelligent analysis. She
examined the article from top to bottom. She
looked for bias and detected coded language.
She noticed missing references and the absenceof bibliographic pointers. She concluded that theauthor is someone who supports the teaching

Figure 10
University of Arizona’s Evaluating Web Resources

STANFORD HISTORY EDUCATION GROUP"The article provides no citations or sourcesor specific references. Additionally the article is tagged under a 'religion' tag rather
than politics or education. This suggeststhat the intended audience for the article
carries a religious bias at the outset. The
article also specifically references the Bible,
not religious texts in general so the article bears a bias toward one religion. The
article's most prominent graphics are thesocial media icons which suggests that thecontent is intended to be attention-grabbing. This is reinforced by the religious bias.
The opening statement 'further the divide
between church and state' implies that theauthor does not support the fundamental
constitutional idea of total separation ofchurch and state, further reinforcing a
biased perspective."15

of religion in public schools. In isolation, none ofthese statements is wrong.
What’s wrong is the entire approach.
The student mistakes this satire as a legitimate
source worthy of sustained attention. She expends
considerable mental effort. It’s as if she’s answeringa school writing prompt: “Analyze this text anddefend your position with evidence from thearticle.” Despite having a live internet connection
at her fingertips, the student never left the screen
displaying The Seattle Tribune prompt.
We should view this student’s response against
the backdrop of traditional schooling, which does
little to prepare students for the challenges of adigital environment. The SAT, long considereda gold standard of academic achievement,
presents students with predigested passages
and orderly multiple-choice questions. Students
are never asked whether the passage should
be believed, whether it’s reliable, or whether its
creators have a hidden agenda.
Advanced Placement exams present students
with “Document-Based Questions” (DBQs)
containing multiple documents from different
points of view. But DBQs never include the kindsof sources that slither in today’s digital swamps:
concocted evidence, Photoshopped images,
and pseudo-scholarly accounts studded withfootnotes that lead to nonexistent archives. No
wonder students are baffled trying to find truth ina digital environment.
Students don’t merely lack the skills they need to
thrive in a digital environment. It’s worse. They’vebeen taught ineffective ones.
Professional Fact Checkers
Research we have conducted with professional
fact checkers reveals a different approach toevaluating digital content.23 Fact checkers

EDUCATING FOR MISUNDERSTANDING

approach their screens knowing that things are
not what they seem. They know that signals likethe top-level domain, links to reputable sources,
or a dispassionate About page provide scantevidence of an organization’s true intent.
They understand that the Web is a web: aninterconnected network of information, where
probing a single node (particularly an unfamiliarnode) requires locating it in a vast matrix oflinked information.24 An entomologist studying
the complexity of a spider’s web would neverrestrict an analysis to a single strand. Similarly
unwise is the act of evaluating an unknown site
without first consulting the wider Web. Failingto do so approaches a 21st-century problem
equipped with rusting tools of the 20th century.
Presented with an unfamiliar site, fact checkers
always turned to other sources before reachinga conclusion. They engaged in a kind of reading
uniquely suited to the internet. Rather thandwelling on a single site, they read laterally,
opening up multiple tabs across the horizontal
axis of their browser to get a fix on the originalsite.25 In fact, sometimes their first stop was a sitemany educators tell students to avoid: Wikipedia.
But fact checkers used Wikipedia differently thanmany students do. They skipped the main article
and dove straight to references, where moreestablished sources can be found. They knewthat the more controversial the topic, the more
likely the entry was to be "protected" throughthe various locks Wikipedia applies to preventchanges by anyone except high-ranking editors.
Further, the fact checkers knew how to use a
Wikipedia article's "Talk" page, the tab hiding
in plain sight next to the main entry—a tab thatfew students know about, let alone consult.
The “Talk” page is where an article's claims are
established, disputed and, when the evidence
merits, altered.
16

Figure 12
Fact Checker’s Screenshot with Seven Open Tabs in 30 Seconds

Screenshot of professional fact checker evaluating the Employment Policies Institute. Landing on the site, the fact checker
immediately leaves it, types the name of the organization into the browser, and in less than thirty seconds has opened sevendifferent tabs, including one that ties EPI to Berman's Washington, D.C., public relations firm.

Miseducation
In medicine, the term iatrogenic disease is aeuphemism for a physician-caused malady: apatient who develops sepsis because the doctor
forgot to wash her hands; an open-heart surgery
patient who dies because an errant latex glove
was sewn into a patient’s chest cavity.
Rarely are educators’ mistakes so grave. But in theaggregate, they have serious consequences for the
choices students make about major social issues.
No student is born distrustful of dot-coms andfavorably predisposed toward dot-orgs. No
student opens up a webpage for the first time
believing that the About page is the go-to place fortrusted information. Students bring such beliefs
to the internet because they have internalized thelessons we have taught them.

EDUCATING FOR MISUNDERSTANDING

As internet theorist and blogger Mike Caulfield
has shown, the roots of the widely-disseminated
CRAAP test predate the internet. The original listof questions was formulated to help librarians
make decisions about which print materials tobuy on a limited budget.26 To retrofit the testfor evaluating websites and to make it easy toremember, a librarian at California State University,
Chico replaced the O for “objectivity” with P for“purpose,” resulting in “CRAAP.” Today, thousandsof college students are given a tool devised in
1978, a time of 8-track tape players and shoeboxsized cellphones, to deal with the challenges of
information overload wrought by high-powered
computers that fit snugly in their back pockets.

17

RECOMMENDATIONS
Anxiety over misinformation around the 2016
presidential race led to a flurry of legislation,
including in California, the nation’s most populous
state. Senate Bill No. 830, signed into law by
Governor Jerry Brown, mandated that by 2019
the California State Board of Education must
provide schools and teachers with “resources
and instructional materials on media literacy . . .
including media literacy professional development
programs for teachers.”27This legislation, however, presumed we alreadyknow what to do. But what if these “bestpractices”—the approaches widely taught by
our nation’s colleges and universities—are not
helpful? Worse, what if they make students more
susceptible to misinformation? The results of thisstudy suggest that may be the case.
We offer the following recommendations for
improving students’ web savvy. Some of theserecommendations are easy to implement andcan be pursued immediately. Others will require
more sustained investment.
What Can Be Done Now?
Cut the CRAAP. Colleges and universities
must make sure that they are not doing harmby dispensing guidelines that make students
susceptible to misinformation.
(a) Provide updated information about thenature of dot-orgs and the dangers of using
top-level domains as proxies for credibility.
(b) Help students understand that by itself,
an organization’s tax-exempt status is no
guarantee that it is working on behalf ofthe public good. Provide vivid and concrete
examples of non-profits that students
recognize as advocacy groups (or better, hate

STANFORD HISTORY EDUCATION GROUP

groups) that have nothing to do with social
benevolence.
(c) Eliminate suggestions that a site can be
judged by its “look.” In an age of inexpensive
web templates, a slick-looking site is
ludicrously easy to produce.
(d) Warn students not to take a site’s About pageat face value. Students understand that theirfriend’s Instagram account is a cultivatedview of how their friend wants to be seen. An
organization’s About page is no different.
(e) Explain that links to authoritative sources do
not guarantee credibility. Links have to beevaluated. Sites may link to sources for the
sheer allure that the source radiates—even
if the link has nothing to do with the original
claim or, in extreme cases, undermines it.
(f) Encourage the responsible use of Wikipedia
as a fact-checking resource. Had students
bothered to go to the Wikipedia entry on the
Employment Policies Institute they would
have found warning signs in the first few
sentences. Wikipedia is not perfect. Yet used
intelligently, it is invaluable to any searcher.
(g) Provide students with simple internet searchskills. Telling college students not to use
Google and to use only librarian-approved
databases, as some information specialists
do, makes as much sense as telling them topractice abstinence-only birth control.28 We
need to teach students basic Google searchskills—the selection of effective keywords,
the ease of doing a reverse image search,or even something as simple as locating asingle word on a dense, text-laden webpage
by using Control or Command-F (something
that, according to Google’s internal research,
90% of users don’t know how to do).29

18

The following three steps will require
substantial planning and investment:(1) At many colleges and universities, firstyear students are required to take “college
success” or “composition and rhetoric”
courses. Many of these courses have notbeen retooled since they were added to thecurriculum. In addition to traditional ways ofreading, today’s students need to be taught
how to read laterally in a digital environment:
how to turn to the broader Web to determine
the credibility of digital information.30(2) Institutions need to follow the example offorward-looking librarians and information
specialists at the vanguard of new approaches
to dealing with misinformation—often on
shoestring budgets at liberal arts colleges
and state universities. Consider the exampleof librarians at Stonehill College, a small
Catholic institution south of Boston. They
issue this sage advice about evaluating theAbout page: “Checking out a website’s ‘About’
page can be useful, if the reader remembersthat the ‘About’ page is written by the authorof the website in question. If the website is
untrustworthy, then what the site says about
itself is most likely untrustworthy too.” Andrea
Baer, PhD, and Daniel Kipnis at New Jersey’s
Rowan University have substantially enhanced
their library website, and at the time of thiswriting are conducting groundbreaking
research to improve their students’ web
savvy. Finally, Robert Detmering and Amber
Willenborg, librarians at the Ekstrom Libraryat the University of Louisville, have produced
a series of polished videos (with just the rightdose of snark) that provoke college students
to reevaluate their online behavior. We hopethese and similar efforts will shine a light on

EDUCATING FOR MISUNDERSTANDING

a path for other colleges and universities tofollow.31(3) Beyond these efforts, a suite of new courses
will need to be developed, piloted, andevaluated. The idea that students will become
effective web searchers after a one-off
presentation is wishful thinking. Any college
or university that claims to prepare students
for civic participation but fails to provide
systematic instruction in web credibility is
engaging in educational malpractice.
Together, these changes will require theassistance of university librarians andinformation scientists. But it would be a grave
mistake to ask these specialists to shoulder this
responsibility alone. At a time when knowledge
is under assault, when facts are assailed, andwhen expertise is discounted, the challenge ofpreparing students for the digital age must beborne by every member of the higher education
community.

19Endnotes
Colleen Flaherty, “And Civics Literacy for All,” Inside Higher Ed, February 24, 2020, https://www.insidehighered.
com/news/2020/02/24/purdue-looks-adopt-civics-knowledge-undergraduate-requirement.
1

Eric Lipton, “Fight Over Minimum Wage Illustrates Web of Industry Ties,” New York Times, February 9, 2014,
https://www.nytimes.com/2014/02/10/us/politics/fight-over-minimum-wage-illustrates-web-of-industry-ties.
html.
2“Evaluating Information — Applying the CRAAP Test,” Meriam Library, California State University, Chico,https://web.archive.org/web/20200825103508/https://library.csuchico.edu/sites/default/files/craap-test.pdf.
3

For more on the problems with the CRAAP test, see Joel Breakstone, Sarah McGrew, Mark Smith, Teresa
Ortega, and Sam Wineburg, “Why We Need a New Approach to Digital Literacy,” Phi Delta Kappan 99, no.6
(March 2018): 27-32, https://kappanonline.org/breakstone-need-new-approach-teaching-digital-literacy/.
4“The ABC’s of Evaluating Web Sites,” San Diego Mesa College, https://web.archive.org/web/20200902211513/
https://www.sdmesa.edu/library/documents/Mesa-Library-Research-Instructions-Evaluating-Websites.pdf.
5

Public Interest Registry Dashboard: January through June 2013, report published by Public Interest Registry,
https://docplayer.net/13026868-Public-interest-registry-dashboard.html.
6“The history of .ORG,” Public Interest Registry, March 31, 2014,
http://web.archive.org/web/20140602095149/https://pir.org/the-history-of-org.
7

Sam Wineburg and Nadav Ziv, “The Meaninglessness of the Dot.Org Domain,” New York Times, December 5,
2019, https://www.nytimes.com/2019/12/05/opinion/dot-org-domain.html.
8

See the Annual Report of the Public Interest Registry for 2018, https://thenew.org/app/uploads/2019/09/PIR2018-Annual-Report.pdf.
9Stanford History Education Group, “Research Note: Dot.Orgs and Hate Groups,” November 23,
2019, https://purl.stanford.edu/vt471sv7857.
10

Rob Reich, Lacey Dorn, and Stephanie Sutton, “Anything Goes: Approval of Nonprofit Status by the IRS,”
Stanford University Center on Philanthropy and Civil Society, October 2009, https://pacscenter.stanford.edu/
wp-content/uploads/2015/08/Anything-Goes-PACS-11-09.pdf. The researchers also compiled a list of themost “Eccentric Public Charities” approved by the IRS, which included the “Grand Canyon Sisters of Perpetual
Indulgence,” an international order of drag nuns; the Metempyrion Foundation, where “people with intuitive
and telepathic potential will be given an opportunity to enhance their skills,” and “Planet Jelly Donut” locatedin Hawaii but registered in California whose goal it is to spread “the common belief that the core essence of thehuman spirit is goodness.”
11“Evaluating Web Sources,” Harvard University, https://web.archive.org/web/20200930182834/https://usingsources.fas.harvard.edu/evaluating-web-sources; “Evaluating Sources (Archive): Evaluating Websites,”
Northwestern University, https://web.archive.org/web/20200930183526/https://libguides.northwestern.edu/c.
php?g=119278&p=946214; “Evaluating Internet Sources: Authority,” Boston College, https://web.archive.org/
web/20200930185239/https://libguides.bc.edu/c.php?g=44018&p=279570; “Evaluating Information Rubric,”
Penn State, https://web.archive.org/web/20200930185611/https://libraries.psu.edu/research/how/evaluatinginformation-rubric; “Savvy Info Consumers: Internet Domains,” University of Washington, https://web.archive.
org/web/20200925202604/https://guides.lib.uw.edu/research/evaluate/domains.
12

STANFORD HISTORY EDUCATION GROUP

20

Northwestern University, https://web.archive.org/web/20200930183526/https://libguides.northwestern.
edu/c.php?g=119278&p=946214; Brandeis University, https://web.archive.org/web/20200902221446/https://
guides.library.brandeis.edu/evaluatinginfo/web-and-social-media; Penn State, https://web.archive.org/
web/20200930185611/https://libraries.psu.edu/research/how/evaluating-information-rubric.
13

See Savannah Technical College’s web evaluation guidelines, https://web.archive.org/web/20200902221938/
https://www.savannahtech.edu/student-affairs/evaluating-websites/.
14

Wynne Davis, “Fake or Real? How to Self-Check the News and Get the Facts,” NPR, December 5, 2016,
https://www.npr.org/sections/alltechconsidered/2016/12/05/503581220/fake-or-real-how-to-self-check-thenews-and-get-the-facts.
15

Suzanne Goldenberg, “Lobbyist Dubbed Dr Evil Behind Front Groups Attacking Obama Power Rules,” The
Guardian, February 23, 2015, https://www.theguardian.com/environment/2015/feb/23/lobbyist-dubbed-drevil-behind-front-groups-attacking-obama-power-rules.
16

Microsoft researchers found that “dwell time” on websites was “no more than 70 seconds on 80% of
the 205,873 pages” that users visited. See Chao Liu, Ryen W. White, and Susan Dumais, “Understanding
Web Browsing Behaviors through Weibull Analysis of Dwell Time” (presentation, Proceedings of the 33rd
International ACM SIGIR Conference on Research and Development in Information Retrieval, Geneva,
Switzerland, 2010).
17“Introduction to Research,” Cornell University, https://web.archive.org/web/20200220055353/https://www.
library.cornell.edu/research/introduction; “Web Page Evaluation,” State University of New York, Binghamton,
https://web.archive.org/web/20200220060107/https://www.binghamton.edu/libraries/research/guides/
web-page-checklist.html; “Evaluating Internet Sources: Currency,” Boston College, https://web.archive.org/
web/20200220060337/https://libguides.bc.edu/c.php?g=44018&p=279573; Library Research and Information
Literacy Tutorial, University of Notre Dame, https://web.archive.org/web/20200220060621/https://potofgold.
library.nd.edu/evaluating/?page=7; “Evaluating News Sources: Steps for Evaluating the News,” Rutgers
University, https://web.archive.org/web/20200220060927/https://libguides.rutgers.edu/fake_news; “Student
Resources: Evaluating Information,” Stockton University, https://web.archive.org/web/20200220061250/
https://library.stockton.edu/c.php?g=830109&p=5926889.
18

Ryan Chittum, “The Minimum Wage and the Danish Big Mac,” Columbia Journalism Review, September 5,
2013, https://archives.cjr.org/the_audit/the_minimum_wage_and_the_danis.php.
19

See University of Arizona’s web evaluation guidelines, https://web.archive.org/web/20200220061527/
http://www.library.arizona.edu/tutorials/evaluating_web_resources/ and https://web.archive.org/
web/20200304155127/http://www.library.arizona.edu/sites/default/files/users/blakisto/Evaluation%20
Checklist.pdf.
20

See Brigham Young University’s web evaluation guidelines, https://web.archive.org/web/20200903034546/
https://guides.lib.byu.edu/c.php?g=216340&p=1428399.
21

See University of Illinois’s web evaluation guidelines, https://web.archive.org/web/20200902224745/https://www.library.illinois.edu/ugl/howdoi/evaluate_sources/.
22

See Sam Wineburg and Sarah McGrew, “Lateral Reading and the Nature of Expertise: Reading Less andLearning More When Evaluating Digital Information,” Teachers College Record 121, no. 11 (2019): 1-40, https://www.tcrecord.org/Content.asp?ContentId=22806. A fuller version is available without a paywall, Sam
Wineburg and Sarah McGrew, “Lateral Reading: Reading Less and Learning More When Evaluating Digital
23

EDUCATING FOR MISUNDERSTANDING

21

Information” (Stanford History Education Group Working Paper No. 2017-A1, October 6, 2017), https://ssrn.
com/abstract=3048994. For a short summary of this research see Katy Steinmetz, “How Your Brain Tricks You
Into Believing Fake News,” Time Magazine, August 9, 2018, https://time.com/5362183/the-real-fakenews-crisis/.
We are indebted to Michael Caulfield for helping us understand this point. See Michael Caulfield, Web Literacy
for Student Fact-Checkers (2017), https://webliteracy.pressbooks.com/.
24

For a brief video on using Wikipedia wisely, see the “Civic Online Reasoning” website of the Stanford History
Education Group, https://cor.stanford.edu/videos/how-to-use-wikipedia-wisely, as well as John Green’s Crash
Course video (done in collaboration with the Stanford History Education Group), https://www.youtube.com/
watch?v=ih4dY9i9JKE&list=PL8dPuuaLjXtN07XYqqWSKpPrtNDiCHTzU&index=6.
25

Michael Caulfield, “A Short History of CRAAP,” September 14, 2018, https://hapgood.us/2018/09/14/a-shorthistory-of-craap/.
26

California Senate Bill No. 830 (September 17, 2018), https://leginfo.legislature.ca.gov/faces/billTextClient.
xhtml?bill_id=201720180SB830.
27

Lisa Rose-Wiles, “Reflections on Fake News, Librarians, and Undergraduate Research,” Reference and User
Services Association of the American Library Association 57, no. 3 (2018), https://journals.ala.org/index.php/
rusq/article/view/6606/8827. “The primary challenge of getting students to use vetted library sources is simply
getting them there, as opposed to using a web search engine such as Google. . . . Assuming we can convince
our students to use library resources.”
28

Daniel M. Russell and Mario Callegaro, “How to Be a Better Web Searcher: Secrets from Google Scientists,”
Scientific American, March 26, 2019, https://blogs.scientificamerican.com/observations/how-to-be-a-betterweb-searcher-secrets-from-google-scientists/.
29

See Nadav Ziv and Adrian Liu, “The Rhetoric of PWR,” Stanford Daily, September 25, 2019,https://www.stanforddaily.com/2019/09/24/the-rhetoric-of-pwr/. For research showing that it is possible
to move the needle on students’ skills see: Sarah McGrew, Joel Breakstone, Mark Smith, Teresa Ortega,and Sam Wineburg, “Improving University Students’ Web Savvy: An Intervention Study,” British Journal of
Educational Psychology 89, no. 3 (2019), https://onlinelibrary.wiley.com/doi/abs/10.1111/bjep.12279; Sarah
McGrew, “Learning to Evaluate: An Intervention in Civic Online Reasoning,” Computers & Education 145 (2019),
https://doi.org/10.1016/j.compedu.2019.103711; Jessica E. Brodsky, Patricia J. Brooks, Donna Scimeca, Ralitsa
Todorova, Peter Galati, Michael Batson, Robert Grosso, Michael Matthews, Victor Miller, Tracy Tachiera, andMichael Caulfield, “Teaching College Students the Four Moves of Expert Fact-Checkers,” (presentation, Annual
Meeting of American Psychological Association).
30

Stonehill College, https://web.archive.org/web/20200902234651/https://libguides.stonehill.edu/lateral;
Andrea Baer and Daniel G. Kipnis, “SIFTing and Four-Moving Online: Opportunities and Challenges with
Teaching Lateral Reading through an Online Module” (presentation, LOEX Conference, May 7, 2020),
https://rdw.rowan.edu/lib_scholarship/19; University of Louisville Libraries, https://web.archive.org/
web/20200930192006/https://library.louisville.edu/citizen-literacy/home and https://youtu.be/GZvsGKvqzDs.
See as well the efforts of Sara Sharun and Erika E. Smith at Calgary, Alberta’s Mount Royal University,“Educational Development Partnerships and Practices: Helping Librarians Move Beyond the One-Shot,”College & Research Libraries News 81, no. 9, (2020): 445, https://crln.acrl.org/index.php/crlnews/article/
view/24644/32464.
31

EDUCATING FOR MISUNDERSTANDING

22