Do Security Toolbars Actually Prevent Phishing Attacks?
Min Wu, Robert C. Miller, Simson L. Garfinkel
MIT Computer Science and Artificial Intelligence Lab
32 Vassar Street, Cambridge, MA 02139{minwu, rcm, simsong}@csail.mit.edu
ABSTRACT

SpoofStick

Security toolbars in a web browser show security-related
information about a website to help users detect phishingattacks. Because the toolbars are designed for humans touse, they should be evaluated for usability – that is, whether
these toolbars really prevent users from being tricked into
providing personal information. We conducted two user
studies of three security toolbars and other browser security
indicators and found them all ineffective at preventing
phishing attacks. Even though subjects were asked to pay
attention to the toolbar, many failed to look at it; others
disregarded or explained away the toolbars’ warnings if thecontent of web pages looked legitimate. We found thatmany subjects do not understand phishing attacks or realize
how sophisticated such attacks can be.

Netcraft Toolbar

TrustBareBay Account Guard

SpoofGuard

Figure 1. Existing security toolbars
Author Keywords

World Wide Web and Hypermedia, E-Commerce, User
Study, User Interface Design.

admitted to having provided personal data to a phishingsite; and US consumers have lost an estimated $500 million
as a result of these attacks. [15]ACM Classification Keywords

APWG has collected and archived many phishing attacks.
A typical example is an attack against eBay customers, firstreported in March 2004. [1] The attack starts with an email
claiming that the recipient’s account information is invalid
and needs to be updated by visiting the provided link. The
message appears to come from S-Harbor@eBay.com, andthe link apparently points to cgi1.ebay.com, but actuallyleads to 210.93.131.250, a server in South Korea withno relationship to eBay. Following the link produces a web
page that looks legitimate, with an eBay logo and page
design, and asks for the victim’s credit card, Social Security
number, eBay username and password. Clicking the submit
button sends the data to the hostile server, where it iscollected and used by the attackers.

H.5.2 User Interfaces, H.1.2 User/Machine Systems, D.4.6
Security and Protection.
INTRODUCTION

Phishing has become a significant threat to Internet users.
Phishing attacks typically use legitimate-looking but fakeemails and websites to deceive users into disclosing
personal or financial information to the attacker. Users canalso be tricked into downloading and installing hostile
software, which searches the user’s computer or monitors
online activities to steal private information.
Phishing attacks are on the rise. According to the AntiPhishing Working Group (APWG), 2870 phishing sites
appeared in March 2005, a 28% increase per month since
July 2004. [2] A survey sponsored by TRUSTe found 70%of the respondents had visited a phishing site; over 15%

Many proposals for stopping phishing attacks rely on asecurity toolbar that displays warnings or security-related
information in the web browser’s interface. Figure 1 showssome existing security toolbars:• SpoofStick [20] displays the website’s real domain
name, in order to expose phishing sites that obscuretheir domain name. An attack might use a legitimatelooking domain name as a sub-domain, e.g.,
www.paypal.com.wws2.us to fool users; SpoofStickwould display this domain as wws2.us.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee.
CHI 2006, April 22-28, 2006, Montréal, Québec, Canada.
Copyright 2006 ACM 1-59593-178-3/06/0004...$5.00.

1• Netcraft Toolbar [16] displays information about thesite, including the domain’s registration date, hosting
country, and popularity among other toolbar users. This
information is thought to be helpful in detecting
phishing sites because most phishing sites are shortlived compared to the legitimate sites they imitate, anda large number of phishing sites spoof US-based
corporations but are registered in other countries.
• Trustbar [13] makes secure web connections (SSL)
more visible by displaying the logos of the website andits certificate authority (CA). This is useful against
phishing because many legitimate websites use SSL toencrypt the user’s sensitive data transmission, but mostphishing sites do not. Attackers avoid SSL because
obtaining an SSL certificate from a well-known CA,
such as VeriSign, requires site identity information thatcan be traced, and because using a CA that is notknown to the browser will trigger a warning and thus
might raise the user’s suspicion.
• eBay’s Account Guard [7] shows a green icon toindicate that the current site belongs to eBay or PayPal,a red icon to indicate a known phishing site found on a
blacklist maintained by eBay, and a gray icon for allother sites.
• SpoofGuard [5] calculates a spoof score for the current
web page using a set of heuristics derived fromprevious phishing attacks. It then translates this score
into a traffic light: red for spoof scores above athreshold, indicating the page is probably hostile;
yellow for scores in the middle; and green for lowscores, indicating the page is probably safe.
In addition to these toolbars, existing browser indicators
can also help users to detect phishing attacks. For example,
the address bar displays the URL of the current web page,
and the status bar displays a lock icon to indicate if the page
was downloaded with SSL. To further differentiate SSLdownloaded pages, Mozilla Firefox changes the address
bar’s background from white to yellow and adds a lock iconin the address bar. Users are commonly advised by online
security tips to pay attention to these kinds of indicators
whenever they access a web site. [9]There are several potential drawbacks to the securitytoolbar approach:• A toolbar is a small display in the peripheral area of thebrowser, compared to the large main window that
displays the web content. Users may not pay enoughattention to the toolbar at the right times to notice anattack.
• A security toolbar shows security-related information, but
security is rarely the user’s primary goal in web
browsing. Users may not care about the toolbar’s display
even if they do notice it.

• If a toolbar sometimes makes mistakes and identifies
legitimate sites as phishing sites, users may learn todistrust the toolbar. Then, when the toolbar correctly
identifies a phishing site, the user may not believe it.
This paper describes two user studies we performed to findout why users get fooled by phishing attacks, to determine
which attacks were more effective than others, and toevaluate the security toolbar approach for fighting phishing.
The rest of this paper is organized as follows. We begin bysurveying related work. Next we discuss how we designed auser study to evaluate the security toolbars, with particularattention to the issues and tradeoffs of a study that
intentionally attacks users. We present the results from thisstudy and discuss why the security toolbars do not work as
expected. The next section presents a follow-up study using
the same methodology that supports our reasoning about
why phishing attacks are effective. We conclude with somedesign principles for anti-phishing solutions based on the
results and observations from our two user studies.
RELATED WORK

A growing number of user studies are investigating why
phishing attacks are so effective against computer users.
Anti-spam firm MailFrontier Inc did a web survey on how
well people can distinguish phishing emails from legitimate
ones. [21] Subjects saw screenshots of ten emails but could
not interact with them. About 28% of the time, subjects
incorrectly identified the phishing emails as legitimate.
In April 2004, a study in London found that 34% of the
respondents would give the researchers their password in
exchange for a bar of chocolate. [4] The researchers did nottest the passwords to see if they were accurate, however.
In April 2005, a study at Indiana University Bloomington
showed that social context can make phishing attacks far
more effective. [14] The researchers sent out phishingemails to university students, claiming to be from a friend,
having mined friendship relations from a social networking
site used on campus. The email led to a phishing site that
asked for the subject’s university username and password.
72% of the subjects provided valid usernames andpasswords to the phishing site.
Whalen and Inkpen used an eye-tracker to study the user’s
attention to browser security indicators when doing secure
online transactions. [22] Their study found that subjects
often looked at the lock icon in the status bar, but rarely
clicked on the lock and thus didn’t learn anything about thesite’s certificate. By contrast, subjects in our studies rated
the status bar least effective at preventing phishing attacks.
We think that the difference is due to the fact that Whalen
and Inkpen’s subjects were explicitly told to pay attention
to the security indicators in the browser, while our subjects
were asked to detect fake websites, so the address bar andthe URL were more useful indicators.

Address bar frame

Neutral-Information toolbar

Security toolbar frame

SSL-Verification toolbar

System-Decision toolbar

Main frame

Status bar frame

Figure 3. Browser simulation using HTML frames
Figure 2. The three simulated toolbars tested in the study

The SSL-Verification toolbar differentiates sites that use
SSL from those that do not. SSL sites are displayed with thesite’s logo and CA; a general warning message is displayedfor other sites. This approach that imitates Trustbar seeks tomake the user suspicious when a non-SSL page asks forsensitive information such as a password or credit card
number.

At least two organizations have initiated phishing attacks
against their own members, with the goal of teaching themto protect themselves. [3] The US Military Academy atWest Point found that more than 80% of its cadets
succumbed to a phishing attack by a fictional colonel. The
State of New York mounted two attacks on its 10,000
employees; 15% were spoofed by the first attack, but only
8% by the second, which came three months later.

The System-Decision toolbar displays a red light and themessage “Potential Fraudulent Site” if it decides that a web
page is actually a phishing attack, an approach that issimilar in design to both eBay Account Guard andSpoofGuard. This display is easy for a user to interpret, but
it requires the user to trust the toolbar’s decision process,which is generally hidden from the user.

Besides the security toolbars we tested, there are other antiphishing solutions that help users to differentiate thelegitimate web sites from the phishing ones. Dynamic
Security Skins [6] proposes to use a randomly generated
visual hash to customize the browser window or web form
elements to indicate the successfully authenticated sites.
PassMark [18] includes a personalized image in a web page
to indicate that the user has set up an account with the site.
Google Safe Browsing for Firefox [12] pops up an alert
when a user is on a web page that Google determines to beillegitimate. The content of the phishing page is also
darkened to make it less convincing. Internet Explorer 7[19] protects against phishing with a dynamically-updated
black list of known phishing web sites, a client-side list of
acceptable sites, and a set of heuristics. It blocks the user's
activity with a detected phishing site. IE7 also has stricter
enforcement of SSL certificates, in that it will not display
websites with certificates that are invalid. A comprehensive
survey of anti-phishing solutions can be found in [8].

Study Implementation

In order to simulate attacks against users, we needed tocompletely control the display of the toolbars and othersecurity indicators. Users in the study interacted with asimulated Internet Explorer built inside an HTMLapplication running in full screen mode (figure 3). Different
HTML frames displayed different browser components,
including the security toolbars. The locations and sizes of
the toolbars were consistent with the existing toolbars thatthey are based on. The Neutral-Information toolbar and the
System-Decision toolbar were located below the address
bar and above the main browsing window. The SSLVerification toolbar was located below the title bar andabove the menu bar. The address bar took the FireFox
approach by using the yellow background and a lock icon toindicate SSL connections. The status bar also displayed alock icon for SSL connections.

STUDY DESIGN

To simplify the study design, we grouped the features of thefive existing toolbars into three simulated toolbars (figure2), based on the three types of information that existing
security toolbars display:

Our study simulated ideal phishing attacks whose content isa perfect copy of the actual website. This is realistic, sincean attacker might not bother mirroring the entire site, but
might simply act as a man-in-the-middle between the user
and the real site. The attackers would pass the real web
pages to the user and the user’s submitted data to the realsite and in the meantime capture the user’s sensitive data
during the online transaction. As such, the main frame inour browser always connected to the real website,

The Neutral Information toolbar shows website
information, such as domain name, hostname, registration
date and hosting country, as SpoofStick and Netcraft
Toolbar do. With this information, users must use their own
judgment and experience to decide whether a site islegitimate or phishing.

3

regardless of whether the site was supposed to be phishingor not. To simulate phishing attacks, we changed theappearance of the HTML frames that displayed thebrowser’s security indicators—including the security
toolbar, the address bar and the status bar—to indicate thatthe web page was served by an unusual source, e.g.,tigermail.co.kr rather than paypal.com.
Study Scenario

Phishing is an attack that directly targets the human being
in the security system. Simulating these kinds of attacks forstudy purposes raises some special problems. Chief among
them is the secondary goal property articulated by Whitten
and Tygar: in real life, security is rarely a user's primarygoal. [23] The user is primarily concerned with other tasks,
such as reading mail, buying a book, or editing a document.
Avoiding disclosure of passwords or personal information
may be important, but it isn’t foremost in the user's mind.
In order to produce generalizable results, a lab study must
be designed to preserve this behavior as much as possible.
If we simply asked subjects to “identify the fake web
pages,” security would become their primary goal andhence lead them to pay attention and take precautions thatthey would be unlikely to take in real life.
We addressed this problem by creating a scenario whichgave the subjects tasks to attend to other than security.
With the given tasks, the subjects have to protect a secretfrom the attack. For ethical reasons we could not use actual
financial data belonging to the subject. On the other hand,fake financial data cannot go through a real web site. One
approach would be to use specially-crafted test bankaccount or credit card numbers provided by a financial
institution. Another approach is to mirror the results of ecommerce transactions, as Whalen and Inkpen did. [22] A
third approach is to set up our own fictional bank site, withwhich subjects could interact using the fake data. Thisapproach is problematical for lab studies, since subjects
would need some time to get used to the testing site and its

transaction procedures.
Our studies took a different approach. We set up dummy
accounts in the name of “John Smith” at various legitimate
e-commerce websites and then asked the subjects to protect
those passwords. With this approach, we could study a wide
variety of existing websites with little setup. The subject
played the role of John Smith’s personal assistant and wasgiven a printout of John’s profile, including his fictitious
personal and financial information and a list of his
usernames and passwords. The task was to process 20 email
messages, most of which were requests by John to handle a
forwarded message from an e-commerce site. Eachmessage contained a link for the user to click. Figure 4
shows a sample message.
Simulating Phishing Attacks

Five of the 20 forwarded emails were attacks, with links
directing the users to a simulated phishing website. Each ofthese attacks represents a real phishing attack technique thathas been recorded by APWG:• Similar-name attack: Since one way that users
authenticate web sites is by examining the URL displayedin the address bar, attackers can use a hostname that bears
a superficial similarity to the imitated site’s hostname.
For example, we used www.bestbuy.com.ww2.us tospoof bestbuy.com.
• IP-address attack: Another way to obscure a server’s
identity is to display it as an IP address, e.g.,
http://212.85.153.6/ to spoof bestbuy.com.
• Hijacked-server attack: Attackers sometimes hijack aserver at a legitimate company and then use the server tohost phishing attacks. For example, we used a hijacked
site www.btinternet.com to spoof bestbuy.com.
• Popup-window attack: A popup-window attack displays
the real site in the browser but puts a borderless window
from the phishing site on top to request the user’s
personal information. Our phishing site displayed thehollywoodvideo.com site in the browser but popped
up a window requesting the username and password.
Although this pop-up window lacked an address bar andstatus bar, it nevertheless included the security toolbar.
• PayPal attack: The email message warns that John’s
account has been misused and needs to be reactivated,and points to a phishing website with hostname
tigermail.co.kr. Unlike the other attacks, which
simulate man-in-the-middle behavior while displaying
the real web site, this attack requests not only a PayPal
username and password, but also credit card and bankaccount information.
We consider the PayPal attack different from the other fourattacks, which we call wish-list attacks because they merely
asked the user to log in and modify a wish-list. First, thePaypal attack is like current phishing attacks that target
online banks and financial services; the wish-list attacks

Figure 4. A sample email in the user study

8th, 16th and 19th emails, with each type of wish-list attack
randomly assigned to one of these four positions. These
fixed positions were chosen to space out the attacks.

target online retailers instead, which is not as common
today, although growing. [10] The PayPal attack is greedy,
asking for lots of sensitive information; the wish-list attacks
can only steal usernames and passwords. The PayPal attack
is far more intimidating, urging users to reactivate theiraccount and threatening to suspend their account if they did
not do so immediately. We expected experienced Internet
users to be more suspicious of the Paypal attack.

Study Hypotheses

We define the spoof rate as the fraction of simulated attacks
that successfully obtain John’s username and password orother sensitive information without raising the subject’s
suspicion. We made two hypotheses: (1) that the spoof ratesof all three toolbars would be substantially greater than 0,
so that none of the toolbars effectively prevents attacks; and(2) that some toolbars would have better spoof rates thanothers. In particular, we expected that the System-Decision
toolbar would have a lower spoof rate than the others
because it used a simple traffic light metaphor, and theselights were always correct (at least in our simulation).

All three toolbars were configured to differentiate thelegitimate sites from the phishing sites. None of thephishing sites used SSL so that the SSL-Verification toolbar
always displayed a warning on them. On the SystemDecision toolbar, all legitimate sites were displayed astrustworthy (green) but all the phishing sites were displayed
as phishing (red) or unsure (yellow). On the NeutralInformation toolbar, the phishing sites and hijacked servers
displayed as a “New Site” and some of them were displayed
as they were hosted in other countries outside the US.

RESULTS AND DISCUSSION

A total of 30 subjects with previous experience in online
shopping, 14 females and 16 males, were recruited byonline and poster advertising at a college campus. Twenty
subjects were college students from 10 different majors. All
subjects had at least a college education. The average age
was 27 (the range, 18 to 50). Each of the three security
toolbars was tested on 10 subjects.

Toolbar Tutorial

Another question in this study design is when and how togive users a tutorial about the security toolbar. Few users
read documentation in the real world; some may read an
introduction when they download and install a security
toolbar, but others may not read anything at all, particularly
if the security features are bundled with the web browser.

To gauge subjects’ experience with online shopping, we
asked them which of our 19 selected e-commerce sites they
had visited. All 30 subjects had used Amazon, and 25 ormore had used PayPal, Travelocity, Bestbuy, and Yahoo.
On average, each subject had used 10 of the sites in ourstudy.

Our pilot study found that the presence or absence of atutorial has a strong effect on performance. When five pilot
subjects received a printed tutorial explaining the security
toolbar, showing how it looked for both legitimate andphishing websites, only 1 out of 15 attacks (7%) was
successful. Another six pilot subjects received no printed
tutorial; instead, we added a “What’s this?” link in eachtoolbar which displayed the tutorial in a popup window.
These subjects succumbed to 17 out of 18 attacks (94%);
not one subject one clicked the “What’s this?” link.

Before the study, subjects were given a consent form which(1) explained that the purpose of the study was to test web
browser security indicators that detect fake web pages thatlook like pages from well-known legitimate websites; (2)indicated that the purpose of these fake websites is to trick
people into making dangerous decisions or taking
dangerous actions; and (3) encouraged the subjects to detect
all the fake web pages and report them by clicking the“report fraud” button in the browser’s toolbar. All thesubjects were required to read the consent form carefully,
especially the study procedure part.

This result was problematic. In the former case, the printed
tutorial gave the pilot subjects too strong a clue that security
was the primary goal in the study. In the latter case,subjects had no idea what the security toolbar meant, or its
role in preventing phishing attacks.
Based on this experience, we introduced the tutorial as partof the scenario. In the experiment, John Smith forwards tothe subject an email from his company’s systemadministrator. The email says that a security toolbar has
been installed on the company’s computers to prevent
phishing attacks. The message contains a link to thetutorial. When John Smith forwarded this email to thesubject, he explicitly requests that they be careful with his
personal information.

After the consent form, the subject was briefed about theJohn Smith scenario and their role as John Smith's assistant.
This briefing did not mention security at all.
We personally observed the subjects’ browsing behaviors
during the study. We did not interrupt the study except
when subjects clicked the “report fraud” button, at which
point we asked them to explain why they reported fraud.
Security Awareness

The tutorial email appeared in the middle of the study, as
the 11th of the 20 emails, where it could serve as a control tosee how users behaved before and after seeing the tutorial.
The PayPal attack was the 10th email because of its
uniqueness. The remaining four attacks occurred at the 5th,

One of the risks of using an artificial scenario is that users
may not care about the fictional John Smith's security at all.
Fortunately, a number of indicators showed our subjects
were behaving as if they did care about the security of John

5

Smith’s accounts. Designing these kinds of secondary
indicators into a security study turned out to be a good idea.
For example, 18 subjects unchecked the “Remember Me”
checkbox on the login page of at least one site. This
checkbox, which is generally checked by default and must
be explicitly unchecked, controls whether John Smith's
login information is recorded in a cookie. Furthermore, 13
subjects explicitly logged out or tried to log out of at least
one web site after finishing a task. These cautious subjects(23 in all) were protective of John Smith’s online identityand did not want the browser to remember the login
sessions. We never told them anything about unchecking“Remember Me” or logging out. The other 7 subjects also
exhibited suspicion and caution at least once in the study,
either by reporting fraud or by trying to carefully explore a
web site to determine if it was legitimate.
Subjects also demonstrated caution with false alarms—
believing that a good site was an attack. Subjects did notfinish tasks at good sites 3.3% of the time (13 out of 390
tasks) because of security concerns. There were six false
alarms before the tutorial and seven after the tutorial. False
alarms were generally due to browser warnings generated
by the legitimate site (such as “You are about to be
redirected to a connection that is not secure”).
The Wish-list Attacks

Figure 5 shows the spoof rates of wish-list attacks for eachtoolbar. These spoof rates, 45% for the Neutral-Information
toolbar, 38% for the SSL-Verification toolbar, and 33% for
the System-Decision toolbar, are all significantly higher
than 0%, the ideal. No significant difference was foundbetween the toolbars by a one-way ANOVA test. But thishardly matters since all the toolbars have high spoof rates.
Among the 30 subjects, 20 were spoofed by at least one
wish-list attack (7 used the Neutral-Information toolbar, 6
used the SSL-Verification toolbar, and 7 used the SystemDecision toolbar). We interviewed these subjects to find out
why they did not recognize the attacks:• 17 subjects (85%) mentioned in the interview that the
web content looked professional or similar to what they
had seen before. They were correct because the content
was the real web site, but a high-quality phishing attack
Spoof Rate by Wish-list Attacks

100%
80%
60%

45%

38%

33%

SSL-Verification
toolbar

System-Decision
toolbar0%
Neutral-Information
toolbar• 12 subjects (60%) used rationalizations to justify theindicators of the attacks that they experienced. Nine
subjects explained away odd URLs with comments like:www.ssl-yahoo.com is a subdirectory of Yahoo!, like
mail.yahoo.com.
sign.travelocity.com.zaga-zaga.us must be an
outsourcing site for travelocity.com.

Sometimes the company [Target] has to register adifferent name [www.mytargets.com] from its brand.
What if target.com has already been taken by anothercompany?
Sometimes I go to a website and the site directs me toanother address which is different from the one that I
have typed.
I have been to other sites that used IP addresses [insteadof domain names].
Four subjects explained away the popup window that
asked for a username and password. One subject
commented that she must have triggered the popup
window herself, by clicking “Register for new account”
instead of “Sign in for existing account”.
One subject explained away a toolbar message showingthat Yahoo! was a “New Site” and located in Brazil byreasoning that Yahoo must have a branch in Brazil.
Another explained away the warning on the SystemDecision toolbar by saying that it was triggered because
the web content is “informal,” just like a spam filter saysthat “this email is probably a spam.”• Nine subjects (45%) said that the reason they werespoofed was that they were focused on finishing the study
tasks—i.e., dealing with John Smith's email requests.
Three explicitly mentioned that, although they noticed the
security warnings, they had to take some risks to get thejob done. Simply warning these subjects that somethingis wrong was not sufficient: they needed to be provided
with a safe alternative way to achieve their goals.
• Five subjects (25%) claimed that they did not notice thetoolbar display at all for some attacks.

40%
20%

or man-in-the-middle can look exactly like the targeted
website as well. Seven of these subjects were observed touse security-related links on the site itself to decide if asite was legitimate or not—for example, clicking on the
Verisign seal, the site’s privacy policy, contact
information, copyright information, or a credit card
security claim. Of course, attackers can and do fake these
indicators. A lot of research has done to improve web
credibility (e.g., [11]), and attackers have clearly adopted
these techniques.

Figure 5. Spoof rates with different toolbars• One subject extensively clicked links on the web pages totest whether the web site worked properly. By relying on
the site’s behavior as an indication of its authenticity, thissubject was fooled by all of the wish-list attacks.

green light at the good sites. The Neutral-Information
toolbar showed all phishing sites as either a “new site” orhosted in a non-US country (or both), but all good sites as
hosted in the US and in existence for several years. But itturned out that 9 of the 18 online stores that we chose forthis study had login pages that were not protected by SSL,
so the SSL-Verification toolbar produced warnings even for
legitimate sites. Thus, the SSL-Verification toolbar failed toadequately distinguish fake sites from good ones.

The similar-name attack had the highest spoof rate, 50%,
among all simulated phishing attack techniques. But nomatter how the phishing URL is presented, the spoof rates
are always high, with 43% for the hijacked-server attackand 33% for the IP-address attack. The popup-window
attack had a relatively low spoof rate of 27%—many
subjects thought that using the popup window for login
information was abnormal and suspicious. The spoof rate
differences were not significant by a one-way ANOVA test.

The tutorial was not the only factor affecting subjects’learning, of course. Another contribution to the decrease in
the spoof rate before and after the tutorial is that the spoof
rate steadily decreased for each attack as the subjects
experienced more wish-list attacks and learned how to
detect them, as shown in figure 7.

Learning Effects

Seven subjects clicked the toolbar’s “What’s this?” link
before the tutorial email (1 using the Neutral-Information
toolbar, 2 using the System-Decision toolbar, and 4 using
the SSL-Verification toolbar.) Subjects using the SSLVerification toolbar clicked “What’s this?” even before thefirst attack. We believe that this is because the toolbar
displayed a warning message on all pages that did not use
SSL, which is the case for many web pages.

The PayPal Attack

As discussed above, the PayPal attack is very different fromthe wish-list attacks. The difference is reflected in the
study. The PayPal attack had a significantly lower spoof
rate (17%) than the wish-list attacks (38%) (two-tail t(56) =-2.63, p = 0.01). Ten subjects said that they had seen similar
phishing emails in the real world, so they could detect the
PayPal attack just by reading the email message, withouteven clicking through to the phishing site. The wish-listattacks have a lower spoof rate (28%) on these 10 subjects
than the other 20 subjects (44%). But the difference is not
significant (one-tail t(23) = -1.36, p = 0.09). Some subjects
did not feel comfortable providing John Smith’s credit cardand bank account information, and eventually noticed thesuspicious signs from the toolbar or the suspicious URL
from the address bar and thus avoided the attack.

We found that the difference in spoof rates for wish-listattacks before and after the subjects saw the tutorial, either
by clicking the “What’s this?” link or by reading the tutorial
email, to be statistically significant (one-tail t(43) = 2.27, p= 0.014). Figure 6 shows the spoof rate before the tutorial
was 52%, while after the tutorial it dropped to 26%.
Although a decrease was found with all three toolbars, thedecrease was significant for the Neutral-Information toolbar(one-tail t(18) = 1.84, p = 0.04), marginally significant for
the System-Decision toolbar (one-tail t(12) = 1.52, p =0.077), and not significant for the SSL-Verification toolbar(one-tail t(7) = 0.61, p = 0.28).
Several subjects mentioned that the tutorial email helped
them to pay more attention to the security toolbar and better
understand its display, explaining the drop in spoof rate
following the tutorial.

However, there were still five subjects out of 30 (17%) who
were tricked by the PayPal attack (at least one using eachtoolbar). Four were PayPal users in real life. They werespoofed because the content of the site looked authentic.
One typical comment was “I've used PayPal before and thissite looks exactly the same. If I trust a site from myexperience, I am not suspicious.” They also justified therequest as being reasonable. One subject said that “theyneed this information [the credit card and the bank accountinformation] to charge me.” Thus, familiar phishing attacks
can continue to be persuasive and effective, even with

Subjects using the Neutral-Information toolbar and the
System-Decision toolbar saw their spoof rates significantly
drop following the tutorial. This was not true of subjects
using the SSL-Verification toolbar. One explanation is thatthe toolbars had different levels of accuracy. We tried tomake every toolbar accurate enough to distinguish phishingsites from legitimate sites. The System-Decision toolbar
displayed a red or yellow light at the phishing sites but a100%
Correlation Coefficient r = -0.83

Before tutorial
After tutorial

80%

80%

Spoof Rate

Spoof Rate by Wish-list Attacks

100%

60%

40%

40%
57%

65%
52%
20%
28%

43%

20%

50%
26%

60%

35%

30%

39%

1

0%
Total

Neutral-Information
toolbar

SSL-Verification
toolbar

23%

0%

15%
System-Decision
toolbar2

3

4

Wish-list Attack Position

Figure 7. Spoof rates for wish-list attacks at different
attack positions

Figure 6. Spoof rates before and after the tutorial

7

security toolbars to warn the user.
Subjective Ratings and Comments on Toolbars

Subjects were asked at the conclusion of the study to rate
the effectiveness of the address bar, status bar, the security
toolbar that they used in differentiating authentic web sites
from phishing sites, on a scale from -2 (very ineffective) to
2 (very effective). Figure 8 shows the mean ratings.
Among the three toolbars, the SSL-Verification toolbar was
rated as less effective, although the difference was not
significant. One reason might be because the SSLVerification toolbar could not distinguish phishing sites
from legitimate sites that do not use SSL. Sadly, many suchsites exist in the wild, and some were used in our study. But
even when the toolbar functioned properly, it was often
ignored. One subject commented that the toolbar looked
like an advertisement banner, so it was unclear whether it
was put there by the browser or by the site.
The other two toolbars were thought more effective than thebrowser’s own address bar. A common remark on the
security toolbar was that the toolbar worked with theaddress bar: the toolbar alerted and warned the subject,causing the subject to pay more attention to the address bar.
Some subjects did not know how to interpret the
information the toolbars displayed—especially the NeutralInformation toolbar. One subject said: “How do I have any
idea about the [registration] time and location of a site?”
Why Don’t the Security Toolbars Work?

Many users relied on the web content to decide if a site isauthentic or phishing. The web content has a large display
area and is in the center of the user’s attention. It can makeitself very convincing. Most of the time, the web
appearance does reflect the site’s identity because of thelow phishing rate in the real world. What’s more, in theearly days of phishing, phishing attacks frequently had poor
grammar and spelling mistakes. In our study, simulated
phishing sites had high-fidelity content. As a result, eventhough the security toolbar and other security indicators inthe browser tried to alert the user, many users disregarded
the security toolbars because the content looked so good.
Poor web practices on the part of e-commerce firms makeSubjectiveRating
RatingSubjective22 (Very(Very Effective)
Effective)

1
1 (Effective)(Effective)0
0 (Neutral)(Neutral)

0.9

0.9

0.6

0.6

0.6

0.9

0.9

0.6

NeutralNeutralInformation
Information
toolbartoolbar

SystemSystemDecision
Decision
toolbartoolbar

SSLSSLVerification
Verification
toolbartoolbar-0.6-1-1 (Ineffective)(Ineffective)-2-2 (Very(Very Ineffective)
Ineffective)-0.6
Address
Address Bar
Bar

StatusStatus Bar
Bar

Figure 8. Subjective ratings of the address bar, the statusbar and the toolbars

Figure 9. A sample blocking warning box at a phishing site

phishing attacks even more likely to succeed. For example,
many legitimate companies do not use SSL to protect theirlogin page; this was a serious problem for the SSLVerification toolbar. Many operators use domain names thatare vague, inconsistent, or otherwise unrelated to theirbrands. Many organizations make their outsourcing
relationships directly visible to Internet users. Such
practices make it even harder for users to distinguishlegitimate websites from malicious attacks.
FOLLOW-UP STUDY

A more effective interface for getting the user’s attention
about a phishing web site is to actually block access to it—
for example, by popping up a modal dialog box when thesite is visited. Several security toolbars, including Netcraft
Toolbar, eBay Account Guard and SpoofGuard, display a
pop-up warning when they have high confidence that thecurrent site is phishing. This warning is likely to get theuser’s attention since it appears in the center of the browser
and impedes progress until it is acknowledged.
A pop-up is a very aggressive warning, disrupting the user'stask, so it must be accurate or it will be disabled. Since
phishing attacks evolve rapidly, we have found that security
toolbars are rarely certain enough about a new phishingattack to display a pop-up. As a result, these toolbars
depend more heavily on the persistent toolbar display towarn users about new dangers. This is why our first study
focused on the toolbar display.
Nevertheless one might expect a pop-up dialog to be more
effective at prevent phishing attacks. To find out, we ran afollow-up study with new subjects to test the pop-up alert
technique. The second study used the same scenario and thesame attacks with the same numbering and positioning of
attacks. Half of the subjects saw a blocking warning box(figure 9) at the phishing sites, which closely resembles thewarning used by the Netcraft Toolbar. The rest acted as acontrol, using only a standard browser interface with theaddress and status bars are the main security indicators.
The follow-up study had 20 subjects aged 19 to 37 (averageage 23). 18 (90%) were college students, 13 male.
It turned out that these subjects exhibited more caution than
the subjects in the first study. We quantified their degree ofcaution as a sum of three indicator variables: 1 point if thesubject ever tried to log out of a site; 1 point if the subject
unchecked “remember me” at a login page; and 1 point if
the subject failed to finish the task at a good site because of• None of the four spoofed subjects offered that the content
of the page was convincing as a reason that they werespoofed—somewhat ironic, since the content was in fact
the real site! Apparently the warning box blocking thepage is a stronger signal than the web content.

security concerns. The second study’s subjects had an
average caution score of 2.0, compared to 1.3 for the firststudy, a significant difference (two-tail t(39) = -3.29, p =0.002). Possible reasons for the difference include thedemographics of the subjects and substantial media
coverage about phishing and identify theft in theintervening three months.

• Two subjects believed the warning and knew the site wasphishing, but still wanted to complete the task. The
subjects claimed that a wish list was not sensitive enoughfor them to decline John’s request. Apparently they did
not realize that revealing John’s shopping password to an
attacker could result in financial loss if John used thesame username and password at another site.

As expected, the blocking warning box dramaticallydecreased the spoof rate of the wish-list attacks in the study,
as shown in figure 10. The decrease was statisticallysignificant (one-tail t(9) = -2.88, p = 0.01).
Seven out of the 10 subjects with the regular browser
interface were spoofed by at least one wish-list attack:• The other two subjects did not trust the blocking warningbox. One said that he had never seen such a warning
before. The other thought that the warning was wrong.
This subject had his own anti-phishing strategy: he typed
a wrong password at the suspicious site’s login page. If
the site accepted the wrong password, he inferred that thesite was phishing. But if the site rejected the wrong
password, he concluded that the site was good and thatthe warning box was making an error. Clearly, thisstrategy does not work against phishing sites executing
man-in-the-middle attacks, for these sites pass usernames
and passwords on to the real site to perform the validity
check. Interestingly, two other subjects in the follow-up
study also used this same strategy to check a site’s
authenticity; we never saw this behavior in the first study.

• Six subjects (86%) said that the web content looked goodor the same as they had seen before.
• Two subjects (29%) rationalized the suspicious URL.
One subject, experiencing a similar-name attack at
www.walmart.com by www.walmart.com.globalupdate2.com, said that “global-update2 is a service todo the website’s global updating and this service isworking for Wal-mart.”• Three subjects (43%) did not look at the URL in theaddress bar at all. One said that “I did not bother to lookat the address bar since the page looked so good.”
Two subjects with the regular browser interface werespoofed by the PayPal attack, both PayPal users in real life.
They mentioned that the site looked just like PayPal’s andthat the phishing email was a reasonable request by PayPal.

The follow-up study confirms that many users do not know
how sophisticated a phishing attack can be. Some subjects
were impressed by our simulated phishing attacks. One
typical comment: “I cannot imagine that an attacker canmake the attack so elegant to mirror a whole site.” Others
wrongly believed that phishing sites cannot check password
validity since they don’t have the correct password.

The results from the 10 subjects who used the regularbrowser interface supported our conclusions from the firstuser study: many users depend on the web content toauthenticate the site’s identity. Even though they are
cautious and notice suspicious signs from the browser’s
security indicators, since these signals are weak comparedto the strong signals from convincing web content, the users
tend to ignore or explain away the security indicators.

CONCLUSIONS AND RECOMMENDATIONS

We evaluated three types of security toolbars, as well as
browser address and status bars, to test their effectiveness at
preventing phishing attacks. All failed to prevent users frombeing spoofed by high-quality phishing attacks.

Of the 10 subjects who used the blocking warning box,
none were spoofed by the PayPal attack but four werespoofed by wish-list attacks:

60%

Users fail to continuously check the browser’s security
indicators, since maintaining security is not the user’s
primary goal. Although users sometimes noticed suspicious
signs coming from the indicators, they either did not know
how to interpret the signs or they explained them away.
Many users had no idea how sophisticated an attack could
be, and do not know good practices for staying safe online.

40%

Design Principles for Anti-phishing Solutions

Spoof Rate by Wish-list Attacks

100%
80%

20%

40%

Based on these studies, we propose the following design
guidelines to make effective anti-phishing solutions. We arealso developing new techniques that use these guidelines.

10%0%Regular browser interface

Blocking warning box

As the follow-up study shows, active interruption like thepopup warnings is far more effective than the passive
warnings displayed in the toolbars. But it's well-known that

Figure 10. Spoof rates with a regular browser interface andthe blocking warning box

9popup confirmations, used indiscriminately, become lesseffective over time: the more often they appear, the lessoften users heed them. [17] In order to make theinterruption effective, it should always appear at the righttime with the right warning message. For example, mostphishing attacks trick users into submitting their personal or
financial information through web forms. Instead of using
generic warnings like "Are you sure you want to continue
sending this information over an unencrypted connection?",the browser should interrupt the user only for a dangerous
action, like submitting one site’s login information toanother site. Knowing the user’s intention makes it easierfor the browser to protect the user.
User intentions should be respected. Simply warning usersthat something is wrong and advising them not to proceed is
not the right approach. Users will take risks to finish thetasks they think worthwhile and are not good at evaluating
the tradeoffs between the claimed benefits and the potential
risks. Warnings that propose an alternative path (e.g.,
directing users to the real intended site) allowing users tofinish the task safely would probably be more effective.
If users must make security-critical decisions, it is best to
integrate the security concerns into the critical path of theirtasks so that they have to deal with it, and can’t simply
ignore it. Do not expect users to keep a separate security
task in mind. Asking users to choose a safe mode to finish
their tasks has been found to be more dependable andeffective than merely reminding them to finish their tasks ina safe mode. [24]Finally, Internet companies need to follow some standardpractices to better distinguish their sites from malicious
phishing attacks. Companies should use a single domainname that matches their brands name rather than using IP
addresses or multiple domain names for servers. They
should use SSL to encrypt every web page on their sites.
SSL certificates should be valid and from widely used CAs.
Acknowledgements

We gratefully acknowledge the help and suggestions of
Ruth Rosenholtz, Srinivas Devadas, the members of the
UID group at CSAIL, and the anonymous reviewers. This
work was supported in part by the National Science
Foundation under grant IIS-0447800. Any opinions,findings, and conclusions or recommendations expressed inthis material are those of the author(s) and do notnecessarily reflect the views of the National Science
Foundation.
REFERENCES
1. Anti-Phishing Working Group. eBay – NOTICE eBay
Obligatory Verifying - Invalid User Information. March 9,
2004. http://www.antiphishing.org/phishing_archive/
eBay_03-09-04.htm
2. Anti-Phishing Working Group. Phishing Activity Trends
Report, March 2005. http://antiphishing.org/
APWG_Phishing_Activity_Report_March_2005.pdf

3. Bank, D. ‘Spear Phishing’ Tests Educate People About Online
Scams. The Wall Street Journal. August 17, 2005.
4. BBC News. Passwords revealed by sweet deal.
http://news.bbc.co.uk/1/hi/technology/3639679.stm
5. Chou, N., Ledesma, R., Teraguchi, Y., Mitchell, J.C. ClientSide Defense Against Web-Based Identity Theft. 11th Annual
Network and Distributed System Security Symposium (2004).
6. Dhamija, R. Tygar, J.D. The Battle Against Phishing:
Dynamic Security Skins. Symposium on Usable Privacy andSecurity (2005), pp. 77-88.
7. eBay Toolbar and Account Guard. http://pages.
ebay.com/help/confidence/account-guard.html
8. Emigh, A. Online Identity Theft: Phishing Technology,
Chokepoints and Countermeasures. ITTC Report on Online
Identity Theft Technology and Countermeasures. October 3,
2005. http://www.antiphishing.org/Phishing-dhs-report.pdf
9. Federal Bureau of Investigation, Department of Justice. FBI
Says Web ‘Spoofing’ Scams are a Growing Problem. 2003.
http://www.fbi.gov/pressrel/pressrel03 /spoofing072103.htm
10. Fluendy, S. Phishing targeting online outlets. Computer Crime
Research Center. March 16, 2005. http://www. crimeresearch.org/news/03.16.2005/1050/
11. Fogg, B.J, et al. What makes Web sites credible?: a report on alarge quantitative study. CHI 2001, pp. 61-68.
12. Google Safe Browsing for Firefox. 2005.
http://www.google.com/tools/firefox/safebrowsing/.
13. Herzberg, A., Gbara, A. TrustBar: Protecting (even Naïve)Web Users from Spoofing and Phishing Attacks. 2004.
http://www.cs.biu.ac.il/~herzbea/Papers/
ecommerce/spoofing.htm.
14. Jagatic, T., Johnson, N., Jakobsson, M., Menczer, F. Social
Phishing. School of Informatics & Dept. of Computer Science,
Indiana University. 2005. http://
informatics.indiana.edu/fil/Net/social_phishing.pdf
15. Leyden, J. US phishing losses hit $500m. The Register.
September 29, 2004.
16. Netcraft Toolbar. 2004. http://toolbar.netcraft.com/.
17. Norman, D. A. Design rules based on analyses of human error.
CACM, v26 n4 (April 1983), pp. 254-258.
18. PassMark. 2005. http://www.passmarksecurity.com/.
19. Sharif, T. Phishing Filter in IE7, September 9, 2006.
http://blogs.msdn.com/ie/archive/2005/09/09/463204.aspx
20. SpoofStick. 2004. http://www.spoofstick.com/.
21. Sullivan, B. Consumers still falling for phish. MSNBC. July28, 2004. http://www.msnbc.msn.com/id/5519990/
22. Whalen, T., Inkpen, K. Gathering Evidence: Use of Visual
Security Cues in Web Browsing. Graphics Interface 2005.
23. Whitten, A., Tygar, J.D. Why Johnny Can’t Encrypt: A
Usability Evaluation of PGP 5.0. 8th Usenix Security
Symposium, 1999, pp. 169-184.

24. Wu, M., Garfinkel, S., Miller, R. Secure Web Authentication
with Mobile Phones. DIMACS Workshop on Usable Privacy
and Security Software, 2004.