COMPARATIVE TEST REPORTWeb Browsers vs. Malware

Overview

Q2 2021

0%
20%
40%
60%
80%
100%
During Q2, 2021, CyberRatings.org performed anindependent test of malware protection offered by webMicrosoft Edge
97.4%
browsers. The tests ran for 20 days with 80 discrete test
2.6%
runs. To protect against malware, Microsoft Edge uses
Microsoft Defender SmartScreen; Google Chrome and
13.7%
Google Chrome
86.3%
Mozilla Firefox use the Google Safe Browsing API.
Microsoft Edge offered the most protection, blocking
97.4% of malware while providing the highest zero-hourprotection rate (97.7%). Google Chrome provided theMozilla Firefox
18.2%
81.8%
second-highest protection, blocking an average of 86.3%,
Block Rate
Missedfollowed by Mozilla Firefox at 81.8%.
The ability to warn potential victims that they are about to stray onto a malicious website puts web browsers in a uniqueposition to combat malware. Websites that trick (socially engineer) users to download malware have short lifespans, so it isessential that the site is discovered and added to the reputation system as quickly as possible. As such, a good reputation
system must be both accurate and fast to realize high catch rates.

Malware Protection Over Time

100%
80%
60%
40%
20%
Microsoft Edge

Google Chrome

Mozilla Firefox0%

Throughout the test, new malware was constantly added. URLs that were either no longer reachable or hosting malware
were removed. Each data point is calculated from measurements recorded at a specific point in time. If the malware wasblocked early on, the browser’s score protection over time improved. Alternatively, if the browser did not block themalware, the score decreased.
Malware Block Rate Over Time
100%

Block Rate

90%
80%
70%
60%

Summary of Results50%
Microsoft Edge
Google Chrome
Mozilla Firefox

0-hr
97.7%
86.4%
82.8%

1d97.9%
90.6%
85.9%

2d97.9%
91.7%
87.1%

3d97.9%
92.5%
88.0%

4d97.9%
92.9%
88.3%

5d97.9%
93.1%
88.4%

6d97.9%
93.1%
88.6%

7d
97.9%
93.1%
88.7%

Total
97.9%
93.6%
89.1%

The figure above shows how long each browser took to block malware once the sample was introduced into the test
cycle. The core protection technology within Microsoft Edge is SmartScreen, which provides URL-based protection fromattacks via an integrated, cloud-based URL-reputation service, as well as application reputation for malicious fileblocking. Google Chrome and Mozilla Firefox use the Google Safe Browsing API for both URL reputation and to block orwarn users about downloading certain types of files.

© 2021 CyberRatings.org. All rights reserved.

1

COMPARATIVE TEST REPORT

Total Number of Malicious Samples Tested

Malware Attacks

Social engineered malware (SEM) attacks use deceptions totrick users into downloading malware: Hijacked email andsocial media accounts take advantage of the implicit trustbetween contacts and deceive victims into believing that links
to malicious files are trustworthy. Other deceptions include
pop-up messages advising users that applications (such as
Adobe Flash Player) need to be installed or warn that a user’s
computer is infected, or that it requires an update.
Once malware is installed, victims are vulnerable to credential
theft, identity theft, bank account compromise, etc.

18,621 raw, unvalidated samples were tested multiple times
with each web browser, over a total of 78 test cycles each,conducted without interruption over 468 hours (every 6 hours
for 20 days). Our engineers removed samples that did not pass
the validation criteria, including those tainted by exploits (notpart of this test). Ultimately, 950 unique, valid malware
samples were included in the final set of 48,672 discrete, valid
malware tests (16,224 tests per web browser), providing amargin of error of less than 3.2 percent (<3.2%) at aconfidence level of 95%.
How We Tested – Malware Samples

Web Browsers Protection Against Malware

To protect against malware, cloud-based reputation systems
scour the Internet for malicious websites and then categorize
content accordingly. Web browsers then ask the cloud-based
reputation systems about specific URLs, files, or applications.
If results indicate that malware is present, the web browser
redirects the user to a warning message explaining that theURL, file, or application is malicious. Some reputation systems
also include additional educational content.
Google Chrome and Mozilla Firefox use the Google Safe
Browsing API for both URL reputation and application
reputation for blocking malicious files. Microsoft Edge uses
Microsoft Defender SmartScreen, which provides protection
from attacks via a cloud-based reputation service for URLreputation, as well as application reputation for malicious fileblocking.
Average Number of Malicious Malware Samples Added Per Day

On average, 49 new validated malware samples were added
to the test set per day; numbers varied on some days as
criminal activity levels fluctuated.

Data in this report spans a testing period of twenty (20) days
between May 11 and May 31, 2021. During the test,
CyberRatings engineers routinely monitored connectivity toensure the browsers under test could access the malware aswell as the reputation services in the cloud.
The emphasis was on freshness with new samples constantly
being added to the test and dead samples removed.
How We Assessed Results

We measured each browser’s ability to block malware as
quickly as they were discovered on the Internet. Engineers
repeated these tests every six hours to determine how long it
would take a vendor to add protection if they did at all.
Each browser’s performance was measured continuously, andthe overall block rate of all malware samples tested with thebrowser was recorded. Each browser’s overall block rate wascalculated as the number of successful blocks divided by thetotal number of test cases. For example, with tests conducted
every 6 hours, a malware sample that was online for 48 hours
was tested eight (8) times. A browser blocking it on 6 (out of amaximum of 8) test runs achieved a block rate of 75%.

Test Environment

Tested Products•••

Microsoft Windows 10 Pro, 21H1

MALWARE PROTECTION 2020•

Google Chrome: Version 90.0.4430.212 - 91.0.4472.19
Microsoft Edge: Version: 91.0.864.19 - 91.0.864.37
Mozilla Firefox: Version 88.0.1 - 88.0.1© 2021 CyberRatings.org. All rights reserved.

Authors
Thomas Skybakmoen, Vikram Phatak

Test Methodology
CyberRatings Web Browser Security Test Methodology v1.0 is available at www.cyberratings.org

Contact Information
CyberRatings.org
2303 Ranch Road 620 South
Suite 160, #501
Austin, TX 78734
info@cyberratings.org
www.cyberratings.org© 2021 CyberRatings.org. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval system,
emailed or otherwise disseminated or transmitted without the express written consent of CyberRatings.org. (“us” or “we”).
1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it.
2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All use of andreliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of any nature whatsoever
arising from any error or omission in this report.
3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED BY US. IN NO
EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT DAMAGES, OR FOR ANY LOSSOF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or thehardware and/or software used in testing the products. The testing does not guarantee that there are no errors or defects in the products orthat the products will meet your expectations, requirements, needs, or specifications, or that they will operate without interruption.
5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report.
6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective
owners.

© 2021 CyberRatings.org. All rights reserved.