VPN Products Performance Benchmarks(Edition 1)
Windows 10March 2021

Document: VPN Products Performance Benchmarks (Edition 1)
Authors:
J. Han, D. Wren
Company: PassMark Software
Date:
3 March 2021
Edition:1
File:
VPN_Products_Performance_Benchmarks_2021_Ed1.pdf

VPN Products Performance Benchmarks 2021

PassMark Software

Table of Contents
REVISION HISTORY ........................................................................................................................................ 3
REFERENCES.................................................................................................................................................. 3
EXECUTIVE SUMMARY .................................................................................................................................. 4
OVERALL SCORE ............................................................................................................................................ 5
PRODUCTS AND VERSIONS ............................................................................................................................ 6PERFORMANCE METRICS............................................................................................................................... 7
VPN LOCATIONS............................................................................................................................................ 9
VPN ENTRY POINT .................................................................................................................................................. 9
VPN EXIT POINT ..................................................................................................................................................... 9
TEST RESULTS.............................................................................................................................................. 10
BENCHMARK 1 – DOWNLOAD SPEED (MBPS) ............................................................................................................. 10
BENCHMARK 2 – UPLOAD SPEED (MBPS) .................................................................................................................. 11
BENCHMARK 3 – LATENCY (MS) ............................................................................................................................... 12
BENCHMARK 4 – JITTER (MS) ................................................................................................................................... 13
BENCHMARK 5 – PACKET LOSS (%) ........................................................................................................................... 14
BENCHMARK 6 – NETWORK THROUGHPUT (SEC) ......................................................................................................... 15
BENCHMARK 7 – FILE DOWNLOAD TIME (SEC) ............................................................................................................ 16
BENCHMARK 8 – DATA TRANSFER RATE (MBPS) ......................................................................................................... 17
DISCLAIMER AND DISCLOSURE .................................................................................................................... 18
CONTACT DETAILS ....................................................................................................................................... 18
APPENDIX 1 – TEST ENVIRONMENT ............................................................................................................. 19
APPENDIX 2 – METHODOLOGY DESCRIPTION .............................................................................................. 20

Edition 1

Page 2 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Revision History
RevEdition 1

Revision History

Date

Initial version of report

3 March 2021

References
Ref #1

Edition 1

Document
How does Speedtest measure my network speeds?

AuthorDate

Overturf, Cn.d.

Page 3 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Executive Summary
PassMark Software® conducted objective performance testing on ten VPN products on Windows platform. Thisreport presents our results and findings as a result of performance benchmark testing conducted on theseproducts.
The aim of the benchmark is to compare the performance of Norton Secure VPN with its competitors’ products.
To test the VPN performance, we used the following eight performance metrics:• Download Speed• Upload Speed• Latency• Jitter• Packet Loss
• Network Throughput• File Download Time• Data Transfer Rate
We conducted the benchmarks on Windows machines located in three different countries, connecting the VPN
servers in five different geographic locations.

Edition 1Page 4 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Overall Score
PassMark Software assigned every product a score depending on its ranking in each metric compared to otherproducts in the same category.
In the following table the highest possible score attainable is 80; in a hypothetical situation where a product has
attained first place in all 8 metrics. The table below shows the VPN products ranked by their overall scores:
Product Name

Edition 1

Overall Score

Norton Secure VPN

70

NordVPN

67

Private Internet Access

56

Surfshark

53

ExpressVPN

45

IPVanish42

Hotspot Shield

31

Avast SecureLine VPN

28

CyberGhost

26

PureVPN22

Page 5 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Products and Versions
The tested VPN products names and versions are as follows:
Product Name
Norton Secure VPN

4.0.0.408

NordVPN

6.33.10.0

Surfshark

2.8.0

ExpressVPN

10.0.9IPVanish

3.6.5.0

Hotspot Shield

10.12.2

Avast SecureLine VPN

5.8.5262

CyberGhost

8.2.1.7176

Private Internet Access
PureVPN

Edition 1

Product Version

2.6.1 (build 05824)
1.1.74.0

Page 6 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Performance Metrics
We have selected a set of objective metrics which provide a comprehensive and realistic indication of a VPN’sperformance. The following section lists the details of each performance metric.

Benchmark 1 – Download Speed
This metric measures the download speed over a VPN connection. The test was run by using Speedtest CLI by
Ookla (Version 1.0.0.2) tool. After a VPN connection with a region was established from the VPN software, theSpeedtest command was executed from command line to test the speed and performance of the Internet
connection. The higher download speed value indicates the faster VPN’s downloading performance.

Benchmark 2 – Upload Speed
Similar to the Download Speed test, the Speedtest CLI by Ookla tool was used to measure the upload speed of theVPN connection. The testing client establishes multiple TCP connections with the specified Speedtest server andtransfers data to the server for a certain amount of time for measuring the uploading speed. The higher upload
speed value indicates the faster VPN’s uploading performance.

Benchmark 3 – Latency
This metric was also measured using the Speedtest CLI by Ookla tool. Latency measures the amount of time ittakes a packet from the client to reach the server, and the server replies to the request and the reply message
arrives at the client. The lower latency value shows the better VPN performance.

Benchmark 4 – Jitter
The delay jitter over a VPN connection was measured by the Speedtest CLI by Ookla tool, and the jitter value
indicates the variance in the packet delay. The lower of the value shows the better VPN performance.

Benchmark 5 – Packet Loss
Packet loss is the packet loss rate of the TCP connection measured by the Speedtest CLI by Ookla tool. It represents
the failure rate of transmitted packets to arrive at the destination server.

Benchmark 6 – Network Throughput
The metric measures the amount of time it takes to download a sample set of files from a Web server using GNU
Wget. The test runs with Hypertext Transfer Protocol (HTTP) which is the main protocol used on the web forbrowsing, linking and data transfer.

Benchmark 7 – File Download Time
This metric measures the amount of time it takes to download a sample set of files from a Web server using
WebClient.DownloadFile method. The test runs with Hypertext Transfer Protocol (HTTP) which is the main
protocol used on the web for browsing, linking and data transfer.
Edition 1

Page 7 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 8 – Data Transfer Rate
The metric measures the data transfer rate over VPN connection. This benchmark testing runs with
PerformanceTest (Version 10.0) and uses the Advanced Network Test module of the software to test the datatransfer rate between two computers communicating to each other using TCP/IP.

Edition 1

Page 8 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark SoftwareVPN Locations
VPN Entry Point
VPN products were installed on the Windows 10 client machines located in three different regions:• United States• United Kingdom• Australia

For the US and UK regions, we used virtual machines hosted on Microsoft Azure to set up the test environment.
In Australia, we ran VMs on a physical machine using VMware software.

VPN Exit Point
The following five countries were selected to set as VPN exit nodes.
• United States• United Kingdom• Germany• Japan• Australia

Speedtest servers located in the above five locations were selected and used for Speedtest CLI by Ookla test. We
created VMs on Microsoft Azure in the above five different locations and ran Web servers on Windows Server
2016 operating systems installed on each VM. PerformanceTest software was also installed on the VMs to run
the Data Transfer Rate test.

Edition 1

Page 9 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Test Results
In the following charts, we have highlighted the results we obtained for Norton Secure VPN in orange color. The
average has also been highlighted in blue for ease of comparison.

Benchmark 1 – Download Speed (Mbps)
The following chart compares the average download speed of VPN connections of each VPN product tested.
Products with higher download speed values are considered better performing products.

PureVPN

82.71

Private Internet Access

99.92

Avast SecureLine VPN

113.81CyberGhost

118.29Hotspot Shield

123.87

ExpressVPN

130.67

Average159.46

SurfShark

194.83

IPVanish200.65

Norton Secure VPN

230.69

NordVPN
0 Mbps

Edition 1

299.1850 Mbps

100 Mbps

150 Mbps

200 Mbps

250 Mbps

300 Mbps

350 Mbps

Page 10 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 2 – Upload Speed (Mbps)
The following chart compares the average upload speed of VPN connections of the VPN products. The higher
upload speed values mean the better performing products.

PureVPN

IPVanish

Avast SecureLine VPN

26.05

38.70

47.76

SurfShark

54.24

CyberGhost

55.30

Hotspot Shield

57.45

Norton Secure VPN

61.87

Average

87.09

ExpressVPN

115.16

Private Internet Access

188.64

NordVPN0 Mbps

Edition 1

225.7650 Mbps

100 Mbps

150 Mbps

200 Mbps

250 Mbps

Page 11 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 3 – Latency (ms)The following chart compares the average packet delay values measured over the VPN connections. Products withlower latency are considered better performing products in this category.

Norton Secure VPN

150.03

IPVanish150.45

Private Internet Access153.75

NordVPN

159.51

SurfShark

161.17

ExpressVPN

162.90Avast SecureLine VPN

163.37

PureVPN

167.48Average200.72CyberGhost

280.80Hotspot Shield

0 ms

Edition 1

457.7350 ms

100 ms

150 ms200 ms

250 ms

300 ms

350 ms

400 ms

450 ms

500 msPage 12 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 4 – Jitter (ms)The following chart compares the average delay jitter over the VPN connections. Products with lower jitter are
considered better performing products in this category.

Private Internet Access

0.38

Norton Secure VPN

0.38

SurfShark

1.79

NordVPN

2.39

Avast SecureLine VPN

3.15

ExpressVPN

6.94

IPVanish

7.38

PureVPN

13.74

Average

54.78

CyberGhost

133.76

Hotspot Shield0 ms

Edition 1

377.9050 ms

100 ms

150 ms200 ms

250 ms

300 ms

350 ms

400 msPage 13 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 5 – Packet Loss (%)
The following chart compares the average packet loss rate of the VPN connections measured in the test using
Speedtest CLI by Ookla. Products with lower packet loss rate are considered better performing products in thiscategory.

PureVPN

0.00

Norton Secure VPN

0.09

SurfShark

0.18

Private Internet Access

0.35

NordVPN

0.40

ExpressVPN

1.03

IPVanish

4.92

Avast SecureLine VPN

5.23

Average

6.34

CyberGhost

13.41

Hotspot Shield

37.78

0%

Edition 1

5%

10 %

15 %

20 %

25 %

30 %

35 %

40 %

Page 14 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 6 – Network Throughput (sec)
The following chart compares the average time taken to download a sample files from the Web servers using VPNs.
Products with lower download times are considered better performing products in this category.

SurfShark

13.65

NordVPN

15.99

Norton Secure VPN

22.40

IPVanish24.81

Hotspot Shield

27.40

Private Internet Access

36.11

ExpressVPN

36.91

Average37.08

Webroot SecureAnywhere AntiVirus

40.19

Avast SecureLine VPN

48.81

PureVPN

104.540s

Edition 120 s

40 s

60 s

80 s

100 s

120 s

Page 15 of 21
3 March 2021VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 7 – File Download Time (sec)
The following chart compares the average file download times measured using FileDownload tool with the VPNs
connected. Products with lower times are considered better performing products in this category.

Norton Secure VPN

13.7

SurfShark

25.7

NordVPN

29.4

Hotspot Shield

30.0

Private Internet Access

68.9

Average

75.7

ExpressVPN

83.0

Avast SecureLine VPN

84.8

IPVanish89.9

CyberGhost

93.0

PureVPN0 sec

Edition 1

238.1

50 sec100 sec

150 sec

200 sec

250 sec

300 sec

Page 16 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 8 – Data Transfer Rate (Mbps)
The following chart compares the average data transfer rate tested using PerformanceTest. Products with higher
results are considered better performing products in this category.

SurfShark

PureVPN

Avast SecureLine VPN

Hotspot Shield

Webroot SecureAnywhere AntiVirus

31.87

45.44

52.66

57.82

69.60

CyberGhost

102.22Average108.27

ExpressVPN

131.56

Norton Secure VPN

146.34

Private Internet Access

193.72

NordVPN
0 ms

Edition 1

251.5250 ms

100 ms

150 ms200 ms

250 ms

300 msPage 17 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Disclaimer and DisclosureThis report only covers the versions that are listed in the “Products and Versions” section of this report.

Disclaimer of Liability
While every effort has been made to ensure that the information presented in this report is accurate, PassMark
Software Pty Ltd assumes no responsibility for errors, omissions, or out-of-date information and shall not be liable
in any manner whatsoever for direct, indirect, incidental, consequential, or punitive damages resulting from theavailability of, use of, access of, or inability to use this information.

Disclosure
NortonLifeLock Inc. funded the production of this report. The products list, performance metrics, and the VPN
locations included in the report were selected by NortonLifeLock.

Trademarks
All trademarks are the property of their respective owners.

Contact Details
PassMark Software Pty Ltd
Level 5, 63 Foveaux St
Surry Hills, 2010
Sydney, Australia
Phone + 61 (2) 9690 0444
Fax+ 61 (2) 9690 0445

Web

www.passmark.com

Edition 1

Page 18 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Appendix 1 – Test Environment
VPN entry nodes virtual machines instance details are as follows:
Azure Region:

West US / UK SouthVM Size:Standard D2s v3

vCPUs:

2RAM:

8 GiB

Storage Type:

Premium SSD

Disk Size:

127 GiB

Network:

Default settings

OS Image:

Windows 10 Pro Version 2004 – Gen1

The VMs in Australian entry point were ran on the physical machine with the following hardware specifications:
CPU:Intel Core i7-6700K @ 4.00GHz

Motherboard:

ASRock Fatal1ty Z170 Gaming K6

Video Card:

ASUS Strix GeForce GTX 960 4GB

RAM:

G.Skill DDR4-2133MHz 2x8GB

Main Boot Drive:

Samsung 950 Pro M.2 NVMe 256GB

2nd Drive:

Crucial BX500 SATA SSD 1TB

Network:

1Gbps (Connected via Gigabit Ethernet cable)

OS:

Windows 10 Pro Version 2004 (OS Build 19041.508)

VPN exit nodes virtual machines instance details are as follows:
Azure Region:

Central US / UK South / Germany West Central / Japan East / Australia East

VM Size:Standard D2s v3vCPUs:

2

RAM:

8 GiB

Storage Type:

Premium SSD

Disk Size:

127 GiB

Network:

Default settings

OS Image:

Windows Server 2016 Datacenter – Gen1

Edition 1Page 19 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Appendix 2 – Methodology Description
We conducted the performance test for five rounds in total. The first three rounds were run from machines locatedin Australia, the fourth round was run from United Kingdom, and the fifth-round test was completed in UnitedStates.
For each VPN product installed on the machine in each location, we established VPN connections with five VPN
servers in different locations United States, United Kingdom, Germany, Japan, and Australia. We ran thebenchmarks three times with each VPN connection and averaged the numbers to obtain a final result for theround.
The final results that used to calculate the over score were from the average of the first three rounds tests fromAustralia, then averaging that average with the fourth and fifth round results from United States and UnitedKingdom.

Benchmark 1 – Download Speed
The download speed over VPN connections was measured by Speedtest CLI by Ookla (Version 1.0.0.2) tool. Aftera VPN connection with a region was established from the VPN software, the Speedtest command was executedfrom command line to test the speed and performance of the Internet connection.
For each VPN region, we listed the nearest Speedtest servers and picked one from them. In case the selected
Speedtest server became not available for any reason, we chose a different one and kept on using the new one
for the remaining tests. When executing the Speedtest command we specified the selected server using its server
id. List of Speedtest servers selected are as follows:
VPN Region (Exit Point)

Speedtest Server ID

United States

37261

United Kingdom7352, 6032

Germany10010, 30907

Japan8407

Australia

2629, 21417

Benchmark 2 – Upload Speed
Similar to the download speed test, the upload speed was measured by Speedtest CLI by Ookla tool.

Benchmark 3 – Latency
Similarly, the latency was measured by Speedtest CLI by Ookla tool.

Benchmark 4 – Jitter
Jitter was measured by Speedtest CLI by Ookla tool.

Benchmark 5 – Packet Loss
Packet loss was also measured by Speedtest CLI by Ookla tool.
Edition 1

Page 20 of 21
3 March 2021

VPN Products Performance Benchmarks 2021

PassMark Software

Benchmark 6 – Network Throughput
The metric measures the amount of time it takes to download a sample set of files using GNU Wget (Version1.10.1). This benchmark uses GNU Wget software package to download the sample files hosted on Web servers
in different regions. The CommandTimer.exe is used in conjunction with GNU Wget to measure the time. Thesample data set details are as shown in the table below:
File Type

NumberSize

Video Files

226.8 MBPDF Files

915.4 MB

Compressed Archive Files

216.1 MBTotal

1358.3 MB

Benchmark 7 – File Download Time
This metric measures the amount of time it takes to download a sample set of files using WebClient.DownloadFilemethod. The test uses FileDownload (by PassMark Software) tool developed using Microsoft Windows
Webclient.DownloadFile() API to download files from the Web servers located in different regions. The
CommandTimer.exe is used in conjunction with the tool to measure the download execution time. The data set
includes the following:
Filename
TeamViewer_Setup.exe

Size
26.7 MB

GeForce_Experience_v3.20.5.70.exe

121 MB

Total

147.7 MB

Benchmark 8 – Data Transfer Rate
PerformanceTest software was installed on both VPN entry and exit machines and the data transfer rate of thetwo computers was measured using the software Advanced Network Test module. In the PerformanceTest
Advanced Network Test settings, the VPN entry node was configured as client mode the VPN exit machine was set
as server mode, the sending data block size was set as fixed 32,768 Bytes, the test duration was set to 40 seconds
with the thread number set to 10, and IPv4 and TCP protocols were selected by default. The test will allow thedata to be sent from the client to server at the highest transfer rate possible for the duration of the test over TCP
connection with VPN enabled.

Edition 1Page 21 of 21
3 March 2021