VPN Products Performance Benchmarks(Edition 1)
Windows 11
November 2023

Document: VPN Products Performance Benchmarks (Edition 1)
Authors:
J. Han, D. Wren
Company: PassMark Software
Date:
22 November 2023
Edition:1
File:
VPN_Products_Performance_Benchmarks_2023_Ed1.pdf

VPN Products Performance Benchmarks 2023

PassMark Software

Table of Contents
REVISION HISTORY............................................................................................................................................ 3
REFERENCES...................................................................................................................................................... 3
EXECUTIVE SUMMARY ...................................................................................................................................... 4
OVERALL SCORE ................................................................................................................................................ 5
PRODUCTS AND VERSIONS ............................................................................................................................... 6PERFORMANCE METRICS .................................................................................................................................. 7
VPN LOCATIONS ............................................................................................................................................... 9
VPN ENTRY POINT .................................................................................................................................................. 9
VPN EXIT POINT ..................................................................................................................................................... 9
TEST RESULTS ................................................................................................................................................. 10
BENCHMARK 1 – DOWNLOAD SPEED (MBPS) ............................................................................................................. 10
BENCHMARK 2 – UPLOAD SPEED (MBPS)................................................................................................................... 11
BENCHMARK 3 – LATENCY (MS) ............................................................................................................................... 12
BENCHMARK 4 – JITTER (MS) ................................................................................................................................... 13
BENCHMARK 5 – PACKET LOSS (%) ........................................................................................................................... 14
BENCHMARK 6 – NETWORK THROUGHPUT (SEC) ......................................................................................................... 15
BENCHMARK 7 – FILE DOWNLOAD TIME (SEC) ............................................................................................................ 16
BENCHMARK 8 – DATA TRANSFER RATE (MBPS) ......................................................................................................... 17
DISCLAIMER AND DISCLOSURE ....................................................................................................................... 18
CONTACT DETAILS .......................................................................................................................................... 18
APPENDIX 1 – DOWNLOAD SPEED BY COUNTRY ............................................................................................. 19
APPENDIX 2 – TEST ENVIRONMENT ................................................................................................................ 23
APPENDIX 3 – METHODOLOGY DESCRIPTION ................................................................................................. 24

Edition 1

Page 2 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Revision History
Rev
Edition 1

Revision History

Date

Initial version of report

22 November 2023

References
Ref #1

Edition 1

Document
How does Speedtest measure my network speeds?

AuthorDate

Overturf, C

n.d.

Page 3 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Executive Summary
PassMark Software® conducted objective performance testing on nine VPN products on Windows platform. This
report presents our results and findings as a result of performance benchmark testing conducted on theseproducts.
The aim of the benchmark is to compare the performance of Norton Secure VPN with its competitors’ products.
To test the VPN performance, we used the following eight performance metrics:• Download Speed• Upload Speed• Latency• Jitter• Packet Loss
• Network Throughput• File Download Time• Data Transfer Rate
We conducted the benchmarks on Windows machines located in three different countries, connecting the VPN
servers in six different geographic locations.

Edition 1

Page 4 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Overall Score
PassMark Software assigned every product a score depending on its ranking in each metric compared to otherproducts in the same category.
In the following table the highest possible score attainable is 100; in a hypothetical situation where a product has
attained first place in each of the test metrics. The table below shows the VPN products ranked by their overall
scores:
Product Name

Edition 1

Overall Score

Norton Secure VPN

79

NordVPN

74

SurfShark

67

ExpressVPN

54

Proton VPN

54

CyberGhost

53

PureVPN

51

Private Internet Access

39

Hotspot Shield

29

Page 5 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Products and Versions
The tested VPN products names and versions are as follows:
Product Name
Norton Secure VPN

6.3.0.39

NordVPN

7.14.3.0

ExpressVPN

12.59.0 (42)Proton VPN

3.2.1

Private Internet Access

Edition 1

Product Version

3.5 (build 07703)

Hotspot Shield

12.3.5

SurfShark

5.1.1

CyberGhost

8.3.12.10413

PureVPN

11.14.0.3

Page 6 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Performance Metrics
We have selected a set of objective metrics which provide a comprehensive and realistic indication of a VPN’s
performance. The following section lists the details of each performance metric.

Benchmark 1 – Download Speed
This metric measures the download speed over a VPN connection. The test was run by using Speedtest CLI by
Ookla (Version 1.2.0.84) tool. After a VPN connection with a region was established from the VPN software, theSpeedtest command was executed from command line to test the speed and performance of the Internet
connection. The higher download speed value indicates the faster VPN’s downloading performance.

Benchmark 2 – Upload Speed
Similar to the Download Speed test, the Speedtest CLI by Ookla tool was used to measure the upload speed of the
VPN connection. The testing client establishes multiple TCP connections with the specified Speedtest server andtransfers data to the server for a certain amount of time for measuring the uploading speed. The higher upload
speed value indicates the faster VPN’s uploading performance.

Benchmark 3 – Latency
This metric was also measured using the Speedtest CLI by Ookla tool. Latency measures the amount of time ittakes a packet from the client to reach the server, and the server replies to the request and the reply message
arrives at the client. The lower latency value shows the better VPN performance.

Benchmark 4 – Jitter
The delay jitter over a VPN connection was measured by the Speedtest CLI by Ookla tool, and the jitter value
indicates the variance in the packet delay. The lower of the value shows the better VPN performance.

Benchmark 5 – Packet Loss
Packet loss is the packet loss rate of the TCP connection measured by the Speedtest CLI by Ookla tool. It represents
the failure rate of transmitted packets to arrive at the destination server.

Benchmark 6 – Network Throughput
The metric measures the amount of time it takes to download a sample set of files from a Web server using GNU
Wget. The test runs with Hypertext Transfer Protocol (HTTP) which is the main protocol used on the web forbrowsing, linking and data transfer.

Benchmark 7 – File Download Time
This metric measures the amount of time it takes to download a sample set of files from a Web server using
WebClient.DownloadFile method. The test runs with Hypertext Transfer Protocol (HTTP) which is the main
protocol used on the web for browsing, linking and data transfer.
Edition 1

Page 7 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Benchmark 8 – Data Transfer Rate
The metric measures the data transfer rate over VPN connection. This benchmark testing runs with
PerformanceTest (Version 11.0) and uses the Advanced Network Test module of the software to test the datatransfer rate between two computers communicating to each other using TCP/IP.

Edition 1

Page 8 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

VPN Locations
VPN Entry Point
VPN products were installed on the Windows 11 client machines located in three different regions:• United States• United Kingdom• Australia

For the US and UK regions, we used virtual machines hosted on Microsoft Azure to set up the test environment.
In Australia, we ran VMs on a physical machine using VirtualBox software.

VPN Exit Point
The following six countries were selected to set as VPN exit nodes.
• United States• United Kingdom• Germany• Japan• Canada• Australia

Speedtest servers located in the above six locations were selected and used for Speedtest CLI by Ookla test. We
created VMs on Microsoft Azure in the above six different locations and ran Web servers on Windows Server
2022 operating systems installed on each VM. PerformanceTest software was also installed on the VMs to run the
Data Transfer Rate test.

Edition 1

Page 9 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Test Results
In the following charts, we have highlighted the results we obtained for Norton Secure VPN in orange color. The
average has also been highlighted in blue for ease of comparison.

Benchmark 1 – Download Speed (Mbps)
The following chart compares the average download speed of VPN connections of each VPN product tested.
Products with higher download speed values are considered better performing products.

SurfShark

452.48

Norton Secure VPN

401.78

NordVPN

275.30

Proton VPN

254.72Average

213.81

PureVPN

155.66CyberGhost

144.07

ExpressVPN

141.39Hotspot Shield

Private Internet Access

56.56

42.32

0 Mbps100 Mbps

200 Mbps

300 Mbps

400 Mbps

500 Mbps

Download speed ranks by region can be found in the Appendix.

Edition 1

Page 10 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Benchmark 2 – Upload Speed (Mbps)
The following chart compares the average upload speed of VPN connections of the VPN products. The higher
upload speed values mean the better performing products.

Norton Secure VPN

277.56

SurfShark

256.45

PureVPN

219.86

Proton VPN

196.33

ExpressVPN

177.01

Average152.28

NordVPN

97.61

CyberGhost

68.88

Private Internet Access

Hotspot Shield

53.68

23.11

0 Mbps

Edition 1

50 Mbps

100 Mbps

150 Mbps

200 Mbps

250 Mbps

300 Mbps

Page 11 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Benchmark 3 – Latency (ms)The following chart compares the average packet delay values measured over the VPN connections. Products withlower latency are considered better performing products in this category.

CyberGhost

174.60

NordVPN

177.42

ExpressVPN

177.44

Norton Secure VPN

178.31

Private Internet Access

182.81

Average184.66

Proton VPN

185.31

SurfShark

188.43

Hotspot Shield

197.28

PureVPN

160 ms

Edition 1

200.34165 ms

170 ms

175 ms

180 ms

185 ms

190 ms

195 ms200 ms

205 msPage 12 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Benchmark 4 – Jitter (ms)The following chart compares the average delay jitter over the VPN connections. Products with lower jitter are
considered better performing products in this category.

Norton Secure VPN

0.89

NordVPN

1.34

CyberGhost

1.44

SurfShark

2.31

PureVPN

4.64

Private Internet Access

5.20

Proton VPN

5.75

Average

ExpressVPN

19.64

31.64

Hotspot Shield

0 ms

Edition 1

123.54

20 ms

40 ms

60 ms

80 ms

100 ms

120 ms

140 msPage 13 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Benchmark 5 – Packet Loss (%)
The following chart compares the average packet loss rate of the VPN connections measured in the test using
Speedtest CLI by Ookla. Products with lower packet loss rate are considered better performing products in thiscategory.

CyberGhost

0.11

Private Internet Access

0.42

PureVPN

1.38

ExpressVPN

2.36

Norton Secure VPN

2.37

SurfShark

3.00

Average

3.92

NordVPN

6.01

Proton VPN

7.19

Hotspot Shield

12.43

0%

Edition 1

2%

4%

6%

8%

10 %

12 %

14 %

Page 14 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Benchmark 6 – Network Throughput (sec)
The following chart compares the average time taken to download sample files from the Web servers using VPNs.
Products with lower download times are considered better performing products in this category.

NordVPN

14.66

SurfShark

16.72

Hotspot Shield

17.70

Proton VPN

21.76

ExpressVPN

23.27

Norton Secure VPN

30.81

Average41.46

Private Internet Access

53.62

PureVPN

55.76

CyberGhost

138.840s

Edition 1

20 s

40 s

60 s

80 s

100 s

120 s

140 s

160 s

Page 15 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Benchmark 7 – File Download Time (sec)
The following chart compares the average file download times measured using FileDownload tool with the VPNs
connected. Products with lower times are considered better performing products in this category.

NordVPN

9.9

SurfShark

10.9

Norton Secure VPN

22.9

Hotspot Shield

25.6

ExpressVPN

38.1

Proton VPN

39.5

Average

63.5

PureVPN

67.4

Private Internet Access

84.6

CyberGhost
0 sec

Edition 1

273.0

50 sec100 sec

150 sec

200 sec

250 sec

300 sec

Page 16 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Benchmark 8 – Data Transfer Rate (Mbps)
The following chart compares the average data transfer rate tested using PerformanceTest. Products with higher
results are considered better performing products in this category.

Norton Secure VPN

343.53

Proton VPN

217.63

PureVPN

208.31

ExpressVPN

171.46

Average141.13

NordVPN

107.95CyberGhost

90.09

Private Internet Access

53.29

SurfShark

Hotspot Shield

45.04

32.92

0 ms

Edition 1

50 ms

100 ms

150 ms200 ms

250 ms

300 ms

350 ms

400 msPage 17 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Disclaimer and DisclosureThis report only covers the versions that are listed in the “Products and Versions” section of this report.

Disclaimer of Liability
While every effort has been made to ensure that the information presented in this report is accurate, PassMark
Software Pty Ltd assumes no responsibility for errors, omissions, or out-of-date information and shall not be liable
in any manner whatsoever for direct, indirect, incidental, consequential, or punitive damages resulting from theavailability of, use of, access of, or inability to use this information.

Disclosure
Gen Digital funded the production of this report. The products list, performance metrics, and the VPN locations
included in the report were selected by Gen Digital.

Trademarks
All trademarks are the property of their respective owners.

Contact Details
PassMark Software Pty Ltd
Level 5, 63 Foveaux St
Surry Hills, 2010
Sydney, Australia
Phone + 61 (2) 9690 0444
Fax

+ 61 (2) 9690 0445

Web

www.passmark.com

Edition 1

Page 18 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Appendix 1 – Download Speed by Country
The following charts show the download speed rankings by country.

Download Speed - United States
Norton Secure VPN

569.36

SurfShark

480.74

NordVPN

275.36

Proton VPN

266.80

Average

230.80

PureVPN

168.97

ExpressVPN

148.70CyberGhost
Hotspot Shield
Private Internet Access

74.67
58.32
34.30

0 Mbps

Edition 1

100 Mbps

200 Mbps

300 Mbps

400 Mbps

500 Mbps

600 Mbps

Page 19 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Download Speed - United Kingdom
SurfShark

529.45

Norton Secure VPN

371.44

NordVPN

267.66

Proton VPN

218.03

Average

216.00CyberGhost

173.46

ExpressVPN

150.02

PureVPN
Hotspot Shield
Private Internet Access

129.3257.76
46.83

0 Mbps100 Mbps

200 Mbps

300 Mbps

400 Mbps

500 Mbps

600 Mbps

Download Speed - Germany
SurfShark

319.31

Norton Secure VPN

276.42

NordVPN

266.24

Proton VPN

215.49Average

178.34CyberGhost

160.54

ExpressVPN

143.46

PureVPN

142.35

Hotspot Shield
Private Internet Access

56.90
24.39

0 Mbps

Edition 1

100 Mbps

200 Mbps

300 Mbps

400 Mbps

Page 20 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Download Speed - Japan
SurfShark

456.92

Norton Secure VPN

264.19

NordVPN

262.68

Proton VPN

205.02Average

193.69

PureVPN

164.43

ExpressVPN

146.93CyberGhost

136.55

Hotspot Shield

55.23

Private Internet Access

51.24

0 Mbps100 Mbps

200 Mbps

300 Mbps

400 Mbps

500 Mbps

Download Speed - Canada
Norton Secure VPN

346.56

NordVPN

273.45

Proton VPN

217.43

Average167.11

SurfShark

159.00CyberGhost

157.87

PureVPN

136.75

ExpressVPN

109.68

Hotspot Shield

52.08

Private Internet Access

51.14

0 Mbps

Edition 1

100 Mbps

200 Mbps

300 Mbps

400 Mbps

Page 21 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Download Speed - Australia
SurfShark

769.44

Norton Secure VPN

582.67

Proton VPN

405.55

NordVPN

306.45

Average296.91

PureVPN

192.15CyberGhost

161.35

ExpressVPN

149.53

Hotspot Shield

59.07

Private Internet Access

46.00

0 Mbps

Edition 1

100 Mbps

200 Mbps

300 Mbps

400 Mbps

500 Mbps

600 Mbps

700 Mbps

800 Mbps

Page 22 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Appendix 2 – Test Environment
VPN entry nodes virtual machines instance details are as follows:
Azure Region:

West US / UK SouthVM Size:Standard D2s v3

vCPUs:

2

RAM:

8 GiB

Storage Type:

Premium SSD LRSDisk Size:

127 GiB

Network:

Default settings

OS Image:

Windows 11 Pro Version 22H2 – Gen2

The VMs in Australian entry point were ran on the physical machine with the following hardware specifications:
CPU:Intel Core i3-12100 CPU @ 3.30GHz

Motherboard:

ASUS PRIME Z690-A

Video Card:Intel UHD Graphics 730

RAM:

Crucial 32GB DDR5-4800

Main Boot Drive:

Samsung 980 PRO PCIe 4.0 NVMe SSD 250GB

2nd Drive:

Samsung SSD 850 PRO 512GB

Network:

1Gbps (Connected via Gigabit Ethernet cable)

OS:

Windows 11 Pro Version 22H2 (OS Build 22621.1105)

VPN exit nodes virtual machines instance details are as follows:
Azure Region:

Central US / UK South / Germany West Central / Japan East / Canada Central / Australia

East
VM Size:Standard D4s v3vCPUs:

4RAM:

16 GiB

Storage Type:

Premium SSD LRS, 500 Max IOPS

Disk Size:

127 GiB

Network:

Default settings

OS Image:

Windows Server 2022 Datacenter – Gen2

Edition 1

Page 23 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Software

Appendix 3 – Methodology Description
We conducted the performance test for five rounds in total. The first three rounds were run from machines locatedin Australia, the fourth round was run from United Kingdom, and the fifth-round test was completed in UnitedStates.
For each VPN product installed on the machine in each location, we established VPN connections with six VPN
servers in different locations United States, United Kingdom, Germany, Japan, Canada and Australia. We ran thebenchmarks three times with each VPN connection and averaged the numbers to obtain a final result for theround.
The final results that used to calculate the over score were from the average of the first three rounds tests from
Australia, then averaging that average with the fourth and fifth round results from United States and United
Kingdom.

Benchmark 1 – Download Speed
The download speed over VPN connections was measured by Speedtest CLI by Ookla tool. After a VPN connection
with a region was established from the VPN software, the Speedtest command was executed from command line
to test the speed and performance of the Internet connection.
For each VPN region, we listed the nearest Speedtest servers and picked one from them. In case the selected
Speedtest server became not available for any reason, we chose a different one and kept on using the new one
for the remaining tests. When executing the Speedtest command we specified the selected server using its server
id.

Benchmark 2 – Upload Speed
Similar to the download speed test, the upload speed was measured by Speedtest CLI by Ookla tool.

Benchmark 3 – Latency
Similarly, the latency was measured by Speedtest CLI by Ookla tool.

Benchmark 4 – Jitter
Jitter was measured by Speedtest CLI by Ookla tool.

Benchmark 5 – Packet Loss
Packet loss was also measured by Speedtest CLI by Ookla tool.

Benchmark 6 – Network Throughput
The metric measures the amount of time it takes to download a sample set of files using GNU Wget (Version
1.10.1). This benchmark uses GNU Wget software package to download the sample files hosted on Web servers
Edition 1

Page 24 of 25
22 November 2023

VPN Products Performance Benchmarks 2023

PassMark Softwarein different regions. The CommandTimer.exe is used in conjunction with GNU Wget to measure the time. The
sample data set details are as shown in the table below:
File Type

NumberVideo Files2

26.8 MBSize
15.4 MBPDF Files

9

Compressed Archive Files2

16.1 MBTotal

13

58.3 MB

Benchmark 7 – File Download Time
This metric measures the amount of time it takes to download a sample set of files using WebClient.DownloadFile
method. The test uses FileDownload (by PassMark Software) tool developed using Microsoft Windows
Webclient.DownloadFile() API to download files from the Web servers located in different regions. The
CommandTimer.exe is used in conjunction with the tool to measure the download execution time. The data set
includes the following:
Filename
TeamViewer_Setup.exe

Size
26.7 MB

GeForce_Experience_v3.20.5.70.exe

121 MB

Total

147.7 MB

Benchmark 8 – Data Transfer Rate
PerformanceTest software was installed on both VPN entry and exit machines and the data transfer rate of thetwo computers was measured using the software Advanced Network Test module. In the PerformanceTest
Advanced Network Test settings, the VPN entry node was configured as client mode the VPN exit machine was set
as server mode, the sending data block size was set as fixed 32,768 Bytes, the test duration was set to 40 seconds
with the thread number set to 10, and IPv4 and TCP protocols were selected by default. The test will allow thedata to be sent from the client to server at the highest transfer rate possible for the duration of the test over TCP
connection with VPN enabled.

Edition 1

Page 25 of 25
22 November 2023