Fast and Effective Endpoint
Security for BusinessComparative Analysis
June 2010

Document: Fast and Effective Endpoint Security for Business – Comparative Analysis
Authors:
K. Lai, D. Wren
Company: PassMark Software
Date:
15 June 2010
Edition:
Edition 1
File:
Fast and Effective Security for Business - Ed1 June 2010.docx

Fast and Effective Endpoint Security for Business

PassMark Software

Table of Contents
TABLE OF CONTENTS........................................................................................................................................... 2
REVISION HISTORY.............................................................................................................................................. 3
REFERENCES........................................................................................................................................................ 3
INTRODUCTION .................................................................................................................................................. 4
RATINGS AND SUMMARY ................................................................................................................................... 5
RATING CATEGORIES .................................................................................................................................................... 6
STAR RATING DESCRIPTION ........................................................................................................................................... 6TASK DESCRIPTION ............................................................................................................................................. 7
HARDWARE ENVIRONMENTS.......................................................................................................................................... 7
PRODUCTS AND VERSIONS TESTED .................................................................................................................................. 7
PERFORMANCE BENCHMARK RESULTS ............................................................................................................... 8
ESET SMART SECURITY BUSINESS EDITION ....................................................................................................... 11
KASPERSKY BUSINESS SPACE SECURITY ............................................................................................................ 14
MCAFEE TOTAL PROTECTION FOR ENDPOINT ................................................................................................... 17
MICROSOFT FOREFRONT CLIENT SECURITY ....................................................................................................... 20
SOPHOS ENDPOINT SECURITY AND DATA PROTECTION .................................................................................... 23
SYMANTEC ENDPOINT PROTECTION ................................................................................................................. 26
TREND MICRO WORRY FREE BUSINESS SECURITY ............................................................................................. 29
DISCLAIMER AND DISCLOSURE ......................................................................................................................... 32
DISCLAIMER OF LIABILITY............................................................................................................................................. 32
DISCLOSURE ............................................................................................................................................................. 32
TRADEMARKS............................................................................................................................................................ 32
CONTACT DETAILS ............................................................................................................................................ 32
APPENDIX A – PERFORMANCE METHODOLOGY ................................................................................................ 33

Comparative Analysis
Edition 1

Page 2 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Revision History
Rev
Edition 1

Revision History

DateFirst edition of the document.
Performance charts and comparative reviews added.

15 Jun 2010

References
Ref #Document

AuthorDate

AV Comparatives

24 Dec 2009

AV Comparatives

17 Mar 2010

AV Comparatives

5 Jun 2010

Virus Bulletin

Jun 2010

Nico Cuppen Software-

AV Comparatives - Summary Report 2009
Current Edition: December 20091

The information in the most recent version of this report by
AV Comparatives was used to determine the effectiveness ofreviewed business security solutions.
AV Comparatives - On-Demand Comparatives
Current Edition: Report Number 25, February 2010

2

The information from the most recent version of this reportby AV Comparatives was used to determine the effectivenessof reviewed business security solutions.
AV Comparatives – Retrospective/ProActive Test
Current Edition: Report Number 26, May 2010

3

Information from Retrospective/ProActive Test reports by AV
Comparatives was used in determining the effectiveness ofreviewed business security solutions.
VB100 Test Results
Current results: Windows Server 2008 R2, June 20104

Overall test results obtained by VB100 were used indetermining the effectiveness of reviewed business security
solutions.
Network Traffic Monitor v2.01

5

A tool used to monitor the amount of inbound and outboundnetwork traffic for the Update Size metric.

Comparative Analysis
Edition 1

Page 3 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Introduction
The importance of effective security for businesses cannot be understated, with malware damage and costs to1businesses escalating every year . The nature of the threat landscape has become more sophisticated, withmalware events not only causing lost productivity and reputation for affected businesses but a staggering2potential for losses from theft of data and other security breaches from more targeted attacks.
In response, businesses have been placing their confidence in a wide range of security solutions to meet theirneeds. With the large amount of software available from many vendors, the challenge for businesses now
becomes determining which security solution is the most effective at mitigating the threat of malware, while
minimizing implementation cost and impact to existing business functions and workflow.
This report presents a comparative analysis on the performance, effectiveness and usability of seven security
solutions from some of the world’s largest security vendors. In this report, PassMark Software has evaluated thefollowing business security products:
•
•
•
•
•
•
•

1
2

ESET Smart Security 4 Business Edition
Kaspersky Business Space Security
McAfee Total Protection for Endpoint
Microsoft Forefront Client Security
Symantec Endpoint Protection
Sophos Endpoint Security and Data Protection
Trend Micro Worry-Free Business Security: Standard Edition

A 2007 survey conducted by ‘Computer Economics Inc’ estimates that worldwide, malware costs businesses over US$13B each year.
The Fifth Annual “Cost of a Data Breach” report by Ponemon Institute, Jan 2010, puts the average price of data breach at US$204 per compromised record.

Comparative Analysis
Edition 1

Page 4 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Ratings and Summary
Passmark Software has given each security product a rating which reflects its overall performance, ease of use, design, features and level of excellence in that category.
Categories represent major functions or feature sets common to the sphere of business security. These ratings represent PassMark Software’s subjective views andexperiences in installing, configuring and use of business security products to manage endpoints.
The following table summarizes ratings in all categories for all products evaluated:
ESET

Kaspersky

McAfeeMicrosoft

SophosSymantec

Trend Micro

Overall Rating
Installation & Configuration
Migration
Default Policies
Client Installation
Interface Design
Client & Policy Management
Remote Management--

UpdatesCommon Use Cases
Effectiveness
Performance

Comparative Analysis
Edition 1

Page 5 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Rating Categories
The table below describes the criteria and factors which were considered for each business security solution ineach category to determine a rating. Evaluation categories were determined prior to testing and were chosen as
a set of expected features or functions which define business security products.
Category
Overall Rating
Installation & Configuration

Category Description
The rating for this category is calculated as an average of all other ratings, with all categories carrying
equal weight.
This category evaluates the speed and relative ease of the installation and configuration process of server
components, including the quality and accuracy of documentation, the ability to install pre-requisites andthe level of installer integration.

Migration

This category rates the relative ease and simplicity of product migration from previous vendor solutionsor third party software solutions. Extra consideration is given to vendors who have documented themigration process well.

Default Policies

This category considers whether the policies created by default during product installation ‘make sense’
from a management perspective, taking into account whether different default choices are available.

Client Installation

This category evaluates the simplicity and ease of client installation, taking into account the speed of
deployment and the level of impact of installation on endpoint users.

Interface Design

This category rates the design of the server console’s user interface for responsiveness, intuitiveness,consistency and functionality.

Client & Policy Management

This category assesses the flexibility and granularity of policy management from the server console,taking into account the level of automation in setting up groups and the depth of configuration options
available to administrators.

Remote Management

This category examines how much support is provided for administrators to access the management
console from a remote terminal.

UpdatesThis category rates the level of configuration required to enable the management server to update acentral repository, as well as the ease of deployment to and the timeliness of retrieval by endpoint
machines.

Common Use Cases

This category examines the relative ease of use and handling for three common management use casesby each security product. These scenarios are: conducting an ‘on-demand’ scan of selected endpoints;
creating and viewing a malware report ‘on-demand’; and creating, assigning and deploying a new policy.

EffectivenessThis category rates the anti-malware effectiveness of a security product based on information fromrecently published material at reputable, third party testing sites. The sources we have used for thiscategory are VB100 and AV Comparatives.

PerformanceThis category assigns an overall rating based on a security product’s performance over ten performancebenchmark tests conducted by PassMark Software.

Star Rating Description
The table below explains the general significance of ratings relative to product performance, usability andfunctionality.
Star Rating--

Rating Description
Unsupported – This category was not supported by the business security solution. Support was not documented in
product guides, the online knowledgebase or help files.
Very Poor – The security solution offered very limited performance in this category. Products with this rating hadsparse or inaccurate documentation, extremely poor usability, or technical issues which severely hampered
product stability, usability and functionality.
Poor – The security solution had inadequate or basic performance in this category, as a result of poor usability orfunctionality. Some products with this rating had bugs which hampered product performance in this category.
Average – The security solution had adequate performance in this category with some room for improvement.
Good– The security solution provides good performance in this category area with useful features and good
documentation.
Exceptional – The security solution provides outstanding performance in a category area, with unique, thoughtful
or well-designed features that streamline usability or functionality and excellent documentation.

Comparative Analysis
Edition 1

Page 6 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Task Description
PassMark Software has conducted performance benchmark testing and subjective comparative analysis on theoverall ease of use, speed and effectiveness on seven (7) business security software products.

Hardware Environments
The following hardware platforms were used in conducting our comparative analysis and performance tests, andare intended to represent a typical server and business deployment:

Server Machine Specification
The following machine ran a virtual machine on which the server components of the security software
were installed:
Operating System:
CPU:
Motherboard:
RAM:
HDD:

Windows Server 2003 32-bit
Intel Xeon CPU @ 3.4GHz
Hewlett-Packard 08B4h Motherboard
4GB ECC RAM
Western Digital Raptor 74GB 10,000RPM

Virtual Machine Specification
Operating System:
RAM:

Windows Server 2003 32-bit
1-2 GB (depending on the product’s requirements)

Client Machine Specification
Operating System:
CPU:
Video Card:
RAM:
HDD:

Windows 7 Ultimate x64
Intel Core i7 720 @ 2.67GHz
nVidia GeForce 8800 GT
6 GB
500 GB

Products and Versions Tested
Product Name

Server Component and Version

Client Component and Version

ESET Smart Security 4
Business Edition

Remote Administrator Console:v4.0.122.0

Smart Security 4 Business Edition:
v4.2.40

Kaspersky BusinessSpace Security

Administration Kit:
v8.0.2090

Kaspersky Anti-virus 6.0:
v6.0.4.1424

McAfee Total Protection
for Endpoint

ePolicy Orchestrator:
v4.5

McAfee Agent:v4.5.0.1270

Microsoft Forefront
Client Security

Forefront Client Security Console:v1.1.1710.90

Forefront Client Security:v1.5.1981.0

Symantec Endpoint
Protection

Endpoint Protection Manager:
v11.0.600.550

Symantec Endpoint Protection:
v11.0.6000.550

Sophos Endpoint Security
and Data Protection

Enterprise Console:v4.0.0.2362

Endpoint Security and Control:
v9.05

Trend Micro Worry-Free Business
Security: Standard Edition

Worry-Free Business Security:v6.0 SP2 build 3025

Client/Server Security Agent:v16.0.3052

Comparative Analysis
Edition 1

Page 7 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Performance Benchmark Results
The following performance categories have been selected as ‘real-life’ metrics which may impact heavily on
endpoint system performance and responsiveness. These benchmarks allow the comparison of the level ofimpact that business security software products may have on endpoint machines. Products with good
performance will have less impact on business activities, workflow and productivity.
More detailed description of the methodology used can be found in Appendix A – Performance Methodology.

Install Size

Protect endpoints without filling up disk space
Newer versions of products often have increased disk
space requirements, ensuring disk space remains criticalfor endpoint systems. Endpoint clients with a larger
installation footprint may be consuming more disk spacethan necessary.
This metric measures the total additional disk spaceconsumed by the endpoint client after installation and amanual update. Our final result is measured in megabytes(MB).

Symantec
McAfeeKaspersky
Trend Micro
ESET
SophosMicrosoft
0 MB

200 MB

400 MB

600 MB

800 MB

Memory usage commit charge

Have more system resources to perform tasks
Extensive memory (or physical RAM) usage by security
products have significant impact on endpoint system
performance and cause more reliance on hard disk drives,
which have slower read and write speeds than RAM.
Business security products which use more memory willvisibly slow performance on affected endpoints.
This metric measures the total additional memory use
consumed by the endpoint machine during a period ofsystem idle where an endpoint security product has been
installed. Our final result is measured in megabytes (MB),and calculated from an average of 40 samples.

Microsoft
McAfeeSophosTrend Micro
Symantec
Kaspersky
ESET
0 MB

60 MB

120 MB180 MB240 MB

300 MB

Word document launch time

Spend less time waiting for documents to launch
Security products may hinder launch times of applications
and documents as a result of poorly performing antimalware functionality, such as real-time file scanning or
behavioural heuristics. Slow endpoint system response
times can bring about avoidable issues for productivity.
The metric measures the total time taken to launch a large
Microsoft Word 2007 document with a system restart prior
to application launch. Our final result is measured inmilliseconds (ms), and calculated from an average of five(5) samples.

Comparative Analysis
Edition 1

McAfeeTrend Micro
SophosMicrosoft
Symantec
Kaspersky
ESET
0 ms

4,000 ms

8,000 ms

12,000 ms

Page 8 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Word document restart time
Restart your applications without the wait
Restarting applications is a common task for endpoint
users, who may quickly close or switch between
documents and applications as they conduct work tasks.
The amount of time taken by applications to restart is a
visible measure of system responsiveness for endpoint
users.
This metric measures the total time taken to re-open alarge, mixed media Microsoft Word 2007 document, wherean endpoint security product has been installed. Our final
result is measured in milliseconds (ms), and calculated
from an average of ten (10) samples.

File copy time for small files
Copy your documents more quickly
Transferring files between devices and drives is a common
activity undertaken by endpoint users. File copy times may
be negatively affected by poor performance of businesssecurity products functionality, such as file scanning or
heuristics.
This metric measures the total time taken to copy a set ofsmall files between directories, where an endpoint security
product has been installed. Our final result is measured inseconds (s), and calculated from an average of five (5)samples.

PassMark Software

McAfeeKaspersky
Symantec
ESET
Trend Micro
SophosMicrosoft
1,800 ms

McAfeeSymantec
Kaspersky
ESET
0s

Kaspersky

10 s

20 s

30 s

40 s

50 s

60 s

Trend Micro
McAfeeSophosSymantec
ESET

Client Signature Database Size

Symantec

Reduce the footprint of endpoint solutions

Trend MicroThis metric measures the total size of anti-malware
signature files on the endpoint machine after a manual
update. Our final result is measured in megabytes.

2,600 ms

SophosMicrosoft

The size of the anti-malware signature database on
endpoint machines gives an indication of a product’s
footprint on the local disk drive.

2,400 ms

Microsoft

Copy your media files more quickly

This metric measures the total time taken to copy a set oflarge files between directories, where an endpoint security
product has been installed. Our final result is measured inseconds (s), and calculated from an average of five (5)samples.

2,200 ms

Trend Micro

File copy time for large files
Copying large files between directories may similarly beaffected by poor performance of anti-malware
functionality in business security products.

2,000 ms

80 s

82 s

0 MB

40 MB

84 s

86 s

88 s

90 s

120 MB

160 MB

McAfeeKaspersky
SophosMicrosoft
ESET

Comparative Analysis
Edition 1

80 MB

Page 9 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Boot time
Spend less time waiting for your computer to start
Many business software suites create start up tasks andprocesses, causing machine boot times to take significantly
longer. End users can do little but wait for their machine tobecome responsive. Better performing products will haveless of an impact on boot time.
This metric measures the time taken to boot the machine
where an endpoint security product has been installed. Our
final result is measured in seconds (s) and calculated froman average of five (5) samples.

Machine restart time
Reduce the time needed to restart your computer
Extra system resources consumed by processes andservices created by security software may delay shut downtime and the restart cycle of endpoint machines.
This metric measures the time taken to restart the machine
where an endpoint security product has been installed. Our
final result is measured in seconds (s) and calculated froman average of five (5) samples.

Daily network traffic

PassMark Software

Microsoft
Trend Micro
Kaspersky
Symantec
McAfeeSophos
ESET
0s

60 s

SophosSymantec
Trend Micro
ESET
Kaspersky
McAfee
70 s

75 s

80 s

85 s

90 s

0 MB

20 MB

40 MB

60 MB

80 MB

McAfeeKaspersky

All security solutions require the latest signatures andengine updates in order to provide managed networks thebest possible protection against malware. However, not all
security software is equal, with some products performing
incremental updates and others downloading much more.

Microsoft

Comparative Analysis
Edition 1

40 s

Microsoft

Minimize impact on the coporate network

This metric measures the total daily inbound and outboundtraffic as a result of security software engine and signature
updates to the repository on the server machine. Our final
result is measured in megabytes and is calculated as anaverage of sixteen (16) days of data.

20 s

Symantec
Trend Micro
ESET
SophosPage 10 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

ESET Smart Security Business Edition
Installation and Configuration

4/5

Installing ESET Remote Administrator Server and Console,
the management components of ESET Smart Security 4
Business Edition, was a relatively fast and streamlined
process. It took approximately an hour to complete theinstallation, updates and basic configuration of management
components.

Review Summary
Overall Rating
Installation & Configuration
Migration
Default Policies
Client Installation
Interface Design
Client & Policy Management
Remote Management
UpdatesCommon Use Cases
Effectiveness
Performance• Very little system impact on endpoint
machines. Fastest, best overall performance.
• Low traffic overhead for updates (3-4MB per
day on average)• Flexibly manage older versions and variantsof ESET client solutions from a single,
compatible console.
• Endpoints can be automatically sorted into
parametric groups as they join the network.
• Signature updates are transportable.
• Mirror server is not automatically createdduring a typical installation.
• Remote Administrator Console interface isfunctional, rather than aestheticallyappealing.
Comparative Analysis
Edition 1

The typical Administrator Server installation required very
little input from the user, seamlessly installing all needed
components including the embedded Access JET database.
ESET Remote Administrator Server also supports otherdatabases such as SQL Server, Oracle and MySQL, and can
automate the creation of an empty database for use where
an “Advanced” installation is selected. During installation,
administrators can set security passwords but this step isentirely optional and passwords may be configured at a laterstage. After installing the Remote Administrator Server,
installing the Remote Administrator Console took under ten
minutes to install.
The ESET Smart Security Business Edition Basic Setup guide
was extremely user-focused. The guide gave step-by-step
instructions for users from purchase and download to
installation and configuration, and even included estimatedtimes for completion of each phase. The guide also provided
useful tips for less technical users. ESET also provides acomprehensive User Guide which documents custom
installations and more advanced product functionality.

Migration from Previous Solutions

4.5/5

The removal of third party software prior to ESET client
deployment can be performed through the ESET Remote
Administrator Console via a push installation. Administrators
will first need to download the relevant third party removal
tool from a list of links provided by ESET, before creating acustom package for push installation.
Some versions of ESET clients are automatically upgraded by
the Remote Administrator. In all cases, the Remote
Administrator Console can manage most different types andversions of ESET security software, making it unnecessary to
remove or upgrade existing ESET client software.
Upgrading to a new version of ESET Remote Administrator iseasy. Administrators can simply install the latest version of
Remote Administrator and the existing database andsettings will be automatically migrated. For businesses withspecific upgrade or migration needs, ESET also provides a“Rip and Replace” service for North American customers.

Page 11 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Default Policies

4.5/5

Default policies are created during the Remote
Administrator installation and generally provide agood balance for most deployments.
The Basic Setup guide gives some starting points toassist in moving away from default settings, such as
the disabling of nonessential notifications forendpoint machines.

Client Install

4/5

PassMark Software

Interface Design

4/5

The Remote Administrator console is designed toappear similar in format to many familiar Windows
applications. Unfortunately, this means the console
interface is a bit bland and grey, with an emphasis
on functionality rather than aesthetics.
Major functions can be accessed from categories oftabs or from the large menu buttons across the topof the interface. Navigating from the interface wasextremely responsive, taking a fraction of a second
to navigate between windows and menus.
An endpoint status dashboard can be viewed fromthe Clients tab, with conditionally formatted panels
providing a quick indication about warnings, errors
or alerts for each endpoint machine.

Client and Policy Management

Screenshot 1: ESET Smart Security 4 Business Edition (Client)

The Basic Setup guide provides a checklist of items
to confirm prior to installation. In our case, the only
configuration change needed to start remote
installation was the enabling of the Remote Registry
service on the endpoint machine.

5/5

Endpoint machines can be arranged into groups
manually or dynamically added to user-definedparametric groups when certain conditions are met.
Parametric groups are extremely powerful,automatically assigning policies to clients as they
connect to servers, or for filtering and reporting
under certain conditions. Businesses with existinggroup infrastructure, such as Active Directory, canalso make use of the Policy Rules Wizard to createuser-defined rules and automate the mapping ofpolicies to existing groups and objects.

After pre-requisites are met, administrators canbrowse to the Remote Install tab and create an ESET
Security Products installation package. Package
creation is reasonably fast, taking around fiveminutes to configure and create by the management
server. Once correctly set up, a single package canbe used to remotely install to both 32-bit and 64-bit
endpoint systems.
Administrators can also use Remote Administrator
to flexibly deploy any custom install package, even
non-ESET applications, as long as the installer is a.msi file.
Target systems can be discovered via NetBIOS, IP
search or through existing Active Directory
infrastructure. Once initiated, the client module tookroughly ten minutes to install and update on theendpoint machine. No machine restart was required
for functional protection.
For larger deployments, client installation can be
performed through Windows Group Policy
Management. This procedure is well-documented in
the User Guide.
Comparative Analysis
Edition 1

Screenshot 2: ESET Remote Administrator Console (Server)

The propagation of policy changes is not immediate,
but clients will update and enforce policies on thenext connection to the server. Should an
administrator need to revert to default policies, they
can simply click on the “Default” button from thepolicy manager.
The Remote Administrator console is highly
compatible with most variants and legacy versions of
ESET security products, all of which can be managed
Page 12 of 36
15 June 2010

Fast and Effective Endpoint Security for Businesssimultaneously from a single platform. The flexibilityof the Remote Administrator Console caters well toscenarios where client upgrades may not be
possible.
All policies can be imported or exported in an XML
format for backup or duplication purposes.

Remote Management

4.5/5

Where installed on a terminal, the Remote
Administrator Console (RAC) allows for fully
functional remote management of server modules.
No additional configuration from the management
server is required for remote access. If the default
settings are selected for the management server
installation, accessing the management server will
not require a password. ESET strongly recommends
that users set a secure password in their installation
guides.

Updates

4/5

Basic server configuration included the setup of themirror server from which endpoint machines will
retrieve signature updates. While the process was
simple, well-documented and requiring only a few
minutes, other products we tested automated thecreation of an update repository during the
installation process.
Administrators are initially required to enter theirusername and password details in order to accessthe ESET Update Servers. These details are
remembered by the management server for future
repository updates.

PassMark Software
All signature files downloaded by ESET Smart
Security are transportable and able to be manually
moved between machines where required. Thisfeature is advantageous for administrators managing
machines on isolated networks.

Effectiveness

5/5

AV Comparatives awarded ESET NOD32 Antivirus(v4.0.474.0) the ADVANCED+ rank in their latest OnDemand Comparative, citing very few false positives
and a total detection rate of 97.7%.
The most recent Summary Report (2009) from AV
Comparatives noted that ESET has high detection
rates, a very good heuristics engine and low system
impact. Overall, this report awarded the ESET engine
third place out of 16 products.
Impressively, ESET is the only security vendor tohave an unbroken record of ADVANCED+ rankings in
the Retrospective/Proactive Test from AV
Comparatives. ESET has been awarded a total of 13
ADVANCED+ rankings since AV Comparatives began
testing in February 2004, which is the maximum
amount of ratings possible out of 13 reports and themost highly awarded vendor in this category.
Overall, ESET products have had a total of 62
successes with only three misses since their
inclusion in VB100 testing. Its flagship product wastested since May 1998 and has the most consecutive
awards in VB100 testing, with the highest successrate of 97%.

Common Use Cases
Task

5/5
Ease of Use

Details

Conducting an ‘on-demand’scan of selected endpoints

SimpleThe on-demand scan is one of several preset tasks in the Remote
Administrator Console. It can be accessed from the Clients tab by
selecting New Task.

Creating and viewing amalware report ‘on-demand’

SimpleMalware reports are accessed from under the Reports tab, and can becustomized with filters for target clients or threats. By default, an HTML
report is generated.

Creating, assigning anddeploying a new policy

SimpleThe Policy Rules wizard walks administrators through policy creation. Thiscan be accessed by right-clicking on the Policies heading.

Comparative Analysis
Edition 1

Page 13 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Kaspersky Business Space Security
Installation and Configuration

3.5/5

The typical installation, configuration and update process
for management components of Kaspersky Business Space
Security took roughly two to three hours. Within a few
screens of initiating the Administration Kit installer, it
detected that Microsoft .NET Framework 2.0 was notinstalled and prompted to install it as a pre-requisite.
Agreeing to install the missing component launched the.NET Framework installer and closed the Kaspersky
Administration Kit wizard. After installing .NET, theAdministration Kit installer needed to be manually restarted
for the installation process to continue.

Review Summary
Overall Rating
Installation & Configuration
Migration
Default Policies
Client Installation
Interface Design
Policy Management
Remote Management
UpdatesCommon Use Cases
Effectiveness
Performance• Low system impact on endpoint machines,
relatively good performance.
• Relatively high amounts of traffic generated
by server repository updates.
• User-focused interface provides good
feedback for task status and progression.
• Mobile policies offer flexible management
based on endpoint status.
• Initial server repository update may be slow
to perform.
• Relatively time-consuming to deploy clients
across the network.

Comparative Analysis
Edition 1

The installation process proceeded without any issue after
this minor disruption. Though it is not integrated into theinstaller, the Administration Kit installer wizard successfully
initiated the installation of SQL Server 2005 Express Edition
with default settings.
The Quick Start wizard appears straight after installation,
prompting for a key file to enable product evaluation. The
wizard created default tasks and policies before initiating
the update process for the Administration server.
Kaspersky offers four technical guides documenting the
installation and use of Business Space Security. The content
in the guides lacked definition with some content duplicatedacross guides. The need to cross-reference between guides
occasionally made it challenging to locate information on a
specific topic. Generally, we found that the Quick Start guide
was concise, but oversimplified the process in some areas.

Migration from Previous Solutions

4.5/5

Migration from version 6 or higher to the current version of
the Administration Kit is reasonably simple compared tosome products. Administrators need to back up the existing
database using the provided utility from Kaspersky, and all
previous settings and data from the previous version will be
migrated on installation. In this case, the installation will
simply upgrade the Administration Kit component without
the need to uninstall.
Client software migration is also simple, with existingversions or variants of Kaspersky client products
automatically uninstalled during client installation.
Administrators can choose to automatically remove third
party security or other incompatible software prior to
installing the Kaspersky client using the Deployment Wizard.

Page 14 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Default Policies

4/5

Two default policies profiles are created by the
Quick Start wizard after installation, one forWindows Servers and the other for Windows
Workstations. The default policies cater well to eachof the needs of both machine roles, with balanced
security settings and pre-defined alert levels fortypes of events.

PassMark Software

Interface Design

4.5/5

The Administration Kit interface is relatively
intuitive, with a directory tree on the left –providing
access to major areas of functionality and a display
pane which closely resembles a web-interface withorganized sections of information.
The dashboard is displayed after logging into the
Administration Kit. The overall health status of the
network is indicated by red, amber and green traffic
symbols over several security categories. Morespecific information about the status of individual
endpoints can be retrieved as a report. Reports are
flexible and able to be generated quickly, making useof pie and bar charts to increase readability.
The progress and status of deployed tasks is wellpresented, and generally appears to update in realtime. At the conclusion of tasks, status is clearlymarked with a large tick or cross symbol.

Client and Policy Management

Screenshot 3: Kaspersky Administration Kit (Server)

Client Install

4/5

The deployment wizard appears after the Quick Start
wizard, making it simple to get started deploying theclient software for even the least preparedindividual.

4/5

The Administration Kit provides some options forpartially automating policy management, forexample, policies can be configured to automatically
activate upon a “Virus Outbreak”. Unfortunately,
these policies will need to be manually reverted backto original policies by administrators when moresecure settings are no longer required.

Out of all tested products, Kaspersky provided thebest administrator feedback for the client
installation process. The Administration Kit displayed
the status of installation in ‘real time’, with a pie
chart showing the status of endpoints and a progressbar. A few minutes later, it was discovered that thereporting mechanism may not be entirely accurate,
as the status of the task took 25 minutes to movefrom the “Waiting for Application Initialization” tothe “Completed” status, despite client deployment
appearing fully functional on the endpoint machine.
Regardless of this potential flaw, it was a welcomechange to have feedback about client progress.

Protection policies for Windows workstations can bedelegated the “Mobile Policy” status. These policy
settings will be enforced when any computer
affected by the policy is disconnected from theserver.

The deployment wizard also provided the flexibility
to add non-Kaspersky applications for deployment,
and options for background installation andautomated machine restarts where required. The
installation of the Kaspersky client tookapproximately fifteen minutes and required theendpoint machine to be restarted to finalize theinstallation.

The Kaspersky Administration Kit console canconnect remotely to any Kaspersky Management
Servers on the network. Once connected, theconsole provides full functionality in managing
endpoints and policy.

Comparative Analysis
Edition 1

Generally, policies are propagated instantly to theclient. Administrators can also view the results andstatus of policy enforcement by viewing a policy’s
properties. Polices can be duplicated, imported orexported to a file in KLP format.

Remote Management

4.5/5

The console appears to be a customized version of
the Microsoft Management Console, offering serverconnection/disconnection functionality which is
unique to Kaspersky Administration Kit.

Page 15 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Updates

2.5/5

Initially downloading updates to the Server
Repository took more than two hours. Fortunately,
administrators are able to proceed throughinstallation while downloading server updates in the
background.

PassMark Software

Effectiveness

4.5/5

The most recent On-Demand Comparative from AV
Comparatives awarded Kaspersky Anti-virus(v9.0.0.736) the ADVANCED+ rank, noting a low
number of false positives and a total detection rateof 97.1%.
The latest Summary Report (2009) from AV
Comparatives noted that Kaspersky had very goodheuristics, good detection rates and many protection
features, awarding the engine second place out of
16 products.
Kaspersky
Anti-virus
further
attainedan
ADVANCED+
rankingintheJune
2010 Retrospective/ProActive Test report. Since
February 2004, Kaspersky has attained a total ofseven ADVANCED+ ratings, making it the second
most awarded vendor in this category.

Screenshot 4: Kaspersky Anti-Virus 6.0 (Client)

The Kaspersky client solution took approximately 20
minutes to update and, somewhat perplexingly,
opted to download the updates from the Kaspersky
update servers by default instead of the up-to-datemanagement server repository. Further investigation
revealed that the creation of an update task was
required for clients to retrieve update from thecentral repository.

Common Use Cases
Task

Kaspersky has passed the VB100 test 54 times andmissed 18 times, resulting in a 75% success rate forall products entered. The most recent failure was byKaspersky Antivirus in April 2010 due to a missed Inthe-Wild virus.

5/5Ease of Use

Details

Conducting an ‘on-demand’ scanof selected endpoints

SimpleA Virus Scan group task is automatically created by the Quick Start wizard
for use in on-demand scanning. This task is accessible from underManaged Computers heading.

Creating and viewing a malware
report ‘on-demand’

SimpleMalware reports are accessible from under the Reports and Notifications
heading. Reports can be generate “on-the-fly” and viewed from theAdministration Kit.

Creating, assigning anddeploying a new policy

SimpleThe New Policy wizard walks administrators through policy creation. Thiscan be accessed by right-clicking on the Policies heading.

Comparative Analysis
Edition 1

Page 16 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

McAfee Total Protection for Endpoint
Installation and Configuration

3.5/5

The total time needed to perform installation, updating andthe basic configuration of the McAfee Total Protection for
Endpoint and ePolicy Orchestrator was approximately twohours. Generally, the installation proceeded well andwithout much hassle, with the most time consuming aspect
proving to be task and policy configuration prior to client
deployment.

Review Summary
Overall Rating
Installation & Configuration
Migration
Default Policies
Client Installation
Interface Design
Policy Management
Remote Management
UpdatesCommon Use Cases
Effectiveness
Performance• Moderate system impact on endpoint
machines with average performance.
• Highest amount of traffic generated by
update client out of all tested products.
• Initial server task and policy configuration
is not straightforward and requires
patience or experience.
• A powerful and granular policy
management solution intended for use bylarge enterprises and more experienced
administrators.

Comparative Analysis
Edition 1

Conveniently, the ePolicy Orchestrator installer did not
require a key to install the evaluation version of thesoftware. The default installation automated the installationof Microsoft SQL 2005 Express including its pre-requisites,
MSXML Parser, Microsoft .NET Framework and MS VisualC++ 2005. All component installations were integrated andexecuted in order as part of the McAfee ePolicy Orchestrator
installer. This component of installation was a bit slow,
taking over half an hour to complete.
At the conclusion of installation, administrators can opt tolaunch ePolicy Orchestrator immediately from the final
prompt. We observed that the initial launch of ePolicy
Orchestrator from this prompt appeared to crash the web
application. Closing and restarting the browser instancemanually fixed the issue.
The Evaluation Guide provided accurate, step-by-step
instructions interspersed with technical notes. Conversely,
the Quick Overview Guide appeared to be a high levelmarketing document rather than the expected “Quick Start”
guide.

Migration from Previous Solutions

2/5

The procedure to upgrade or migrate to the latest version ofePolicy Orchestrator is not documented anywhere in official
guides or in the McAfee Corporate Knowledgebase.
The ePolicy Orchestrator product guide included a few
sentences about server tasks which appear to be used in
migrating historical data (such as Host IPS Policy Migration,
Event Migration), but provided no further information on
the migration process.
On the client-side, McAfee VirusScan Enterprise will checkfor and attempt to remove existing third party security
software on client machines by invoking software removal
through Windows. McAfee VirusScan will also automaticallyremove older versions and types of McAfee client solutions
prior to installation. While this migration functionality is
well-documented in the Evaluation guide, it is not referred
to or explained during the client deployment process via
ePolicy Orchestrator.

Page 17 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Default Policies

3.5/5

Some default policies created on installation are on
the more restrictive side of the security spectrum.
Until they are changed, these restrictive policies may
impact some business functions and endpoint users.
Fortunately, the Evaluation Guide suggests somereasonable policy changes which can be made to
improve default policies, such as making the default
e-mail policy less restrictive by turning off the e-mailport blocking rule for e-mail servers.
Other policies required further configuration toensure balanced security, such as adding a password
to secure the client console from tampering.

PassMark Software
The deployment of McAfee Agent and thesubsequent installation of modules took roughly
fifteen minutes to install and update.

Interface Design

3.5/5

Overall, the ePolicy Orchestrator web interface ispolished and well-designed. The product is ideal forinternational use, as the administrator may selectfrom one of ten language choices from the ePolicy
Orchestrator login page without needing to
download additional components. After logon,
administrators are presented with a dashboard
showing network status using charts and quick links
to relevant security categories. The dashboard canbe customized according to preferences using
premade monitors, or through user-defined
database queries.
The user interface is relatively responsive and stable
for a web interface, pausing for a mere second or
two while navigating between categories. In
contrast, the help files for ePolicy Orchestrator
appeared somewhat slow to browse or searchthrough.
Major functions can be accessed from the menu
button on the top left, with each category item
represented by an easily recognized, unique icon.

Screenshot 5: McAfee ePolicy Orchestrator (Server)

Client Install

3/5

Setting up the ePolicy Orchestrator to deploy
McAfee client software to endpoint machines is
initially complex, especially for administrators who
are unfamiliar with ePolicy Orchestrator. However,after the procedure has been correctly set up, theprocess becomes more or less automated forunmanaged machines.
Unmanaged endpoints must first be organized into
groups in the system tree. Administrators may opt toperform this task manually at first, or configure
criteria-based filtering to automatically sortdiscovered endpoints into groups.
The administrator must then create a custom
product deployment task which specifies products to
be deployed to endpoints. This task will be executedby the McAfee Agent after it is deployed to
endpoints and commences policy enforcement. With
the exception of Microsoft Forefront, this is the only
product that requires additional tasks or policies tobe created in order to install a client product. The
task then needs to be applied via policy to selectedgroups or endpoints.

Comparative Analysis
Edition 1

Client and Policy Management

5/5

While initial policy and task configuration may provedaunting for less experienced administrators,
McAfee Total Protection for Endpoint is able toautomate client sorting and policy mapping after
correct configuration.
ePolicy Orchestrator is able to sort discovered
endpoints into groups and subgroups in the systemtree using an algorithm. This algorithm is based on
criteria such as the IP address range and tags
defined by users. Policies control is granular, withadministrators able to control security behavior to afine level using a myriad of settings for eachcomponent. Compared to other reviewed products,
the depth of control over components is impressive
and likely to be more suitable for experienced
administrators managing larger deployments.
A feature that is unique to ePolicy Orchestrator is its
ability to open “issues” as a response to virus
security, monitoring or other events. Issues can thenbe manually or automatically assigned to different
remote users, who manage the issue by opening, reassigning or closing tickets.

Page 18 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Remote Management

4.5/5

ePolicy Orchestrator is easy to access remotely fromany terminal via the web interface. All management
functionality is available from the web interface
without additional configuration from the browser
or ActiveX downloads.

PassMark Software
amount of sense for very large deployments,
administrators managing smaller deployments
should seriously consider configuring more frequent
updates to ensure comprehensive protection.
The initial update of the McAfee client softwarefrom the ePolicy Orchestrator server was conductedmanually from the client machine. The process wasquick, taking roughly three minutes to complete.
Systems which are temporarily disconnected fromthe network are still able to run their update tasks,
but retrieve updates from the McAfee Update Server
rather than the ePolicy Orchestrator Server.

Effectiveness

Screenshot 6: McAfee Agent Monitor (Client)

Accessing the management server remotely issmooth and responsive, with the interface able tonavigate between functions or retrieve data
relatively quickly.

Updates

4/5

Repository and client updates are simple toconfigure and manage from ePolicy Orchestrator.
The “Update Master Repository” task is createdduring the installation of ePolicy Orchestrator, whichschedules the download of repository updates daily
by default. The client task “Update Client
Protection” is also set up during installation andchecks for downloads once a day withrandomization in place to minimize network impact.
While these default settings make a reasonable

Common Use Cases
Task

3.5/5

The most recent On-Demand Comparative from AV
Comparatives
awarded
McAfeeAntiVirus(v14.0.306) the ADVANCED rank, noting a highnumber of false positives but a total detection rateof 98.9%.
The latest Summary Report (2009) from AV
Comparatives reiterates these findings, noting very
high malware detection rates but unfortunately ahigh number of false positives.
The latest Retroactive/ProActive Test awarded
McAfee Antivirus with a STANDARD ranking. To
date, McAfee has been awarded the ADVANCED+
rating two times since February 2004.
McAfee has passed the VB100 test 48 times andmissed 21 times, attaining an overall 70% successrate. In three of these tests, McAfee has entered
two products and placed #6 out of 7 reviewed
products in this report.

3/5
Ease of Use

Details

Conducting an ‘on-demand’scan of selected endpoints

Moderate

Administrators can edit the default On-Demand Scan task to scan all
endpoints, or endpoints with certain criteria. The schedule must be set to“Run immediately”, and there isn’t much feedback on the status of thetask. After the scan task has been issued, administrators may wish tochange this task back to default settings.

Creating and viewing amalware report ‘on-demand’

Moderate

Reports are run from Queries, there are several dozen default queries
which may be edited or run. Reports are generated and viewed from
ePolicy Orchestrator.

Creating, assigning anddeploying a new policy

Moderate

Policies can be created from the Policy Catalog or through the System
Tree. The depth of configuration makes selecting desired configuration
settings and assigning a policy somewhat complex.

Comparative Analysis
Edition 1

Page 19 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Microsoft Forefront Client Security
Installation and Configuration

1/5

A typical, single-server installation of Microsoft Forefront
Client Security took more than a day to complete. The
installation was challenging and convoluted, with asuccessful installation relying upon deep knowledge of andexperience with a variety of Windows server components.
Overall, installing and configuring Microsoft Forefront was
the least user-friendly experience of all software reviewed.
As an exception to the relatively neat and self-contained
installers from other security vendors, the Microsoft
Forefront installer came in two parts with the main installer
provided on an ISO image, which needed to be transferred
to a CD prior to installation. The product update to Forefront
SP1 was provided as a separate executable.

Review Summary

Microsoft Forefront did not install any pre-requisites,
instead requiring all necessary components to beresearched, separately downloaded and installed inadvance. Microsoft Forefront had the largest number of prerequisites necessary for installation, some of which required
other pre-requisites to be downloaded and installed.

Overall Rating
Installation & Configuration
Migration
Default Policies
Client Installation
Interface Design
Policy Management
Remote Management--

UpdatesCommon Use Cases
Effectiveness
Performance• Relatively high system impact on endpoint
machines, below average performance.
• Relatively high amounts of traffic
generated by server updates.
• Convoluted installation, configuration andclient deployment may challenge evenexperienced administrators.
• Troubleshooting is time-consuming as aresult of independent error logging by eachcomponent.
• Limited ability to perform on-demand tasks
from the Management Server.

Comparative Analysis
Edition 1

It appeared to be possible to install Forefront without
necessary Windows components (such as the Windows
Server Update Services), resulting in a partially functional
installation.
Documentation for Forefront was provided online fromMicrosoft TechNet. Documentation was average and, atworst, unhelpful. Occasionally, support articles would omit
important steps or details in the installation andconfiguration process.

Migration from Previous Solutions

1/5

No documentation was provided from Microsoft TechNet inregards to migration from older versions of Forefront, orfrom third party security solutions. Some documentation isavailable describing the removal of an existing installation,or the migration process between different operating
systems or server topologies.
The limited, migration procedures which are documented
appear awkward, comprising of many steps of manual
configuration or application removal in a strict order,
depending on your current topology or configuration. Somesteps required the performing of low level system tasks,
such as manually observing or altering values in system
registry.

Page 20 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Default Policies

2/5

Default policies are not automatically generated by
the installation of Microsoft Forefront Client
Security.
A policy with default settings needs to be created bynavigating to the Policy Management tab andselecting “New” under Policy. If deployed, thedefault policy provides endpoint machines with realtime
malware protection, and scheduledvulnerability and malware scans.

Client Installation

1/5

Forefront Client Security supports remote
deployment to clients via several independent
Windows components and services. Administrators
are required to create and deploy policies to initiate
the installation of the client solution. Some of thecomponents involved in client installation,
configuration and updating include the WSUS
(Windows Server Update Services) or SMS (System
Management Server), the GPMC (Group Policy
Management Console), GPO (Group Policy Objects)
and the MOM (Microsoft Operations Manager).
The disadvantages of a deployment process which
relies on so many different components became
evident during our testing. Error reporting lacked
integration and clarity, with each component making
use of an independent error logging system andoutput location. Any single issue with client
deployment may involve a number of different
modules; documentation for which was scatteredover different areas of the Microsoft TechNet
Library.

PassMark Software
exceptions which would seem to defeat the purposeof such a design. For example, servers with anexisting MOM infrastructure cannot use the existing
infrastructure for Forefront but instead will need toobtain a customized version of MOM for thispurpose.

Interface Design

2.5/5

The interface for the Microsoft Client Security
console lacks many functions seen in other security
solutions, and appears to be very minimalistic. The
simplicity of the console does not assist
administrators in managing the underlying
complexity of components.
The interface uses the Microsoft Management
Console framework, with two tabs enabling accessto the Dashboard and Policy Management in thedisplay pane. The dashboard provides links to
Summary Reports and a list of issues affecting
endpoints.

Screenshot 8: Microsoft Forefront Security Console (Server)

Client and Policy Management

1.5/5

Most products have several options to manage clientgroups from within security solution, but Microsoft
Forefront Client Security relies solely on the use ofexisting Windows server infrastructure to manage
endpoints, such as through Group Policy
Management (GPM) and the Active Directorydomain. Endpoint machines must be organized into
groups or Organizational Units (OU) within the
Active Directory Domain before Forefront Client
Security policies can be deployed and enforced.
Screenshot 7: Microsoft Forefront Client Security (Client)

The installation procedure may be more familiar foran administrator who is extremely experienced with
Windows modules involved, but some of the“common components” have special rules andComparative Analysis
Edition 1

Forefront Client Security policy management
appears somewhat basic, with limited options forsecurity management. The apparent lack of granular
control is surprising given the product’s complexity
and intended large-scale corporate audience.

Page 21 of 36
15 June 2010

Fast and Effective Endpoint Security for Business
Deployed policies do not come into enforcement
until the Group Policy refresh takes place, which
may take hours. Administrators can manually force
an update by issuing a command through thecommand prompt on endpoint machines, but
Forefront Client Security does not provide thisfunctionality.

Remote Management

0/5

Microsoft Forefront Client Security does not appear
to support Remote Management. If remotemanagement is supported, it has not been
documented in Forefront Client Security help files or
the TechNet Library. Other products in the Microsoft
Forefront range such as Forefront Threat
Management Gateway document support remotemanagement.
It’s possible that Microsoft documentation authors
assumed that administrators would make use ofexisting remote desktop infrastructure to remotely
access their Forefront Client Security server. Whilemost modern servers will already have Remote
Desktop services implemented for other purposes,
the Microsoft TechNet library should at least
document the possibility of using it for remotemanagement.

Updates

2/5

In order to delegate the management server as acentral repository for scheduled Forefront Client
Security updates, the Windows Server Update
Services (WSUS) component must be installed.
However, since client synchronization is staggered
across the network, it is difficult to predict when
specific clients will update with the latest definitions
and upgrades. For this reason, the distribution ofvirus definition updates and scan engine updates to

Common Use Cases
Task

PassMark Software
client machines is difficult to perform on-demand
from the management server.
The client component of Forefront Client Security
has a button which users can click to manually“Update now”. However, clicking on this button
provides no functional response. This issue wasresolved by manually downloading the first updatefrom Microsoft Update. As the first update forms
part of the deployment process, this is a further
troubling issue caused by Microsoft Forefront’s
reliance on independent components.

Effectiveness

4/5

The most recent On-Demand Comparative from AV
Comparatives awarded Microsoft Business Essentials(v1.0.1611.0) the ADVANCED rank, noting very few
false positives and a total detection rate of 96.3%.
The latest Summary Report (2009) from AV
Comparatives notes that Microsoft has highproactive detection rates and good removal
capabilities.
The latest Retroactive/ProActive Test report gives
Microsoft Business Essentials the ADVANCED+
rating. To date, Microsoft has received three
ADVANCED+ awards, making it the third mostawarded vendor in our testing overall.
Microsoft's score in VB100 testing including all theirproducts is 17 successes and two misses giving theman 89% success rate, though it entered one test withtwo products. Out of all reviewed products,
Microsoft has achieved second place in VB100
testing.

3/5
Ease of Use

Details

Conducting an ‘on-demand’scan of selected endpoints

Moderate

Administrators can click the “Scan Now” button to conduct an ondemand scan. Only two options are provided for targets, either all
managed computers are scanned or a single target computer is scanned.
Administrators can, however, choose between Quick and Full scans.

Creating and viewing amalware report ‘on-demand’

SimpleAdministrators can create and view a malware report from thedashboard which provides a list of report links. The reports are generated
in HTML by default.

ComplexCreating, assigning and deploying a new policy can be accomplished fromthe “Policy Management” tab. Enforcement of the policy may or may notoccur at the scheduled interval, administrators may need to refer to logs
to ensure policy has deployed correctly.

Creating, assigning anddeploying a new policy

Comparative Analysis
Edition 1

Page 22 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Sophos Endpoint Security and Data Protection
Installation and Configuration

3.5/5

A ‘complete’ server installation of the management
components in Sophos Endpoint Security and Data
Protection took roughly two to three hours to complete.
While the installation itself was straightforward, updating
the server repository took a significant amount of time due
to unexplained timeouts from the Sophos update server.
After file extraction, the Enterprise Console installer
detected that the Microsoft .NET Framework was missing
and prompted for its installation as a pre-requisite. Clicking“Yes” activated the Microsoft installer for .NET Framework
while the Enterprise Console installer paused in the
background. This was also the case for the MS XML and SQL
Server Setup, with the Enterprise Console installer guiding
the installation process by initiating necessary installers. At
the conclusion of installation, Enterprise Console required
us to log out of Windows to complete the process.

Review Summary
Overall Rating
Installation & Configuration
Migration
Default Policies
Client Installation
Interface Design
Policy Management
Remote Management
UpdatesCommon Use Cases
Effectiveness
Performance• Moderate system impact on endpoint
machines. Good performance in some areas,poor performance in others.
• Migration process from legacy variants andthird party solutions is well-documented
and straightforward.
• Updating the server repository can beproblematic.
• Some default policies are visibly restrictive,
and may disrupt business functions if notchanged.
• Technical documentation is organized intonine separate technical guides.

Comparative Analysis
Edition 1

Sophos provides nine separate technical guides
documenting the installation and use of Endpoint Security
and Data Protection. While the advanced guides were
useful and comprehensive, the sheer number of guides
made referencing information beyond the brevity of
instructions in the Quick Startup Guide clumsy and difficult.

Migration from Previous Solutions

5/5

The migration process is very straightforward and welldocumented. Sophos offers two guides which document the
upgrade process; the Quick Upgrade Guide describes the
upgrade procedure from a single-server scenario while theAdvanced Guide covers more complex implementations.
In a typical installation, Sophos recommends backing up
database components prior to migrating from a supported
variant of Sophos. Installation packages for different
components requiring upgrades can be downloaded andactivated much like a clean installation. All previous
settings, computer groups and policies will be detected by
the installation wizard and remain unchanged in the new
installation.
Where selected during client deployment, the Sophos
Endpoint Security can automate the removal third party
security software where it is discovered on endpoints. By
default, the client solution uses a standard removal tool, but
a non-standard removal tool is also provided which
supports the removal of more third party software than thestandard tool. The non-standard tool requires user
configuration prior to use. The removal process is
documented in the Third Party Security Software Removal
Guide.
Page 23 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Default Policies

3.5/5

Where the firewall component is installed on target
machines, the default settings are visibly restrictivewith a heavy impact on network connectivity for
endpoints. Default firewall policies block all nonessential connections, which include most inboundand outbound connections. Thankfully, Sophos
advises administrators about the restrictiveness ofthe firewall policy in the Advanced Startup Guide.

PassMark Software
indicated that the endpoint required a restart for
updates to become effective.

Interface Design

3.5/5

Otherwise, default policies make sense and provide
reasonable protection for most managed networks.
Some of the more specialized security management
and compliance features are disabled by default,
such as data control, device control and application
control.
Screenshot 10: Sophos Enterprise Console (Server)

The Enterprise Console appears to have been
designed with practicality in mind. The dashboard isa consistent element of the interface, enabling theadministrator to monitor network health regardlessof the task being performed. The height of thedashboard can be adjusted where required.

Screenshot 9: Sophos Endpoint Security and Control (Client)

Client Install

3/5

Our client installation experience was indicative of
the problems associated with providing nine
separate technical guides for users. After referencing
the Quick Start Guide, we initially attempted to
deploy the client via network installation. Client
installation failed twice without a resulting error
message and we referred to the Troubleshooting
Guide without much luck. Finally, referring to the
Advanced Startup Guide revealed that enabling theRemote Registry service on the endpoint machine
was required to deploy over the network.
Unfortunately, this pre-requisite was not mentioned
in the Quick Start Guide, which is likely to be the firstplace administrators will look.
The Protect Computers wizard guided us through therest of the installation process, giving a choice ofcomponents to install on target endpoints. The
installation over the network was fast, taking lessthan five minutes for a silent installation. Afterinstallation, the client automatically updated
without manual intervention, after which the server
Comparative Analysis
Edition 1

The layout of the interface is straightforward. By
default, administrators can check the status of
security components of individual computers below
the dashboard, or switch this view by clicking“Update Managers” to show software subscriptions.
Group structure and policies are accessible to theleft, arranged into tree directories.
Despite the available mechanisms for monitoring,
there appeared to be no obvious way to check thestatus of on-demand scanning.

Client and Policy Management

3/5

Client and policy management using Sophos
Endpoint Security was easy to learn but ultimately
lacked the depth of policy automation or locationbased that some security products offered.
The only automated group management featureoffered by the Enterprise Console was the Download
Security Software wizard. When executed, thiswizard could import and replicate existing domain
infrastructure, automatically placing discovered
computers into groups. Otherwise, administrators
can find new computers via Active Directory,
network or IP range.
Without importing existing Active Directory
structure, there is currently no method to automate
the filtering of discovered endpoints into groups
Page 24 of 36
15 June 2010

Fast and Effective Endpoint Security for Businesswith defined policies. Unassigned computers needbe manually dragged and dropped into groups.

Remote Management

4/5

Remotely managing the Sophos server required
some additional configuration through the existing
Enterprise Console on the management server. The
administrator needs to enable remote access on an
account by adding it to the correct administrative
groups.
After this, administrators should select a “custom”
installation of the Management Console at theterminal from which they wish to remotely accessthe Sophos server. If the account is configured
correctly with database access, the remote versionof the Enterprise Console should have remote accessto all management features.

Updates

1.5/5

The initial update of the server repository as part of
installation was problematic.
The Download Security software wizard dialog
popped up after the initial launch of the Enterprise
Console, prompting us to enter the key andpassword provided by Sophos to access thedownload server. Initially, the wizard was unable toconnect to the Sophos update server after repeatedattempts citing server timeouts. Based on feedbackin the Sophos forum, we attempted to repair theUpdate Manager component using the installer
which appeared to resolve the issue.
The initial update to the server repository was quite
large, taking over an hour and 200MB of bandwidth.

Common Use Cases

PassMark Software
At some point, the Download Security software
prompt had reported an ambiguous error that it was
unable to retrieve the update. The error message
suggested that users wait for the next automatic
download to complete the download. After closing
the error message, the “Run the Download Security
Software Wizard” item had vanished from theActions menu and there appeared to be no way tomanually retry the update.
Conversely, client updates were automaticallydownloaded after client installation and applied
without issue.

Effectiveness

3.5/5

The most recent On-Demand Comparative from AV
Comparatives awarded Sophos Antivirus (v9.03) the
ADVANCED rank, noting few false positives and total
detection rate of 93.7%.
The latest Summary Report (2009) from AV
Comparatives noted that Sophos had relatively good
detection rates. We believe that, compared to otherproducts, a detection rate of 93.7% is relatively low
and mean that roughly one in twenty viruses may
not be detected.
The Retroactive/ProActive Test in June 2010
awarded Sophos Anti-Virus with an ADVANCED
rating. Sophos has not achieved an ADVANCED+
rating in this category to date.
Sophos has passed the VB100 test 52 times andmissed16
times,
attaininganoverall
77% success rate. Since the first test in January 1998,
Sophos has only entered one product into VB100
testing.

3.5/5

Task

Ease of Use

Details

Conducting an ‘on-demand’scan of selected endpoints

Moderate

Administrators can right-click on target endpoints or groups to initiate afull system scan. The console provides no immediate feedback on
performing this action. No choices on the depth of scanning.

Creating and viewing amalware report ‘on-demand’
on-demand

SimpleClicking on the Reports action opens the Report Manager. To create areport on-demand, users can choose a report and select “Run”

Creating, assigning anddeploying a new policy

SimpleCreate a new policy by right-clicking on a security category and choosing“New”. Double-click the new policy to activate the Policy Wizard andselect new settings.

Comparative Analysis
Edition 1

Page 25 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Symantec Endpoint Protection
Installation and Configuration

3.5/5

Installing and configuring the management components of
Symantec Endpoint Protection took roughly two hours fromstart to finish. The Symantec Endpoint Protection Manager
installer required the server to have the IIS World Wide
Web Publishing Service (WS3SVC) enabled, which wasadded as a Server Role from Windows Server Management.
We then manually restarted the Endpoint Protection
Manager installer, and it guided us through the rest of theprocess without issue. Symantec Endpoint Protection did
not even require the entering of a registration key tocomplete the installation. After installation, theManagement Server Configuration Wizard created and set
up an embedded database via an integrated installer.

Review Summary
Overall Summary
Installation & Configuration
Migration
Default Policies
Client Installation
Interface Design
Policy Management
Remote Management
UpdatesCommon Use Cases
Effectiveness
Performance• Below average system impact on endpoint
machines.
• Moderate amounts of traffic generated byserver repository updates.
• Malware detection rates were very highly
rated by third party testing firms.
• Flexible policy management anddeployment of endpoint clients andlocations.
• Retrieving and distributing signature
updates to endpoints is easy to setup.
• Manager Console user interface issomewhat sluggish.
• No support is provided for migrating fromthird party products.
Comparative Analysis
Edition 1

After basic installation, administrators may choose to run
the Migration and Deployment wizard or proceed directly
to the Manager Console.
Overall, the documentation for the installation procedure
was well-written, providing accurate step-by-step
instructions regarding user options in each phase of
installation, along with recommendations for typicalinstallations.

Migration from Previous Solutions

3.5/5

Migration from other versions or variants of Symantec
products is reasonably well supported. Some of this is
documented on the Symantec support website in high-level
through FAQs and online presentations. The website gives
administrators quick access to information about migration
product paths, server “before and after” hierarchy
snapshots, post and pre-migration tasks and some
information about pricing.
A low level walkthrough of the migration process for theserver is available in the Installation Guide. For compatible
variants of older Symantec server software, the installation
will detect the existing and export all existing settings
during the installation of the new product version. It is notnecessary to remove existing server components in thissituation. Once the server has been upgraded, the server
will automatically upgrade the endpoint machines using the
AutoUpgrade function.
For organizations using an older version of Symantec
Antivirus, existing groups and policies can be migratedusing the Migration and Deployment wizard.
Unfortunately, no documentation or support is available for
upgrading from third party products.
Page 26 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Default Policies

3.5/5

Most protection features are enabled by default
policies. Default protection of endpoints is balanced
and intended to be ‘silent’ for endpoint users, forexample, quarantined items on client machines will
be repaired and restored where possible.
Default firewall policies are not overtly restrictive,
with endpoint users able to access the internet andinternal network as usual. The only policy areas
where default policies are not created on installation
is for Application and Device Control and Centralized
Exceptions.
Administrators can choose from three preset policy
profiles for Antivirus and Antispyware protection:balanced protection, high-performance protection
or high-security protection. Balanced protection isenabled by default.
For new policies to be deployed to endpoint
machines, the administrator must wait for the next
network ‘refresh’ cycle, which pushes new policies
to endpoints. We discovered that the default autorefresh setting in Console Manager is set to “Never”
by default. Until an administrator changes thisdefault setting, new policies will not be downloaded
and enforced on endpoint machines.

Client Install

3/5

The Migration and Deployment wizard can create an
installation package to deploy across the network,
giving administrators a choice between either an‘unattended’ or a silent installation.

PassMark Software
It takes under three minutes to create theinstallation package prior to deployment. After
package creation, the Push Deployment Wizard
allows discovery of unmanaged endpoints through
NetBIOS, existing group structures such as Active
Directory, IP address or host name. Unmanaged
computers can also be discovered through theManager Console by IP address or host name.
Some instability was encountered during the
deployment of the client to the endpoint machine,where the remote deployment application(ClientRemote.exe)
quietlycrashedwithout
providing any feedback to the administrator via the
Troubleshooter log. This issue was resolved byrestarting the management server.
Once deployed, the client package took roughly four
minutes, requiring a machine restart to finalize theinstallation.

Interface Design

3.5/5

The Symantec Endpoint Protection console is letdown by its sluggishness and slow response times.
Generally, it is slow to navigate around the interface,taking a few seconds to retrieve data or to move
between categories and perform tasks. The initialload of a category is always the slowest, withsubsequent accesses improving responsiveness. We
believe that, despite appearing to be a Windows
application, one possible explanation for theManager Console’s poor data retrieval speed is that
it is using somewhat slower web technologies.
The tab categories are succinct and task headings
are descriptive or task-based, making it relatively
intuitive to perform common administrative tasks. A
summarized dashboard is visible from the Home tab,or administrators can look under the Monitors tab
for pie charts about specific items.
As a minor issue, the task wizards, while helpful instreamlining migration, client deployment andbackups, are not accessible from the Manager
Console directly.

Screenshot 11: Symantec Endpoint Protection (Client)

The names used for these two network deployment
options can be confusing. When an ‘unattended’
installation is selected, the client software installer isvisible on the endpoint machine, showing
installation progress before prompting the endpointuser to restart the machine. Conversely, a silent
installation progresses in the background with norestart prompt or end-user input required.
Comparative Analysis
Edition 1

Client and Policy Management

4/5

The Manager Console provides relatively granular
policy management options for endpoint machines
and locations. Shared policies are standard and areable to be applied to any group or physical location.
Conversely, non-shared policies are more specialized
and apply only to a group when it is in a specificlocation.
Groups can be imported from existing organizational
structures, such as LDAP or an Active Directory
Page 27 of 36
15 June 2010

Fast and Effective Endpoint Security for Businessserver. Otherwise groups must be manually added
to the client tree prior to client installation.
Symantec Endpoint Protection Manager Console
gives administrators flexibility between push or pullmodes for policy deployment. Different modes canbe set for different locations. This feature is great forallowing administrators to adapt the deployment ofpolicies to network conditions and the size of
deployment.

PassMark Software

Updates

5/5

Downloading and distributing updates to clients iseffortless in Endpoint Protection, requiring little
setup.
The management server is designated as the central
repository during installation and clients retrieve
updates from the management server by default. In
some cases, the default setting is ideal and requires
no further configuration from the administrator. For
larger deployments, administrators can designate
other servers as Group Update Providers which act
as an intermediary in retrieving updates from the
management server to distribute to nodes.
Updating the Symantec client solution from the
management server was fast compared to othersolutions.

Effectiveness

Screenshot 12: Symantec Manager Console (Server)

Remote Management

4.5/5

The management server can be remotely accessedusing the Symantec Protection Manager Web Accessinterface. With the correct login and password,
access to the management server remotely requires
no additional configuration of user accounts or
installation of browser components.
The Web Access remote console is fully functional
and looks identical to the Manager Console on the
management server. Similar to the Manager
Console, navigating between categories andheadings is somewhat slow.

Common Use Cases
Task

5/5

The most recent On-Demand Comparative from AV
Comparatives
awardedNortonAntivirus(v17.5.0.127) the ADVANCED+ rank, noting few false
positives and a total detection rate of 98.6%.
The latest Summary Report (2009) from AV
Comparatives noted that Symantec had very highdetection rates, low system impact and was easy touse. In this report, Symantec were awarded firstplace out of 16 products.
The Retroactive/ProActive Test in June 2010
awarded Norton Anti-Virus with an ADVANCED
rating. To date, Symantec has not yet received an
ADVANCED+ rating in this category.
Overall, Symantec has passed the VB100 test 54
times and missed seven times. With an 88% successrate, Symantec has the third best overall rate ofsuccess in VB100 testing.

4/5
Ease of Use

Details

Conducting an ‘on-demand’scan of selected endpoints

SimpleFrom the Clients tab, right-click on the group or target computer andactivate “Run Command on Group > Scan”. Manager Console offers threescan options: Active, full or custom. Command process can be viewed
from Command Status.

Creating and viewing amalware report ‘on-demand’

Moderate

From the Reports tab, administrators can select the type of report andthe period of time for reporting. Reports are generated in HTML format
by default, and take some time to generate on-demand.

Creating, assigning anddeploying a new policy

SimpleFrom the Policies tab, choose the relevant security category and rightclick to “Add” a new policy. The new policy can then be assigned to alocation or group.

Comparative Analysis
Edition 1

Page 28 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Trend Micro Worry Free Business Security
Installation and Configuration

3.5/5

A typical “evaluation version” Installation of Trend Micro
Worry-Free Business Security management components
took roughly two hours to complete.
While the installation progressed without issue, theinstaller required a moderate amount of input from theuser about items which could be either excluded from theinstallation process or moved to the configuration phase,for example, prompting the user about conducting a prescan of the machine before installation and asking about
the user’s professional industry on the telemetric prompt
screen. One or two installation screens lacked detail, forinstance, the “Select Components” prompt asked thatcomponents be selected for installation, but provided no
explanation about the functionality of components.

Review Summary
Overall Summary
Installation & Configuration
Migration
Default Policies
Client Installation
Interface Design
Policy Management
Remote Management
UpdatesCommon Use Cases
Effectiveness
Performance• Relatively high system impact on endpoint
machines, below average performance.
• Relatively low amounts of traffic generated
by server repository updates.
• Migration from previous solutions is well
supported.
• Somewhat simplistic client and policy
management which may be better suited forsmaller organizations.
• Outbreak defense manages possible virus
outbreaks on a network without
intervention.
• Graphical glitches detract from an otherwise
streamline interface.
Comparative Analysis
Edition 1

The Worry-Free Business Security installer automaticallyinstalled the .NET framework and Apache Web Server as
an integrated part of the main installation. The evaluation
version of Worry-Free also did not require a license key to
be entered in order to proceed with installation.
While Trend Micro does not provide a “Quick Start” guide,
the Installation Guide has all the information required,with step by step guides for all types of installations.

Migration from Previous Solutions

4.5/5

The migration process is reasonably well supported byTrend Micro Worry Free Business Security.
Only some older versions of Trend Micro business security
products are supported for upgrade to Worry-FreeBusiness Security. For supported product variants,
upgrading is as simple as running the standard installationand identifying the existing Security Server when
prompted.
Client-side migration is supported for most variants ofTrend Micro products as well as most third party antivirus
products. Where a Trend Micro product is detected by theclient deployment, endpoints will be upgraded to the
Trend Micro Client/Server Security Agent. Existing third
party antivirus software will be removed by Worry-FreeBusiness Security prior to replacement with the Trend
Micro client.

Page 29 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

Default Policies

4/5

Worry-Free Business Security comes with default
policies which are well balanced for standard usage.
Depending on whether a typical installation was
selected or an upgrade was implemented, default
policies may vary slightly.
The default setting of policy options for security
components differ depending on a group’s businessrole or physical location, for example, desktop
groups will have Behavior Monitoring enabled while
Server groups will have this protection disabled.
The firewall and POP3 Mail Scanning is disabled bydefault. Where enabled, the Firewall comes withdefault settings which take into account common
scenarios encountered by endpoint users, such as
the need to access the web.

Client Install

3.5/5

Deployment of the Client/Server Security Agent
through the network to endpoints was relatively
easy, but documentation for the process wasn’t
entirely accurate which made it more challenging totroubleshoot. The Administrator’s Guide mentions
the need to enable the Remote Registry Service on
Windows Vista clients prior to proceeding, but we
discovered that it was also necessary to enable thisservice manually on Windows 7 machines.

PassMark Software
log onto the domain using a Login script created by
Worry-Free Business Security. Other methods relyon the participation of end users, for example, by
instructing users to download and install the Agent
package from the Worry-Free Business Security
internal site created during setup or from an e-maillink.

Interface Design

2.5/5

As a web interface, the Trend Micro Worry-FreeBusiness Security console has a streamlined“webpage-like” feel. Unfortunately, we experienced
graphical glitches while interacting with someelements of the console, which detracted from itsotherwise slick and responsive design. The worst
example of this was when we attempted toconfigure Security Settings; after clicking on a few
items, the interface was littered with broken
navigation tabs and drop-down graphics.
Another disadvantage of the Worry-Free Business
Security web interface was the requirement toinstall an ActiveX control named “AtxEnc” fromTrend Micro before the console could be accessed.
The requirement to install this ActiveX control isn’t
documented explicitly by Trend Micro, though it ismentioned that the Web Console uses ActiveX as aninternet technology.
Management functionality is intuitively accessible
from the toolbar above the display pane, withcategories dropping down to reveal more specificoptions. The “Live Status” dashboard is shown as themain page from logon and displays network status
for systems and threats. Dashboard status isn’t inreal-time, but administrators are free to manuallyrefresh the dashboard if they require information
on-demand.

Client and Policy Management

Screenshot 13: Trend Micro Client/Server Security Agent (Client)

Target machines are first manually added from theInstallation from Security Settings. Endpoint
machines can be discovered through NetBIOS, Active
Directory or by searching for a computer name. Our
deployment through the network took under fiveminutes to complete. Worry-Free Business Security
has a number of methods of client deployment,
making it reasonably flexible to an organization’s
needs. Automated deployment methods include
agent installation for unprotected computers that
Comparative Analysis
Edition 1

3.5/5

Worry-Free Business Security has a reasonable
depth of security configuration options, but is letdown by ‘group only’ policy assignments, theinability to import existing node infrastructure andthe lack of capacity to automate sorting of endpoints
into relevant groups.
However, Worry-Free Business Security does have aunique client management feature called ‘Outbreak
Defense’. Outbreak Defense acts as an automatedresponse to a world-wide virus outbreak detected byTrend Micro. Where an outbreak occurs, Worry-FreeBusiness Security takes pre-emptive measures toprevent the network from being infected by the
specific malware threat. Preventative steps may
include raising security by blocking shared file andPage 30 of 36
15 June 2010

Fast and Effective Endpoint Security for Businessfolder access, blocking ports, denying write access orassessing endpoint clients for vulnerabilities.

PassMark Software

Updates

4.5/5

Updating Worry-Free Business Security is simple.
Installing the management software configures theserver to check and download updates from theTrend Micro ActiveUpdate Server every hour, and toact as a central repository for retrieval of updates byendpoints.
Updating clients the first time after installation issimilarly simple. By default, a Client/Server Agent
automatically updates virus definitions on
installation, prior to the first launch. This ensuresthat a client is fully protected from the first launch.

Screenshot 14: Trend Micro Worry-Free Business Security (Server)

Each security component in Worry-Free Business
Security has a large amount of configuration options
available. There is also some policy flexibility in theform of “Out of Office” policies for the Web
Reputation and TrendSecure modules. Where
configured correctly, different policies will be
enforced for these modules where endpoints havebeen disconnected from the server.

Remote Management

3.5/5

Remotely accessing the Worry-Free Business
Security server is simple. The management server
can be remotely accessed using the web interface
with no further configuration required from the
management server. The interface performs
moderately well across the network, with allfunctionality enabled and reasonably fast response
times between categories and functions.
As previously noted, an ActiveX add-on is required toaccess the web console. The need to install thisActiveX control for access needs to be better
documented. No other security solution which weevaluated required an additional browser
installation for access.

The Worry-Free Business Security Console alsoenables administrators to synchronize clients to thelatest update, or rollback a patch by one version ondemand.

Effectiveness

2/5

The most recent On-Demand Comparative from AV
Comparatives awarded Trend Micro Antivirus(v17.50.1366.0) the TESTED rank, noting a great
number of false positives and a total detection rateof 90.7%. This total detection rate was the lowest
compared to all other evaluated vendors andproducts.
The latest Summary Report (2009) from AV
Comparatives did not include results for any Trend
Micro products.
The Retroactive/ProActive Test in June 2010
awarded Sophos Anti-Virus with a STANDARD rating.
Since the introduction of this test in February 2004,
Sophos has not yet achieved an ADVANCED+ ratingin this category.
Trend Micro has passed the VB100 test 16 times andmissed 11 times, attaining an overall 59% successrate. Trend Micro failed the last three tests they
participated in and since April 2008, have withdrawn
their products from participation in VB100 tests.

Common Use Cases
Task

3/5
Ease of Use

Details

Executing an on-demand scanfor endpoints

SimpleOn-demand scans can be conducted by clicking on the Scan category andselecting a Manual Scan. While administrators can only select groups ofcomputers to scan, they can choose from a range of scanning options andactions.

Creating and viewing amalware report ‘on-demand’

Moderate

A one-time report can be generated from the Reports menu item.
Unfortunately, reports can only be generated in the PDF format.

Creating, assigning anddeploying a new policy

Moderate

Editing existing policy for a group proves to be much simpler thanCreating, assigning and deploying a new policy.

Comparative Analysis
Edition 1

Page 31 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Disclaimer and DisclosureThis report covers selected Enterprise Security products that were available at the time of testing. Version
numbers of software reviewed within this document are provided in the “Product Versions Tested” section ofthis report. With the exception of the performance test results and product effectiveness which are quantitative
measures, the product reviews herein represent PassMark Software’s subjective opinions and experiences ininstalling, configuring or operating each product.

Disclaimer of Liability
While every effort has been made to ensure that the information presented in this report is accurate, PassMark
Software Pty Ltd assumes no responsibility for errors, omissions, or out-of-date information and shall not beliable in any manner whatsoever for direct, indirect, incidental, consequential, or punitive damages resulting
from the availability of, use of, access of, or inability to use this information.

Disclosure
The production of this report was funded by ESET LLC.

Trademarks
All trademarks are the property of their respective owners.

Contact Details
PassMark Software Pty Ltd
Suite 202, Level 2
35 Buckingham St.
Surry Hills, 2010
Sydney, Australia
Phone + 61 (2) 9690 0444
Fax
+ 61 (2) 9690 0445
Web
www.passmark.com

Comparative Analysis
Edition 1

Page 32 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Appendix A – Performance Methodology
Client Image Creation
After installation of the client image, the following Windows services/features were disabled to minimize theimpact of Windows background activity and ensure the consistency of test results:• Windows SuperFetch – Disabled for all tests. Re-enabled to allow boot time optimization functionality, andfor boot and restart time tests.
• Microsoft Search Indexer – Disabled for all tests.
• Windows Sidebar – Disabled for all tests.
• Windows Defender – Disabled for all tests.
• Windows Updates – Initially, Windows updates will be installed. Prior to baseline tests, Windows Updates
will be disabled and remain for the duration of testing.
After installation of each security product on the endpoint machine, variability of client activity has been limited
by disabling client product updates and automatic or scheduled scans.
Microsoft Word will also be installed on endpoint machines prior to testing, in order to facilitate the Word
Document Launch Time test

Install size (Endpoint)
Description

This metric measured the total additional disk space consumed by the endpoint client component
after installation via the network.

Test Tool(s)OSForensics (formerly called OSCheck)
By: PassMark Software
New version of software currently being developed
by PassMark Software. Currently publicallyavailable as OSCheck at the PassMark Website:
http://www.passmark.com/products/oscheck.htm
OSCheck is used to capture and compare
signatures of disks, folders and files. Filecomparisons can be made between two signatures
to determine newly created files, modified ordeleted files.

Methodology

•
•
•
•
•
•
•

ResultComparative Analysis
Edition 1

Screenshot 15: PassMark OSForensics

OSForensics was used to take an initial disk signature prior to each installation to establish abaseline value.
The client component of the security product was installed on the endpoint machine.
Where possible, the package was installed via the network, i.e. deployed from the server
component through the network to the client machine. Deploying through the network more
adequately reflects a user’s ‘real’ installation experience.
Where applicable, PassMark has selected default options during client installation.
After installation, PassMark will initiate a manual update for the client software and thenrestart the endpoint machine to clear away temporary files.
After the manual update and machine restart, a second disk signature was taken.
Disk signatures were compared using OSForensics to discover the size and amount of files
added or modified during client installation.

The final result was calculated as the total size of additional files and modified files (files that werelarger) after client installation, manual update and machine restart.

Page 33 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Memory usage commit charge (Endpoint)
Description

This metric measured the total additional memory use consumed by the endpoint machine during aperiod of system idle where an endpoint security product has been installed.

Test Tool(s)SysinfoAvg
by: PassMark Software
A command-line utility developed by PassMark software which retrieved and logged memorycommit charge values (e.g. Total commit charge, Peak commit charge, etc) from Windows.

Methodology•••

ResultPrior to testing, the endpoint machine was left idle for five minutes to minimize the impact ofstartup or background processes.
SysinfoAvg is configured to retrieve and log the system memory commit charge every 15
seconds for five minutes, for a total of 20 samples per test.
This test will be run on the first and fifth test cycle, for a total of two runs and 40 test samples.

The final result was measured in megabytes (MB) and calculated as an average of 40 samples. Thisaverage was subtracted from the baseline to obtain the total amount of additional memory
consumed by the security solution.

Word document launch and restart time (Endpoint)
Description

This metric measured the total time taken to open or re-open a large, mixed media Microsoft Word
2007 document, where an endpoint security product has been installed.

Test Tool(s)AppTimerby: PassMark Software
A test utility developed by PassMark software which measured and logged the time taken to launch
an application in Windows. AppTimer can be downloaded from:http://www.passmark.com/products/apptimer.htm

Methodology

•
•
•
•
•
•
•

Test File(s)ResultOne 980 KB Microsoft Office Word Document in DOCX format.
Our final results were measured in milliseconds (ms), and calculated from an average of ten (10)samples. This test provides results for two separate metrics:••

Comparative Analysis
Edition 1

AppTimer was configured to launch a ‘large’ word document.
We have defined ‘large’ as a mixed media Word document roughly 980 KB in size.
The document was in *.docx format and was launched in Microsoft Word 2007
(v1.2.0.4518.1014).
At the start of each test cycle, the machine was left idle for five minutes to minimize theimpact of startup or background processes.
After five minutes, AppTimer was used to launch and close the word document three times.
The test was performed over five cycles with a machine restart between each cycle to limit
possible caching behavior by the Windows operating system. Three test results were obtained
per cycle, for a total of 15 test results.
Out of fifteen results:• Five results represent the time taken to launch the word document after restarting themachine. These results tend to be slower because Windows has not yet cached thest th thapplication/document being launched. Out of fifteen results, these are the 1 , 4 , 7 ,thth
10 and 13 results; and• Ten results represent the time taken to launch the word document by restarting thend rdapplication without restarting the machine. Out of fifteen results, these are the 2 , 3 ,th th th th
th
th
th
th
5 , 6 , 8 , 9 , 11 , 12 , 14 and 15 results.

Word document launch time: The final result is calculated as an average of five results (out ofst th ththth
fifteen), specifically the 1 , 4 , 7 , 10 and 13 results.
Word document restart time: The final result is calculated as an average of ten results (out ofnd rd th th th th
th
th
th
th
fifteen), specifically the 2 , 3 , 5 , 6 , 8 , 9 , 11 , 12 , 14 and 15 results.

Page 34 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

File copy time for small files (Endpoint)
Description

This metric measured the total time taken to copy a set of small files between directories, wherean endpoint security product was installed.

Test Tool(s)CommandTimer
by: PassMark Software
A command line utility developed by PassMark Software which measured and logged the time
taken to perform a task in a command prompt.

Methodology•••

Test File(s)At the start of each test cycle, the machine was left idle for five minutes to minimize theimpact of startup or background processes.
CommandTimer measured the amount of time taken to copy a large set of small files toanother directory on a local drive using the xcopy command.
This test was performed five times with a machine restart between each cycle.

A file set containing 821 files totaling roughly 659 MB.
This file set was intended to represent a large set of small files found on an average user PC. It
includes a range of file formats such as Windows system files (DLL, EXE, CPL, UCE, etc), image files(GIF, JPG, PNG), movie files (AVI, WM, RM), music files (MP3), Office documents (PPT, PPTX,DOCX, DOC, XLS) and PDFs.

ResultOur final result was measured in seconds (s) and calculated from an average of five (5) samples.

File copy time for large files (Endpoint)
Description
Test Tool

This metric measured the total time taken to copy a set of large files between directories, wherean endpoint security product was installed.
CommandTimer
by: PassMark Software
A command line utility developed by PassMark Software which measured and logged the time
taken to perform a task in a command prompt.

Methodology•••

Test File(s)At the start of each test cycle, the machine was left idle for five minutes to minimize theimpact of startup or background processes.
CommandTimer measured the amount of time taken to copy a small set of large files toanother directory on a local drive using the xcopy command.
This test was performed five times with a machine restart between each cycle.

A file set containing 11 files totaling roughly 2.5GB in size.
This file set was intended to represent a small set of large files found on an average user PC. It
includes a range of file formats such as data files (DAT), large image files (BMP, JPG), document
files (PDF, TXT), large archive files (ZIP) and installation executable files (EXE).

ResultComparative Analysis
Edition 1

Our final result was measured in seconds (s) and calculated from an average of five (5) samples.

Page 35 of 36
15 June 2010

Fast and Effective Endpoint Security for Business

PassMark Software

Boot time (Endpoint)
Description

This metric measured the time taken to boot the machine, where an endpoint security product was
installed.

Test Tool(s)xbootmgr and xperf
by: Microsoft
These tools are available from the Windows Performance Toolkit version 4.6 (as part of theMicrosoft Windows 7 SDK, obtainable from the Microsoft Website). Xbootmgr was used to optimize
the boot process, as well as to benchmark the time taken to boot the machine. Xperf was used toparse the detailed boot traces outputted by xbootmgr.

Methodology

•
•
•
•
•
•

ResultNetwork connections were disabled prior to and during this test to ensure consistent results
and minimize network interference.
The Windows service SuperFetch will be enabled prior to testing, in order for boot optimization
to function correctly. This process is disabled for the rest of testing to minimize background
activity.
Prior to boot time testing, xbootmgr was used to perform boot time optimization using thecommand “xbootmgr.exe -trace boot –prepSystem”. Optimization ensures consistent results.
After boot optimization, the benchmark test was conducted using the command "xbootmgr.exe-trace boot -numruns 5”.
The xbootmgr command boots the system five times in succession, taking detailed boot traces
for each boot cycle.
xperf was used to parse the boot traces and obtain the BootTimeViaPostBoot value. This value
reflects the amount of time it takes the system to complete all (and only) system startup
processes.

Our final result is measured in seconds (s) and calculated from an average of five (5) samples.

Machine Restart Time (Client)
Description

This metric measured the time taken for the machine complete the entire restart cycle. We
measured the time taken from the execution of a “Restart” command from within the operating
system, through shutdown and system startup, to when the operating system becomes
responsive to user interaction.

Test Tool(s)Rebooterby: PassMark Software
A test utility developed by PassMark software
which measured the time taken to complete a
restart cycle starting from the operating
system. Rebooted can be downloaded from our
website at:
http://www.passmark.com/products/rebooter.
htm
Screenshot 16: PassMark Rebooter

Method••••ResultComparative Analysis
Edition 1

Network connections were disabled prior to and during this test to ensure consistency andminimize network interference.
Prior to this test, we executed the boot time test. Prior to the boot time test, xbootmgr.exe
was used to perform boot time optimization to ensure consistent results. The effects of
optimization were in effect during the Restart Time test. SuperFetch was also enabled forthis test and for boot optimization to function correctly.
Rebooter was used to restart the machine from the operating system and measure the time
taken to complete each restart cycle.
The machine was restarted five times to obtain five test results.

Our final result was measured in seconds (s) and calculated from an average of five (5) samples.

Page 36 of 36
15 June 2010