Anti-Virus Testing and AMTSO
PatrikPatrik Runald,
Runald, Security
Security Response
Response Manager
Manager

Helsinki ● Kuala Lumpur ● San Jose ● Tokyo ● London ● Paris ● Milan ● Hong Kong
Mumbay ● Warsaw ● Munich ● Copenhagen ● Brussels ● Oulu ● Utrecht ● Oslo ● Stockholm

3

MalwareGrowth of malware
700,000
650,000
600,000
550,000
500,000
450,000
400,000
350,000
300,000
250,000
200,000
150,000
100,000
50,000086 87 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08

Data source: F-Secure
平成20年3月27日

456

Attack Response Attack Response
ATTACKRESPONSE

Malicious email attachments

Dropping EXEs and PIFs

Drive-by-Downloads to exploit
Windows vulnerabilities

Automatic Windows Updates

Attacking Web browser
vulnerabilities

Browsers automatically prompt
for new versions

Attacking browser plugins andAdd-ons (think Quicktime, Flash,Acrobat)

Nothing much yet…

平成20年3月27日

7Disclaimer

Material and comments in this presentation doesn’t necessarily
represent the actions or methods used by
F-Secure Corporation.

8Problem with AV testing
Doesn’t always state testing methodology
Requires deep knowledge about antivirus technologies
Requires extensive collection or malware
Doesn’t test all parts of an antivirus solution
Engines can be modified/tweaked to do well in tests
Heavily biased toward static file scanning
Testing same as 10 years ago

9Testing today
On-demand testing• WildList/WildCore• Large (zoo)Retrospective• Test product after x amount of timeOn-access• Barely happens, too difficult

Reaction tests• How long did it take for vendor x to detect sample y10

WildList/WildCore• A collection of malware samples in-the-wild“WildCore is a set of replicated virus samples that represents thereal threat to computer users”“When a virus is reported by two or more reporters it goes into theWildCore”• Easy to pass as the list is known

11

Zoo collections
Large collection of samples
Shouldn’t contain trash but often does
Can be tricked by setting paranoid settings/heuristics
Could mean detection based on another vendor
False positives is a problem but can be whitelisted12Retrospective• Scan a sample collection (WildCore or Zoo) after x amount of time
• Define a date and freeze the updates at that time• Check detection rates on new samples• Doesn’t take into account generics on new families• Unless correlated with false positive testing will benefit highparanoid heuristics

13

On-access• Every file would have to be executed. Enough said…

14

Reaction testing• How long does it take for a vendor to detect a specific sample• Less interesting today as there are so many new samples• Not only that, they are polymorphing15

Bias toward file scanning• Why? Because it’s easy and fast
• Today's products are multi-layered (firewall, HIPS, buffer overflow,
IDS etc)

16

Let’s look atsome tests
17

18

19

AV-Test March 2008
Vendordetected 3/2008% detected“A comprehensive
review should not only
concentrate on
AVK (G Data)
1129521
99.9%detection
WebWasher-GW
1128900
99.9%
scoresoftheon-demandscanner,asthiswouldgiveauser only aTrustPort
1126425
99.6%
AntiVirmisleading(Avira)1122874
99.3%
veryand limited view of the
product's capabilities.”
Avast! (Alwil)
Trend Micro

11170991116139

98.8%
98.7%

Sophos1109020
98.1%“Whencomparing the security of cars,1107753
we would not only
focus onIkarus98.0%
Nod32 (Eset)belts, but also check the ABS
1106236
97.8% braking
the safetysystem (anti-lockMicrosoft1105564
97.8%system),
one
or
more
airbags,
crush
zones,
theESP(electronicBitDefender1105397
97.8%
Kaspersky
1098533
97.2%
stabilizationprogram) as well as constructionalchangesand many
F-Secure
109418896.8%
a car secure.” 96.7%
F-Prot (Frisk) other features which make1093516eScan
ZoneAlarm
AVG
Norton (Symantec)
Panda
McAfee10932471089404
1088422108197810809861080580

96.7%
96.4%
96.3%
95.7%
95.6%
95.6%

Norman1049019

92.8%

In order to get a more comprehensive impression of the products,
one should not only look at this test, but also compare the results ofvarious tests and the products' performance over time and their
Rising1063990
94.1%
on-going development.

20

AV-Comparatives Feb 200821

AV-Comparatives Feb 200822

VirusInfo

23

VirusTotal

24

So what canbe done?
25

26

27

28

AMTSO
ALWIL Software
IBM
AV-Comparatives
Kaspersky Lab
25
member organizations
AV-Test.org
McAfee, Inc.
Started Feb 4
AVG Technologies
Microsoft Corp.
Providea forum to discuss testing
Avira GmbH
Norman ASA
Bit9
Panda
Security
Developand publish standards fortestingincl. best practices
BitDefender
PC Tools
Providetools andDoctor Web,
Ltd. resources
Sana Security
ESET analysis and review current
SecureProvidetestsComputing
F-Secure Corporation
Sophos Plc
Next meeting April 30
G DATA Software
Symantec Corporation
Hispasec Sistemas
Trend Micro Inc.
Virusbuster Ltd.

29

Finally we’remakingchanges
30Patrik Runald
Security Response Manager
F-Secure Corporationwww.f-secure.com

31