Document: Profil Technology – Image & Video Detection Technology Assessment.docxAuthors:
M. Baquiran, D. Wren
Company: PassMark Software
Date:
13 November 2015
Edition:1Image & Video Detection - Technology Assessment
Edition 4Page 1 of 1426 November 2015

TABLE OF CONTENTS......................................................................................................................................... 2
REVISION HISTORY............................................................................................................................................ 3
EXECUTIVE SUMMARY ...................................................................................................................................... 4
IMAGES - RESULTS ............................................................................................................................................ 5
IMAGES - CHARTS ............................................................................................................................................. 6
VIDEOS - RESULTS ............................................................................................................................................. 7
VIDEOS – CHARTS ............................................................................................................................................. 8
RESULTS SUMMARY .......................................................................................................................................... 9
DISCLAIMER AND DISCLOSURE ....................................................................................................................... 11
CONTACT DETAILS .......................................................................................................................................... 11
APPENDIX 1 – TEST ENVIRONMENT ................................................................................................................ 12
APPENDIX 2 – METHODOLOGY DESCRIPTION ................................................................................................. 13

Image & Video Detection - Technology Assessment
Edition 4

Page 2 of 14
26 November 2015

RevRevision History

Date

Edition 1

Initial version of this report.

11 November 2015

Edition 2

Added Optimal Threshold Analysis.

12 November 2015

Edition 3

Added version information.

12 November 2015

Edition 4

Revised name, thresholds used in results, added company logo.

13 November 2015

Image & Video Detection - Technology Assessment
Edition 4

Page 3 of 14
26 November 2015

PassMark Software® conducted an objective accuracy evaluation on Profil Technology®’s detection technology:VISIA Technology®. For the purpose of this test, we will use the term “explicit” to define content containing male
and/or female genitals and/or female breasts, and the term “normal” content to define content not containing
these. VISIA technology is used to scan a folder’s contents for explicit images and/or videos. When a file is scanned,
it is determined to be either explicit (logged as “EXPLICIT”) or normal (logged as “OK”) at different levels ofsensitivity.
PassMark compiled the following four data sets, and a separate scan was performed on each: Normal (no nudity) Videos (1019 files)
 Explicit (contains nudity) Videos (1018 files) Normal (no nudity) Images (3059 files)
 Explicit (contains nudity) Images (3060 files)
Each file set contains data from various sources of various file types and sizes. Images were of a minimum
resolution of 320x240 pixels and videos were of a minimum length of 3 minutes.
The results for each scan were collected and analyzed in order to calculate the accuracy of detection. The
technology tests for explicit content, meaning that a “Positive” result is given when a file is identified as“Explicit”. Conversely, a “Negative” result is given when a file is identified as normal. Each result is further
described as either “True” or “False” depending on whether the identification was accurate or not.
The chart below illustrates this terminology:

Test Result

File Status

Explicit

NormalExplicit
True Positive – An

NormalFalse Positive – A

explicit file that has
been correctly
identified as explicit
False Negative – An
explicit file that has
been incorrectly
identified as normal

normal file that has
been incorrectly
identified as explicit
True Negative – A
normal file that has
been correctly
identified as normal

A small proportion of files were not scanned by the software and logged with the result “ERROR”. Since thisreport deals with the tool’s detection capabilities, these skipped files have been omitted from the following
values.

Image & Video Detection - Technology Assessment
Edition 4

Page 4 of 14
26 November 2015

The following tables contain results for the images at three different sensitivity levels (low, medium, and high):

Sensitivity (Threshold)Low (0.8)

Middle (0.73)

High (0.65)# Correctly Identified (True Negatives)

2680

2527

2327

# False Positives

310

463

663

Total

2990

2990

2990

Low (0.8)

Middle (0.73)

High (0.65)# Not Detected (False Negatives)

595

448

304

# Detected (True Positives)

2420

2567

2771

Total

3015

3015

3015

Sensitivity (Threshold)Image & Video Detection - Technology Assessment
Edition 4

Page 5 of 14
26 November 2015

The following charts are a graphical interpretation of the above data, showing the proportions of normal andexplicit images that have been correctly identified at three sensitivity levels:

Normal Images

Sensitivity

HighMiddle

Low
0%

20%

40%60%80%

100%80%100%

Axis Title
Correctly Identified

False Positives

Explicit Images

Sensitivity

HighMiddle

Low
0%

20%

40%60%

Proportion of Images
Detected

Image & Video Detection - Technology Assessment
Edition 4

Not Detected

Page 6 of 14
26 November 2015

The following tables contain results for the videos at three different sensitivity levels (low, medium, and high):

Sensitivity (Threshold)Low (0.74)

Middle (0.46)

High (0.38)# Correctly Identified (True Negatives)

964

903

873

# False Positives

39

100

130

Total

1003

1003

1003

Low (0.74)

Middle (0.46)

High (0.38)# Not Detected (False Negatives)

147

100

80

# Detected (True Positives)

869

916

936

Total

1016

1016

1016

Sensitivity (Threshold)Image & Video Detection - Technology Assessment
Edition 4

Page 7 of 14
26 November 2015

The following charts are a graphical interpretation of the above data, showing the proportions of normal andexplicit videos that have been correctly identified at three sensitivity levels:

Normal Videos

Sensitivity

HighMiddle

Low
80%

85%90%

95%

100%

Percentage of Videos
Correctly Identified

False Positives

Explicit Videos

Sensitivity

HighMiddle

Low
75%

80%

85%

90%

95%

100%

Percentage of Videos
Detected

Image & Video Detection - Technology Assessment
Edition 4

Not Detected

Page 8 of 14
26 November 2015

The following analysis was conducted in order to determine the optimal sensitivity thresholds for the data. TheDetection rate and False Positive rate was logged for a range of 200 threshold levels between -1.0 to 1.0 withincrements of 0.01. The least sensitive, or “lowest” threshold corresponds to 1.0 and the most sensitive (“highest”)
threshold corresponds to -1.0. The higher the sensitivity, the more the software is likely to determine that a file isexplicit. The following charts illustrate our findings for each file set:ThresholdThresholdImage & Video Detection - Technology Assessment
Edition 4

Page 9 of 14
26 November 2015

We can thus synthetize the above information to give us appropriate thresholds (see the marked values above)for low, medium, and high sensitivity levels. This gives the following detection rate and false positive rate forimages and videos:Threshold
Detection Rate
False Positive Rate

ImagesLow (0.8)
Medium (0.73)
High (0.65)
80.3%
85.1%
89.9%
10.4%
15.5%
22.2%

Threshold
Detection Rate
False Positive RateVideos
Low (0.74) Medium (0.46)
High (0.38)
85.5%
90.2%
92.1%
3.9%
10.0%
13.0%

Image & Video Detection - Technology Assessment
Edition 4

Page 10 of 14
26 November 2015

While every effort has been made to ensure that the information presented in this report is accurate, PassMark
Software Pty Ltd assumes no responsibility for errors, omissions, or out-of-date information and shall not be liable
in any manner whatsoever for direct, indirect, incidental, consequential, or punitive damages resulting from theavailability of, use of, access of, or inability to use this information. Additionally, the results obtained in this report
are unique to the methodologies and data sets described and are not necessarily representative of consistentbehavior on other data sets and other builds of the software.

Profil Technology funded the production of this report, defined the test criteria and provided the software builds
for the tests.

All trademarks are the property of their respective owners.

PassMark Software Pty Ltd
Level 5, 63 Foveaux St.
Surry Hills, 2010
Sydney, Australia
Phone + 61 (2) 9690 0444
Fax+ 61 (2) 9690 0445

Web

www.passmark.com

Image & Video Detection - Technology Assessment
Edition 4Page 11 of 14
26 November 2015

For our testing, PassMark Software used a test environment with the following hardware specifications:
O/S:
CPU:
Video Card:
Motherboard:
RAM:
Boot Drive:
Data Drive:
Network:
Video:

Windows 7 Professional Edition Service Pack 1 build 7601 (64-bit)Intel Core i5-4570 CPU @ 3.20 GHz
Intel HD Graphics 4600
ASUSTeK H87M-PRO
16GB RAMIntel SSDSC2CW180A3 ATA Device 167.68GB
WDC WD20EZRX-00D8PB0 ATA Device 186.01GB (This drive contains the softwareapplication)
Gigabit (1GB/s)Intel HD Graphics 4600

Image & Video Detection - Technology Assessment
Edition 4Page 12 of 14
26 November 2015

Profil Technology provided a build of their VISIA Technology software for this test. The build details are as follows:
Filename:
File version:
Date:

IVEvalTool
1.0.0.0
3 November 2015

A scan was run on each of the below file sets and the results were exported to a .csv file. These files were thenanalyzed to give false/true positive/negative rates.

The files used in this test were obtained from a variety of sources online, including (but not limited to) open source
and creative commons libraries. Images were of a minimum resolution of 320x240 px and videos were of aminimum length of 3 minutes. Extra-long (>30 minutes) videos as well as high resolution images were also
included.
“Explicit” videos and images contained male and/or female genitals and/or female breasts, whereas “normal”
content did not containing these. Explicit data contained a balanced amount of male and female genitals, female
breasts, varied skin tones, as well as varied themes. Normal data contained a mix of both humans, animals, andobjects across a variety of themes.
Below is a breakdown of the data sets used in this test.

File Type

Number of files

Total File Size.jpg

2938

1.6 GB.png

105

69.6 MB.gif

15

6.7 MB.bmp

1

2.2 MB

Total

3059

1.68 GB

File Type

Number of files

Total File Size

.jpg

2971

610.9 MB.png

4529.2 MB.gif

4447.6 MB

Total

3060

687.7 MB

Image & Video Detection - Technology Assessment
Edition 4Page 13 of 14
26 November 2015

File Type

Number of files

Total File Size

.avi

258

176.0 GB

.mp4

527

78.7 GB.mkv

148

22.7 GB.webm

28

7.7 GB.m4v

34

2.5 GB.ogv

6988.5 MB.wmv

3101.4 MB.swf

1255.3 MB.mov

3

30.4 MB

Total

1019

288 GB

File Type

Number of files

Total File Size

.avi

281

73.3 GB.flv

328

37.5 GB.mkv

1

6.6 GB.mov

25

12.3 GB.mp4

260

99.9 GB.ogv

1291.5 MB

.rmvb

4

1.2 GB.wmv

118

61.9 GB

Total

1018

293 GB

Image & Video Detection - Technology Assessment
Edition 4

Page 14 of 14
26 November 2015