Evaluating 

Examiner/Operator-Le

Biometric Application

Biometric Tech

National Policing Im
What is an Examiner/Operator-Led 

Application? 

 A biometric system that is reliant on the interaction and skill of a hu

one or more stages of the overall biometric search process – that isthrough to enrolment, template generation and final decision 

Capture 

Edit 

Encode 

Search 

Parameters 

View 

& Decision 

Who is the Operator? • An expert or examiner • An advanced user • Makes the final decision 

www.npia.police.uk 
Where would these be used? 

• Non real time /offline • Back Office Checking 

Systems 

• Forensic Applications 

– E.g. Fingerprint, Face, 

Voice/Speaker, Signature analysis 

• 1:many or 1:1 

searches 

Capture 

Edit 

Encode 

www.npia.police.uk 
Example: Forensic Fingerprint Systems 

Only bit that is fully 

Capture Edit/Encode Search 

Parameters 

Match 

Pre processing images for 

searching or encoding 

automated 

Selection, Cropping and 

Extraction 

All fingerprint evidence that goes to court must be validated by at least 2 human experts 

www.npia.police.uk 
A Real World Example: 

IDENT1 – National Palm and Fingerprint searching across UK 

• 8.1 million subjects • 17.8 million fingerprint sets (rolls, 

flats and multiples) 

• 7.9 million palm prints • 1.8 million un-id crime scene marks, 

160,000 from palms 

• 48 000 crime scene mark id’s 

within last 6 months 

• Over 1200 fingerprint expert 

operators across the UK 

• All fingerprint evidence used for court 

checked by at least 2 experts 

www.npia.police.uk 
What is the value of such an approach? 

www.npia.police.uk 

Minimises the 

labour of manual 

processes 

Help manage and 

present the data 

C

p

Expertise – 

perceptio

Improves and compliments 

the performance of both 

Political/ legall

requirem

All fingerprint evidence used for court checked by at least 2 experts 
Why is it important to address such systems 

specifically? 

• Understand the role of the operator to

the system to influence its design an

performance. 

– What value does the system provide to the – What value does the operator’s skill provide– Not measuring their skill but how they emp

• 

“Educated Users” - Expert’s perceptio

system performance may affect their odecision making 

• Key to obtaining user acceptance of 

technology 

– No confidence – no use for the application! 

www.npia.police.uk 
Examples of unique factors to consider 

• Measuring Accuracy Performance 

• Reporting Accuracy Performance (for 1:Many a

• Controlling Test Variables 

• Example of Designing a test 

www.npia.police.uk 
1. How Accuracy Performance is Measured 

Measuring the true accuracy must inthe operator’s decision! 

Match View & 

Decision 

However, the Omay not detecon the list! 

The system mate in the responses t

x x x 

www.npia.police.uk 
 

A: Depends on how far the Operator is looking

Rank (Position Returned)

2. How Accuracy Performance is Reported(1:Many Closed set) 

0 

10 

20 30 40 

50 

60 

70 

80 90 

100 

Cumulative Frequency (% ) 

Cumulative Match Curves 

Q: Which CM Curve shows 

better performance? 

Rank 

www.npia.police.uk 
 

2. How Accuracy Performance is Reported

(1:Many Closed set) 

0 

10 

20 30 40 

50 

60 

70 

80 90 

100 

Cumulative Frequency (% ) 

Rank (Position Returned) 

Cumulative Match Curves 

Q: Which CM Curve shows 

better performance? 

www.npia.police.uk 
 

2. How Accuracy Performance is Reported 

(1:Many Closed set) 

Q: Which CM Curve shows 

better performance? Cumulative Match Curves 

100 

90 80 

70 

60 

50 

40 30 20 

10 

0 

Rank (Position Returned) 

A: Depends on how far 

Cumulative Frequency (% ) the Operator is looking

www.npia.police.uk 
2. How Accuracy Performance is Reported 

(1:Many) 

Paired functions = {Rel(n) , Sel(n)} 

Reliability (Rel)(%) = True Mates detected within the list of lenSelectivity (Sel) = Mean number of responses compared by an 

of length (n) 

1 2 3 4 5 .... Rank 

0 

10 

20 30 40 

50 

60 

70 

80 90 

100 

www.npia.police.uk 
2. How Accuracy Performance is Reported 

Example 

CMCurve 

0 

20 40 60 80 

100 

0 5 10 15 20 

Rank 

Cumulative Frequency 

(Rel, Sel) = (77%, 3), (75%, 4) 

(Rel, Sel) = (79%, 4), (95%, 5) 

(Rel, Sel) = 

(100%, 4), (100%, 5) 

Reliabil(Rel)(%detecteSelectiv(Sel) = A#Resposearch 

www.npia.police.uk 
Examples of unique factors to consider 

• Measuring Accuracy Performance 

• Reporting Accuracy Performance (for 1:Many a

• Controlling Test Variables 

• Example of Designing a test 

www.npia.police.uk 
Common Considerations when Controlling Variables 

Data: 

–Biometric Search and Match Pairs –Biometric data pre-processed with operator input –Background data 

Operators: 

–Required Expertise –Available as Test Operators? (Day job?) –Training - to use the technology –Perception/Behaviour/ Subjective judgement 

Test Environment 

–Live System? Test Bed? –Operational Scale –Operator Behaviour Altered? 

Scope & Design 

–End to end solution –Sub-process/component e.g. HCI –Testing human or machine or both? –Controlling variables – Repeatable/Operationally reflective? –White/Black box 

Results Reporting Most significant and dif

–Interpretation of Data - Operator Decision control are those relate–Quantify Value added benefit to manual process –Comparable 

www.npia.police.uk 
Operator Induced Variables 

– Input/interactions will differ between operators well as for the same operator – Varying Expertise 

• Specify Search parameters/filters – e.g. Palm ROI, finger mask, pattern • Decisions/Judgements – subjective (not always accurate) • Make use of other data (e.g. application form, case notes) 

– Varying Approaches/Behavior 

• May alter when tested • Not testing their expertise but how they employ it! 

– Training 

• Proficiency 

• HCI toolsets 

– Business procedures 

• Prioritise searches differently • Variations in Operator effort 

– Test Environments 

• Confidence in system 

• Perception 

as 

Ed

www.npia.police.uk 
Controlling Variables 

Types Testing • Application 

e.g. Design and develop components or processes within the system? – Repeatable Tests – Biometric data pre-processed with operator input 

• Scenarios to define business/operator workflows 

e.g. Monitoring of operations throughout life of service – Need operators to reflect reality but the solution and how they behave may change in reality 

• Operational performance 

e.g. benchmarking different solutions – Realistic – Real data – Live System – Real Operators 

Variables? 

White Box Black box 

Variables? 

Test 

Ed

For each testing approach one needs to know what variables to control and how it affects the measured performance. 

www.npia.police.uk 
Examples of unique factors to consider 

• Measuring Accuracy Performance 

• Reporting Accuracy Performance (for 1:Many a

• Controlling Test Variables 

• Example of Designing a test 

www.npia.police.uk 
Design a Test! 

Background: 

An existing Operator led forensic finger and palm searto be replaced by a new solution provided by a differen

Task: 

There is a need to demonstrate the accuracy of the repsolution against the existing AFIS 

– to show no loss in performance to users – And baseline the performance of the new service. 

Approach: 

To design and execute a fair benchmark test to compaof a legacy system against a new solution that will replcontract award. 

www.npia.police.uk 
Design A Test! 

Testing New Vs Legacy System 

New solution is partially developed only -Legacy system is complete. Limited to Algorithm tests? Design of operator business process to be defined based on new solution 

e.g. HCI functions to be developed with user 

Operators already familiar with existing toolsets. Need to maintain a level playing field Need to discriminate between solutions Gain user acceptance 

Limitation

Algorithm/Application TestinDoesn’t account for the operCannot guarantee performanUnlikely to be based on operCannot compare solutions Scenario Testing: Must reflect approaches of dNot testing like for like if testNew Solution tested may chaLimited scope Operational Testing: Realistic solution - after contReal Data, Real Operators, LiEverything is Uncontrolled - between systems! What is thattributed to? The Operator o

www.npia.police.uk 
Volume of Searches depends on... 

• ...Constraints 

– How many searches are required to significa

discriminate between the performance of the solutions? 

– How many test operators can be spared from 

work to do these tests? 

– How long does it take for the operators to pr

searches? 

– How much time is there to execute the tests– Cost - proportionate to the purpose/benefit? These constraints limit the scale of the test an

infer from the results. 

www.npia.police.uk 
Summary 

• The performance of Operator led systems is equally relian

algorithm and interaction of a human operator 

• Where poor data quality and other algorithmic limitations 

challenge Operator led approaches are necessary. 

• Forensic AFIS and Back Office comparisons are key exam• Addressing the unique challenges described for these app

important to the design, development and user acceptancapproaches across modalities. 

• Operator induced variables are the most significant and d

– Impact on how performance is measured and reported 

• E.g. Reliability and Selectivity 

– Operators’ Perception – Managing varying operator behaviours, expertise, training, in

• Practical limitations with test approaches – application, sc

operational tests 

• Example of designing test – highlight challenges and give 

thought! 

– though no test is identical. No rule fits all! 

www.npia.police.uk 
Contact Details: 

Ambika Suman Biometric Technologies Manager National Policing Improvement Agency New King’s Beam House 22 Upper Ground London SE1 9QY Tel: +44(0) 20 8358 5446 Mobile :+44(0) 7881 673 890 Email: ambika.suman@npia.pnn.police.uk