Scientific Development 

Branch Towards a framework for 

the testing of biometric systems: 

- in procurement, design and operational use

Marek Rejman-Greene

2 March 2010 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

Testing through an SE lens 

– Systems Engineering approach to Testing 

– reduce distraction of novel technology and ‘apparent’ benefits 

– clear focus on stakeholder objectives 

– business benefits, user concerns, audit procedures, ... 

– potential cost reduction 

– testing what’s needed, and when it’s needed .... 

– input into modelling and simulation activities 

– reuse datasets through lifecycle? 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

Systems Engineering 

– ‘An interdisciplinary approach and means to enable the realisation of 

successful systems... 

– ‘it focuses on defining customer needs and required functionality early 

in the development lifecycle, documenting requirements, then 

proceeding with design synthesis and system validation while 

considering the whole problem... 

– ‘Systems Engineering integrates all the disciplines and speciality 

groups into a team effort forming a structured development process 

that proceeds from concept to production to operation ... 

– ‘Systems Engineering considers both the business and technical 

needs of all customers with the goal of providing quality products that 

meet user needs’ 

– (International Council on Systems Engineering, INCOSE, 2006) 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

Application 

– For complex projects in military, non-military governmental 

applications, and commercial operations 

– Approach could be used for smaller systems as well 

– Aim to reduce integration and interoperability risks 

 as well as deliver applications that work first time at lowest cost 

– Testing: 

– Verification – ‘building the service right’ – fulfilling requirements 

– Validation – ‘building the right service’ – fulfilling intended use 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use

SPEAKER Marek Rejman-Greene 

http://www.aof.mod.uk/aofcontent/tactical/engineering/downloads/20090630-TheSystemsEngineeringHandbookVE-u.doc 

1 

2 

3 

4 

5 

6 

7 

8 

9 10 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use

SPEAKER Marek Rejman-Greene 

Supplier context 

sensor 

supplier 

matching 

software 

supplier 

Biometric middleware supplier 

specialist 

software 

supplier 

Specialist biometric integrator 

System integrator 

mobile unit 

supplier 

Test house 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

Stakeholder context 

– Operating Agency / Authority 

– Enrolment Agency / Authority 

– Subject 

– Subject’s Interest Group (trade union, lobby organisation) 

– Media 

– Auditing Authority 

– Regulator 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

1 

http://www.aof.mod.uk/aofcontent/tactical/engineering/downloads/20090630-TheSystemsEngineeringHandbookVE-u.doc 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

Initial review (1) 

– Key performance indicators? 

– Does the technology address this problem space? 

– Do you need biometrics? Why? 

– If YES, relative maturity/performance/sourcing of modality/suppliers 

– Seek technology / scenario / operational data in similar contexts 

– How different is the proposed application from reference context? 

– environment, demographics, throughput requirements, exception handling, 

expectations of stakeholders, ... 

– How long ago was the testing done? 

– Who undertook it? 

– Testing and reporting in accordance with standards? 

– Comparison with KPIs 

– RFP 

– How to validate that objectives met (9)? 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use

SPEAKER Marek Rejman-Greene 

http://www.aof.mod.uk/aofcontent/tactical/engineering/downloads/20090630-TheSystemsEngineeringHandbookVE-u.doc 

1 

2 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

Requirements capture (2) 

– Focus of SE approach 

– Functional and non-functional requirements 

– Relating application KPIs to biometric metrics 

– Environmental context 

– Error rates 

– Usability indices 

– Acceptability criteria 

– Throughput rates 

– Interface/Conformance 

– Security and safety 

– Design for verification (8) and in-life testing (10) 

– Clarity of definitions (ISO vocabulary) 

– Reference implementation 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use

SPEAKER Marek Rejman-Greene 

http://www.aof.mod.uk/aofcontent/tactical/engineering/downloads/20090630-TheSystemsEngineeringHandbookVE-u.doc 

1 

2 

3 

4 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

Design and Specification (3 and 4) 

– Work towards acceptance and integration testing (6 + 7) 

– What mix of technology, scenario and quasi-operational approaches? 

– Specially designed harnesses? 

– In parallel, option for modelling and simulation to examine trade-offs? 

– Agreement on datasets to be used in acceptance and integration testing? 

– How representative and challenging should these datasets be? 

– Who supplies? 

– Off-the-shelf standards and interfaces? 

– May require application profiles and adaptation of interface specifications 

– Dispute resolution procedures 

– Impact of change control if adjustments needed after delivery 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use

SPEAKER Marek Rejman-Greene 

http://www.aof.mod.uk/aofcontent/tactical/engineering/downloads/20090630-TheSystemsEngineeringHandbookVE-u.doc 

1 

2 

3 

4 

5 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

Procurement, Design, Development (5) 

– Testing of bidder’s claims during procurement 

– focussing on the unknowns and highest risks 

– alternatives 

– development of reference model + independent tests? 

– simulation or modelling – sensitivity analysis + spot checks? 

– benchmarking / assessment of reference deployments 

– Demonstration of effectiveness of consortium working together 

– Costs 

– Shared? Benefits to both parties 

– Involvement of stakeholders other than Authority and Bidder? 

– Proposals to adjust during design and development 

– based on internal testing and additional knowledge 

– how involved is the customer in change management? 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use

SPEAKER Marek Rejman-Greene 

http://www.aof.mod.uk/aofcontent/tactical/engineering/downloads/20090630-TheSystemsEngineeringHandbookVE-u.doc 

1 

2 

3 

4 

5 

6 

7 

8 

9 10 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

– Comparative performance 

– across time 

– across sites and terminals 

– as introduce upgrades 

– Collect both Application KPIs and Biometric Metrics 

– Aim: 

– assurance of performance 

– early warning of problems 

– support for continual improvement and cost reduction 

Operational testing (10) 
SCIENTIFIC DEVELOPMENT BRANCH 

INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE: Towards a framework for the testing of biometric 

systems: in procurement, design and operational use 

SPEAKER Marek Rejman-Greene 

http://www.aof.mod.uk/aofcontent/tactical/engineering/downloads/20090630-TheSystemsEngineeringHandbookVE-u.doc