Initiatives Toward 

Re-usable Testing

Rick Lazarick 

Computer Sciences Corporation 

Patty Wolfhope 

DHS Science and Technology

IBPC March 8, 2012
Why Pursue Re-usable Testing?

Testing can get expensive

• Scenario testing with sufficient scale to justify 

declaration of low biometric matching error rates 

are time and resource consuming

Repetitive testing is wasteful

• Conformance to a standard (e.g. Data 

Interchange Format for Fingerprint Images) need 

only be done once for a version of a format 

generator

A little motherhood goes a long way
Are there examples of re-used test results?

• Appendix F Certification -fingerprint sensor image quality

– Public list of certified products

– Acquisition with confidence

• MINEX interoperability - fingerprint template generators/matchers

– Open, public test report (quantitative)

– Cited by GSA, TWIC,...

Lead by example
What are the prerequisites for 

reusable testing?

• Agreement on the method/procedure for 

conducting test and reporting results

• Methods should be Open (shared across the 

trio of tester/supplier/consumer)

• Some testing results are binary (Pass/Fail), 

others may be quantitative (tend to be more 

shareable)

Proceed directly to GO, collect $200
Comparison of Test Types

Test Type Characteristics Re-usable? Relative Cost

Conformance to 

Standards 

Automated,

Repeatable

YES Low

Performance - 

Technology

Repeatable, 

Complex

YES 

(with care)

Moderate

Performance - 

Scenario

End-to-end, 

NOT Repeatable

Graded YES 

Pass/Fail Maybe

High

Common Criteria Few biometrics, 

complex lengthy

YES Very High

Do you get what you pay for?
What makes sharing/re-use work? 

• Willingness and authority to share

• Tested by a trusted source (independent or accredited)

• Tested to common/compatible requirement

• Tested at the same level of integration

– Integration levels such as component (template 

generator), subsystem (fingerprint capture device) or 

system (mobile handheld)

– Less sharable as level of integration increases

• Capability to disseminate information

– Open when practical

– Controlled when necessary

Share, share alike
What is the USG Motivation?

Motivation: 

•Section 6 of the 2011 NSTC National Biometrics 

Challenge document cites the benefits of 

standards related to biometric testing including 

reduced product costs. One of the 

contributing factors stated is the reuse of 

testing procedures and certifications. 

Nothing quite like a good Challenge
USG Response? “BITES”

Biometric Interagency Testing and Evaluation 

Schema

– Team formed in December 2011 with multi-agency 

participation

– Spawned by NSTC Subcommittee on Biometrics & 

Identity Management, Standards & Conformity 

Assessment Working Group 

– Effort to develop a repository for sharing biometrics 

test methods and successfully completed test results.

Taking a bite out of “information silos”
What is the Vision of the End-State?

(excerpt from BITES report used to establish the effort)

“When the process for sharing of biometrics testing information and 

resources has been established and institutionalized across the United 

States Government (USG), several of the points identified in the 

National Biometrics Challenge document will be met. The benefits of 

BITES to the government and commercial providers of biometrics 

goods and services will be in the form of fewer required tests, a design 

for reusable results, open and well defined testing methods, and cost 

savings on the part of government implementers as well as a reduced 

testing cost burden to the product suppliers.”

Keeping an eye on the target
What is the Scope of BITES?

• central repository to list successful biometric 

technology test results

– including pointers to externally-managed and agency- specific test methodologies and product lists

– limited to biometric products

• governance structure (to control contents)

• oversight and management of testing-related 

documentation and activities 

– lead the development of unified, USG-desired, 

conformance, performance, interoperability, and quality 

test methodologies

• used for information sharing and acquisition by federal 

government agencies

Don’t bite off more than you can chew
What is the Purpose of BITES?

• Facilitate the advancements of biometric technology at 

a lower cost to the USG

• Capability to collaborate and reuse the results of tests 

• Launch an electronic repository for collaboration

• Develop a test framework that ties requirement and 

approved test methods together

• Establish a governance structure represented by 

interagency members

• Establish a focal point for dissemination of biometric 

testing information to the public

Motherhood revisited
What are the Benefits of BITES?

• Cost avoidance for both government and product suppliers by listing of 

registered testing results from testing performed by a credible source 

(third or second party, unbiased, accredited)

• Improved program testing efficiency and thoroughness (particularly for 

newly developed programs, leveraging the prior work of others)

• Increased interoperability between USG agencies and programs

• Provision of open or controlled access to testing documentation for use by 

the general public, suppliers, acquisition, academia, test labs through the 

establishment of a web-based mechanism

• Uniformity of test methods to support a repository to identify areas where 

new/additional specific test methods (e.g. Conformance, Performance, 

Quality, Interoperability) are necessary

• Increased confidence in the testing activities provided by accredited

testing laboratories

• Assist law enforcement in satisfying the federal rules of evidence (the 

Daubert factors) 

For all the right reasons...
BITES Next Steps?

• USG team creating foundational documents

• Charter

• Project Plan and Roadmap

• Business Plan

• Governance and Policy Plan

• Establishing a web-based project portal for the 

use by general public, suppliers, acquisition, 

academia, test laboratories.

One step at a time...
Thanks for your attention

Rick Lazarick 

Chief Scientist 

CSC Identity Labs

rlazarick@csc.com

Patty Wolfhope

Biometrics Transition PM

DHS HSARPA

Human Factors Division

patricia.wolfhope@dhs.gov