1

Charles Fenimore & Wo Chang

fenimore@nist.gov

wchang@nist.gov

01-301-975-2428

Developing a Forensic Image Examination 

Quality Rating Metric

11/30/2012
2

The Challenge

Forensic imagery as stills or CCTV is constantly captured 

for expert examination in support of public safety. Often, it 

is “from the wild” without fine control of illumination, position, 

attitude, or obscuration of persons, vehicles etc. Data volumes 

are huge; quality (and hence utility) unknown.

An FBI DVD, “Caught on Camera”, addresses some of the quality issues.
Photographic comparison and content 

analysis tasks on imagery “in the wild”

Lighting dramatically affects quality for facial comparison task

Localize measurement of quality factors: resolution, 

3

	
4

The Challenge

The image examiner’s knowledge and judgment is essential 

in law enforcement forensics – frequently with 

computational assist.

– Automatic methods read license plates, do biometric-based ID, or other 

well-defined tasks but generally requires high quality imagery.

– The examiner may find unique image elements to solve a case; see LA 

arson bomber or Germantown, Maryland, 7-11 flash mob
5

Quality of CCTV images – a use case

CCTV imagery ‘from the wild’ can have highly variable 

quality depending on: 

– illumination,

– camera resolution, distance to target, focus, other acquisition 

parameters,

– scene effects, obscuration, object motion, etc.

Quality metric to:

– semi-automate image selection or prioritization

– quality rating defined by examination potential of imagery.

Hypothesis: Forensics can benefit from application of the 

National Image Interpretability Rating Scale methodology 

(NIIRS) to law enforcement needs.
6

Outline

1. The National Imagery Interpretability Rating Scale (NIIRS) 

– measures the quality of an image as interpretability or intelligence 

potential for use by a trained image analyst.

– Assigns rating to tasks and images. Simple model predicts image 

rating based on image and target specific parameters.

– High level of agreement among trained image analysts on image 

quality rating.

2. Metric development for a forensic image examination 

rating scale (FIERS):

– Images

– quality factors

– Modeling

– assessment.
7

Surveillance image interpretation

open bay door

sedan

five steps

Rating level 5:

Detect open bay doors of 

vehicle storage buildings

Rating Level 6:

Identify automobiles as 

sedans or station wagons.

Rating level 7:

Detect individual steps on 

stairway.

Rating Level 8:

Identify grill detailing and/or 

the license plate on a 

passenger/truck type vehicle
8

National Imagery Interpretability Rating Scale (NIIRS)

• Measures image interpretability or intelligence potential of an image 

to an analyst.

• Mapped images and interpretation tasks to a scale, 1 – 9 (and up)

– Higher NIIRS level interpretation tasks are more demanding

– Task are structured: detect /distinguish between/ identify

Image quality equation maps acquisition parameters to scale level

Quality = F ( resolution at object, edge blur, SNR, edge overshoot ) 

where F is log-linear

Supports reliability measurement for quality rating, system design and 

image collection planning

Identify bicycle details (e.g., frame, wheel/tire, etc.).

Recognition Level Object Qualifier
9

Civil interpretation tasks for 

forensic image exam scale development

Interpretation Tasks (Criteria)

NIIRS

Rating

Identify individual bunches of pine needles. 9

Identify tree species based on leaf size and shape. 8.4

Identify facial features on an individual

(i.e., at least partial discrimination of some facial features). 

8.2

Identify windshield wipers on a vehicle. 8

Identify grill detailing and/or the license plate on a passenger/truck type 

vehicle. 8

Detect outside rearview mirrors on passenger cars. 7.7

Identify bicycle details (e.g., frame, wheel/tire, etc.). 7.7

Identify limbs (arms, legs) on an individual. 7.3

Detect small road signs (e.g., stop, yield, speed limit) in an urban area. 6.6

Detect individuals, when not in a group 6.2
10

Forensic image Examination Rating Scale 

(FIERS) development 

• Implemented Civil NIIRS quality model. Exercise model.

Still Image General Quality Equation (GIQE)

 Q = c0 + c1log10 (TSD) + c2log10 (RER) + c3 G / SNR + c4 H (1)

Q = NIIRS quality measure for a deployed imaging system,

TSD = target sampled distance, pixel spacing projected to the target,

RER = relative edge response (slope of the normalized edge response measured

between two points one-half pixels on opposing sides of an edge.)

SNR = signal to noise ratio [5, p. 306],

G = noise gain arising from modulation transfer function compensation 

(MTFC) In the absence of compensation, G = 1.

H = edge overshoot is the height of edge overshoot on the normalized edge 

response. H arises from modulation transfer function compensation.

See Leachtenauer, JC and Driggs, RG, Surveillance and Reconnaissance Imaging 

Systems, Artech, Boston, [2001]
FIERS Quality Model Implementation

11
12

Forensic image Examination Rating Scale 

(FIERS) development 

• Implemented Civil NIIRS quality model. Initially compare 

Quality model scores on images with published task scores.

• Formal evaluation of imagery and tasks await institutional 

approval.

• NIST is interested in working with practitioners; participation 

is essential to FIERS development. Interpretation tasks must 

be domain specific.
NIST CCTV Image Collection 

High definition collection for NIIRS scale development.

13

Interpretation Task

Civil

NIIRS

Image 

Rating

Identify grill detailing and/or the license plate on a vehicle. 8 7.9

Detect small road signs (e.g., stop, yield, speed) in urban area. 6.6 6.55
NIST CCTV Image Collection 

High definition collection for NIIRS scale development.

14

Interpretation Task

Civil

NIIRS

Image 

Rating

Identify grill detailing and/or the license plate on a vehicle. 8 9.5

Detect outside rearview mirrors on passenger cars. 7.7 7.8
NIST CCTV Image Collection 

High definition collection for NIIRS scale development.

15

Interpretation Task

Civil

NIIRS

Image 

Rating

Identify facial features on an individual

(i.e., at least partial discrimination of some facial features). 8.2 10.3

Identify limbs (arms, legs) on an individual. 7.3 8.3
Imagery sources for FIERS, forensic image 

examination rating scale development

Imagery: current sources ‘from the wild’

• Scripted: iLIDS, future collections

• Scripted NIST Biometrics collections. 

• Modified (corrupted/meliorated) images –

blurring or noise added/filtered to mimic 

imagery ‘in the wild’

• Synthetic (faces from the British Home Office, 

NIST ++ )

• Imagery from the wild, available to 

practitioners

16
17

Conclusions: Forensic image scale development 

• We have taken initial steps to assess the applicability of an 

image interpretation quality model; need a more formal eval.

• It remains to determine forensics-specific tasks, to 

incorporate other aspects of forensics work flow, and to build 

an optimal forensic image quality model.

• We are working with 2 forensic practitioners and invite you to 

view a demo of the FIERS implementation at the break.

• Potential payoffs for forensic science:

- quantify image quality requirements

- potential for automated ratings of large image collections, could 

support prioritized image examination

- reliability measures for forensic examination process

- quantify value of images as evidence (pathology). The capability to 

quantify examination potential of any image, may reduce the risk of 

conflicting expert testimony.