April 2018 NIST Gene Editing Workshop

How to Achieve Confidence in Complicated 

Measurements

Measurement Assurance For Regenerative Medicine and 

Advanced Therapies

Anne Plant

Biosystems and Biomaterials Division

National Institute of Standards and Technology

Gaithersburg MD

anne.plant@nist.gov
Challenges for Regenerative Medicine and Advanced 

Therapies Products 

1. Characterization of product Quality Attributes

• Identity

• Quantity

• Purity

• Sterility

• Potency

2. Control of the manufacturing process

assure consistency of product during Scale Up, 

Change in personnel, process, location

Improved methods, Changes in raw materials
Ground Truth
• What is the measurand?

• Is your assay measuring what you intend?

• Are there assay variables (personnel, reagents, 

unknown factors) that are influencing the assay 

result?

• Can other labs get the same result?

• Is the measurement biologically meaningful? 

Ground Truth

Hard to find in biology 

...it is important to understand .....Compared to what?

your measurements:
 Qualify the assay

• ACCURACY: Orthogonal method 

• PRECISION: Reproducibility: same day 

replicates, day to day, different technicians 

• ROBUSTNESS: sensitivity to assay parameters 

• SPECIFICITY: sensitivity to matrix effects

• DYNAMIC RANGE AND RESPONSE FUNCTION: 

Instrument benchmarking. +/- controls. 

Calibration curve. Limit of detection

Ground Truth

Hard to find in biology 

Understand your measurements: .....Compared to what?
Measurement Qualification

 Are the results the same?

• Can’t tell precision without 

sufficient replicates that 

demonstrate dispersion in 

the measurement.

 Are the results accurate?

• Can’t tell accuracy without 

something to compare it 

to.

NIST, Salit/Deuwer

.....Compared to what?
Easy to imagine for measurement of lead in water

Comparability through reference materials

Harder to envision for measuring complex biologicals 

and biological function
What is measurement 

assurance?

Knowing the level of confidence you have in the data that you are 

using to make a decision.

Having the data that provide credibility of the measurement 

result. 

There are many strategies for achieving measurement assurance
Differences in absolute 

absorbance

Differences in 

response 

functions

Differences in 

measurement 

uncertainties

Assuring comparability: Interlaboratory studies, 

Design of Experiment for Robust Protocols

DoE

John Elliott

Result: 

robust 

protocol and 

comparable 

results
Identifying Sources of Uncertainty – Reportable Parameters?
Identifying sources of measurement uncertainty

Zook, et al., Nature Biotechnology 2014
Evaluating the performance of a cell counting method: 

Experimental design and statistical analysis

S. Sarkar et al.
Different Manufacturers’

calibration beads

NIST SRM 1934/ Calibrated fluorimeter

Fluorescein

Nile Red

Allophycocyanin (APC)

Coumarin 30

Light obscuration flow instrument

For accurate bead concentration

Equivalent Reference 

Fluorophore

(ERF) Number

Assuring comparability in instrumentation: 

traceability to a reference material

Flow Cytometry Quantitation Consortium

81 Federal Register 136 (15 July 2016), pp. 46054-46055

ERF Value Assignment to Cytometer Calibration Microbeads Submitted 

by Consortium Members

Lili Wang



o Is the assay sufficiently 

sensitive to detect a 

problem? 

o If you detect a problem, 

are you sure the assay 

results are correct?

How well do we have to measure?
The measurement process

•Trypsin/ 

collagenase / 

EDTA treatment

Sample 

Collection

•Disaggregation 

of cell clumps

Sample 

Preparation

•Dilutions

Sampling

•Viability or Ab 

stain

Counting 

Preparation

•Load 

hemocytometer

•Collect data via 

microscope

•Flow cytometer

Measurement

/Data 

Collection

Count 

remaining 

cells

Light scattering or 

microscopy to 

assess extent of 

disaggregation

Check 

accuracy 

of volumes 

by weight Orthogonal 

Measurements Process 

Controls

Performance 

Specifications

Measurement 

Process Flow 

Diagram

Ishikawa 

(Causeand-Effect) 

Diagram Reference 

Materials
Some tools for achieving measurement 

assurance:*

• Ishikawa (cause/effect) diagram to 

identify sources of variability 

• Design of Experiment

• Process Controls

• Charting 

• Validation specifications

• Interlaboratory comparisons

• Reference Materials for traceability 

* There are many different ways of 

realizing confidence in measurements. 

Measurement Assurance is Necessary for Confident Decisions

Addressing these challenges will be a 

community effort:

Tool/methods development

Interlaboratory comparisons

Data sharing
Thank You