Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Targeted Potency

David Siev

Statistics Section Leader

Center for Veterinary Biologics

April 21, 2015

“Public Service is a Public Trust”

5CFR §2635.101
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Data-driven Criteria

• Consistent Principles

• Coherent System

Core
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Release Throughout-Dating

Dating Period
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

“Minimum Protective Dose” (MPD)
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

‘Dose’ has at least four different usages:

• Concentration (e.g. 8mg/dL dose)

• Volume (e.g. 1mL dose)

• Treatment (e.g. 2-dose vaccine)

• Potency (as implied by MPD)

“Minimum Protective Dose” (MPD)
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

“Minimum Protective Dose” (MPD)

Not designed for minimum
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

“Minimum Protective Dose” (MPD)

Often not protective but reductive

Non-dichotomous outcome

Term not meaningful
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

“Minimum Protective Dose” (MPD)
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

Targeted Potency

“Minimum Protective Dose” (MPD)
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

“Antigen Overage”
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

“Antigen Overage”

OMG
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

“Antigen Overage”

Antigen ≠ Potency
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

Overage? 

Outdated 9CFR term
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

Overage? 

Outdated 9CFR term

Intent is not adding excess antigen
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

Increment Overage?

Measurement Uncertainty
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Measurement Uncertainty

Guide to the Expression of Uncertainty in 

Measurement. 

ISO/IEC Guide 98-3 (“The GUM”). International 

Organization for Standardization, 2008.
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Measurement Uncertainty

• ISO/IEC Guide 98-3, Guide to the Expression of Uncertainty in

Measurement. International Organization for Standardization, 2008.

(“GUM”)

• NIST Technical Note 1297, Guidelines for Evaluating and Expressing the

Uncertainty of NIST Measurement Results. National Institute of Standards

and Technology, United States Department of Commerce, 1994.

• G104 – Guide for Estimation of Measurement Uncertainty in Testing.

American Association of Laboratory Accreditation, 2014.

• G108 – Guidelines for Estimating Uncertainty for Microbiological Counting

Methods. American Association of Laboratory Accreditation, 2014.

• UKAS Publication ref: Lab 12. The Expression of Uncertainty in Testing.

United Kingdom Accreditation Service, 2000.
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Targeted Potency
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Targeted Potency
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Targeted Potency
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Targeted Potency

Observed in pivotal efficacy study (PES)

+

Increment due to measurement uncertainty
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

The Impact of Terminology

Expiry = Minimum Protective + Overage

Target = PES Observed + Uncertainty
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

The Impact of Terminology

Expiry = Minimum Protective + Overage

Target = PES Observed + Uncertainty

? 
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

The Impact of Terminology

Expiry = Minimum Protective + Overage

Target = PES Observed + Uncertainty

Determine target by estimating an unknown quantity

Add extra to a known target just to be extra sure
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

The Impact of Terminology

Expiry = Minimum Protective + Overage

Target = PES Observed + Uncertainty

Determine target by estimating an unknown quantity

Add extra to a known target just to be extra sure
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Efficacy Uncertainty
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Efficacy Uncertainty

• Subjects not necessarily representative of target

population
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Efficacy Uncertainty

• Subjects not necessarily representative of target

population

• Vaccine lot may not match production serial
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Efficacy Uncertainty

• Subjects not necessarily representative of target 

population

• Vaccine lot may not match production serial

• Challenge method does not usually emulate 

natural exposure
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Efficacy Uncertainty

• Subjects not necessarily representative of target

population

• Vaccine lot may not match production serial

• Challenge method does not usually emulate

natural exposure

• Experimental outcome often differs from clinical

outcome
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Efficacy Uncertainty

• Subjects not necessarily representative of target 

population

• Vaccine lot may not match production serial

• Challenge method does not usually emulate 

natural exposure

• Experimental outcome often differs from clinical 

outcome

• Clinical course of the disease not well matched 

by observation period, etc., etc.
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Efficacy Uncertainty

• Licensure may be based on a single study,

and there is no limit on number of tries
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Schoenfeld JD, Ioannidis JPA, 2012. Is everything we eat associated with cancer? American Journal of Clinical Nutrition, 97: 127-134

Do common foods prevent or cause cancer?
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Efficacy Uncertainty

DHHS-FDA USDA-CVB

Target Population Design Not required

Trial Location Clinical Manufacturer

Trials Inspected Usually Rarely

Serious Problems 2%* ?

*Saife C, 2015. Research misconduct identified by the US Food and Drug Administration.

JAMA Intern Med, online 2015.02.09
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Observed Efficacy

• Random variable

 Not a constant
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Observed Efficacy

• Random variable

 Not a constant

• Measured with uncertainty

 Not omniscience
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

PES Observed Efficacy

• Random variable

 Not a constant

• Measured with uncertainty

 Not omniscience

 Measured only once
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Uncertainty in Stability Profile

• No stability-indicating assay

• Stability study may have only 

included pre-licensing serials

• No stability requirements for bulk 

vaccine preparations

Some Other Sources of Uncertainty

Uncertainty in PES Vaccine Potency

• Potency-efficacy relationship not 

established

• Assay may not be the same as 

release assay

Uncertainty in the Lot Release Test

• Serial release testing may be done 

on a single vial

• Biased retesting may be done

• Manufacturing consistency

And many others ...

Uncertainty in Potency Assay

• Assay probably not validated

• Potency may be function of more 

than one element

• Assay imprecision
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Can we quantify the uncertainty?
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Error = Systematic + Random

Can we quantify the uncertainty?

(BTW statistical error is only a part of uncertainty)
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Error = Systematic + Random

Bias2 Variance

Can we quantify the uncertainty?
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Quantify variance and bias?

Can only quantify variance and bias

by assuming a distribution for X

X ~ f(x)
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

But many of the uncertainties here 

are about the distributional form of X

Quantify variance and bias?

X ~ f(x)

?
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Uncertainty about the

nature of the uncertainty

Quantify variance and bias?

But many of the uncertainties here 

are about the distributional form of X
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

And ...

There are many random variables

..., W, X, Y, ...

not just one

Quantify variance and bias?
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Industry Proposal

Eliminate increment over PES altogether

Observed in pivotal efficacy study (PES)

+

Increment due to measurement uncertainty
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Proposal to Eliminate Increment
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Proposal to Eliminate Increment

0.06%
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

You don’t see this
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

You see this
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Poor relationship 

to reality
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Poor relationship 

to reality

No probabilities without sampling
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Lot Release – Draft 440 does not

use the concept of confidence or 

refer to confidence intervals
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Interval Estimation?

• Lot Release – Draft 440 does not

use the concept of confidence or 

refer to confidence intervals

• Stability

Draft 155 – prediction intervals

PQRI – tolerance intervals
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Targeted Potency

PES Observed + Uncertainty Increment
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Targeted Potency
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Aiming at the Target

Release
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Trajectory

Where is there uncertainty?

Target (throughout-dating spec)

Aim (release spec)

Trajectory (shelf life stability)
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Target Potency

Where is uncertainty considered?

Target (throughout-dating spec)

Aim (release spec)

Trajectory (shelf life stability)
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

RP ELISA

In the past, target required no increment
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

RP ELISA

In the past, target required no increment

New reference qualified in host species 

vaccination-challenge study every 2-3 years
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

RP ELISA

Extended reference dating instituted in 

2011 (VSM 800.211)

“To facilitate the development of welldesigned and rigorously validated assays”
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

RP ELISA

Extended reference dating instituted in 

2011 (VSM 800.211)

“To facilitate the development of welldesigned and rigorously validated assays”

15 years –

legacy assays with minimal monitoring

Indefinite dating –

assays with rigorous monitoring
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

RP ELISA

Extended reference dating instituted in 

2011 (VSM 800.211)

Sound assay design –

Eliminates old burdensome requirement

Through modern validation methods
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Potency too high?
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Potency too high?

• May be an important consideration
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Potency too high?

• May be an important consideration

• Lower limit can’t control for 

uncertainty at the upper limit
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Summary

Terminology

Targeted Potency

Increment

Measurement Uncertainty

Target (the only place where it is considered)

Release

Stability Profile

Data based target

All assay types

including RP ELISAs

Let’s stay real

Please don’t use unwarranted jargon
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Data-driven Criteria

• Consistent Principles

• Coherent System

Core
Targeted Potency

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

“Public Service is a Public Trust”

5CFR §2635.101