Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

CVB Potency Assay

Policy Development

Overview

David Siev

Statistics Section Leader

Center for Veterinary Biologics

April 21, 2015

“Public Service is a Public Trust”

5CFR §2635.101
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Core
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Core

• Data-driven

• Consistent

• Coherent
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Historical Background

2003
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Historical Background

2003

Archaic hodge-podge Unified framework
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Historical Background Example
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Historical Background

Efficacy?

Don’t bother, we 

know it works.

Example
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Historical Background

Efficacy?

Don’t bother, we 

know it works.

Potency?

Don’t bother, you 

couldn’t do it 

anyway.

Example
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Data-driven – The specific methods should set 

testing criteria based on experimental and 

observed data, rather than arbitrary standards.

Core
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Data-driven – The specific methods should set 

testing criteria based on experimental and 

observed data, rather than arbitrary standards.

Core

e.g. 

throughout-dating spec for live vaccines 

fixed constant

not based on data
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Consistent – The general principles should be 

consistent across all classes of assays and 

products.

Core
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Consistent – The general principles should be 

consistent across all classes of assays and 

products.

Core

e.g. 

throughout-dating spec for live vaccines

fixed constant 

2X for count assays

5X for titration assays
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Consistent – The general principles should be 

consistent across all classes of assays and 

products.

Core

e.g. 

throughout-dating spec for live vaccines

fixed constant 

2X for count assays

5X for titration assays

bacterial vaccines

viral vaccines

vid. e.g. 9 CFR §113.71, 9 CFR §113.330
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Consistent – The general principles should be 

consistent across all classes of assays and 

products.

Core

Consistent principles ≠ Inflexible application
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Coherent – The various elements of a 

potency testing system must work together 

in a coordinated way.

Core
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Core

e.g. 

No assay validation requirement before mid-2008

• Approximately 75% products licensed before 

validation 

• Many of the 25% licensed since then have been 

grandfathered in with existing assays

• Coherent – The various elements of a 

potency testing system must work together 

in a coordinated way.
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

e.g. 

No assay validation requirement before mid-2008

• Approximately 75% products licensed before 

validation 

• Many of the 25% licensed since then have been 

grandfathered in with existing assays

Approximately 

1,850 licensed products

 700 distinct antigenic fractions

 90 validated potency assays

(many more incrementally improved)
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Core

Immortal vaccine licenses

Contemporary science

• Coherent – The various elements of a 

potency testing system must work together 

in a coordinated way.
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Core

• Data-driven Criteria

• Consistent Principles

• Coherent System
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal – Current – Proposed

Lot release specifications

(aka serial release)
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal – Current – Proposed

• Potency-efficacy relationship

• Targeted potency

• Acceptance criteria

Lot release specifications

(aka serial release)
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal

Establish functional 

relationship between the 

potency measurement 

and the efficacy 

response.

The potency-efficacy relationship 

can only be properly estimated from 

a set of studies that is designed for 

that purpose.

Potency-Efficacy Relationship
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal

Establish functional 

relationship between the 

potency measurement

and the efficacy 

response.

The potency-efficacy relationship 

can only be properly estimated from 

a set of studies that is designed for 

that purpose.

Potency-Efficacy Relationship
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal

Establish functional 

relationship between the 

potency measurement 

and the efficacy 

response.

The potency-efficacy relationship 

can only be properly estimated from 

a set of studies that is designed for 

that purpose.

Potency-Efficacy Relationship
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current

Establish functional 

relationship between the 

potency measurement 

and the efficacy 

response.

Assume there is a 

relationship of some kind 

between the potency test 

and efficacy. 

The potency-efficacy relationship 

can only be properly estimated from 

a set of studies that is designed for 

that purpose.

Potency measurement not explicitly 

associated with a specific level of 

efficacy - no basis for quantitative 

conclusions.

Potency-Efficacy Relationship
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current Proposed

Establish functional 

relationship between the 

potency measurement 

and the efficacy 

response.

Assume there is a 

relationship of some kind 

between the potency test 

and efficacy. 

Show a plausible 

conceptual presentation 

of the relationship, but 

not a mathematical 

characterization. 

The potency-efficacy relationship 

can only be properly estimated from 

a set of studies that is designed for 

that purpose.

Potency measurement not explicitly 

associated with a specific level of 

efficacy - no basis for quantitative 

conclusions.

First step in potency assay validation 

according to VSM 800.112.

Potency-Efficacy Relationship
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current Proposed

Establish functional 

relationship between the 

potency measurement 

and the efficacy 

response.

Assume there is a 

relationship of some kind 

between the potency test 

and efficacy. 

Show a plausible 

conceptual presentation 

of the relationship, but 

not a mathematical 

characterization. 

The potency-efficacy relationship 

can only be properly estimated from 

a set of studies that is designed for 

that purpose.

Potency measurement not explicitly 

associated with a specific level of 

efficacy - no basis for quantitative 

conclusions.

First step in potency assay validation 

according to VSM 800.112.

Potency-Efficacy Relationship

Which guidance applies: old or new? Coherence
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal

If a functional potencyefficacy relationship 

exists and is well 

described, it can be used 

to mathematically relate 

a measurement of 

potency to a level of 

efficacy.

Cf. VSM 800.209 and Draft 

122 describe such methods.

Targeted Potency
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current

If a functional potencyefficacy relationship 

exists and is well 

described, it can be used 

to mathematically relate 

a measurement of 

potency to a level of 

efficacy.

Take the observed 

potency of the vaccine 

formulation used in the 

pivotal efficacy study and 

add a standard quantity 

to it.

Cf. VSM 800.209 and Draft 

122 describe such methods.

Targeted Potency
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current Proposed

If a functional potencyefficacy relationship 

exists and is well 

described, it can be used 

to mathematically relate 

a measurement of 

potency to a level of 

efficacy.

Take the observed 

potency of the vaccine 

formulation used in the 

pivotal efficacy study and 

add a standard quantity

to it.

Take the observed 

potency of the vaccine 

formulation used in the 

pivotal efficacy study and 

add an amount based on 

data observed in potency 

testing rather than an 

arbitrary amount.

Cf. VSM 800.209 and Draft 

122 describe such methods.

Targeted Potency
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal

Formulated so that a 

specified fraction of a lot 

meets a minimum 

specification. 

Large enough sample of the vials in 

each lot tested to reasonably depict 

the distribution of potencies among 

them. 

Acceptance sampling system makes 

it possible to formulate explicit 

probability statements about the 

vaccine.

Acceptance Criteria
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current

Formulated so that a 

specified fraction of a lot 

meets a minimum 

specification. 

Varies; often based on 

arbitrary quantities 

unrelated to manufacturing 

or assay performance.

Large enough sample of the vials in 

each lot tested to reasonably depict 

the distribution of potencies among 

them. 

Acceptance sampling system makes 

it possible to formulate explicit 

probability statements about the 

vaccine.

Lot release based on single test of 

single vial. 

Biased retesting of unsatisfactory 

test results, but not satisfactory 

ones. 

No probability statements 

warranted.

Acceptance Criteria
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current Proposed

Formulated so that a 

specified fraction of a lot 

meets a minimum 

specification. 

Varies; often based on 

arbitrary quantities 

unrelated to manufacturing 

or assay performance.

Target based on potency 

test performance to account 

for assay precision and 

manufacturing consistency. 

Release above target by an 

amount estimated from 

average potency loss during 

storage.

Large enough sample of the vials in 

each lot tested to reasonably depict 

the distribution of potencies among 

them. 

Acceptance sampling system makes 

it possible to formulate explicit 

probability statements about the 

vaccine.

Lot release based on single test of 

single vial. 

Biased retesting of unsatisfactory 

test results, but not satisfactory 

ones. 

No probability statements 

warranted.

Rewarding precise assays and 

consistent manufacturing gives the 

manufacturer greater degree of 

control over its potency 

specifications.

Acceptance Criteria
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal – Current – Proposed

Large type version
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current Proposed

Establish 

functional 

potency-efficacy

relationship

Assume there is a 

relationship of 

some kind 

between potency 

and efficacy

Show plausible 

conceptual 

potency-efficacy 

relationship

The potency-efficacy 

relationship can only be 

properly estimated from a 

set of studies that is 

designed for that purpose

Potency measurement not 

explicitly associated with a 

specific level of efficacy - no 

basis for quantitative 

conclusions

First step in potency assay 

validation according to VSM 

800.112

Potency-Efficacy Relationship
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current Proposed

potency-efficacy 

relationship 

exists and is used

to relate potency

measurement to

efficacy level

add a standard 

amount to 

observed potency

in pivotal efficacy 

study

add an amount 

based on data to 

observed potency

in pivotal efficacy 

study

Cf. VSM 800.209 and Draft 

122 describe such methods.

Targeted Potency
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal Current Proposed

Specified fraction 

of a lot meets a 

minimum 

specification

Varies; often 

based on arbitrary 

quantities

Target – potency 

test performance

Release – average 

potency loss in 

Must sample enough vials in storage

each lot to depict the 

distribution of potencies 

among them. Can make 

explicit probability 

statements about the lot.

Release based on single test 

of single vial. Biased 

retesting of unsatisfactory 

test results. No probability 

statements warranted.

Rewards precise assays and 

consistent manufacturing; 

manufacturer control over 

potency specifications.

Acceptance Criteria
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal – Current – Proposed

Why is proposed less than ideal?
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Ideal – Current – Proposed

• Regulatory pragmatism

• Number of product licenses

• Historical inertia

• Manufacturer reluctance

Why is proposed less than ideal?
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Comparison of Regulatory Systems
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Comparison of Regulatory Systems

“The current concept of the quality assurance of vaccines 

is based on the overall consistency of production, 

involving several in-process controls, rather than being 

based simply on a single lot release assay. The adherence 

to good manufacturing practices is therefore of critical 

importance...”

WHO Annex 3; also vid. 21 CFR §211.100(a)
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Comparison of Regulatory Systems

“The current concept of the quality assurance of vaccines 

is based on the overall consistency of production, 

involving several in-process controls, rather than being 

based simply on a single lot release assay. The 

adherence to good manufacturing practicesis 

therefore of critical importance...”

WHO Annex 3; also vid. 21 CFR §211.100(a)
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Comparison of Regulatory Systems
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Comparison of Regulatory Systems
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Guidelines for Potency Assays

EMA CVB

Active substance Correlation between 

potency and efficacy Crude association

Adjuvant Adjuvant testing for 

quality and quantity

Adjuvant testing not 

required

Production GMP and process 

validation

No GMP or process 

validation

Assay validation Follow VICH guidelines VSM 800.112, 

June 2008

EMA/CVMP/IWP/582970/2009, EMA/CVMP/IWP/206555/2010
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Guidelines for Setting Potency Specs

ICH/VICH CVB

Clinical

Specifications linked to 

preclinical and clinical 

studies 

Tenuous link

Assay Specifications linked to 

analytical procedures Not yet (Draft 440)

Manufacturing Specifications linked to 

manufacturing process Not yet (Draft 440)

Stability Specifications account 

for stability Informal (Draft 155)

ICH Q6B
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Guidelines for Stability

FDA, ICH, VICH CVB

Stability-Indicating 

Assay

Stability-Indicating assay

required “Validated quantitative 

analytical procedure that can detect 

changes ...”

Not yet 

(Draft 155)

Stability Protocol

Many characteristicsincluding 

potency, physicochemical measurements, 

pH, bioburden, pyrogenicity, moisture if

lyophilized, stability following 

reconstitution if lyophilized, stability 

through freeze-thaw if frozen, ...

Potency only

(Draft 155)

Stability

Monitoring On-going stability monitoring Draft 155

ICH Q1A(R2), ICH Q1E, VICH GL3(R), VICH GL51; FDA Guidance for Industry: Content and Format of Chemistry, Manufacturing and 

Controls Information and Establishment Description Information for Vaccines
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Number of Products

FDA-CBER USDA-CVB

Vaccines 79 1,211

Antibody 49 54

Diseases 26 220

• 20 of 79 CBER vaccines are influenza

• 1,211 CVB vaccines does not include 145 FFMs

Data accessed 2015.02.27

http://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProducts/ucm093833.htm

http://www.fda.gov/downloads/BiologicsBloodVaccines/UCM149970.pdf
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Risk Assessment – Decision Analysis
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Risk Assessment – Decision Analysis

Example

Sub-potent Serial

Vaccine Manufacturer

Low, because all vaccine 

is gone long before end 

of dating.

(But don’t shorten my dating 

period, because we need it to 

market the product.)
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Risk Assessment – Decision Analysis

Example

Sub-potent Serial

Vaccine Manufacturer Vaccine User

Low, because all vaccine 

is gone long before end 

of dating.

High, because vaccine is 

used throughout its dating 

period.

(But don’t shorten my dating 

period, because we need it to 

market the product.)

(Check your medicine cabinet 

tonight.)
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Risk Assessment – Decision Analysis

Events

Value

Action
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Risk Assessment – Decision Analysis

Events Probability Distribution

Value Utility Function (Cost / Benefit)

Action Decision Rule (Optimization)
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Risk Assessment – Decision Analysis

Events Probability Same for All

Value Utility Function Different for Each

Action Decision Rule Different for Each
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Terminology

Potency is a quantitative measure of the activity of a 

vaccine lot as measured in a validated potency assay. 
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Potency is a quantitative measure of the activity of a 

vaccine lot as measured in a validated potency assay. 

• Activity may be measured in various ways

• Immunochemical

• Cellular

• Clinical

• Component concentration

Terminology
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Potency is a quantitative measure of the activity of a 

vaccine lot as measured in a validated potency assay. 

• Activity may be measured in various ways

• Immunochemical

• Cellular

• Clinical

• Component concentration

Terminology

• Ideally related to efficacy
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

• Often assumed to be a function of a single input 

for simplicity

• Vaccine activity may actually be a function of 

several inputs, e.g.

• Antigenic epitopes

• Innate immunity stimulants

• Adjuvants

Terminology

Potency is a quantitative measure of the activity of a 

vaccine lot as measured in a validated potency assay. 
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Core

• Data-driven Criteria

• Consistent Principles

• Coherent System
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

Summary

• Core

• Context – framework

• Ideal-Current-Proposed

• Other regulatory systems

• Risk assessment

• Potency
Overview of Potency Assay Policy Development

D. Siev “Public Service is a Public Trust” – 5CFR §2635.101 2015-04-21

End

“Public Service is a Public Trust”

5CFR §2635.101