Meeting Reports

Strategies for Achieving Measurement Assurance for

Cell Therapy Products

CARL G. SIMON JR., SHENG LIN-GIBSON, JOHN T. ELLIOTT, SUMONA SARKAR, ANNE L. PLANT

Key Words. Biological assay x Cell- and tissue-based therapy x Quality control x Reference standards

ABSTRACT

The cell therapy industry has identified the inability to reliably characterize cells as possibly its greatest challenge and has called for standards and reference materials to provide assurance for measurements of cell properties. The challenges in characterization of cell therapy products can be largely

addressed with systematic approaches for assessing sources of uncertainty and improving confidence

in key measurements. This article presents the many strategies that can be used to ensure measurement confidence and discusses them in terms of how they can be applied to characterization of cell

therapy products. Application of these strategies to cell measurements will help to establish qualified

assays for cell characterization, which may help streamline regulatory approval and enable more efficient development of cell therapy products. STEM CELLSTRANSLATIONALMEDICINE 2016;5:705–708

SIGNIFICANCE

The regenerative medicine industry has identified the lack of reliable methods for measuring critical

cell attributes as possibly the single greatest challenge facing the field. There are many strategies for

achieving measurement assurance, or confidence in cell assays, which can streamline regulatory approval and enable more efficient development of cell-based therapies.

INTRODUCTION

A recent survey conducted by the Alliance for Regenerative Medicine found that “product consistency

and lack of standards is possibly the single greatest

challenge facing the field” [1]. Similarly, a Regenerative Medicine Foundation survey concluded that the

“lack of reference materials to benchmark measurements and validation criteria for critical assays” is the

biggest roadblock for the field [2], and the National

Cell Manufacturing Consortium identified the need

for “product quality standards” as critical to the success of theindustry [3].At the root of these challenges

is the need for quantitative, validated, and robust assays for characterizing cell therapy products (CTPs).

All cell therapy products, whether mesenchymal stem cells, T cells, or embryonic or induced

pluripotent-derived cells, require reliable, robust

measurements for product characteristics that

support informed decision-making during their

development, manufacturing, and regulation. A

workshop, “Strategies to Achieve Measurement

Assurance for Cell Therapy Products,” was held

at the National Institute of Standards and Technology (NIST) in Gaithersburg, Maryland, on May

11–12, 2015 [4], to convene industry, academia,

regulators, and funding agencies (supplemental

online Fig. 1a) and begin to examine approaches

for improving confidence in themeasurements that

are necessary for bringing CTPs tomarket (agenda is

shown in supplemental online File 1 and breakout

session templates in supplemental online File 2).

The workshop focused on strategies for identifying,

monitoring, and mitigating the sources of variability

in the measurement process of cell assays. These

strategies are the basis of what we refer to as measurement assurance. These measurement assurance

approaches align with US Food and Drug Administration guidance on assay development [5], and an

understanding of these principles can help to improve the quality and reliability of cell therapy

product characterization.

STANDARDS ARE UNDERPINNED BY

MEASUREMENT ASSURANCE

There is general agreement that the development

and regulatory approval process of CTPs can be accelerated by standardized measurement protocols

forassays.Standard testmethodswill have themost

benefit to the wider industry if they have broad application to many CTPs. In many cases, there is no

need for wider standardization of product-specific

methods that can be locally optimized and validated.

This would hold true for a highly specific potency

assay that applies to a single product. For broadly applicable methods that are candidates for wider standardization, the protocols must be amenable to

Biosystems and Biomaterials

Division, National Institute of

Standards and Technology,

Gaithersburg, Maryland, USA

Correspondence: Carl G. Simon

Jr., Ph.D., National Institute of

Standards and Technology,

Biosystems and Biomaterials

Division, 20899 Bureau Drive,

Gaithersburg, Maryland 20832,

USA. Telephone: 301-975-8574;

E-Mail: carl.simon@nist.gov

Received September 28, 2015;

accepted for publication January

18, 2016; published Online First

on April 21, 2016.

©AlphaMed Press

1066-5099/2016/$20.00/0

http://dx.doi.org/

10.5966/sctm.2015-0269

STEM CELLS TRANSLATIONAL MEDICINE 2016;5:705–708 www.StemCellsTM.com ©AlphaMed Press 2016

MEETING REPORTS
validation with sound measurement science. However, it is often unclear how to apply measurement science tools to cell assays, given

the dynamic, complex, and heterogeneous nature of cells. Therefore,

it may be premature to declare specific protocols as standards for

the CTP field.In theabsence of standardized protocols,measurement

assurance strategies and tools can be applied to any measurement

process to assess the confidence in the assay result. The successful

use of measurement assurance strategies to establish highly qualified assays is the first step to realizing “standardized assays.”

TOOLS FOR ACHIEVING MEASUREMENT ASSURANCE

The concepts of measurement assurance have been applied in many

chemical and physical measurements [6], and herein the application

of these concepts to complex biological systems is discussed. Many

analytical measurements are underpinned by stable reference materials (such as a known amount of lead in an aqueous solution), which

provide a true value for an analyte. A laboratory can use that reference material to determine the amount of substance in their unknown sample. Such reference materials with a known value of an

analyteare tools thatprovidea benchmark foraninstrument reading,

for determining limit of detection, or for an assay response function.

Because the usefulness of a reference material is that it provides a

result that is constant over time, inert materials are most often used.

When cells are being considered as reference materials [7], the

potential instability of living materials presents a challenge. By definition, living systems are capable of “adaption to the environment”

[8], which is a disadvantageous property for a referencematerial that

should have stable properties [9]. In addition, the lack of reliable cell

measurements highlighted by the CTP industry [1–3]may complicate

the process forestablishing the referencevalues for the cell reference

material. Cell preparations often serve as reference materials for inhousemeasurements,and celllines have been proposedas reference

materials to enable comparability of cell preparations [9] across laboratories.Foracademiclaboratoriesengagedin cell therapy research,

a cell reference material could be useful for serving as a reference

point between different laboratories for comparing data. This would

require that the cell reference material respond according to specifications under the diverse culture conditionsin differentlaboratories

and after shipping, storing, thawing, expansion, and passaging.

For clinical manufacturing, cell reference materials could be useful for reducing variability in the CTP characterization assays that use

cellsas part of themeasurement system.Anexample discussedat the

workshop is the tubulogenesis assay [10], in which the in vitro ability

of primary vascular endothelial cells to form tubules when incubated

in CTP-conditioned medium is measured as an indicator of the angiogenic potential of the CTP. The variability in the performance of the

vascular endothelial cells in this assay impedes use of the tubulogenesis assay for CTP characterization. A cell reference material that reproducibly and reliably formed tubules may help reduce variability in

this assay so that it could be useful for CTP characterization. Thus,

development of a cell reference material could be advantageous

for a number of uses, although careful consideration will be required

to develop cell reference materials that are fit for purpose.

In the absence of reference materials, other strategies can be

used to achieve measurement assurance in cell assays, such as

recognizing and mitigating the sources of uncertainty, generating

performance specifications through experimental process controls and other control experiments, and collecting objective evidence that adds to the confidence in the measurement process

and the final assay result. These strategies provide a strong basis

for documentary standards and for decision-making (Fig. 1A).

There is no one tool, method, or process for achieving measurement assurance; many approaches can be used. The following

presents some of the tools for achieving measurement assurance.

c Measurement process flow diagram: By breaking down the measurement process into its component steps at an appropriate

level of granularity, each step is identified and considered as a

contributor to variability in the assay result. Ideally, this tool allows a developer to consider how step in a protocol contributes

to assay variability and how controls applied at each step may be

used to monitor and reduce variability in the assay result.

c Ishikawa (cause-and-effect) diagram: This graphical tool allows

the identification of the parts of the process that are potential

sources of variability. These sources of variability can influence

several steps in a flow diagram (e.g., pipetting) or be specific to

a particular step in an assay, such as the use of an instrument.

c Design of experiments: Not all of the likely sources of variability

will contribute equally, and the relative effect of the various

sources may be determined via a systematic design of experiments. Performing experiments in which assay parameters are

varied can be called “sensitivity testing” or “ruggedness testing” because the goal is to identify sensitive assay parameters

that must be controlled to make the assay more reproducible

under different laboratory conditions.

c Reference materials: Reference materials enable traceability

between measurements at different times and places. Reference materials can be of nonbiological or biological origin

and can be used to calibrate or benchmark an instrument, or

to provide an expected value in the measurement process.

c Process controls: Process controls are procedures to monitor

critical control points in an assay and to check that assay steps

are performing according to specifications. These control values should be documented and charted.

c Performance specifications: When an assay is well characterized,

its characteristics, such as the process control results, the standard

deviation between replicates, the lower limit of detection, and

range of linearity of response,must be within a user-defined range

of values to be acceptable. A suitable acceptance range (e.g., performance specifications) is established by tracking assay performance over time (charting). Documenting that specifications

aremet each time an assay is performed is evidence for validation.

If theassay characteristicsare outside of the specifications, the test

resultmay not bevalid for usein decision-making (e.g.,lot release).

c Interlaboratory comparison study: Sources of variability may be

difficult to recognizewhen a procedure is followed rigorously by a

small number of individuals in a confined environment. One way

to determine the robustness of an assay protocol to unexpected

changes in assay conditions is through interlaboratory comparison studies. These studies can be designed to test various aspects

of the measurement process, such as incubation times, manual

agitation, cell culture procedures, reagent source, or storage conditions, by having different laboratories perform the same measurement. The data from an interlaboratory comparison study

can result in refinement of an assay protocol to account for newly

recognized sources of uncertainty, inclusion of a reference material that provides traceability between laboratories, or establishment of performance specifications. Interlaboratory studies

are typically used toassess the reproducibility of a protocol across

laboratories during the development of documentary standard test methods. CTP companies may use interlaboratory

706 Measurement Assurance for Cell Therapy Products

©AlphaMed Press 2016 STEM CELLS TRANSLATIONAL MEDICINE
comparability studies when trying to achieve reproducibility

across multiple manufacturing/analytical sites.

c Orthogonal measurements: When two or more different measurement methods are used to make a measurement of the

same sample and the results agree, this provides confidence

in the accuracy of each method.

APPLYING MEASUREMENT ASSURANCE TOOLS

Cell counting is the most fundamental cell measurement, and all cell

therapymanufacturersconductcellcountingmeasurements.Furthermore, cell counting/viability was the cell measurement that was the

most frequently identified as “most in need of improvement” by an

informal survey of 23 key stakeholders (supplemental online Fig.

1b). An example of how to use a flow diagram for a cell counting

and/or viability assay is shown in Figure 1B. The assay is depicted

as a series of discrete steps in a measurement process. Each step

has the potential to be a source of variability. At each step, process

control measurements can be used to assess the operation of that

step. For example, a “sample collection” step could include processes

such as trypsinization, collection from microcarriers, sampling from a

bioreactor, collection from cryo-storage, or clinical sample collection.

If a method for cell counting involves removing cells from a substrate,

the details of the process for carrying out this step will likely affect the

efficiency of cell removal, which can influence the cell number, and

therefore can be a significant source of variability. Figure 1B indicates

thatanintermediatemetric for this step could beanevaluation of how

many cells remain attached to the substrate after trypsin treatment.

The process of disaggregation of cell aggregates during a

“sample preparation” step can also strongly affect the accuracy

and reproducibility of the measurement process; poor disaggregation might result in undercounting, and too vigorous a disaggregation step can damage cell integrity. If the cell counting assay is

coupled with cell viability, then this step is likely to be a highly significant source of variability in the assay result. For example, a

control experiment could determine the relationship between the

disaggregation procedure and the number of viable cells counted.

Light scattering or microscopy could be used to determine the

range of sizes of cell aggregates in the solution under conditions

where the maximum number of viable cells was present. This

measurement could be applied as a process control to provide assurance that the disaggregation step was executed as expected.

A “sampling” step might include dilution or concentration of

the sample. A process control for preparing dilutions could involve weighing the aliquots to provide assurance of the accuracy

and precision of that pipetting step. This would allow assessment

Figure 1. Many tools are available for achieving measurement assurance. (A): Measurement assurance is achieved by identifying and mitigating the sources of uncertainty in the measurement process. This enables informed decision-making and underpins the development of documentary standards. (B): Flow diagram of generic and specific steps in a cell counting assay. Examples of process controls are shown between

several steps. Others can be added.

Simon, Lin-Gibson, Elliott et al. 707

www.StemCellsTM.com ©AlphaMed Press 2016
of the contribution of the variability in the dilution procedure

to the total variability in the assay.

Although control experiments such as these are probably used

in most laboratory settings, formalizing the process acknowledges

the need for dedicating time and resources for developing an assay

protocol and helps to ensure that significant sources of variability

will not be overlooked. Control measurements such as these are

useful for developing a new assay or examining a unique cell sample, but practical considerations will determine how often they will

be applied. If an assay is particularly critical and is highly sensitive to

a particular step in the protocol, then using a process control measurement for that step every time the assay is run may be advised.

Reporting ancillary experimental data and the use of reference materials and process controls in addition to the assay results enable

evaluation of the confidence in the result of the assay. These data

also provide confidence to others (such as regulators or investors)

that the measurement process is well controlled.

The potential sources of variability for the cell counting process

can also be organized in an Ishikawa (cause-and-effect) diagram

(supplemental online Fig. 2), which provides a graphical method to

examine the factors thataremajor sources of variabilityand how they

are related to one another. Applying a formal process for assessing

sources of variability in an assay can help to reveal weakness in protocols. For example, an assay such as the tubulogenesis assay for testing in vitro angiogenic activity entails culturing vascular endothelial

cells on an extracellularmatrix and assessing the formation of tubules

in response to the conditioned culture medium from the CTP [10].

One step in the process involves preparation of extracellular matrix,

another step involves culturing endothelial cells on the matrix, a

third step is treatment with conditioned medium, and a fourth step

is assessment of tubulogenesis. The source of extracellular matrix

material is challenged with inconsistent composition and activity.

The sourcing of the endothelial cells is problematic because of variability in the tubule-forming activity of different cell preparations.

The scoring of tubule formation is a manual process that is inconsistent because of operator bias and variability. At each step, there

are significant challenges to controlling the sources of variability

and achieving adequate control over the measurement process.

FUTURE DIRECTIONS

Additional discussions within the cell therapy development community could help to develop measurement assurance strategies

and protocols for specific assays that can be validated. A potential

role for NIST could be to help coordinate these discussions. Cell

counting and cell viability, which are fundamental to characterizing almost any CTP, may be appropriate topics for broad collaboration in precompetitive space. Standards organizations, such

as ISO Technical Committee 276 Biotechnology and ASTM International Committee F04 Tissue Engineered Medical Products, are

potential partners for converting these community-developed

protocols into documentary standards for CTPs.

ACKNOWLEDGMENTS

We are grateful to our steering committee for helping us to plan the

workshop: Fouad Atouf (U.S. Pharmacopeial Convention), Kyle

Kolaja (Cellular Dynamics International), Michael McCaman (Lonza),

Steven Oh (U.S. Food and Drug Administration), Darwin Prockop

(Texas A&M), Pamela Robey (National Institutes of Health), and

Jon Rowley (RoosterBio). We also thank the speakers, breakout session organizers, and panel members for their thoughtful insight:

Eytan Abraham (Lonza), Jessica Carmen (Alliance for Regenerative

Medicine), Janet Davis (Janssen Research and Development),

Michael Halter (NIST), Sandro Matosevic (Akron Biotech), Michael

Mendicino (Mesoblast), Sagi Nahum (Pluristem), Yael Porat (BioGenCell), Mahendra Rao (Q-Therapeutics), Ivan Rich (Hemogenix),

Marc Salit (NIST), Christopher Wiwi (CelGene), Juliana Woda (Juventas Therapeutics), and Claudia Zylberberg (Akron Biotech).

AUTHOR CONTRIBUTIONS

C.G.S., S.L.-G., J.T.E., S.S., and A.L.P.: conception and design,

manuscript writing.

DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST

The authors indicated no potential conflicts of interest.

REFERENCES

1 Alliance for Regenerative Medicine. Pharma

and Biotech Survey, 2014. Available at http://

alliancerm.org/sites/default/files/ARM_Pharma_

SurveyRept_Mar2014_e.pdf.AccessedSeptember

23, 2015.

2 Hunsberger J, Harrysson O, Shirwaiker R

et al. Manufacturing road map for tissue engineering and regenerative medicine technologies. STEM CELLS TRANSLATIONAL MEDICINE 2015;

4:130–135.

3 National Cell Manufacturing Consortium.

Achieving Large-Scale, Cost-Effective, Reproducible Manufacturing of High-Quality Cells: A

Technology Roadmap to 2025, 2016. Available

at https://cdn.shopify.com/s/files/1/0515/

3253/files/CMC_Roadmap_021716_150ppi.

pdf?2424417672465838090. Accessed March

28, 2016.

4 U.S. National Institute of Standards and

Technology.Workshop on Strategies to Achieve

Measurement Assurance for Cell Therapy Products, Gaithersburg, MD, 2015. Available at http://www.nist.gov/mml/bbd/

biomaterials/measurement-assurance-for-celltherapy-products.cfm. Accessed September

23, 2015.

5 U.S. Food and Drug Administration. Guidance for industry potency tests for cellular

and gene therapy products, 2011. Available at

http://www.fda.gov/BiologicsBloodVaccines/

GuidanceComplianceRegulatoryInformation/

Guidances/default.htm. Accessed September

23, 2015.

6 Hibbert DB. Quality Assurance for the Analytical Chemistry Laboratory. New York, NY:

Oxford University Press, 2007.

7 Viswanathan S, Keating A, Deans R et al.

Soliciting strategies for developing cell-based

reference materials to advance MSC research

and clinical translation. Stem Cells Dev 2014;

23:1157–1167.

8 The American Heritage Stedman’s Medical

Dictionary. Boston: Houghton Mifflin Co., 1995.

9 International Vocabulary of Metrology—

Basic and General Concepts and Associated

Terms (VIM), 3rd ed. Paris, France: Bureau International des Poids et Mesures, 2012:50.

10 Porat Y, Abraham E, Karnieli O et al. Critical elements in the development of cell therapy

potency assays for ischemic conditions. Cytotherapy 2015;17:817–831.

See www.StemCellsTM.com for supporting information available online.

708 Measurement Assurance for Cell Therapy Products

©AlphaMed Press 2016 STEM CELLS TRANSLATIONAL MEDICINE