U.S. Department of 

Health and Human Services

Agency for Healthcare Research and Quality

540 Gaither Road

Rockville, MD 20850

AHRQ Pub. No. 13(14)-EHC111

April 2014

ISBN: 978-1-58763-432-1
The Effective Health Care Program of the Agency for Healthcare Research and Quality (AHRQ) conducts 

and supports research focused on the outcomes, effectiveness, comparative clinical effectiveness, and 

appropriateness of pharmaceuticals, devices, and health care services. More information on the Effective 

Health Care Program can be found at www.effectivehealthcare.ahrq.gov. This report was produced under 

contract to AHRQ by the Outcome DEcIDE Center (Developing Evidence to Inform Decisions about 

Effectiveness) under Contract No. 290-2005-00351 TO7. The AHRQ Task Order Officer for this project 

was Elise Berliner, Ph.D. 

The findings and conclusions in this document are those of the authors, who are responsible for its 

contents; the findings and conclusions do not necessarily represent the views of AHRQ or the U.S. 

Department of Health and Human Services. Therefore, no statement in this report should be construed as 

an official position of AHRQ or the U.S. Department of Health and Human Services.

Copyright Information:

©2014 United States Government, as represented by the Secretary of the Department of Health and 

Human Services, by assignment. All rights reserved. The Agency for Healthcare Research and Quality 

(AHRQ) permits members of the public to reproduce, redistribute, publicly display, and incorporate this 

work into other materials provided that it must be reproduced without any changes to the work or portions 

thereof, except as permitted as fair use under the U.S. Copyright Act. This work contains certain tables 

and figures noted herein that are subject to copyright owned by third parties. These tables and figures 

may not be reproduced, redistributed, or incorporated into other materials independent of this work 

without permission of the third-party copyright owner(s). This work may not reproduced, reprinted, or 

redistributed for a fee, nor may the work be sold for profit or incorporated into a profit-making venture 

without the express written permission of AHRQ. This work is subject to the restrictions of Section 

1140 of the Social Security Act, 42 U.S.C. § 1320b-10. When parts of this work are used or quoted, the 

following citation should be used:

Citation:

Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third 

edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles 

company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, 

MD: Agency for Healthcare Research and Quality. April 2014. http://www.effectivehealthcare.ahrq.gov/

registries-guide-3.cfm.
Registries for Evaluating Patient Outcomes: A User’s Guide

Third Edition 

Volume 2

Prepared for:

Agency for Healthcare Research and Quality

U.S. Department of Health and Human Services

540 Gaither Road

Rockville, MD 20850

www.ahrq.gov

Contract No. 290-2005-00351 TO7

Prepared by:

Outcome Sciences, Inc., A Quintiles Company

Cambridge, MA

Senior Editors

Richard E. Gliklich, M.D.

Nancy A. Dreyer, M.P.H., Ph.D.

Editor

Michelle B. Leavy, M.P.H.

AHRQ Publication No. 13(14)-EHC111

April 2014

iii

Contents Overview

Volume 1

Executive Summary ................................................................................................................................... 1

Section I. Creating Registries.................................................................................................................. 11

Chapter 1. Patient Registries...................................................................................................................... 13

Chapter 2. Planning a Registry .................................................................................................................. 29

Chapter 3. Registry Design........................................................................................................................ 47

Chapter 4. Data Elements for Registries.................................................................................................... 73

Chapter 5. Use of Patient-Reported Outcomes in Registries..................................................................... 93

Chapter 6. Data Sources for Registries.................................................................................................... 127

Section II. Legal and Ethical Considerations for Registries.............................................................. 143

Chapter 7. Principles of Registry Ethics, Data Ownership, and Privacy ................................................. 145

Chapter 8. Informed Consent for Registries............................................................................................ 187

Chapter 9. Protecting Data: Confidentiality and Legal Concerns of Providers, Manufacturers, 

and Health Plans............................................................................................................................... 213

Section III. Operating Registries.......................................................................................................... 233

Chapter 10. Recruiting and Retaining Participants in the Registry ......................................................... 235

Chapter 11. Data Collection and Quality Assurance ............................................................................... 251

Chapter 12. Adverse Event Detection, Processing, and Reporting.......................................................... 277

Chapter 13. Analysis, Interpretation, and Reporting of Registry Data To Evaluate Outcomes............... 291

Chapter 14. Modifying and Stopping Registries...................................................................................... 315

Volume 2

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry Data With 

Other Data Sources............................................................................................................................. 1 

Chapter 15. Interfacing Registries With Electronic Health Records ........................................................... 3

Chapter 16. Linking Registry Data With Other Data Sources To Support New Studies........................... 23

Chapter 17. Managing Patient Identity Across Data Sources.................................................................... 47

Chapter 18. Analysis of Linked Registry Data Sets................................................................................... 65

Section V. Special Applications in Patient Registries............................................................................. 89

Chapter 19. Use of Registries in Product Safety Assessment.................................................................... 91

Chapter 20. Rare Disease Registries........................................................................................................ 113

Chapter 21. Pregnancy Registries............................................................................................................ 135

Chapter 22. Quality Improvement Registries........................................................................................... 171

Chapter 23. Registries for Medical Devices ............................................................................................ 199

Chapter 24. Public-Private Partnerships .................................................................................................. 215

Authors
iv

Section VI. Evaluating Registries............................................................................................................239

Chapter 25. Assessing Quality ...................................................................................................................241

Contributors.............................................................................................................................................253

Reviewers..................................................................................................................................................255

Case Example Contributors....................................................................................................................259

Contributor and Reviewer Affiliations ..................................................................................................263

Appendixes................................................................................................................................................281

Appendix A. An Illustration of Sample Size Calculations.........................................................................283

Appendix B. Copyright Law ......................................................................................................................289

Appendix C. Relevant Entities in Health Information Technology Standards...........................................291

Appendix D. Linking Clinical Registry Data With Insurance Claims Files ..............................................295

Developing an Observational CER Protocol: A User’s Guide
v

Contents—Volume 2

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry Data With 

Other Data Sources............................................................................................................................. 1 

Chapter 15. Interfacing Registries With Electronic Health Records.................................................... 3

1. Introduction ...................................................................................................................................... 3

2. EHRs and Patient Registries............................................................................................................. 4

3. EHRs and Evidence Development.................................................................................................... 5

4. Current Challenges in a Preinteroperable Environment ................................................................... 5

5. The Vision of EHR-Registry Interoperability................................................................................... 6

6. Interoperability Challenges............................................................................................................... 7

7. Partial and Potential Solutions.......................................................................................................... 8

8. Momentum Toward a Functional Interoperability Solution.............................................................. 9

9. The Next Increment ........................................................................................................................ 12

10. What Has Been Done.................................................................................................................... 13

11. Distributed Networks.................................................................................................................... 14

12. Summary....................................................................................................................................... 14

Case Examples for Chapter 15 ............................................................................................................... 15

Case Example 32. Using System Integration Software To Capture Registry Data From Electronic 

Health Records ................................................................................................................................... 15

Case Example 33. Creating a Registry Interface To Incorporate Data From Multiple Electronic 

Health Records ................................................................................................................................... 16

Case Example 34. Technical and Security Issues in Creating a Health Information Exchange......... 17

Case Example 35. Developing a New Model for Gathering and Reporting Adverse Drug Events ... 19

References for Chapter 15....................................................................................................................... 21

Chapter 16. Linking Registry Data With Other Data Sources To Support New Studies.................. 23

1. Introduction .................................................................................................................................... 23

2. Technical Aspects of Data Linkage Projects................................................................................... 24

3. Legal Aspects of Data Linkage Projects......................................................................................... 28

4. Risk Mitigation for Data Linkage Projects..................................................................................... 31

5. Legal and Technical Planning Questions........................................................................................ 35

6. Summary......................................................................................................................................... 38

Case Examples for Chapter 16 ............................................................................................................... 38

Case Example 36. Linking Registries at the International Level ....................................................... 38

Case Example 37. Linking a Procedure-Based Registry With Claims Data To Study Long-Term 

Outcomes............................................................................................................................................ 40

Case Example 38. Linking Registry Data To Examine Long-Term Survival..................................... 41

Case Example 39. Linking Longitudinal Registry Data to Medicaid Analytical Extract Files.......... 42

Contents
vi

References for Chapter 16....................................................................................................................... 43

Chapter 17. Managing Patient Identity Across Data Sources.............................................................. 47

1. Introduction .................................................................................................................................... 47

2. PIM Strategies ................................................................................................................................ 48

3. Emerging Strategies and Related Ideas .......................................................................................... 51

4. Major Challenges and Barriers....................................................................................................... 55

5. Summary......................................................................................................................................... 57

Case Examples for Chapter 17 ............................................................................................................... 58

Case Example 40. Integrating Data From Multiple Sources With Patient ID Matching.................... 58

Case Example 41. Using Patient Identity Management Methods To Combine Health System Data .... 60

References for Chapter 17....................................................................................................................... 61

Chapter 18. Analysis of Linked Registry Data sets............................................................................... 65

1. Introduction .................................................................................................................................... 65

2. Fundamentals of Design and Analysis in Retrospective Database Research ................................. 65

3. Important Considerations ............................................................................................................... 69

4. Special Opportunities ..................................................................................................................... 76

5. Summary......................................................................................................................................... 80

Case Examples for Chapter 18 ............................................................................................................... 81

Case Example 42. Combining De-Identified Data From Multiple Registries To Study Long-Term 

Outcomes in a Rare Disease ............................................................................................................... 81

Case Example 43. Understanding Baseline Characteristics of Combined Data Sets Prior 

to Analysis .......................................................................................................................................... 83

References for Chapter 18....................................................................................................................... 84

Section V. Special Applications in Patient Registries............................................................................. 89

Chapter 19. Use of Registries in Product Safety Assessment............................................................... 91

1. Introduction .................................................................................................................................... 91

2. Registries Specifically Designed for Safety Assessment................................................................ 94

3. Registries Designed for Purposes Other Than Safety................................................................... 101

4. Ad Hoc Data Pooling .................................................................................................................... 102

5. Signal Detection in Registries and Observational Studies ........................................................... 103

6. Potential Obligations for Registry Developers in Reporting Safety Issues.................................. 105

7. Summary....................................................................................................................................... 106

Case Examples for Chapter 19 ............................................................................................................. 107

Case Example 44. Using a Registry To Assess Long-Term Product Safety..................................... 107

Case Example 45. Using a Registry To Monitor Long-Term Product Safety................................... 109
Contents

vii

References for Chapter 19..................................................................................................................... 110

Chapter 20. Rare Disease Registries..................................................................................................... 113

1. Introduction .................................................................................................................................. 113

2. Genesis of a Rare Disease Registry.............................................................................................. 114

3. Implementation of a Rare Disease Registry ................................................................................. 118

4. The Future of Rare Disease Registries.......................................................................................... 125

Case Examples for Chapter 20 ............................................................................................................. 127

Case Example 46. Using Registries To Understand Rare Diseases.................................................. 127

Case Example 47. Studying Rare Diseases in an Existing Registry Population .............................. 130

Case Example 48. Site Motivation and Retention in Rare Disease Registries................................. 132

References for Chapter 20..................................................................................................................... 133

Chapter 21. Pregnancy Registries ........................................................................................................ 135

1. Introduction .................................................................................................................................. 135

2. Justification................................................................................................................................... 135

3. Pregnancy Registry Objectives..................................................................................................... 137

4. Design........................................................................................................................................... 137

5. Study Population: Who and When................................................................................................ 138

6. Enrollment and Followup.............................................................................................................. 138

7. Exposure Ascertainment............................................................................................................... 139

8. Exposure Definition...................................................................................................................... 140

9. Covariates: What Else To Collect?................................................................................................ 140

10. Outcome Ascertainment ............................................................................................................. 142

11. Outcome(s) Definition ................................................................................................................ 142

12. Reference Group(s): Internal or External, Exposed or Unexposed? .......................................... 144

13. Analysis of Registry Data ........................................................................................................... 146

14. Statistical Power, Registry Size, and Duration ........................................................................... 146

15. Biases.......................................................................................................................................... 147

16. External Validity or Generalizability .......................................................................................... 149

17. Operations................................................................................................................................... 149

18. Advantages of Pregnancy Registries........................................................................................... 155

19. Limitations of Pregnancy Registries........................................................................................... 156

20. Evaluation of Reports From Pregnancy Registries..................................................................... 157

21. Summary..................................................................................................................................... 158

Case Examples for Chapter 21 ............................................................................................................. 159

Case Example 49. Expanding an Ongoing Pregnancy Registry....................................................... 159

Case Example 50. Using a Pregnancy Registry To Detect Major Teratogenicity............................. 161
viii

Case Example 51. Implementing a Non-Mandated Pregnancy Registry ......................................... 163

Case Example 52. Using Proactive Awareness Activities To Recruit Patients for a Pregnancy 

Exposure Registry............................................................................................................................. 165

References for Chapter 21..................................................................................................................... 168

Chapter 22. Quality Improvement Registries...................................................................................... 171

1. Introduction .................................................................................................................................. 171

2. Planning ........................................................................................................................................ 172

3. Legal and Institutional Review Board Issues................................................................................ 174

4. Design........................................................................................................................................... 175

5. Operational Considerations .......................................................................................................... 177

6. Quality Improvement Tools .......................................................................................................... 178

7. Quality Assurance......................................................................................................................... 179

8. Analytical Considerations............................................................................................................. 180

9. Reporting to Providers and the Public .......................................................................................... 182

10. Use of QI Registry Data for Research Studies........................................................................... 184

11. Limitations of Current QI Registries.......................................................................................... 184

12. Summary..................................................................................................................................... 185

Case Examples for Chapter 22 ............................................................................................................. 186

Case Example 53. Using Recognition Measures To Develop a Data Set ......................................... 186

Case Example 54. Managing Care and Quality Improvement for Chronic Diseases....................... 187

Case Example 55. Use of Reporting Tools To Promote Quality Improvement ................................ 188

Case Example 56. Using Registries To Drive Quality Improvement in Chronic Conditions........... 190

Case Example 57. Clarifying the Federal Regulatory Requirements for Quality Improvement 

Registries .......................................................................................................................................... 191

References for Chapter 22..................................................................................................................... 193

Chapter 23. Registries for Medical Devices......................................................................................... 199

1. Introduction .................................................................................................................................. 199

2. Differences Between Drugs and Devices ..................................................................................... 199

3. Design and Data Collection Considerations................................................................................. 200

4. Regulatory Uses and Considerations............................................................................................ 204

5. Potential Uses of Emerging Technology....................................................................................... 205

6. Summary....................................................................................................................................... 205
Contents

ix

Case Examples for Chapter 23 ............................................................................................................. 206

Case Example 58. Designing a Registry To Study the Effectiveness of a Device Training Program 

for Providers ..................................................................................................................................... 206

Case Example 59. Identifying and Responding to Adverse Events Found in a Registry Database ..... 208

Case Example 60. Receiving Data From Medical Imaging Devices................................................ 210

Case Example 61. Combining Registry Data With EHR Data To Measure Real-World Outcomes 

of Implantable Devices..................................................................................................................... 211

References for Chapter 23..................................................................................................................... 213

Chapter 24. Public-Private Partnerships............................................................................................. 215

1. Introduction .................................................................................................................................. 215

2. Definition of a Public-Private Partnership .................................................................................... 215

3. PPP Models................................................................................................................................... 215

4. Considerations for Setting Up a PPPs .......................................................................................... 223

5. Evolution of PPPs......................................................................................................................... 226

6. Considerations for Managing a PPP............................................................................................. 227

7. Special Considerations for International PPPs............................................................................. 228

8. Key Factors for Success and Potential Challenges....................................................................... 228

9. Summary....................................................................................................................................... 230

Case Examples for Chapter 24 ............................................................................................................. 231

Case Example 62. Developing a Public-Private Partnership for Comparative Effectiveness 

Research............................................................................................................................................ 231

Case Example 63. Leveraging a Public-Private Partnership for a Postmarketing Commitment ...... 232

Case Example 64. Public-Private Partnerships for Rare Diseases.................................................... 234

References for Chapter 24..................................................................................................................... 236

Section VI. Evaluating Registries.......................................................................................................... 239

Chapter 25. Assessing Quality............................................................................................................... 241

1. Introduction .................................................................................................................................. 241

2. Defining Quality ........................................................................................................................... 242

3. Measuring Quality ........................................................................................................................ 242

4. Quality Domains........................................................................................................................... 243

References for Chapter 25..................................................................................................................... 249

Contributors........................................................................................................................................... 253

Reviewers................................................................................................................................................ 255

Case Example Contributors.................................................................................................................. 259

Contributor and Reviewer Affiliations................................................................................................. 263
Developing an Observational CER Protocol: A User’s Guide

x

Appendixes.............................................................................................................................................. 281

Appendix A. An Illustration of Sample Size Calculations....................................................................... 283

Appendix B. Copyright Law .................................................................................................................... 289

Appendix C. Relevant Entities in Health Information Technology Standards......................................... 291

Appendix D. Linking Clinical Registry Data With Insurance Claims Files ............................................ 295

Tables

Table 16–1. Legal Planning Questions....................................................................................................... 36

Table 16–2. Technical Planning Questions ................................................................................................ 37

Table 18–1. Types of Databases Used for Retrospective Database Studies, and Their Typical Advantages 

and Disadvantages................................................................................................................................... 71

Table 20–1. Role of Stakeholders in Rare Disease Registries................................................................. 117

Table 21–1. Variables Commonly Collected in Exposure Pregnancy Registries..................................... 141

Table 21–2. Issues To Consider When Evaluating Reports From Pregnancy Registries......................... 157

Table 22–1. Common Quality Improvement Tools.................................................................................. 178

Table 22–2. Quality Improvement Tools Implemented in Three Registries............................................. 179

Table 24–1. Regulatory, Data Quality, and Scientific Components of a Typical FDA Clinical Trial and 

INTERMACS........................................................................................................................................ 216

Table 25–1. Overview of Registry Purposes............................................................................................ 243

Table 25–2. Research Quality—Essential Elements of Good Practice for Establishing and Operating 

Registries............................................................................................................................................... 244

Table 25–3. Research Quality—Further Indicators of Quality for Establishing and Operating 

Registries (Optional)............................................................................................................................. 245

Table 25–4. Essential Indicators of Good Evidence Quality for Registries............................................. 247

Table 25–5. Further Indicators of Registry Evidence Quality (Optional)................................................ 248

Table C-1. Relevant Entities in Health Information Technology Standards ............................................ 293

Figures

Figure 15–1. A Building-Block Approach to Interoperability ................................................................... 10

Figure 15–2. Retrieve Form for Data Capture Diagram ............................................................................ 11

Figure 16–1. Relationships Among Confidentiality, Disclosure, and Harm.............................................. 29

Figure 17–1. Basic Process Flow With Patient Identifier Cross-Referencing............................................ 53

Figure 17–2. Data Flow Through a Health Information Exchange............................................................ 54

Figure 24–1. Overall Survival of Adult INTERMACS Subjects Receiving Primary Left 

Ventricular Assist Devices, by Pump Type............................................................................................ 218

Figure 24–2. Structure of INTERMACS Partnerships ............................................................................ 219
Section IV

Technical, Legal, 

and Analytic 

Considerations for 

Combining Registry 

Data With Other 

Data Sources

3

Chapter 15. Interfacing Registries With Electronic 

Health Records

1. Introduction

With national efforts to invest in electronic health 

record (EHR) systems and advance the evidence 

base in areas such as effectiveness, safety, and 

quality through registries and other studies, 

interfacing registries with EHRs will become more 

important over the next few years. While both 

EHRs and registries use clinical information at the 

patient level, registries are population focused, 

purpose driven, and designed to derive information 

on health outcomes defined before the data are 

collected and analyzed. On the other hand, EHRs 

are focused on the collection and use of an 

individual patient’s health-related information. 

While in practice there may be some overlap in 

functionality between EHRs and registries, their 

roles are distinct, and both are very important to 

the health care system. This chapter explores 

issues of interoperability and a pragmatic 

“building-block approach” toward a functional, 

open-standards–based solution. (In this context, 

“open standards” means nonproprietary standards 

developed through a transparent process with 

participation from many stakeholders. “Open” 

does not mean “free of charge” in this context—

there may be fees associated with the use of 

certain standards.)

An important value of this approach is that EHR 

vendors can implement it without major effort or 

impact on their current systems. While the focus 

of this guide is on patient registries, the same 

approach described in this chapter is applicable to 

clinical research studies, safety reporting, 

biosurveillance, public health, and quality 

reporting. This chapter also includes case 

examples (Case Examples 32, 33, 34, and 35) 

describing some of the challenges and approaches 

to interfacing registries with EHRs.

An EHR refers to an individual patient’s medical 

record in digital format. EHRs can be 

comprehensive systems that manage both clinical 

and administrative data; for example, an EHR may 

collect medical histories, laboratory data, and 

physician notes, and may assist with billing, 

interpractice referrals, appointment scheduling, 

and prescription refills. EHRs can also be targeted 

in their capabilities; many practices choose to 

implement EHRs that offer a subset of these 

capabilities, or they may implement multiple 

systems to fulfill different needs. According to the 

Institute of Medicine (IOM), an EHR has four core 

functionalities: health information and data, results 

management, order entry and support, and 

decision support.1

The current EHR market in the United States is 

highly fragmented.2 Until recently, the term 

“EHR” was broadly applied to systems falling 

within a range of capabilities. However, since the 

passage of the American Recovery and 

Reinvestment Act of 2009 (ARRA), a 

transformative change has been underway, with a 

rapid increase in EHR adoption and a strong 

emphasis on standards and certification. Under 

ARRA, approximately $27 billion will be spent on 

incentives and other projects to support the 

adoption of EHRs over the next several years.3

These incentives have spurred an increase in EHR 

implementation from 17 percent of U.S. officebased physicians in 2003 to 72 percent in 2012.4

To ensure that the EHRs implemented under the 

ARRA incentive program contain basic 

functionalities, new standards and a certification 

process have been developed. ARRA emphasizes 

the “meaningful use” of EHRs by office-based 

physicians and hospitals. Meaningful use refers to 

the use of certified EHR technology to “improve 

quality, safety, efficiency, and reduce health 

disparities; engage patients and families in their 

health care; improve care coordination; and 

improve population and public health while 

maintaining privacy and security.”5 ARRA 

describes the three main components of 

meaningful use as (1) the use of a certified EHR in 

a meaningful manner, such as e-prescribing; 

(2) the use of certified EHR technology for 

electronic exchange of health information to 

improve quality of health care, such as promoting 
4

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

care coordination; and (3) the use of certified EHR 

technology to submit clinical quality and other 

measures.6

The Office of the Secretary of Health and Human 

Services (HHS) has been charged under ARRA 

with setting standards and certification criteria for 

EHRs, with interoperability a core goal. Within 

HHS, the Office of the National Coordinator of 

Health Information Technology (ONC) is 

responsible for developing the standards and 

certification criteria for the meaningful use of 

EHRs. ONC is using a three-stage approach to 

developing criteria for meaningful use. Stage 1, 

released in 2011, sets basic standards for capturing 

data in an EHR and sharing data between systems. 

Stage 2, which is under development and 

scheduled for finalization in 2012, expands the 

basic standards to include additional functionality 

and require reporting of more measures (e.g., 

quality of care measures, base functionality 

measures). Finally, Stage 3, to be released in 2015, 

will continue to expand on the standards in Stage 

2. ONC is also developing an EHR certification 

program that will allow EHR vendors to 

demonstrate that their products contain sufficient 

functionality to support meaningful use.

Even with increasing standardization of EHRs, 

there are many issues and obstacles to achieving 

interoperability (meaningful communication 

between systems, as described further below) 

between EHRs and registries or other clinical 

research activities. Among these obstacles are 

limitations to the ability to use and exchange 

information; issues in confidentiality, privacy, 

security, and data access; and issues in regulatory 

compliance. For example, in terms of information 

interoperability and exchange, the Clinical 

Research Value Case Workgroup has observed that 

clinical research data standards are developing 

independently from certain standards being 

developed for clinical care data; that currently the 

interface between the EHR and clinical research 

data is ad hoc and can be prone to errors and 

redundancy; that there is a wide variety of modes 

of research and medical specialties involved in 

clinical studies, thus making standards difficult to 

identify; and that there are differences among 

standards developing organizations with respect to 

health care data standards and how they are 

designed and implemented (including some 

proprietary standards for clinical research within 

certain organizations). With respect to 

confidentiality, privacy, security, and data access, 

the Workgroup has pointed out that secondary use 

of data may violate patient privacy, and that 

protections need to be put in place before data 

access can be automated. In the area of regulatory 

compliance, it notes that for some research 

purposes there is a need to comply with 

regulations for electronic systems (e.g., 21 CFR 

Part 11) and other rules (e.g., the Common Rule 

for human subjects research).7

The new Federal oversight of EHR standards is 

clearly guided by the need to ensure that the EHRs 

that benefit from the market-building impact of the 

provider incentives will serve the broader public 

purposes for which the ARRA funds are intended.8

Specifically, the elusive goal that has not been 

satisfied in the current paradigm is the creation of 

an interoperable health information technology 

(HIT) infrastructure. Without interoperability, the 

HIT investment under ARRA may actually be 

counterproductive to other ARRA goals, including 

the generation and dissemination of information 

on the comparative effectiveness of therapies and 

the efficient and transparent measurement of 

quality in the health care system. Ideally, EHR 

standards will lay the groundwork for what the 

Institute of Medicine has called the “learning 

health care system.”9 The goal of a learning health 

care system is a transformation of the way 

evidence is generated and used to improve health 

and health care—a system in which patient 

registries and similar, real-world study methods 

are expected to play a very important role. 

Ultimately, the HIT standards that are adopted, 

including vocabularies, data elements, data sets, 

and technical standards, may have a far-reaching 

impact on how transformative ARRA will be from 

an HIT perspective.

2. EHRs and Patient Registries

Prior to exploring how EHRs and registries might 

interface, it is useful to clearly differentiate one 

from the other. While EHRs may assist in certain 

functions that a patient registry requires (e.g., data 
5

Chapter 15. Interfacing Registries With Electronic Health Records

collection, data cleaning, data storage), and a 

registry may augment the value of the information 

collected in an EHR (e.g., population views, 

quality reporting), an EHR is not a registry and a 

registry is not an EHR. Simply stated, an EHR is 

an electronic record of health-related information 

on an individual that conforms to nationally 

recognized interoperability standards, and that can 

be created, managed, and consulted by authorized 

clinicians and staff across more than one health 

care organization.10 As defined in Chapter 1, a 

registry is an organized system that uses 

observational study methods to collect uniform 

data (clinical and other) to evaluate specified 

outcomes for a population defined by a particular 

disease, condition, or exposure, and that serves 

one or more predetermined scientific, clinical, or 

policy purposes. Registries are focused on 

populations and are designed to fulfill specific 

purposes defined before the data are collected and 

analyzed. EHRs are focused on individuals and are 

designed to collect, share, and use that information 

for the benefit of that individual.

3. EHRs and Evidence 

Development

The true promise of EHRs in evidence 

development is in facilitating the achievement of a 

practical, scalable, and efficient means of 

collecting, analyzing, and disseminating evidence. 

Digitizing information can dramatically reduce 

many of the scalability constraints of patient 

registries and other clinical research activities. 

Paper records are inherently limited because of the 

difficulty of systematically finding or sampling 

eligible patients for research activities and the 

effort required to re-enter information into a 

database. Digitized information has the capacity to 

improve both of these requirements for registries, 

enabling larger, more diverse patient populations, 

and avoiding duplication of effort for participating 

clinicians and patients. However, duplication of 

effort can be reduced only to the extent that EHRs 

capture data elements and outcomes with specific, 

consistent, and interoperable definitions—or that 

data can be found and transformed by other 

processes and technologies (e.g., natural language 

processing) into standardized formats that match 

registry specifications. Besides enabling health 

care information to be more readily available for 

registries and other evidence development 

purposes, bidirectionally interoperable EHRs may 

also serve an efferent role of delivering relevant 

information from a registry back to a clinician 

(e.g., information about natural history of disease, 

safety, effectiveness, and quality).

4. Current Challenges in a 

Preinteroperable Environment

Data capture for research purposes, in general, can 

be challenging for clinicians. Many hospitals, 

health care facilities, and clinicians’ offices that 

participate in studies use more than one data 

capture system, and change their workflow to 

accommodate nonharmonized research demands. 

In other words, hospitals and practices are 

changing their workflow to accommodate 

nonharmonized research demands. As a result, 

data capture can be awkward and time consuming 

for clinicians and their staff, especially for a 

registry in which a large number of patients may fit 

into a broad set of enrollment criteria. While some 

of this can be overcome without interoperable 

systems by means of uploads from these systems 

to registries of certain standard file formats, such 

as hospital or clinician office billing files, the need 

to re-enter data from one system to another; train 

staff on new systems; and juggle multiple user 

names, passwords, and devices presents a high 

barrier to participation, especially for clinicians, 

whose primary interest is patient care and who are 

often resistant to change. The widespread 

implementation of EHRs that are not truly 

interoperable, coupled with the growth in current 

and future evidence development activities, such 

as patient registries, may ironically create 

significant barriers to achieving the vision of a 

national, learning health care system. In many 

respects, clinicians may be part of the problem, if 

they seek EHRs with highly customized interfaces 

and database schema rather than those that may be 

more amenable to interoperability.

Most EHRs are not fully interoperable in the core 

functions that would enable them to participate in 
6

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

the learning health care system envisioned by the 

IOM. This deficiency is directly related to a 

combination of technical and economic barriers to 

EHRs’ adoption and deployment of standardsbased interoperability solutions. There are more 

than 600 EHR vendors,11 many of which provide 

heavily customized versions of their systems for 

each client. For some time there was significant 

interest in adding clinical research capabilities to 

already implemented EHR systems,12 but this 

so-called “Swiss army knife” approach did not 

prove to be technically or commercially effective. 

Issues ranged from standardization of core data 

sets to achieving compliance with U.S. Food and 

Drug Administration (FDA) requirements for 

electronic systems used in clinical research. And 

because there is no single national EHR, even if 

this were achievable it would not meet many 

registry purposes, since registries seek data across 

large, generalizable populations. In recent years, 

the industry has primarily turned back to pursuing 

an open-standards approach to interacting with, 

rather than becoming, specialized systems.13

Appendix C describes many of the relevant 

standards and standards-setting organizations.

Even though many EHR systems are technically 

uniform, the actual software implementations are 

different in many ways. As a result, achieving 

interoperability goals (across the myriad of 

installed EHRs and current and future registries) 

through custom interfaces is a mathematical, and 

therefore economic, impossibility. (See Section 5 

below.) An open-standards approach may be the 

most viable. In addition, as has been tested in 

many demonstrations and is slowly being 

incorporated by some vendors into commercial 

offerings, a user-configurable mechanism to enable 

the provider to link to any number of registries 

without requiring customization by the EHR 

vendor is also an important aspect of a scalable 

solution.

5. The Vision of EHR-Registry 

Interoperability

As the EHR becomes the primary desktop 

interface for physicians and other health care 

workers, it is clear that registries must work 

through EHRs in order for interoperability to be 

feasible. At the same time, there is a rapidly 

growing need for clinicians to participate in 

registries to manage safety, evaluate effectiveness, 

and measure and improve quality of care. As a 

result, an EHR will need to serve as an interface 

for more than one registry simultaneously. In 

considering the need to interface EHRs with 

patient registries, it is useful to consider the 

specific purpose for which the patient registry is 

designed, and how an EHR that is interoperable 

with one or more registries might lessen the 

burden, barriers, or costs of managing the 

registries and other data collection programs. The 

following potential functions can be thought of 

with respect to a registry purpose:

• Natural history of disease: Identify patients 

who meet eligibility criteria, alert clinicians, 

present the relevant forms and instructions, 

capture uniform data, review the data prior to 

transmission, transmit data to the registry, and 

receive and present information from the 

registry (e.g., population views).

• Effectiveness: Identify patients who meet 

eligibility criteria, execute sampling algorithms, 

alert clinicians, present the relevant forms and 

instructions, capture uniform data, review the 

data prior to transmission, transmit data or 

analytics, and receive and present information 

from the registry (e.g., followup schedules, 

registrywide results).

• Safety: Identify events for reporting through 

triggers, capture uniform data, review the data 

prior to transmission, transmit data, receive and 

present requests for additional information, and 

receive and present safety information from the 

registry.

• Quality: Identify patients who meet eligibility 

criteria, present the relevant forms and 

instructions, capture uniform data, review the 

data prior to transmission, transmit data to the 

registry for reporting, and receive and present 

quality measure information and comparators 

from the registry.

In a truly interoperable system, registry-specific 

functionality could be presented in a software-asa-service or middleware model, interacting with 
7

Chapter 15. Interfacing Registries With Electronic Health Records

the EHR as the presentation layer on one end and 

the registry database on the other. In this model, 

the EHR is a gateway to multiple registries and 

clinical research activities through an open 

architecture that leverages best-in-class 

functionality and connectivity. Registries interact 

across multiple EHRs, and EHRs interact with 

multiple registries.

6. Interoperability Challenges

Interoperability for health information systems 

requires communication, accurate and consistent 

data exchange, and use of the information that has 

been exchanged. The two core constructs, related 

to communication and content, are syntactic and 

semantic interoperability.

6.1 Syntactic Interoperability

Syntactic interoperability is the ability of 

heterogeneous health information systems to 

exchange data. There are several layers of syntactic 

interoperability. First, the physical wiring must be 

in place, and the TCP/IP (Internet) is the de facto 

standard. On top of this, an application protocol is 

needed such as HTTP or SMTP. The third layer is 

a standard messaging protocol such as SOAP 

(Simple Object Access Protocol).14 The message 

must have a standard sequence, structure, and data 

items in order to be processed correctly by the 

receiving system.

When proprietary systems and formats are used, 

the complexity of the task grows dramatically. For 

n systems, n(n-1)/2 interfaces are needed for each 

system to communicate with every other one.15

For this reason, message standards are preferred. 

While this seems straightforward, an example 

portrays how, even for EHR-to-EHR 

communication, barriers still exist. Currently, the 

Health Level Seven (HL7) Version 2 message 

standard (HL7 v2.5) is the most widely 

implemented standard among EHRs, but this 

version has no explicit information model; instead, 

it rather vaguely defines many data fields and has 

many optional fields. To address this problem, the 

Reference Information Model (RIM) was 

developed as part of HL7 v3, but v3 is not fully 

adopted and there is no well-defined mapping 

between v2.x and v3 messages.

Syntactic interoperability assures that the message 

will be delivered. Of the challenges to 

interoperability, this is the one most frequently 

solved. However, solving the delivery problem 

does not guarantee that the content of the message 

can be processed and interpreted at the receiving 

end with the meaning for which it was intended.

6.2 Semantic Interoperability

Semantic interoperability implies that the systems 

understand the data exchanged at the level of 

defined domain concepts. This “understanding” 

requires shared data models that, in turn, depend 

on standard vocabularies and common data 

elements.16

The National Cancer Institute’s (NCI) Cancer 

Bioinformatics Grid (caBIG) breaks down the core 

components of semantic interoperability into 

information or data models, which describe the 

relationships between common data elements in a 

domain; controlled vocabularies, which are an 

agreed-upon set of standard terminology; and 

common data elements, which use shared 

vocabularies and standard values and formats to 

define how data are to be collected.

The standardization of what is collected, how it is 

collected, and what it means is a vast undertaking 

across health care. Much work has been done and 

is continuing currently, although efforts are not 

centralized nor are they equally advanced for 

different medical conditions. One effort, called the 

CDASH (Clinical Data Acquisition Standards 

Harmonization) Initiative, led by the Clinical Data 

Interchange Standards Consortium (CDISC), aims 

to describe recommended basic standards for the 

collection of clinical trial data.17 It provides 

guidance for the creation of data collection 

instruments, including recommended case report 

form (CRF) data points, classified by domain 

(e.g., adverse events, inclusion/exclusion criteria, 

vital signs), and a core designation (highly 

recommended, recommended/conditional, or 

optional). Version 1.0 was published in October 

2008; v1.1 was published in January 2011 and 

included implementation guidelines, best practice 

recommendations, and regulatory references. It 

remains to be seen how widely this standard will 

be implemented in the planning and operation of 
8

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

registries, clinical trials, and postmarketing 

studies, but it is nonetheless an excellent step in 

the definition of a common set of data elements to 

be used in registries and clinical research.

Other examples of information models used for 

data exchange are the ASTM Continuity of Care 

Record (CCR) and HL7’s Continuity of Care 

Document (CCD), which have standardized certain 

commonly reported components of a medical 

encounter, including diagnoses, allergies, 

medications, and procedures. The CCD standard is 

particularly relevant because it is one that has been 

adopted as part of CCHIT certification. The 

Biomedical Research Integrated Domain Group 

(BRIDG) model is an effort to bridge health care 

and clinical research standards and organizations 

with stakeholders from CDISC, HL7, NCI, and 

FDA. Participating organizations are collaborating 

to produce a shared view of the dynamic and static 

semantics that collectively define a shared domain 

of interest, (i.e., the domain of clinical and 

preclinical protocol-driven research and its 

associated regulatory artifacts).18

Even with some standardization in the structure 

and content of the message, issues exist in the use 

of common coding systems. For any EHR and any 

registry system to be able to semantically 

interoperate, there needs to be uniformity around 

which coding systems are to be used. At this time, 

there are some differences between coding systems 

adopted by EHR vendors and registry vendors. 

While it is still possible to translate these coding 

systems and/or “recode” them, the possibility of 

achieving full semantic interoperability is limited 

until uniformity is achieved.

The collection of uniform data, including data 

elements for risk factors and outcomes, is a core 

characteristic of patient registries. If a functionally 

complete standard dictionary existed, it would also 

greatly improve the value of the information 

contained within the EHR. But, while tremendous 

progress has been made in some areas such as 

cancer19 and cardiology,20 the reality is that full 

semantic interoperability will not be achieved in 

the near future.

Beyond syntactic and semantic interoperability, 

other issues require robust, standardized solutions, 

including how best to authenticate users across 

multiple applications. Another issue is permission 

or authorization management. At a high level, how 

does the system enforce and implement varying 

levels of authorization? A health care authorization 

is specific to authorized purposes. A particular 

patient may have provided different authorizations 

to disclose information differently to different 

registries interacting with a single EHR at the 

same time, and the specificity of that permission 

needs to be retained and in some way linked with 

the data as they transit between applications. For 

privacy purposes, an audit trail also needs to be 

maintained and viewable across all the paths 

through which the data move. Security must also 

be ensured across all of the nodes in the 

interoperable system.

A third key challenge to interoperability is 

managing patient identities among different health 

care applications. See Chapter 17 for further 

discussion.

7. Partial and Potential 

Solutions

Achieving true, bidirectional interoperability, so 

that all of the required functions for EHRs and 

patient registries function seamlessly with one 

another, is unlikely to be accomplished for many 

years. However, as noted above, it is critical that a 

level of interoperability be achieved to prevent the 

creation of silos of information within proprietary 

informatics systems that make it difficult or 

impossible to conduct large registries or other 

evidence development research across diverse 

practices and populations. Given the lack of a 

holistic and definitive interoperability model, an 

incremental approach to the successive 

development, testing, and adoption of open, 

standard building blocks toward an interoperable 

solution is the likely path forward. In fact, much 

has been done in the area of interoperability, and if 

fully leveraged, these advances can already provide 

at least a level of functional interoperability that 

could significantly ameliorate this potential 

problem.
9

Chapter 15. Interfacing Registries With Electronic Health Records

From an EHR/registry perspective, functional 

interoperability could be described as a standardsbased solution that achieves the following set of 

requirements:

The ability of any EHR to exchange valid and 

useful information with any registry, on behalf 

of any willing provider, at any time, in a 

manner that improves the efficiency of registry 

participation for the provider and the patient, 

and does not require significant customization 

to the EHR or the registry system.

Useful information exchange constitutes both 

general activities (e.g., patient identification, 

accurate/uniform data collection and processing) 

and specific additional elements, depending on the 

purpose of the registry (e.g., quality reporting). 

Such a definition implies an open-standards 

approach where participation is controlled by the 

provider/investigator. To be viable, such a model 

would require that EHRs become certified to meet 

open standards for basic functional interoperability 

(the requirements of which would advance over 

time), but also allow EHRs the opportunity to 

further differentiate their services by how much 

they can improve the efficiency of participation.

While the goal of functional interoperability likely 

requires the creation and adoption of effective 

open standards, there have been several approaches 

to partially addressing these same issues in the 

absence of a unified approach. HIT systems, 

including some EHRs, have been used to populate 

registry databases for some time. The Society of 

Thoracic Surgeons, the American College of 

Cardiology, and others use models that are based 

on a central data repository that receives data from 

multiple conforming systems, on a periodic basis, 

through batch transfers. Syntactic interoperability 

is achieved through a clear specification that is 

custom-programmed by the HIT systems vendor. 

Semantic interoperability is achieved by the 

publication of specifications for the data collection 

elements and definitions on a regular cycle, and 

incorporation of these by the systems vendors. 

Each systems vendor pays a fee for the 

specifications and for testing their implementation 

following custom programming. In some cases, an 

additional fee is levied for the ongoing use of the 

interface by the systems vendor. Periodically, as 

data elements are modified, new specifications are 

published and the cycle of custom programming 

and testing is repeated. While there is incremental 

benefit to the provider organizations in that they do 

not have to use multiple systems to participate in 

these registries, the initial and periodic custom 

programming efforts and the need to support 

custom interface requirements make this approach 

unscalable. Furthermore, participation in one 

registry actually makes participation in other, 

similar registries more difficult, since the data 

elements are customized and not usable in the next 

program.

The American Heart Association’s Get With The 

Guidelines® program uses a Web services model 

for a similar purpose. The advantage of the Web 

services model is that the data are transferred to 

the patient registry database on a transactional 

basis (immediately), but the other drawbacks in 

custom programming and change management still 

apply. This program also offers an open standards 

approach through IHE RFD21 or Healthcare 

Information Technology Standards Panel (HITSP) 

TP50, both described below. These examples 

describe two models for using EHRs to populate 

registry databases; other models exist.

8. Momentum Toward a 

Functional Interoperability 

Solution

Significant momentum is already building toward 

adopting open-standard building blocks that will 

lead incrementally to functional interoperability 

solutions. For example, the EHR Clinical Research 

Value Case Workgroup has focused its use cases 

on two activities: achieving the ability (1) to 

communicate study parameters (e.g., eligibility 

information, CRFs) and (2) to exchange a core 

data set from the EHR.22 Others in the standards 

development community have taken a stepwise 

approach to creating the components for a firstgeneration, functional interoperability solution. As 

described below, this solution has already 

overcome several of the key barriers to creating an 

open, scalable model that can work simultaneously 

between multiple EHR systems and registries. 

Some issues addressed through these efforts 
10

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

include: the need for flexibility in presenting a 

uniform data collection set that can be modified 

from time to time without custom programming by 

the EHR vendor; the need to leverage existing, 

standardized EHR data to populate portions of the 

data collection set; and the need to be able to 

submit the data on a transactional basis to a 

registry, clinical trial, or other data recipient in a 

standard format.

A building-block approach to the technical side of 

this issue is an effective and pragmatic way to 

build in increments and allow all players in the 

industry to focus on specific components of 

interoperability; early successes can then be 

recognized and used as the basis for the next step 

in the solution. This is a change from the earlier 

approaches to this issue, where the problem (and 

the solution) was defined so broadly that complete 

semantic interoperability seemed to be the only 

way to solve the problem; this proved 

overwhelming and unsupportable. Instead, a 

working set of industry-accepted standards and 

specifications that already exist can focus tightly 

on one aspect of interfacing multiple data capture 

systems, rather than considering the entire spread 

of issues that confound the seamless interchange 

between health care and research systems.

There are many different standards focused on 

different levels of this interface, and several 

different key stakeholders that create, work with, 

and depend on these standards (see Appendix C). 

A useful way to visualize these technical standards 

is to consider a stack in which each building block 

is designed to facilitate one aspect of the technical 

interface between an EHR and a data collection 

system (Figure 15–1). The building blocks are 

modest but incremental changes that move two 

specific systems toward interoperability and are 

scalable to different platforms.

Figure 15–1. A building-block approach to interoperability

HL7 = Health Level Seven; CDISC = Clinical Data Interchange Standards Consortium; CRD = Clinical Research 

Data Capture; DSC = Drug Safety Content; RFD = Retrieve Form for Data Capture; RPE = Retrieve Protocol for 

Execution; RSP = Redaction Services Profile.
11

Chapter 15. Interfacing Registries With Electronic Health Records

This theoretical stack starts with the most basic 

technical components as the ground layers. 

Physical network connections, followed by Web 

services, secure hypertext transfer protocol (http), 

secure socket layer (SSL) communications 

protocol, and Web browsers create the foundation 

of the interoperability structure. These standard 

technologies are compatible across most systems 

already.

A standard integration profile, Retrieve Form for 

Data Capture (RFD), is the base of specific 

interoperability for health care data transfer, and it 

takes advantage of the Web standards as a way to 

integrate EHRs and registry systems. RFD is a 

generic way for systems to interact. In a sense, 

RFD opens a circuit or provides a “dial tone” to 

allow an EHR to exchange information with a 

registry or other clinical research system. RFD 

was created and is maintained by Integrating the 

Healthcare Enterprise (IHE). It is also accepted 

under HITSP as TP50. Specifically, RFD provides 

a method for gathering data within a user’s current 

application to meet the requirements of an external 

system (e.g., a registry). In RFD, as Figure 15–2 

shows, this is accomplished by retrieving a registry 

or other data collection form from a source; 

displaying it within the EHR system to allow 

completion of the form, with data validation 

checks, either through direct user entry or 

automated population from the EHR database; and 

then returning an instance of the data back to the 

registry system. Importantly, the EHR initiates the 

transaction.

Figure 15–2. Retrieve form for data capture diagram 

CAP = Capability; CCD = Continuity of Care Document; CRD = Clinical Research Data Capture; DSC = Drug 

Safety Content; EHR = Electronic Health Record; HITSP = Healthcare Information Technology Standards Panel; 

IHE = Integrating the Healthcare Enterprise; RFD = Retrieve Form for Data Capture; TP = Transaction Package.

Once an EHR is RFD-enabled, it can be used for 

multiple use-cases. RFD opens a circuit and allows 

for information exchanges of different purposes, 

including registries and clinical trials, quality 

initiatives, safety, and public health reporting.23

Content profiles such as Clinical Research Data 

Capture (CRD) build the next level, allowing 

standard content defined within an EHR to be 

mapped into the data collection elements for the 

registry, eliminating duplicate entry for these 

defined elements. CRD and the Drug Safety 

Content (DSC) profiles, managed by IHE, build 

upon the IHE RFD integration profile. 

Correspondingly, HITSP C76, or Case Report 

Pre-Populate Component (for Drug Safety), 

leverages the HITSP TP50 retrieve form for data 

capture (RFD) transaction package.

CRD allows the functional interoperability 

solution to leverage standardized content as it 

becomes defined and available within EHRs. In 

other words, it is an incremental approach to 

leveraging whatever content has been rigorously 

Open Standards: HITSP: TP 50, C 76, C 151, CAP 135 | IHE: RFD, CRD, DSC

CCD

EHR Module

Form Library

Secondary Use

Database
12

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

defined and resident within the EHR and is also 

usable and acceptable to the registry (i.e., content 

that matches some portion of the registry’s defined 

data elements and definitions). To the extent that 

these data reside in a common format, they can be 

used for autopopulation of the registry forms 

without custom programming. CRD leverages the 

Continuity of Care Document (CCD), an HL7 

standard. In this scenario, the EHR generates the 

CCD to populate a case report form. The registry 

uses only the relevant data from the CCD, as 

determined by the registry system presenting the 

form. Alternatively, CRD specifies that CDASH, a 

CDISC standard for data collection elements, may 

be used as the content message to prepopulate the 

case report form.

9. The Next Increment

As the basic components of functional 

interoperability are being tested and implemented, 

more attention is being focused on the next 

increments of the building-block approach. The 

important challenges to be addressed include: 

patient identification/privacy protection; the 

potential and appropriate use of digital signatures; 

other related and emerging profiles, such as 

querying the EHR for existing data through the 

Query for Existing Data (QED) profile; and 

transferring process-related study information as 

captured in the study protocol (Retrieve Protocol 

for Execution [RPE]). More extensive work in data 

mapping and the development of use cases around 

content are also needed.

9.1 Patient Identification/Privacy 

Protection

Patients within the context of clinical care are 

identified by a patient identifier, usually referred to 

as a medical record number. When these patients 

participate in a registry, they will also have a 

patient identifier within the context of the registry’s 

programs. In some cases, where explicit 

authorization has been obtained, the medical 

record number may be shared across programs and 

can be used as a common identifier that links the 

patient across systems. In other cases, there is a 

need to anonymize the patient identifier. In the 

latter situation, infrastructure can be deployed to 

create unique, anonymized patient identifiers that 

serve to protect the patients’ identity and facilitate 

secure patient identity management (e.g., Patient 

Identifier Cross-Referencing [PIX]).21

Beyond anonymizing, it also may be desirable to 

maintain a cross-referencing of patient identifiers 

or aliases across multiple systems so that the 

medical record number within the EHR can be 

linked back to the identifier within the registry or 

clinical trial without revealing the patient identity. 

Pseudonymization is a procedure by which all 

person-related data are replaced with one artificial 

identifier that maps one-to-one to the person.22

Pseudonymization allows for additional use cases 

where it is necessary to link a patient seen in 

different settings (such as linking back to source 

records for additional information or 

monitoring).23 See Chapter 17 for a more detailed 

discussion of this topic.

9.2 Digital Signatures

Certain registry purposes (such as regulatory 

reporting) require electronic signatures—for 

example, when the clinician or investigator attests 

to the completeness and accuracy of information 

being submitted for a research purpose. The 

current paradigm is the investigator’s physical or 

electronic signature on a paper or electronic case 

report form. The potential and appropriate use of 

digital signatures may further broaden the set of 

use cases by which EHRs may be used for 

secondary purposes. Other approaches to 

facilitating identity management, signing, and 

verification, such as Private Key Infrastructure 

(PKI), provide advantages in terms of 

nonrepudiation and detection of tampering. In the 

next wave of the interoperability effort, it will be 

important to define those scenarios that will 

require the strength of an enhanced digital 

signature.

9.3 Other Related and Emerging 

Efforts

As the building blocks of interoperability develop, 

additional flexibility will be gained as the registry 

and EHR can more fully communicate in a 

common language, both to request more clinical 

data and to provide the EHR with more 
13

Chapter 15. Interfacing Registries With Electronic Health Records

information on the workflow requirements of the 

registry or other study protocol. These 

requirements point to other work being done to 

address these issues. Below are three examples 

from IHE profiles, some of which are under 

development by the Quality, Research, and Public 

Health (QRPH) Domain:

• Retrieve Protocol for Execution (RPE): This 

integration profile allows an EHR to retrieve a 

protocol or a complex set of clinical research 

instructions necessary to fulfill the specified 

requirements of a protocol. The availability of 

these definitions and a set of transactions 

defined by RPE can provide an EHR with 

content that may be used to identify patients for 

a research program based on defined inclusion/

exclusion criteria; manage the patient visit 

schedule and appropriate case report forms or 

assessments that need to be completed in the 

appropriate sequence; and/or assist with other 

clinical activities such as ordering protocolspecified tests or laboratory reports.24 RPE 

eliminates the need to manually enter data in 

two places (an EHR and an electronic data 

capture system collecting data for clinical 

research), resulting in a lower user burden on 

sites participating in research, as they are able 

to contribute EHR data to research protocols 

without leaving their EHR session. 

• Redaction Services Profile (RSP): This 

integration profile addresses the privacy 

concerns around the exchange of electronic 

health data. It provides a way to redact certain 

data (e.g., personal identifiers) before 

transmitting that data from one system to 

another (e.g., from an EHR to a QRPH system), 

and acts as a “safety net” by ensuring that only 

the necessary and specified data is transmitted. 

In addition to this function, it also records and 

maintains an audit trail of the transmissions it 

facilitates, to support data quality processes.25

• Drug Safety Content (DSC): This content 

profile from the QRPH Domain details which 

data (and in what format) should be used in the 

RFD prepopulation transaction between the 

Form Manager and Form Filler. It is specifically 

used for reporting adverse events and other data 

related to drug safety.26

9.4 Data Mapping and Constraints

While the efforts described above continue to 

expand the use of electronic medical record data 

for a variety of secondary purposes, it is clear that 

clinical and research teams, standards, and 

terminologies need to be further harmonized to 

maximize the benefits of information sharing 

across the variety of clinical and research systems. 

Effective and efficient management requires that 

harmonization efforts are furthered among vendors 

and standards organizations. It also requires that 

use cases continue to be honed and explicitly 

defined so that new clinical document constraints 

can be applied as necessary for each specified use 

case. Use cases will range across study types and 

across purposes, including drug safety, 

biosurveillance, and public health. Each clinical 

document constraint should strive to capture and 

deliver the information necessary to fully support 

the level of information sharing required by the 

scenario that maximizes both the efficiency of the 

clinical care/research workflow and the value of 

previously collected relevant data.

10. What Has Been Done

A number of efforts have demonstrated success in 

implementing several of the aforementioned 

building-block standards to achieve functional 

interoperability for registry purposes, including 

safety, effectiveness, and quality measurement. In 

one case, a registry that focused on effectiveness in 

pain management was made interoperable with a 

commercial EHR using RFD communication.27 In 

a second case, the Adverse Drug Event 

Spontaneous Triggered Event Reporting (ASTER) 

project,28 interoperability was achieved for the 

purpose of reporting adverse event information to 

FDA. (See Case Example 35.) In a third case, a 

commercial EHR was made interoperable with a 

quality reporting initiative for the American 

College of Rheumatology (ACR),29 and to a 

Physician Quality Reporting Initiative (PQRI) 

Registry for reporting data to the Centers for 

Medicare & Medicaid Services (CMS).30 In each 

case, both the registry and the EHRs were able to 

exchange useful information and decrease the 

effort required by the participating physicians.
14

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

11. Distributed Networks

It should be noted that the models of 

interoperability discussed above presume that data 

are shared between a distributed EHR and a patient 

registry (or another recipient such as a regulatory 

authority or a study sponsor). Alternative models 

may leave all data within the EHR but execute 

analyses in a distributed fashion and aggregate 

only results. To effectively accomplish distributed 

analyses requires either semantic interoperability 

or the ability to map to a conforming database 

structure and content, as well as the sophistication 

of a large number of EHR systems to run these 

types of queries in a manner that does not require 

providers to customize or program their systems. 

Several groups are advancing these concepts, and 

they may eventually prove to be very suitable for 

particular registry purposes (e.g., safety or public 

health surveillance).

PopMedNetTM is one example of a distributed 

network model.31-33 It is a software application 

that enables the creation of a distributed health 

data networks and supports the operation and 

governance of these networks.34 Through the 

application, researchers can create and distribute 

queries to network data partners, who can then 

execute the queries and return the aggregate results 

to the researchers. Data partners retain control of 

their data and can review queries before 

responding. The PopMedNet application is 

designed to support a variety of data networks; 

therefore, the application does not use a specific 

data model or governance structure, but instead 

allows each data network to customize its 

implementation. 

Currently, the PopMedNet application is being 

used for several research projects, including FDA’s 

Mini-Sentinel project; the Agency for Healthcare 

Research and Quality’s (AHRQ) Scalable 

PArtnering Network for CER: Across Lifespan, 

Conditions, and Settings (SPAN); and the 

Population-based Effectiveness in Asthma and 

Lung Diseases (PEAL) project. The Mini-Sentinel 

project is designed to facilitate the development of 

an active surveillance system for monitoring the 

safety of medical products. SPAN uses the 

application to conduct comparative effectiveness 

research in obesity and attention deficit 

hyperactivity disorder. The PEAL project aims to 

understand factors that affect prescribing and 

adherence to asthma medications.35 The software 

application was initially developed by the HMO 

Research Network Center for Education and 

Research on Therapeutics and the University of 

Pennsylvania under contract to AHRQ as part of 

the Developing Evidence to Inform Decisions 

about Effectiveness (DEcIDE) program. 

Additional development was supported by AHRQ 

under the SPAN project and by FDA under the 

Mini-Sentinel project.36

12. Summary

Achieving EHR-registry interoperability will be 

increasingly important as adoption of EHRs and 

the use of patient registries for many purposes 

both grow significantly. The linkage of registries 

with health information exchanges (HIEs) is also 

important, as HIEs may serve as data collection 

assistants with which registries may need to 

interact.37 Achieving interoperability between 

these data sources is critical to ensuring that the 

massive HIT investment under ARRA does not 

create silos of information that cannot be joined 

for the public good.38 Such interoperability should 

be based on open standards that enable any willing 

provider to interface with any applicable registry 

without requiring customization or permission of 

the EHR vendor. Interoperability for health 

information systems requires accurate and 

consistent data exchange, along with use of the 

information that has been exchanged. In addition, 

care must be taken to ensure that integration 

efforts comply with legal and regulatory 

requirements for the protection of patient privacy. 

While full semantic interoperability remains 

distant, a great deal of useful work has been and is 

being done. For example, the adoption of open 

standards such as HITSP TP50, C76 and IHE 

RFD, CRD, and DSC greatly enhance the ability of 

EHRs and registries to function together and 

reduce duplication of effort. Functional 

interoperability is a goal that can be achieved in 

the near term with significant gains in improving 

workflow and reducing duplication of effort for 

providers and patients participating in registries. 
15

Chapter 15. Interfacing Registries With Electronic Health Records

The successive development, testing, and adoption 

of open-standard building blocks, which improve 

functional interoperability and move us 

incrementally toward a fully interoperable solution, 

is a bridging strategy that provides benefits to 

providers, patients, EHR vendors, and registry 

developers today.

Case Example 32. Using system integration 

software to capture registry data from 

electronic health records

Description The PINNACLE Registry is an 

office-based, ambulatory 

cardiology quality improvement 

registry. The registry collects 

data to facilitate performance 

metric evaluation in coronary 

artery disease, atrial fibrillation, 

hypertension, and heart failure.

Sponsor American College of Cardiology 

Foundation (ACCF)

Year Started 2007

Year Ended Ongoing

No. of Sites Over 500

No. of Patients Over 2,000,000 patient records

Challenge

Collection of registry data in an outpatient setting 

can be challenging. Sites wishing to participate in 

the PINNACLE Registry can choose to collect 

and submit their data on paper or electronically. 

Paper data collection (i.e., having a dedicated 

clinical abstractor abstract data manually from an 

existing medical record into a data collection 

form) can be disruptive to practice workflow. This 

method also requires such a significant 

investment in human resources (from both the 

site and the registry) that the PINNACLE 

Registry is no longer accepting new sites that 

submit data on paper.

Electronic data submission involves directly 

abstracting relevant registry information from 

electronic health records (EHRs). The registry 

certified two EHR vendors as fully compatible 

and able to submit data automatically to the 

registry, which minimizes the data entry burden 

on sites. However, many potential sites use other 

EHRs, and the lack of standardized terminology 

and data collection formats among the many EHR 

options available to practices makes it 

challenging to provide an integration solution that 

serves the largest possible number of sites. 

Proposed Solution

Recognizing these challenges, the American 

College of Cardiology Foundation (ACCF) 

partnered with a technology partner to develop 

the PINNACLE Registry System Integration 

Solution (SI). The SI is comprised of (1) a 

Microsoft SQL–based database, which stores 

registry measures and the data mapping 

specifications for the relevant EHR, (2) a .NET 

4.0-based Windows Service, which interfaces 

with the EHR and extracts the relevant registry 

data, and (3) a .NET 4.0-based Windows Client, 

which configures the data extractions and adjusts 

mappings to suit the practice’s specific use of the 

EHR. The SI is compatible with any EHR system, 

including those that have been highly customized 

at the practice level. 

The registry team works with potential sites to 

complete a technical questionnaire, providing 

details about the practice’s technical environment 

and EHR system. The SI software is then 

installed on the practice’s server, is programmed 

to collect registry data elements that are already 

captured in the existing EHR system, and exports 

the data directly to the registry database. The 

primary human resource requirement is from the 

practice’s information technology team who work 

with the technology partner to install the solution 

on the practice’s server.

Results

Currently, 80 percent of sites participating in the 

PINNACLE Registry use the SI to submit their 

data. The SI software has been successfully 

Case Examples for Chapter 15
16

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

Case Example 32. Using system integration 

software to capture registry data from 

electronic health records (continued)

Results (continued)

installed and implemented at 396 sites, which 

combined use 19 different EHR products. 

Installation and data mapping is underway at sites 

using 14 other EHR products. 

The PINNACLE Registry System Integration 

Solution allows for the collection of registry data 

with minimal disruption of practice workflow. By 

eliminating the need for manual chart abstraction 

and data entry, some barriers to practice 

participation are removed. However, this means 

that if data are missing in the EHR, the same data 

will be missing in the registry record. Because of 

the lack of standardization in EHR systems, the 

SI solution does require time and resources 

during the startup phase to implement in a 

particular practice. Until such standards exist, the 

SI solution is a viable solution for capturing 

registry data with minimum workflow disruption 

and minimum human capital commitment. 

Key Point

Extracting registry data directly from ambulatory 

EHRs can reduce the data entry burden on 

participating sites. A software solution that 

executes this extraction automatically may take 

time to set up initially, but minimizes workflow 

disruption during continued registry participation. 

An integrated solution that is flexible enough to 

accommodate many different EHR vendors and 

levels of customization can reduce barriers to 

registry participation for many sites.

For More Information

https://pinnacleregistry.org/Pinnacle/

PINNACLERegistry/DataCollection.aspx

Case Example 33. Creating a registry 

interface to incorporate data from multiple 

electronic health records

Description The MaineHealth Clinical 

Improvement Registry (CIR) is a 

secure Web-based database 

system that provides a tool for 

primary care physicians in the 

outpatient setting to consolidate 

and track key clinical 

information for preventive health 

measures and patients with 

common chronic illnesses.

Sponsor The project is the result of a 

collaboration between Maine 

Medical Center (MMC) 

Physician-Hospital Organization, 

MaineHealth, and MMC 

Information Services.

Year Started 2003

Year Ended Ongoing

No. of Sites 106 primary care practices (450 

providers)

No. of Patients More than 200,000

Challenge

A physician-hospital organization (PHO) 

developed a Web-based patient registry to 

improve quality of care and track patient 

outcomes across a large network of physicians. 

Many practices in the network used EHRs and 

did not have sufficient staff to enter patient data a 

second time into a registry. In addition, the 

practices used a wide range of electronic health 

records (EHRs), and each had unique technical 

specifications. The registry needed a technical 

integration solution to reduce the data entry 

burden on practices that used EHRs, but, due to 

resource limitations, it could not develop 

customized interfaces for each of the many 

different EHRs in use.

Proposed Solution

The registry elected to allow practices to submit 

data from their EHRs to the registry in a one-way 

data transfer. An interface was written against an 

XML specification. Practices wishing to 

participate in the registry without doing direct 

data entry must be able to export their data in a 

file that conforms to this specification (although 

HL7 files are accepted when necessary). Data 
17

Chapter 15. Interfacing Registries With Electronic Health Records

Case Example 33. Creating a registry 

interface to incorporate data from multiple 

electronic health records (continued)

Proposed Solution (continued)

transfers occur on a schedule determined by each 

site—some send their data in real time while 

others send on a monthly basis.

Once the registry receives data files, registry staff 

members review each portion of the data 

(demographics, vaccinations, office visits, etc.) 

before signing off on the file and incorporating 

the data into the registry. Extensive error 

checking and validation are completed during the 

initial specification phase to minimize the amount 

of manual data checking needed during each 

transfer. The validation phase involves both 

technical staff and quality improvement staff at 

the practices to ensure that the data are 

transferred and mapped correctly into the registry 

database.

Results

Of the 106 primary care practices participating in 

the registry, about 60 percent enter data directly 

into the registry, and about 40 percent contribute 

data via XML transfer. The organization and 

management of this initiative have required 

strong internal support from the registry and from 

participating practices. Management teams and 

technical resources were needed during the 

startup phase and continue to be essential as more 

practices contribute data via XML transfer.

Key Point

Technical interface solutions between registries 

and EHRs can be successful, but require a robust 

organizational commitment from the registry 

sponsor and participating sites to provide the 

necessary resources during the setup and launch 

phases.

For More Information

http://www.mmcpho.org/technology/

mainehealth_clinical_improvement_registry_cir/

Case Example 34. Technical and security 

issues in creating a health information 

exchange

Description The Oakland Southfield 

Physicians Quality Registry is a 

practice-based registry designed 

to promote health outcomes and 

office efficiencies, and to 

identify early interventions and 

best practices in primary care 

practices. The registry integrates 

and exchanges health 

information from many sources 

through the Oakland Southfield 

Physicians Health Information 

Exchange (OSPHIE).

Sponsor Oakland Southfield Physicians

Year Started 2006

Year Ended Ongoing

No. of Sites 150

No. of Patients Network covers more than 

250,000 patients

Challenge

In 2006, the practice association launched a 

registry to improve the quality of care in its 

primary care practices. However, the association 

quickly realized that it needed to integrate and 

exchange health information from multiple 

sources, such as payer claims, pharmacy claims, 

practice management systems, laboratory 

databases, and other registry systems, on behalf 

of more than 150 primary care practices.

Proposed Solution

To support this requirement, the practice 

association constructed an HIE. The HIE is a data 

warehouse made up of multiple data sources that 

facilitates the collaborative exchange of health 

information with a network of trading partners 

and then integrates the patient disease registry 

data with a wide range of supplemental clinical 

information. The HIE allows the registry to 

securely exchange data with trading partners 

(third-party payers, laboratories, hospitals, 

registry systems, etc.) via a variety of methods 

and in a variety of structures. By pushing 
18

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

Case Example 34. Technical and security 

issues in creating a health information 

exchange (continued)

Proposed Solution (continued)

information both to the registry system and to 

other systems, the HIE eliminates duplicate data 

entry. Data transfers occur at established 

intervals, based on record updates or availability 

of information.

A key aspect of the system is the master patient 

and physician index, which allows data from 

various sources to be linked to the proper patient. 

Prior to import, data received in the registry are 

validated against a master patient and physician 

index for accuracy.

Results

Through data sharing with the Oakland 

Southfield Physicians registry, the practice 

association has been able to facilitate the 

alignment of multiple data sources, with 

evidence-based care guidelines available at point 

of care—a value partnership striving to improve 

health outcomes as well as the efficient access to 

key health care data points. This solution relies on 

building trust between trading partners in support 

of both the secure transfer of information and 

recommended use.

The HIE has successfully incorporated data from 

practice management systems, laboratory 

providers, an e-prescribing system, a registry 

system, and third-party payers (medical and 

pharmacy claims detail). Relevant data are 

currently transmitted on behalf of the 

participating physicians in a real-time capacity 

from the HIE to both the registry system and the 

e-prescribing system. The data warehouse also 

generates monthly “gaps-in-care reports” for 

physician clinical quality review and patient 

outreach.

Key Point

An HIE may be a useful tool for integrating and 

exchanging data between registries and other 

systems. When integrating data from many 

sources, a master patient and physician index can 

be a critically important tool for ensuring that the 

incoming data are linked to the appropriate 

patient.

For More Information

http://www.ospdocs.com/OSP+Advantage/

Clinical+Quality+Registry-21.html
19

Chapter 15. Interfacing Registries With Electronic Health Records

Case Example 35. Developing a new model for 

gathering and reporting adverse drug events

Description The Adverse Drug Event 

Spontaneous Triggered Event 

Reporting (ASTER) study uses 

a new approach to the gathering 

and reporting of spontaneous 

adverse drug events (ADEs). 

The study was developed as a 

proof of concept for the model 

of using data from electronic 

health records to generate 

automated safety reports, 

replacing the current system of 

manual ADE reporting. The 

goals are to reduce the burden 

sof reporting and provide timely 

reporting of ADEs to regulators.

Sponsor Brigham and Women’s Hospital, 

Partners Healthcare, CDISC, 

CRIX International, Claricode, 

and Pfizer Inc.

Year Started Pilot launched in 2008

Year Ended Pilot ended in 2009

No. of Sites N/A

No. of Patients N/A

Challenge

Health care data are rapidly being translated into 

electronic formats; however, to date, safety 

reporting has not taken full advantage of these 

electronic data sources. The spontaneous adverse 

event reporting system, which relies on reports 

submitted manually by health care professionals, 

is still the primary source of data on potential 

ADEs. However, the availability of large amounts 

of data in electronic formats presents the 

opportunity to rethink the spontaneous adverse 

event reporting system. A new model could take 

advantage of the increasing availability of 

electronic data and improving technology to 

automate the process of gathering and reporting 

ADEs. The goals of automated ADE reporting are 

to reduce the burden of reporting on physicians, 

improve the frequency with which ADEs are 

reported, and increase the timeliness and quality 

of ADE reports.

An automated model, however, must overcome 

many challenges. The system must be scalable, 

must incorporate data from many sources, and 

must be flexible enough to adapt to the needs of 

many diverse groups. The model must address 

point-of-care issues (such as burden of reporting), 

data exchange standards (so that the data are 

interpretable and valid), and processes for 

reviewing the ADE reports.

Proposed Solution

The ASTER study attempted to address these 

challenges and demonstrate the potential viability 

of an automated model for facilitating the 

gathering and reporting of ADEs. ASTER 

allowed data to be transferred from an electronic 

health record (EHR) to an adverse event (AE) 

case report form and submitted directly to the 

U.S. Food and Drug Administration (FDA) in the 

format of an individual case safety report. The 

process of gathering and reporting ADEs through 

ASTER involves four steps based on the openstandard “Retrieve Form for Data Capture 

(RFD)”:

1. A physician indicates in the EHR that a drug 

was discontinued due to an ADE.

2. The system immediately generates an ADE 

report form that is prepopulated with 

demographic, medication, vital signs, and 

laboratory data. The physician sees the form 

in the EHR.

3. The physician enters a small amount of 

additional data, such as outcomes of the 

adverse event, to complete the ADE report 

form.

4. The form is then processed by a third-party 

forms manager, who sends it to FDA as a 

reported spontaneous AE from the physician, 

in a standard format.

Results

The pilot phase of ASTER began in 2008. The 

goal of this phase was to demonstrate proof of 

concept for the new model. Specifically, it was 
20

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

Case Example 35. Developing a new model for 

gathering and reporting adverse drug events 

(continued)

Results (continued)

hypothesized that (1) if an EHR could help a 

clinician identify potential AEs, and (2) if the 

burden of completing an AE form was 

significantly reduced, then the rate of reporting of 

spontaneous AEs to FDA could be significantly 

increased. ASTER recruited 26 physicians, 91 

percent of whom had not reported an AE to FDA 

in the prior year. Following implementation, more 

than 200 events were reported over a period of 3 

months.

Many questions need to be answered before the 

ASTER model can become more widely used in 

the United States. For example, initial findings 

from ASTER suggest that an increased number of 

events are being reported using this model; this 

creates a need for the receiver of the reports (i.e., 

FDA) to have sufficient capacity to respond to the 

reports. Also, the fields captured in the ASTER 

model are based on the paper form fields. Moving 

to a truly digital system may require a change in 

the data collected to better align with the way 

data are collected in electronic formats. In 2012, 

FDA published the results of a quality assessment 

of the data they received during the ASTER pilot. 

While the assessment noted the potential value of 

such an automated reporting system, it also 

provided suggestions for improving the quality 

and utility of the data. In the pilot, users selected 

an ADE description from a predefined list of 

relatively broad terms; the authors of the FDA 

report suggested that either this list be amended 

to include standardized terms for these clinical 

events, or users enter free text to describe the 

ADE, which could later be coded. Other 

suggestions included the implementation of 

real-time edit checks to catch illogical data such 

as an ADE date that precedes the initiation date 

of the suspected drug. 

This ADE reporting model is now being 

expanded to include AEs related to medical 

devices. The “ASTER-D” project, focused on 

device safety reporting, builds upon the ASTER 

concepts. A pilot study is currently underway, 

sponsored by FDA’s Center for Devices and 

Radiological Health (CDRH). 

Key Point

New models for gathering and reporting ADEs 

may be able to leverage electronic health data and 

emerging technologies to both improve the 

timeliness of reporting and reduce the burden of 

reporting on health care professionals.

For More Information

http://www.asterstudy.com/

Brajovic S, Piazza-Hepp T, Swartz L, et al. 

Quality assessment of spontaneous triggered 

adverse event reports received by the Food and 

Drug Administration. Pharmacoepidemiol Drug 

Saf. 2012 Jun;21(6):565-70.

Rockoff JD. Pfizer project looks at side effects. 

The Wall Street Journal, January 2, 2009. 

http://online.wsj.com/news/articles/

SB123085142405347511

Linder JA, Haas JS, Iyer A, et al. Secondary use 

of electronic health record data: spontaneous 

triggered adverse drug event reporting. 

Pharmacoepidemiol Drug Saf. 2010 

Dec;19(12):1211-5.
21

Chapter 15. Interfacing Registries With Electronic Health Records

References for Chapter 15

1. Institute of Medicine. Key capabilities of an 

electronic health record system. Washington DC: 

National Academy Press; 2003.

2. Electronic Medical Records: Slow but Steady 

Growth in Ambulatory Care. HIMSS. http://www.

himssanalytics.org/about/pr_10292008.aspx. 

Accessed September 4, 2013.

3. Electronic Health Records at a Glance Fact Sheet. 

Centers for Medicare and Medicaid Services. 

http://www.cms.gov/apps/media/press/factsheet.as

p?Counter=3788&intNumPerPage=10&checkDate

=&checkKey=&srchType=1&numDays=3500&

srchOpt=0&srchData=&keywordType=All&

chkNewsType=6&intPage=&showAll=&pYear=

&year=&desc=&cboOrder=date. Accessed 

August 16, 2012.

4. Hsiao CJ, Hing E. Use and Characteristics of 

Electronic Health Record Systems Among 

Office-based Physician Practices: United States, 

2001–2012. Hyattsville, MD: National Center for 

Health Statistics; 2012.

5. Medicare & Medicaid EHR Incentive Program 

Meaningful Use Stage 1 Requirements Overview. 

Centers for Medicare and Medicaid Services. 

http://www.cms.gov/Regulations-and-Guidance/

Legislation/EHRIncentivePrograms/Downloads/

MU_Stage1_ReqOverview.pdf. Accessed August 

16, 2012.

6. American Recovery and Reinvestment Act of 

2009. Section 4101. http://www.gpo.gov/fdsys/

pkg/BILLS-111hr1enr/pdf/BILLS-111hr1enr.pdf. 

Accessed September 4, 2013.

7. EHR Clinical Research. ANSI Public Document 

Library. http://publicaa.ansi.org/sites/apdl/

EHR%20Clinical%20Research/Forms/AllItems.

aspx. Accessed August 15, 2012.

8. American Recovery and Reinvestment Act of 

2009. Section 3004. http://frwebgate.access.gpo.

gov/cgi-bin/getdoc.cgi?dbname=111_cong_

bills&docid=f:h1enr.pdf. Accessed August 15, 

2012.

9. Olsen L, Aisner D, McGinnis JM. IOM 

Roundtable on Evidence-Based Medicine. The 

Learning Healthcare System. Washington, DC: 

National Academies Press; 2007.

10. EHR for non-owned clinicians - Coming to terms. 

Life as a Healthcare CIO (blog) Jun 10, 2008. 

http://geekdoctor.blogspot.com/2008/06/ehr-fornon-owned-clinicians-coming-to.html. Accessed 

August 15, 2012.

11. EMR Comparison. EHRScope. http://www.

ehrscope.com/emr-comparison/. Accessed August 

15, 2012.

12. Embi PJ, Jain A, Clark J, et al. Development of an 

electronic health record-based Clinical Trial Alert 

system to enhance recruitment at the point of care. 

AMIA Annu Symp Proc. 2005:231-5. 

PMID: 16779036. PMCID: 1560758.

13. CDISC. Letter from the HIMSS Electronic Health 

Record (EHR) Association to CDISC. October 9, 

2008. http://www.cdisc.org/stuff/contentmgr/

files/0/f5a0121d251a348a87466028e156d3c3/

miscdocs/ehra_cdisc_endorsement_letter_100908.

pdf. Accessed September 4, 2013.

14. SOAP Version 1.2 Part 1: Messaging Framework 

(Second Edition). http://www.w3.org/TR/soap12-

part1/. Accessed September 11, 2013.

15. Enterprise application integration. Wikipedia. 

http://en.wikipedia.org/wiki/Enterprise_

application_integration. Accessed August 15, 

2012.

16. Adapting a Tool To Be caBIG® Compatible. 

National Cancer Institute. https://cabig.nci.nih.

gov/sharable/compatible/. Accessed August 15, 

2012.

17. Clinical Data Acquisition Standards 

Harmonization (CDASH). Oct 12008. http://www.

cdisc.org/stuff/contentmgr/files/0/9b32bc345908a

c4c31ce72b529a3d995/misc/cdash_

std_1_0_2008_10_01.pdf. Accessed August 15, 

2012.

18. Biomedical Research Integrated Domain Group 

(BRIDG). http://bridgmodel.org/. Accessed 

August 15, 2012.

19. The Common Data Element Dictionary-A 

Standard Nomenclature for the Reporting of Phase 

3 Cancer Clinical Trial Data. 14th IEEE 

Symposium on Computer-Based Medical Systems 

(CMBS’01); 2001. http://www.computer.org/csdl/

proceedings/cbms/2001/1004/00/index.html. 

Accessed August 15, 2012.
22

20. McNamara RL, Brass LM, Drozda JP, Jr., et al. 

ACC/AHA key data elements and definitions for 

measuring the clinical management and outcomes 

of patients with atrial fibrillation: a report of the 

American College of Cardiology/American Heart 

Association Task Force on Clinical Data Standards 

(Writing Committee to Develop Data Standards 

on Atrial Fibrillation). Circulation. 2004 Jun 

29;109(25):3223-43. PMID: 15226233.

21. IHE IT Infrastructure Technical Framework 

Supplement 2006–2007. Retrieve Form for Data 

Capture (RFD). http://www.ihe.net/Technical_

Framework/upload/IHE_ITI_TF_Suppl_RFD_

TI_2006_09_25.pdf. Accessed August 15, 2012.

22. Neuer A. Workgroup Sets Priorities to Harmonize 

Standards for EHRs and Research. eCliniqua. Jan 

52009. http://www.bio-itworld.com/news/

2009/01/05/workgroup-sets-EHR-standards.

html?terms=r+kush. Accessed August 15, 2012.

23. El Fadly A, Rance B, Lucas N, et al. Integrating 

clinical research with the Healthcare Enterprise: 

from the RE-USE project to the EHR4CR 

platform. J Biomed Inform. 2011 Dec;44 

Suppl 1:S94-102. PMID: 21888989.

24. Integrating the Healthcare Enterprise. Profile 

Abstract: Retrieve Protocol for Execution. http://

wiki.ihe.net/index.php?title=Retrieve_Protocol_

for_Execution#Profile_Abstract. Accessed 

August 15, 2012.

25. Gabriel D, Bain L, Popat A. IHE Quality, Research 

and Public Health (QRPH) Technical Framework 

Supplement. Trial Implementation. Redaction 

Services (RSP). August 30, 2010. http://www.ihe.

net/Technical_Framework/upload/IHE_QRPH_

Suppl_RSP_Rev1-1_TI_2010-08-30.pdf. 

Accessed September 3, 2013.

26. IHE Quality, Research and Public Health (QRPH) 

Technical Framework Supplement 2009-2010. 

Drug Safety Content Profile (DSC). August 10, 

2009. http://www.ihe.net/Technical_Framework/

upload/IHE_QRPH_TF_Supplement_Drug_

Safety_Content_DSC_TI_2009-08-10.pdf. 

Accessed September 4, 2013.

27. Clinical Research Healthcare Link. CDISC. http://

www.cdisc.org/stuff/contentmgr/files/0/f5a0121d2

51a348a87466028e156d3c3/miscdocs/himss08_

flyer_final.pdf. Accessed August 15, 2012.

28. ASTER. A Collaborative Study to Improve Drug 

Safety. http://www.asterstudy.com. Accessed 

August 15, 2012.

29. Pisetsky D, Antoline D, editors. Measuring quality 

of care is here to stay—and the ACR can help. 1. 

Vol. 3. The Rheumatologist; 2009. From the 

College: News From the ACR and the ARHP; pp. 

7–11. http://www.the-rheumatologist.org/details/

article/873517/Measuring_Quality_of_Care_Is_

Here_to_Stayand_the_ACR_Can_Help.html. 

Accessed August 20, 2012.

30. athenahealth. Medical Practices Maximize 

Incentive Payments Using athenahealth’s National 

Physician Network for the Physician Quality 

Reporting Initiative. Press Release, October 20, 

2008. http://markets.financialcontent.com/stocks/

news/read/6903848/Medical_Practices_

Maximize_Incentive_Payments_Using_

athenahealth’s_National_Physician_Network_for_

the_Physician_Quality_Reporting_Initiative. 

Accessed August 15, 2012.

31. Behrman RE, Benner JS, Brown JS, et al. 

Developing the Sentinel System--a national 

resource for evidence development. N Engl J Med. 

2011 Feb 10;364(6):498-9. PMID: 21226658.

32. Brown JS, Holmes JH, Shah K, et al. Distributed 

health data networks: a practical and preferred 

approach to multi-institutional evaluations of 

comparative effectiveness, safety, and quality of 

care. Med Care. 2010 Jun;48(6 Suppl):S45-51. 

PMID: 20473204.

33. Maro JC, Platt R, Holmes JH, et al. Design of a 

national distributed health data network. Ann 

Intern Med. 2009 Sep 1;151(5):341-4. 

PMID: 19638403.

34. PopMedNet. http://www.popmednet.org. Accessed 

August 16, 2012.

35. PopMedNet. Initiatives & Networks. 

http://www.popmednet.org/?page_id=41. 

Accessed September 3, 2013.

36. PopMedNet. FAQs. http://www.popmednet.

org/?page_id=45. Accessed September 3, 2013.

37. Hinman AR, Ross DA. Immunization registries 

can be building blocks for national health 

information systems. Health Aff (Millwood). 2010 

Apr;29(4):676-82. PMID: 20368598.

38. HIT Standards Committee. Summary of the 

September 15, 2009 Meeting. https://www.aamc.

org/download/90264/data/hit_standards_

committee_meeting_20090915.pdf. Accessed 

September 3, 2013.

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources
23

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

1. Introduction

The purpose of this chapter is to identify important 

technical and legal considerations for researchers 

and research sponsors interested in linking data in 

a patient registry with additional data, such as data 

from claims or other administrative files or from 

another registry. Its goal is to help these 

researchers find an appropriate way to address 

their critical research questions, remain faithful to 

the conditions under which the data were 

originally collected, and protect individual patients 

by safeguarding their privacy and maintaining the 

confidentiality of the data under applicable law.

There are two equally important questions to 

address in the planning process: (1) What is a 

feasible technical approach to linking the data, and 

(2) is the linkage legally feasible under the 

permissions, terms, and conditions that applied to 

the original compilations of each data set? Legal 

feasibility depends on the applicability to the 

specific purpose of the data linkage of Federal and 

State legal protections for the confidentiality of 

health information and participation in human 

research, and also on any specific permissions 

obtained from individual patients for the use of 

their health information. Indeed, these projects 

require a great deal of analysis and planning, as 

the technical approach chosen may be influenced 

by permitted uses of the data under applicable 

regulations, while the legal assessment may 

change depending on how the linkage needs to be 

performed and the nature and purpose of the 

resulting linked data set. Tables 16-1 and 16-2, 

respectively, list regulatory and technical questions 

for the consideration of data linkage project 

leaders during the planning of a project. The 

questions are intended to assist in organizing the 

resources needed to implement the project, 

including the statistical, regulatory, and collegial 

advice that might prove helpful in navigating the 

complexities of data linkage projects. This chapter 

presumes that investigators have identified an 

explicit purpose for the data linkage in the form of 

a scientific question they are trying to answer. The 

nature of this objective is critical to an assessment 

of the applicable regulatory requirements for uses 

of the data. For example, to the extent the Health 

Insurance Portability and Accountability Act 

(HIPAA) Privacy Rule applies, the use or 

disclosure of protected health information (PHI) 

for the registry purpose will need to fall into one 

of the specific regulatory permissions and comply 

with the relevant requirements for the permission 

(e.g., health care quality–related activities, public 

health practice, research, or some combination of 

these purposes), or individual authorization must 

be obtained. If research is one purpose of the 

project, then the Common Rule (Federal human 

subjects protection regulations) is also likely to 

apply to the project. More information on HIPAA 

and the Common Rule is provided in Chapter 7.

The application of the HIPAA Privacy and 

Security Rules depends on the origins of the data 

sets being linked, and such origins may also 

influence the feasibility of making the data 

linkage. Investigators should know the source of 

the original data, the conditions under which they 

were compiled, and what kinds of permissions, 

from both individual patients and the custodial 

institutions, apply to the data. Health information 

is most often data with two sources, individual and 

institutional; these sources may have legal rights 

and continuing interests in the use of the data.

It is important to be aware that the legal 

requirements may change, and that, in fact, the 

protections limiting the research use of health 

information are likely to change in response to 

continued development of electronic health 

information technologies.

This chapter has six sections focusing on core 

issues in three major parts: Technical Aspects of 

Data Linkage Projects, Legal Aspects of Data 

Linkage Projects, and Risk Mitigation for Data 

Linkage Projects. The Technical Aspects section 

discusses the reasons for and technical methods of 

linking data sets containing health information, 

including data held in registries. It should be noted 

that this list of techniques is not intended to be 
24

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

comprehensive, and these techniques have 

limitations for certain types of studies. The Legal 

Aspects section defines important concepts, 

including the different definitions of “disclosure” 

as used by statisticians and in the HIPAA Privacy 

Rule. This section also discusses the risks of 

identification of individuals inherent in data 

linkage projects and describes the legal standards 

of the HIPAA Privacy Rule that pertain to these 

risks. Finally, the Risk Mitigation section 

summarizes both recognized and developing 

technical methods for mitigating the risks of 

identification. Appendix D consists of a 

hypothetical data linkage project intended to 

provide context for the technical and legal 

information presented below. Case Examples 36, 

37, 38, and 39 describe registry-related data 

linkage activities. Chapter 18 provides information 

on analyzing linked data sets. While some of the 

concepts presented are applicable to other 

important nonpatient identities that might be at 

risk in data linkage, such as provider identities, 

those issues are beyond the scope of the discussion 

below.

2. Technical Aspects of Data 

Linkage Projects

2.1 Linking Records for Research and 

Improving Public Health

Data in registries regarding the health of 

individuals come in many forms. Most of these 

data were originally gathered for the delivery of 

clinical services or payment for those services, and 

under promises or legal guarantees of 

confidentiality, privacy, and security. The sources 

of data may include individual doctors’ records, 

billing information, vital statistics on births and 

deaths, health surveys, and data associated with 

biospecimens, among other sources. 

The broad goal of registries is to amass data from 

potentially diverse sources to allow researchers to 

explore and evaluate alternative health outcomes 

in a systematic fashion. This goal is usually 

accomplished by gathering data from multiple 

sources and linking the data across sources, either 

with explicit identifiers designed for linking, or in 

a probabilistic fashion via the characteristics of the 

individuals to whom the data correspond. From the 

research perspective, the more data included the 

better, both in terms of the number of cases and 

the details and the extent of the health information. 

The richer the database, the more likely it is that 

data analysts will be able to discover relationships 

that might affect or improve health care. On the 

other hand, many discussions about privacy 

protection focus on limiting the level of detail 

available in data to which others have access.

There is an ethical obligation to protect patient 

interests when collecting, sharing, and studying 

person-specific biomedical information.1 Many 

people fear that information derived from their 

medical or biological records will be used against 

them in employment decisions, result in limitations 

to their access to health or life insurance, or cause 

social stigma.2, 3 These fears are not unfounded, 

and there have been various cases in which it was 

found that an individual’s genetic characteristics or 

clinical manifestations were used in a manner 

inconsistent with an individual’s expectations of 

privacy and fair use.4 If individuals are afraid that 

their health-related information may be associated 

with them or used against them, they may be less 

likely to seek treatment in a clinical context or 

participate in research studies.5

A tension exists between the broad goals of 

registries and regulations protecting individually 

identifiable information. Approaches and formal 

methodologies that help mediate this tension are 

the principal technical focus of this chapter. To 

understand the extent to which these tools can 

assist data linkages involving registry data, one 

needs to understand the risks of identification in 

different types of data.

There is a large body of Federal law relating to 

privacy. A recent comprehensive review of privacy 

law and its effects on biomedical research 

identified no fewer than 15 separate Federal laws 

pertaining to health information privacy.6 There 

are also special Federal laws governing health 

information related to substance abuse.7 A full 

review of all laws related to privacy, 

confidentiality, and security of health information 

would also consider separate State privacy 

protections as well as State laws pertaining to the 
25

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

confidentiality of data. Nevertheless, the legal 

aspects of this chapter focus only on the Federal 

regulations commonly referred to as the HIPAA 

Privacy Rule.

2.2 What Do Privacy, Confidentiality, 

and Disclosure Mean?

Privacy is a term whose definition varies with 

context.8 In the HIPAA Privacy Rule, the term 

refers to protected health information (PHI)—

individually identifiable health information 

transmitted or maintained by a covered entity or 

business associate—that is to be used or disclosed 

only as expressly permitted or required by the 

Rule, and that is safeguarded against inappropriate 

uses and disclosures. The Privacy Rule addresses 

to whom the custodian of PHI, a covered entity or 

its business associate, may transmit the 

information and under what conditions. The Rule 

also establishes three levels of identifiability of 

health information: (1) fully identifiable data; 

(2) data that lack certain direct identifiers, 

otherwise known as a limited data set; and 

(3) de-identified data. Registries commonly 

acquire identifiable data and may create the last 

two categories of data in accordance with the 

Privacy Rule. Along this spectrum of data 

identifiability, the HIPAA Privacy Rule applies 

different legal standards and protections,5

extending the most stringent protections to data 

containing direct identifiers and none for deidentified information, which is not considered 

PHI. Not all registries contain PHI; Chapter 7 

provides more information on how PHI is defined 

under HIPAA. 

Confidentiality broadly refers to a quality or 

condition of protection accorded to statistical 

information as an obligation not to permit the 

transfer of that information to an unauthorized 

party.5 Confidentiality can be afforded to both 

individuals and health care organizations. A 

different notion of confidentiality, arising from the 

special relationship between a clinician and 

patient, refers to the ethical, legal, and professional 

obligation of those who receive information in the 

context of a clinical relationship to respect the 

privacy interests of their patients. Most often the 

term is used in the former sense and not in the 

latter, but these two meanings inevitably overlap in 

a discussion of health information as data. The 

methods for disclosure limitation described here 

have been developed largely in the context of 

confidentiality protection, as defined by laws, 

regulations, and especially by the practices of 

statistical agencies.

Disclosure for purposes of this discussion has two 

different meanings: one is technical and the other 

is regulatory and contained in the HIPAA Privacy 

Rule.

In the field of statistics, disclosure relates to the 

attribution of information to the source of the data, 

regardless of whether the data source is an 

individual or an organization. Three types of 

disclosure of data possess the capacity to make the 

identity of particular individuals known: identity 

disclosure, attribute disclosure, and inferential 

disclosure.

Identity disclosure occurs when the data source 

becomes known from the data release itself.9, 10

Attribute disclosure occurs when the released data 

make it possible to infer the characteristics of an 

individual data source more accurately than would 

have otherwise been possible.8, 9 The usual way to 

achieve attribute disclosure is through identity 

disclosure. First, one identifies an individual 

through some combination of variables and then 

learns about the values of additional variables 

included in the released data. Attribute disclosure 

may occur, however, without identity disclosure, 

such as when all people from a population 

subgroup share a characteristic and this quantity 

becomes known for any individual in the 

subgroup. 

Inferential disclosure relates to the probability of 

identifying a particular attribute of a data source. 

Because almost any data release can be expected 

to increase the likelihood of an attribute being 

associated with a data source, the only way to 

guarantee protection is to release no data at all. It 

is for this reason that researchers use certain 

methods not to prevent disclosure, but to limit or 

control the nature of the disclosure. These methods 

are known as disclosure limitation methods or 

statistical disclosure control.11
26

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

Disclosure under the HIPAA Privacy Rule means 

the release, transfer, provision of, access to, or 

divulging in any other manner, of information 

outside of the entity holding the information.12

2.3 Linking Records and Probabilistic 

Matching

Computer-assisted record linkage goes back to the 

1950s, and was put on a firm statistical foundation 

by Fellegi and Sunter.13 Most common techniques 

for record linkage either rely on the existence of 

unique identifiers or use a structure similar to the 

one Fellegi and Sunter described with the 

incorporation of formal statistical modeling and 

methods, as well as new and efficient 

computational tools.14, 15 The simplest way to 

match records from separate databases is to use a 

so-called “deterministic” method of linking 

databases in which unique identifiers exist for each 

record. In the United States, when these identifiers 

exist they might be names or Social Security 

numbers; however, these particular identifiers may 

not in fact be unique.16 As a result, some form of 

probabilistic approach is typically used to match 

the records. Thus, there is little actual difference 

between methods using deterministic versus 

probabilistic linkage, except for the explicit 

representation of uncertainty in the matching 

process in the latter.

The now-standard approach to record linkage is 

built on five key components for identifying 

matching pairs of records across two databases:13

1. Represent every pair of records using a vector 

of features (variables) that describe similarity 

between individual record fields. Features can 

be Boolean, discrete, or continuous.

2. Place feature vectors for record pairs into three 

classes: matches (M), non-matches (U), and 

possible matches (P). These correspond to 

“equivalent,” “nonequivalent,” and “possibly 

equivalent” (e.g., requiring human review) 

record pairs, respectively.

3. Perform record-pair classification by 

calculating the ratio (P ( | M)) / (P ( | U)) for 

each candidate record pair, where  is a feature 

vector for the pair and P ( | M) and P ( | U) 

are the probabilities of observing that feature 

vector for a matched and non-matched pair, 

respectively. Two thresholds based on desired 

error levels—Tμ and T—optimally separate 

the ratio values for equivalent, possibly 

equivalent, and nonequivalent record pairs.

4. When no training data in the form of duplicate 

and nonduplicate record pairs are available, 

matching can be unsupervised; that is, 

conditional probabilities for feature values are 

estimated using observed frequencies in the 

records to be linked.

5. Most record pairs are clearly nonmatches, so 

one need not consider them for matching. This 

situation is managed by “blocking,” or 

partitioning the databases based on geography 

or some other variable in both databases, so 

that only records in comparable blocks are 

compared. Such a strategy significantly 

improves efficiency.

The first four components lay the groundwork for 

accurate record-pair matching using statistical or 

machine-learning prediction models such as 

logistic regression. The fewer identifiers used in 

steps 1 and 2, the poorer the match is likely to be. 

Accuracy is well known to be high when there is a 

1–1 match between records in the two databases, 

and deteriorates as the overlap between the files 

decreases and the measurement error in the feature 

values consequently increases. 

The fifth component provides for efficient 

processing of large databases, but to the extent that 

blocking is approximate and possibly inaccurate, 

its use decreases the accuracy of record-pair 

matching. The less accurate the matching, the 

more error (i.e., records not matched or matched 

inappropriately) there will be in the merged 

registry files. This error will impede the quality of 

analyses and findings from the resulting data.17-19

This standard approach has problems when 

(1) there are lists or files with little overlap, 

(2) there are undetected duplications within files, 

and (3) one needs to link three or more lists. In the 

latter case, one essentially matches all lists in 

pairs, and then resolves discrepancies. 

Unfortunately, there is no single agreed-upon way 

to do this, but some principled approaches have 

recently been suggested.20 Record linkage 

methodology has been widely used by statistical 

agencies, especially the U.S. Census Bureau. The 
27

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

methodology has been combined with disclosure 

limitation techniques such as the addition of 

“noise” to variables in order to produce public use 

files that the agencies believe cannot be linked 

back to the original databases used for the record 

linkage. Another technique involves protecting 

individual databases by stripping out identifiers 

and then attempting record linkage. This procedure 

has two disadvantages: first, the quality of matches 

is likely to decrease markedly; and second, the 

resulting merged records will still need to be 

protected by some form of disclosure limitation. 

Therefore, as long as there are no legal restrictions 

against the use of identifiers for record linkage 

purposes, it is preferable to use detailed identifiers 

to the extent possible and to remove them 

following the matching procedure.

Currently there are no special features of registry 

data known to enhance or inhibit matching. 

Registry data may be easier targets for reidentification because the specifics of diseases or 

conditions usually help to define the registries. In 

the United States, efforts are often made to match 

records using Social Security numbers. There are 

large numbers of entry errors for these numbers in 

many databases, and there are problems associated 

with multiple people using one number and some 

people using multiple numbers.16 Lyons and 

colleagues describe a very large-scale matching 

exercise in the United Kingdom linking multiple 

health care and social services data sets using 

National Health Service numbers and various 

alternative sets of matching variables in the spirit 

of the record linkage methods described above. 

They report achieving accurate matching at rates 

of only about 95 percent.21

2.4 Procedural Issues in Linking Data 

Sets

It is important to understand that neither data nor 

link can be unambiguously defined. For instance, a 

data set may be altered by the application of tools 

for statistical disclosure limitation, in which case it 

is no longer the same data set. Linkage need not 

mean, as it is customarily construed, “bringing the 

two (or more) data sets together on a single 

computer.” Many analyses of interest can be 

performed using technologies that do not require 

literal integration of the data sets. 

Even the relationship between data sets can vary. 

Two data sets can hold the same attributes for 

different individuals (horizontal partitioning); for 

example, one data set may contain information for 

individuals born before a certain date, while a 

second data set contains the same information for 

individuals born after that date. Or, two data sets 

may contain different attributes for the same 

individuals (vertical partitioning); for example, 

one data set may contain clinician-reported 

information for a set of individuals, while a second 

data set contains laboratory data for the same 

individuals. Finally, some data sets may contain a 

complex combination of different attributes for 

different individuals.

The process of linking horizontally partitioned 

data sets engenders little incremental risk of 

re-identification. There is, in almost all cases, no 

more information about a record on the combined 

data set than was present in the individual data set 

containing it. Moreover, any analysis requiring 

only data summaries (i.e., in technical terms, 

sufficient statistics) that are additive across the 

data sets can be performed using tools based on 

the computer science concept of secure 

summation.22 Examples of analyses for which this 

approach works include creation of contingency 

tables, linear regression, and some forms of 

maximum likelihood estimation.

Only in a few cases have comparable techniques 

for vertically partitioned data been well enough 

understood to be employed in practice.23 Instead, it 

is usually necessary to actually link individual 

subjects’ records that are contained in two or more 

data sets. This process is inherently and 

unavoidably risky because the combined data set 

contains more information about each subject than 

either of the components.

Suppose that each of the two data sets to be linked 

contains the same unique identifiers (for 

individuals, an example is Social Security 

numbers) in all of the records. In this case, 

techniques based on cryptography (e.g., 

homomorphic encryption24 and hash functions) 

enable secure determination of individuals 

common to both data sets and assignment of 

unique but uninformative identifiers to the shared 

records. The combined data set can then be purged 
28

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

of individual identifiers and altered to further limit 

re-identification. These alterations will of necessity 

reduce the accuracy of standard statistical analyses 

compared with an unaltered data set.

Such linkage techniques are computationally very 

complex, and may need to involve trusted third 

parties without access to information in either data 

set other than the common identifier.25 Therefore, 

in many cases the database custodian may prefer to 

remove identifiers and carry out statistical 

disclosure limitation prior to linkage. It is 

important to understand that this latter approach 

compromises, perhaps irrevocably, the linkage 

process, and may introduce substantial errors into 

the linked data set that later— perhaps 

dramatically—alter the results of statistical 

analyses.

Many techniques for record linkage depend at 

some level on the presence of combinations of 

attributes in both databases that are unique to 

individuals but do not lead to re-identification—a 

combination that may be difficult to find. For 

instance, the combination of date of birth, gender, 

and ZIP Code of residence might be present in 

both databases. It is estimated that this 

combination of attributes uniquely characterizes a 

significant portion of the U.S. population—

somewhere between 65 and 87 percent, or even 

higher for certain subpopulations—so that reidentification would only require access to a 

suitable external database.26, 27 Other techniques 

such as the Fellegi-Sunter record linkage methods 

described above are more probabilistic in nature. 

They can be effective, but they also introduce data 

quality effects that cannot readily be characterized, 

and the intrinsic error associated with the 

matching will need to be accounted for in some 

fashion when the linked data set is analyzed. 

Simulations and sensitivity analyses may help 

clarify the extent of the issues here, but will rarely 

be sufficient.

No matter how linkage is performed, a number of 

other issues should be addressed. For instance, 

comparable attributes should be expressed in the 

same units of measure in both data sets 

(e.g., English or metric values for weight). Also, 

conflicting values of attributes for each individual 

common to both databases need reconciliation. 

Another issue involves the management of patient 

records that appear in only one database; the most 

common decision is to drop them. Data quality 

provides another example; it is one of the least 

understood statistical problems and has multiple 

manifestations.28 Even assuming some limited 

capability to characterize data quality, the 

relationship between the quality of the linked data 

set and the quality of each component should be 

considered. The linkage itself can produce quality 

degradation. For example, there is reason to 

believe that the quality of a linked data set is 

strictly less than that of either component, and not, 

as might be supposed, somewhere between the 

two.

Finally, it is important to understand that there 

exist endemic risks to data linkage. Anyone with 

access to one of the original data sets and the 

linked data set may learn, even if imperfectly, the 

values of attributes in the other. It may not be 

possible to determine what knowledge the linkage 

will create without actually executing the linkage. 

For these reasons, strong consideration should be 

given to forms of data protection such as licensing 

and restricted access in research data centers, 

where both analyses and results can be controlled.

3. Legal Aspects of Data 

Linkage Projects

3.1 Risks of Identification

The HIPAA Privacy Rule describes two methods 

for de-identifying health information.29 One 

method requires a formal determination by a 

qualified expert (e.g., a qualified statistician) that 

the risk is very small that an individual could be 

identified. The other method requires the removal 

of 18 specified identifiers of the individual and of 

the individual’s relatives, household members, and 

employers, as well as no actual knowledge that the 

remaining information could be used alone or in 

combination with other information to identify the 

individual. (See Chapter 7 for more information.) 

For more information about methods of deidentification under the HIPAA Privacy Rule, see 

the recent HHS guidance published on this topic.30

The data removal process alone may not be 

sufficient to remove risks of re-identification. 
29

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

Disclosure

Confidentiality 

obligations

Harm

Residual data especially vulnerable to disclosure 

threats include (1) geographic detail, 

(2) longitudinal information, and (3) extreme 

values (e.g., income). In addition, variables that 

are available in other accessible databases pose 

special risks. 

Statistical organizations such as the National 

Center for Health Statistics have traditionally 

focused on the issue of identity disclosure and thus 

refused to report information in which individuals 

or institutions can be identified. Concerns about 

identity disclosure arise, for example, when a data 

source is unique in the population for the 

characteristics under study, and is directly 

identifiable in the database to be released. But such 

uniqueness and subsequent identity disclosure may 

not reveal any information other than the 

association of the source with the data collected in 

the study. In this sense, identity disclosure may 

only be a technical violation of a promise of 

confidentiality. Thus, uniqueness only raises the 

issue of possible confidentiality problems resulting 

from identification. A separate issue is whether the 

release of information is one that is permitted by 

the HIPAA Privacy Rule or is authorized by the 

data source.

The foregoing discussion implicitly introduces the 

notion of “harm,” which is not the same as a 

breach of confidentiality. For example, it is 

possible for a pledge of confidentiality to be 

technically violated, but produce no harm to the 

data source because the information is “generally 

known” to the public. In this case, some would 

argue that additional data protection is not 

required. Conversely, information on individuals 

or organizations in a release of sample statistical 

data may well increase the information about 

characteristics of individuals or organizations not 

in the sample. This information may produce an 

inferential disclosure for such individuals or 

organizations and cause them harm, even though 

there was no confidentiality obligation. Skinner31

suggests the separation of assessment of disclosure 

potential from harm.

 Figure 16–1 depicts the overlapping relationships 

among confidentiality, disclosure, and harm.

Figure 16–1. Relationships among confidentiality, disclosure, and harm 

Some people believe that the way to ensure 

confidentiality and prevent identity disclosure is to 

arrange for individuals to participate in a study 

anonymously. In many circumstances, such a 

belief is misguided, because there is a key 

distinction between collecting information 

anonymously and ensuring that personal identifiers 

are not inappropriately made available. Moreover, 

clinical health care data are simply not collected 

anonymously. Not only do patient records come 
30

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

with multiple identifiers crucial to ensuring patient 

safety for clinical care, but they also contain other 

information that may allow the identification of 

patients even if direct identifiers are stripped from 

the records.

Moreover, health- or medicine-related data may 

also come from sample surveys in which the 

participants have been promised that their data will 

not be released in ways that would allow them to 

be individually identified. Disclosure of such data 

can produce substantial harm to the personal 

reputations or financial interests of the 

participants, their families, and others with whom 

they have personal relationships. For example, in 

the pilot surveys for the National Household 

Seroprevalence Survey, the National Center for 

Health Statistics moved to make responses during 

the data collection phase of the study anonymous 

because of the harm that could potentially result 

from information that an individual had an HIV 

infection or engaged in high-risk behavior. But 

such efforts still could not guarantee that one 

could not identify a participant in the survey 

database. 

The question about the confidentiality of registry 

data persists after an individual’s death, in part 

because of the potential for harm to others. The 

health information of decedents is subject to the 

HIPAA Privacy Rule until 50 years after their 

death (see Chapter 7 for more information), and 

several statistical agencies explicitly treat the 

identification of a deceased individual as a 

violation of their confidentiality obligations.

3.1.1 Examples of Patient Re-Identification

For years, the confidentiality of health information 

has been protected through a process of “deidentification.” This protection entails the removal 

of person-specific features such as names, 

residential street addresses, phone numbers, and 

Social Security numbers. However, as discussed 

above, de-identification does not guarantee that 

individuals may not be identified from the 

resulting data. On multiple occasions, it has been 

shown that de-identified health information can be 

“re-identified” to a particular patient without 

hacking or breaking into a private health 

information system. For instance, before the 

HIPAA de-identification standards were created, 

Latanya Sweeney, a graduate student at the 

Massachusetts Institute of Technology in the 

mid-1990s, showed that de-identified hospital 

discharge records, which were made publicly 

available at the State level, could be linked to 

identifiable public records in the form of voter 

registration lists. Her demonstration received 

notoriety because it led to the re-identification of 

the medical status of the then-governor in the 

Commonwealth of Massachusetts.32 This result 

was achieved by linking the data resources on their 

common fields of patient’s date of birth, gender, 

and ZIP Code. As noted earlier, this combination 

identifies unique individuals in the United States at 

a rate estimated at somewhere between 65 and 87 

percent or even higher in certain subpopulations.27

3.1.2 High-Risk Identifiers

One response to the Sweeney demonstration was 

the HIPAA Privacy Rule method for deidentification by removal of data elements. This 

process requires the removal of 18 explicit 

identifiers from patient information before it is 

considered de-identified, including dates of birth 

and ZIP Codes. (See Chapter 7.)33 Nonetheless, 

even the removal of these data elements may fail to 

prevent re-identification, as there may be residual 

features that can lead to identification. The extent 

to which residual features can be used for reidentification depends on the availability of 

relevant data fields. Thus, one can roughly 

partition identifiers into “high-risk” and relatively 

“low-risk” features. The high-risk features are 

documented in multiple environments and publicly 

available. These features could be exploited by any 

recipient of such records. For instance, patient 

demographics are high-risk identifiers. Even 

de-identified health information permitted under 

the HIPAA Privacy Rule may leave certain 

individuals at risk for identification if the data are 

combined with public data resources containing 

similar features, such as public records containing 

birth, death, marriage, voter registration, and 

property assessment information.30, 34-36

3.1.3 Relatively Low-Risk Identifiers

In contrast, lower risk data elements do not appear 

in public records and are less available. For 

instance, clinical features, such as an individual’s 
31

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

diagnosis and treatments, are relatively static 

because they are often mapped to standard codes 

for billing purposes. These features might appear 

in de-identified information, such as hospital 

discharge databases, as well as in identified 

resources such as electronic medical records. 

While combinations of diagnostic and treatment 

codes might uniquely describe an individual 

patient in a population, the identifiable records are 

available to a much smaller group than the general 

public. Moreover, these select individuals, such as 

the clinicians and business associates of the 

custodial organization for the records, are 

ordinarily considered to be trustworthy, because 

they owe independent ethical, professional, and 

legal duties of confidentiality to the patients.

3.1.4 Special Issues With Linkages to 

Biospecimens

Health care is increasingly moving towards 

evidence-based and personalized systems. In 

support of this trend, there is a growing focus on 

associations between clinical and biological 

phenomena. In particular, the decreasing cost of 

genome sequencing technology has facilitated a 

rapid growth in the volume of biospecimens and 

derived DNA sequence data. As much of this 

research is sponsored through Federal funding, it is 

subject to Federal data sharing requirements. 

However, biospecimens, and DNA in particular, 

are inherently unique and there are a number of 

routes by which DNA information can be 

identified to an individual.37 For instance, there are 

over 1 million single nucleotide polymorphisms 

(SNPs) in the human genome; these little snippets 

of DNA are often used to make genetic 

correlations with clinical conditions. Yet it is 

estimated that fewer than 100 SNPs can uniquely 

represent an individual.38 Thus, if de-identified 

biological information is tied to sensitive clinical 

information, it may provide a match to the 

identified biological information—as, for example, 

in a forensic setting.39

Biospecimens and information derived from them 

are of particular concern because they can convey 

knowledge not only about the individual from 

whom they are derived, but also about other related 

individuals. For instance, it is possible to derive 

estimates about the DNA sequence of relatives.40

If the genetic information is predictive or 

diagnostic, it can adversely affect the ability of 

family members to obtain insurance and 

employment, or it may cause social 

stigmatization.41-43 The Genetic Information 

Nondiscrimination Act of 2008 (GINA) prohibits 

health insurers from using genetic information 

about individuals or their family members, 

whether collected intentionally or incidentally, in 

determining eligibility and coverage, or in 

underwriting and setting premiums.44 Insurers, in 

collaboration with external research entities, may 

request that policyholders undergo genetic testing, 

but a refusal to do so cannot be permitted to affect 

the premium or result in medical underwriting. 

4. Risk Mitigation for Data 

Linkage Projects 

4.1 Methodology for Mitigating the 

Risk of Re-Identification

The disclosure limitation methods briefly 

described in this section are designed to protect 

against identification of individuals in statistical 

databases, and are among the techniques that data 

linkage projects involving registries are most likely 

to use. One problem these methods do not address 

is the simultaneous protection of individual and 

institutional data sources. The discussion here also 

relates to the problems addressed by secure 

computation methodologies, which are explored in 

the next section.

4.1.1 Basic Methodology for Statistical 

Disclosure Limitation

Duncan45 categorizes the methodologies used for 

disclosure limitation in terms of disclosurelimiting masks, i.e., transformations of the data 

where there is a specific functional relationship 

(possibly stochastic) between the masked values 

and the original data. The basic idea of masking 

involves data transformations. The goal is to 

transform an n x p data matrix Z through pre- and 

post-multiplication and the possible addition of 

noise, such as depicted in Equation (1): 

Z AZB+C (1) 

where A is a matrix that operates on cases, B is a 

matrix that operates on variables, and C is a matrix 
32

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

that adds perturbations or noise to the original 

information. Matrix masking includes a wide 

variety of standard approaches to disclosure 

limitation:

• Addition of noise

• Release of a subset of observations (deleting 

rows from Z)

• Cell suppression for cross-classifications

• Inclusion of simulated data (addition of rows 

to Z)

• Release of a subset of variables (deletion of 

columns from Z)

• Switching of selected column values for pairs 

of rows (data swapping)

This list also omits some methods, such as microaggregation and doubly random swapping, but it 

provides a general idea of the types of techniques 

being developed and applied in a variety of 

contexts, including medicine and public health. 

The possibilities of both identity and attribute 

disclosure remain even when a mask is applied to 

a data set, although the risks may be substantially 

diminished.

Duncan suggests that we can categorize most 

disclosure-limiting masks as suppressions 

(e.g., cell suppression), recodings (e.g., collapsing 

of rows or columns, or swapping), or samplings 

(e.g., release of subsets), although he also allows 

for simulations as discussed below. Further, some 

masking methods alter the data in systematic ways 

(e.g., through aggregation or through cell 

suppression), whereas others do it through random 

perturbations, often subject to constraints for 

aggregates. Examples of perturbation methods are 

controlled random rounding, data swapping, and 

the post-randomization method (PRAM) of 

Gouweleeuw,46 which has been generalized by 

Duncan and others. One way to think about 

random perturbation methods is as restricted 

simulation tools. This characterization connects 

them to other types of simulation approaches.

Various authors pursue simulation strategies and 

present general approaches to “simulating” from a 

constrained version of the cumulative, empirical 

distribution function of the data. In 1993, Rubin 

asserted that the risk of identity disclosure could 

be eliminated by the use of synthetic data (in his 

case using Bayesian methodology and multiple 

imputation techniques) because there is no direct 

function link between the original data and the 

released data.47 Said another way, the data remain 

confidential because simulated individuals have 

replaced all of the real ones. Raghunathan, Reiter, 

and Rubin48 provide details on the implementation 

of this approach. Abowd and Woodcock (in their 

chapter in Doyle et al., 2001)49 describe a detailed 

application of multiple imputation and related 

simulation technology for a longitudinally linked 

individual and work history data set. With both 

simulation and multiple-imputation methodology, 

however, it is still possible that the data values of 

some simulated individuals remain virtually 

identical to those in the original sample, or at least 

close enough that the possibility of both identity 

and attribute disclosure remain. As a result, checks 

should be made for the possibility of unacceptable 

disclosure risk.

Another important feature of the statistical 

simulation approach is that information on the 

variability of the data set is directly accessible to 

the user. For example, in the Fienberg, Makov, and 

Steele50 approach for categorical data, the data 

user can begin with the reported table and 

information about the margins that are held fixed, 

and then run the Diaconis-Sturmfels Monte Carlo 

Markov chain algorithm to regenerate the full 

distribution of all possible tables with those 

margins. This technique allows the user to make 

inferences about the added variability in a 

modeling context that is similar to the approach to 

inference in Gouweleeuw and colleagues.46

Similarly, Raghunathan and colleagues proposed 

the use of multiple imputations to directly measure 

the variability associated with the posterior 

distribution of the quantities of interest.48 As a 

consequence, Rubin showed that simulation and 

perturbation methods represent a major 

improvement in access to data over cell 

suppression and data swapping without sacrificing 

confidentiality. These methods also conform to the 

statistical principle allowing the user of released 

data to apply standard statistical operations 

without being misled. 

There has been considerable research on disclosure 

limitation methods for tabular data, especially in 

the form of multidimensional tables of counts 
33

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

(contingency tables). The most popular methods 

include a process known as cell suppression, 

which systematically deletes the values in selected 

cells in the table and collapses categories. This 

process is a form of aggregation. While cell 

suppression methods have been very popular 

among the U.S. Government statistical agencies, 

and are useful for tables with nonnegative entries 

rather than simple counts, they also have major 

drawbacks. First, good algorithms do not yet exist 

for the methodology when it is associated with 

high-dimensional tables. More importantly, the 

methodology systematically distorts the 

information about the cells in the table for users, 

and, as a consequence, makes it difficult for 

secondary users to draw correct statistical 

inferences about the relationships among the 

variables in the table. For further discussion of cell 

suppression, and for extensive references, see the 

various chapters in Doyle et al.,49 notably the one 

by Duncan and his collaborators.

A special example of collapsing categories 

involves summing over variables to produce 

marginal tables. Instead of reporting the full 

multidimensional contingency table, one or more 

collapsed versions of it might be reported. The 

release of multiple sets of marginal totals has the 

virtue of allowing statistical inferences about the 

relationships among the variables in the original 

table using log-linear model methods (e.g., see 

Bishop, Fienberg, and Holland).51 With multiple 

collapsed versions, statistical theory makes it clear 

that one may have highly accurate information 

about the actual cell entries in the original table. 

As a result, the possibility of disclosures still 

requires investigation. In part to address this 

problem, a number of researchers have recently 

worked on the problem of determining upper and 

lower bounds for the cells of a multi-way table 

given a set of margins; however, other measures of 

risk may clearly be of interest. The problem of 

computing bounds is in one sense an old one, at 

least for two-way tables, but it is also deeply linked 

to recent mathematical developments in statistics 

and has generated a flurry of new research.52, 53

4.1.2 The Risk-Utility Tradeoff

Common to virtually all the methodologies 

discussed in the preceding section is the notion of 

a risk-utility tradeoff, in which the risk of 

disclosure is balanced with the utility of the 

released data (e.g., see Duncan,45 Fienberg,54 and 

their chapter with others in Doyle et al.49). To keep 

this risk at a low level requires applying more 

extensive data masking, which limits the utility of 

what is released. Advocates for the use of 

simulated data often claim that this use eliminates 

the risk of disclosure, but still others dispute this 

claim. See also the recent discussion of risk-utility 

paradigms by Cox and colleagues.55

4.1.3 Privacy-Preserving Data Mining 

Methodologies

With advances in data mining and machine 

learning over the past two decades, a large number 

of methods have been introduced under the banner 

of privacy-preserving computation. The 

methodologies vary, and many of them focus on 

standard tools such as the addition of noise or data 

swapping of one sort or another. But the claims of 

identity protection in this literature are often 

exaggerated or unverifiable. For a discussion of 

some of these ideas and methods, see Fienberg and 

Slavkovic53 and El Emam and colleagues.34 For 

two recent interesting examples explicitly set in 

the context of medical data, see Malin and 

Sweeney56 and Boyens, Krishnan, and Padman.57

The common message of this literature is that 

privacy protection has costs measured in the lack 

of availability of research data. To increase the 

utility of released data for research, some measure 

of privacy protection, however small, needs to be 

sacrificed. It is nonetheless still possible to 

optimize utility, subject to predefined upper 

bounds on what is considered to be acceptable risk 

of identification. See a related discussion in 

Fienberg.58

4.1.4 Cryptographic Approaches to Privacy 

Protection

While the current risks of identification in modern 

databases are similar for statistical agencies and 

biomedical researchers, there are also new 

challenges: from contemporary information 

repositories that store social network data 

(e.g., cell phone, Twitter, and Facebook data), 

product preferences data (e.g., Amazon), Web 

search data, and other sources of information not 

previously archived in a digital format. A recent 

literature emanating from cryptography focuses on 
34

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

algorithmic aspects of this problem with an 

emphasis on automation and scalability of a 

process for conferring anonymity. Automation, in 

turn, presents a fundamentally different 

perspective on how privacy is defined and provides 

for both a formal definition of privacy and proofs 

for how it can be protected. By focusing on the 

properties of the algorithm for anonymity, it is 

possible to formally guarantee the degree of 

privacy protection and the quality of the outputs in 

advance of data collection and publication. 

This new approach, known as differential privacy, 

limits the incremental information a data user 

might learn beyond that which is known before 

exposure to the released statistics. No matter what 

external information is available, the differential 

privacy approach guarantees that the same 

information is learned about an individual, 

whether or not information about the individual is 

present in the database. The papers by Dwork and 

colleagues59, 60 provide an entry point to this 

literature. Differential privacy, as these authors 

describe it, works primarily through the addition of 

specific forms of noise to all data elements and the 

summary information reported, but it does not 

address issues of sampling or access to individuallevel microdata. While these methods are 

intriguing, their utility for data linkages with 

registry data remains an open issue.61

4.1.5 Security Practices, Standards, and 

Technologies

In general, people adopt two different 

philosophical positions about how the 

confidentiality associated with individual-level 

data should be preserved: (1) by “restricted or 

limited information,” that is, restrictions on the 

amount or format of the data released, and (2) by 

“restricted or limited access,” that is, restrictions 

on the access to the information itself. 

If registry data are a public health good, then 

restricted access is justifiable only in situations 

where the confidentiality of data in the possession 

of a researcher cannot be protected through some 

form of restriction on the information released. 

Restricted access is intended to allow use of 

unaltered data by imposing certain conditions on 

users, analyses, and results that limit disclosure 

risk. There are two primary forms of restricted 

access. The first is through licensing, whereby 

users are legally bound by certain conditions, such 

as agreeing not to use data for re-identification and 

to accept advance review of publications. The 

licensure approach allows users to transfer data to 

their sites and use the software of their choice. The 

second approach is exemplified by research data 

centers, discussed in more detail below, and 

remote analysis servers, which are conceptually 

similar to data centers: users, and sometimes 

analyses, are evaluated in advance. The results are 

reviewed, and often limited, in order to limit risk of 

disclosure. The data remain at the holder’s site and 

computers; the difference between a research data 

center and a remote analysis server is whether 

access is in person at a data center or using a 

remote analysis center via the Internet.

4.1.6 Registries as Data Enclaves

Many statistical agencies have built enclaves, often 

referred to as research data centers, where users 

can access and use data in a regulated 

environment. In such settings, the security of 

computer systems is controlled and managed by 

the agency providing the data. Such environments 

may maximize data security. For a more extensive 

discussion of the benefits of restricted access, see 

the chapter by Dunne in Doyle et al.49

These enclaves incur considerable costs associated 

with their establishment and upkeep. A further 

limitation is that the enclave may require the 

physical presence of the data user, which also 

increases the overall cost to researchers working 

with the data. Moreover, such environments often 

prevent users from executing specialized data 

analyses, which may require programming and 

other software development beyond the scope of 

traditional statistical software packages made 

available in the enclave.

The process for granting access to data in enclaves 

or restricted centers involves an examination of the 

research credentials of those wishing to do so. In 

addition, these centers control the physical access 

to confidential data files and they review the 

materials that data users wish to take from the 

centers and to publish. Researchers who are 

accustomed to reporting residual plots and other 

information that allows for a partial reconstruction 

of the original data, at least for some variables, 
35

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

will encounter difficulties, because restricted data 

centers typically do not allow users to remove such 

information.

4.1.7 Accountability

To limit the possibility of re-identification, data 

can be manipulated by the above techniques to 

mitigate risk. At the same time, it is important to 

ensure that researchers are accountable for the use 

of the data sets that are made available to them. 

Best practices in data security should be adopted 

with specific emphasis on authentication, 

authorization, access control, and auditing. In 

particular, each data recipient should be assigned a 

unique login identification, or, if the data are made 

available online, access may be provided through a 

query response server. Prior to each session of data 

access, data custodians should authenticate the 

user’s identity. Access to information should be 

controlled either in a role-based or informationbased manner. Each user access and query to the 

data should be logged to enable auditing functions. 

If there is a breach in data protection, the data 

custodian can investigate the potential cause and 

make any required notifications.

4.1.8 Layered Restricted Access to Databases

In many countries, the traditional arrangement for 

data use involves restrictions on both information 

and access, with only highly aggregated data and 

summary statistics released for public use.

One potential strategy for privacy protection for 

the linkage of registries to other confidential data 

is a form of layered restrictions that combines two 

approaches with differing levels of access at 

different levels of detail in the data. The registry 

might function as an enclave, similar to those 

described above, and in addition, public access 

might be limited to only aggregate data. Between 

these two extremes there might be several layers of 

restricted access. An example is licensing that 

includes privacy protection, requiring greater 

protection as the potential for disclosure risk 

increases.

5. Legal and Technical 

Planning Questions

The questions in Tables 16–1 and 16–2 are 

intended to assist in the planning of data linkage 

projects that involve using registry data plus other 

files. Registry operators should use the answers to 

these questions to assemble necessary information 

and other resources to guide planning for their data 

linkage projects. Like the preceding discussion, 

this section considers regulatory and technical 

questions.

Several assumptions underlie the regulatory 

questions that follow in Table 16–1. Their 

application to the proposed data linkage project 

should be confirmed or determined. These 

assumptions are listed here:

• The HIPAA Privacy Rule applies to the entities 

that first collect data from individuals/subjects.

• Other laws may restrict access or use of the 

initial data sources.

• The Common Rule or FDA regulations may or 

may not apply to data linkage.

• The Common Rule or FDA regulations may or 

may not apply to the original data sets.

Different regulatory concerns arise depending on 

the answers to each category of the following 

questions. Consult as necessary with experienced 

health services, social science, or statistician 

colleagues; and with regulatory personnel (e.g., 

the agency Privacy Officer) or legal counsel to 

clarify answers for specific data linkage projects.
36

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

Table 16–1. Legal planning questions

1. Purpose(s) for data linkage • Research?

• Public health?

• Quality improvement?

• Required for postmarketing safety studies?

• Determining effectiveness of a product or service?

• Other purpose?

• Combination of purposes?

2. Conditions under 

which data (plus or minus 

biospecimens) were 

originally collected

• Collection required by law (e.g., Federal regulatory mandate, State public 

health mandate)?

• For treatment, payment, or health care operations, as defined by the HIPAA 

Privacy Rule?

• With documented consent from each individual to research participation and 

authorization for research use of PHI?

• With an IRB alteration or waiver of consent and authorization?

• With permission of health care provider or plan?

• With contractual conditions or limitations on future use or disclosure (release)?

• What are the reasonable expectations, held by the subjects of the information 

and the entities that first collected individuals’ information, of privacy or 

confidentiality for future uses of the data?

3. Data • Is sensitive information involved (e.g., about children, infectious disease, 

mental health conditions)?

• Do the data contain direct identifiers? Indirect identifiers?

• Is PHI involved?

• Is a limited data set (LDS), and thus a data use agreement (DUA), involved?

• Are the data de-identified in accordance with the HIPAA Privacy Rule?

• Do the data contain a code to identifiers?

• Who holds the key to the code?

• Is a neutral third party (an honest broker) involved?

• Does the code to identifiers conform to the re-identification standard in the 

HIPAA Privacy Rule?

• Is re-identification needed prior to performing the data linkage?

• Will the data linkage increase the risk that the data may be identifiable?

• What is the minimally acceptable cell size to avoid identifying individuals?

4. Person or institution 

holding the data for the 

linkage

• Is this person or institution a covered entity or a business associate of a 

covered entity under the HIPAA Privacy Rule?

5. Person or institution 

performing data linkage

• Is this person or institution a covered entity or a business associate of a 

covered entity under the HIPAA Privacy Rule or the American Recovery and 

Reinvestment Act of 2009?

6. Other laws or policies that 

may apply to data use or 

disclosure (release)

• Are government data involved?

• Are NIH data sharing policies involved?

• Does State law apply? Which State?

7. Terms and conditions 

that apply to data disclosure 

(release) and use 

• Are there consent and authorization documents that contain limitations on data 

use—unless the data have been sufficiently de-identified?

• Is there a DUA or other contract that applies to data use by any subsequent 

holder of the data?
37

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

8. Anticipated needs for data 

validation and verification

• Initially for the data linkage processes?

• In the future?

9. Future need to protect the 

privacy and security of the 

data 

• What will happen to data resulting from the linkage once the analyses have 

been completed? How will the data be stored?

• What measures will be in place to protect the security of the data?

10. Anticipated future uses of 

the linked data

• Will the data resulting from the linkage be maintained for multiple analyses? 

For the same or different purposes?

• Will the data resulting from the linkage be used for other linkages?

• What permissions are necessary for, or restrictions apply to, planned future 

uses of the data?

• Are there currently requirements for tracking uses and disclosures of the data?

DUA = data use agreement; HIPAA = Health Insurance Portability and Accountability Act; IRB = institutional 

review board; LDS = limited data set; NIH = National Institutes of Health; PHI = protected health information.

• Who is performing the linkage? Are the individuals performing the linkage permitted access to identifiers or 

restricted sets of identifiers? Are they neutral agents (“honest brokers”) or the source of one of the data sets to 

be linked?

• Is there a common feature or pseudonym (sets of attributes in both databases that are unique to individuals but 

do not lead to re-identification) available across the data sets being linked?

• Is the registry a flat file or a relational database?* The latter is more difficult to manage unless a primary key is 

applied.

• Is the registry relatively static or dynamic? The latter is harder to manage if data are being added over time, 

because the risk of identification increases.

• How many attributes are included in the registry? The more attributes, the harder it will be to manage the risk of 

identification associated with the registry. 

• How will conflicting values of attributes common to both databases be resolved? (Comparable attributes such as 

weight should be converted to the same units of measurement in data sets that will be linked.)

• Does the registry contain information that makes the risk of identification intrinsic to the registry? Direct 

identifiers such as names and Social Security numbers are problematic, as is fine-scale geography. 

• Is there a sound data dictionary?

• How many external databases will be linked to the registry data? How readily available and costly is each 

external database?

• How will records that appear in only one database be managed?

• How will the accuracy of the linked data set relate to the accuracy of its components? The accuracy is only as 

good as that of the least accurate component.

Table 16–1. Legal planning questions (continued)

Table 16–2. Technical planning questions

*In a relational database, information is presented in tables with rows and columns. Data within a table may be 

related by a common concept, and the related data may be retrieved from the database. (From A Relational Database 

Overview. http://docs.oracle.com/javase/tutorial/jdbc/overview/database.html. Accessed July 16, 2013).
38

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

6. Summary

This chapter describes technical and current legal 

considerations for researchers interested in 

creating data linkage projects involving registry 

data. In addition, the chapter presents typical 

methods for record linkage that are likely to form 

the basis for the construction of data linkage 

projects. It also discusses both the hazards for 

re-identification created by data linkage projects, 

and the statistical methods used to minimize the 

risk of re-identification. Two topics not covered in 

this chapter are (1) considerations about linking 

data from public and private sectors, where 

different, perhaps conflicting, ethical and legal 

restrictions may apply, and (2) the risks involved in 

identifying the health care providers that collect 

and provide data.

Data set linkage entails the risks of loss of reliable 

confidential data management and of identification 

or re-identification of individuals and institutions. 

Recognized and developing statistical methods and 

secure computation may limit these risks and allow 

the public the health benefits that registries linked 

to other data sets have the potential to contribute.

Case Example 36. Linking registries at the 

international level

Description Psonet is an investigatorinitiated, international scientific 

network of coordinated 

population-based registries; its 

aim is to monitor the long-term 

effectiveness and safety of 

systemic agents in the treatment 

of psoriasis.

Sponsor Supported initially by a grant 

from the Italian Medicines 

Agency (AIFA); supported since 

2011 by a grant from the 

European Academy of Dermato 

Venereology (EADV) and 

coordinated by the Centro Studi 

GISED.

Year Started 2005

Year Ended Ongoing

No. of Sites 9 registries across Europe and an 

Australasian registry

No. of Patients 27,800

Challenge

The number of options for systemic treatment of 

psoriasis has greatly increased in recent years. 

Because psoriasis is a chronic disease requiring 

lifelong treatment, data on long-term 

effectiveness and safety are needed for both old 

and new treatments. Several European countries 

have established patient registries for surveillance 

of psoriasis treatments and outcomes. However, 

these registries tend to have small patient 

populations and little geographic diversity, 

limiting their strength as surveillance tools for 

rare or delayed adverse events.

Proposed Solution

Combining the results from nation-based 

registries would increase statistical power and 

may enable investigators to conduct analyses that 

would not be feasible at a single-country level. 

Psonet was established in 2005 as a network of 

European registries of psoriasis patients being 

treated with systemic agents. The goal of the 

network is to improve clinical knowledge of 

prognostic factors and patient outcomes, thus 

improving treatment of psoriasis patients. An 

International Coordinating Committee (ICC), 

including representatives of the national registries 

and some national pharmacovigilance centers, 

oversees the network activities, including data 

management, publications, and ethical or privacy 

issues. The ICC has appointed an International 

Safety Review Board, whose job is to review 

safety data, prepare periodic safety reports, and 

set up procedures for the prompt identification 

and investigation of unexpected adverse events. 

Informed consent for data sharing is obtained 

before patients are enrolled in participating 

registries.

Case Examples for Chapter 16
39

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

Case Example 36. Linking registries at the 

international level (continued)

Proposed Solution (continued)

When drafting the registry protocol, member 

registries agreed to a common set of variables and 

procedures to be included and implemented in the 

national registries. However, some registries were 

already active at the time the draft was written, 

and harmonization is not perfect. Although 

inclusion criteria, major outcomes, and followup 

schedules are quite similar among registries, there 

are some differences. There are also differences 

in terms of software used, data coding, and data 

ownership arrangements. These factors made 

sharing individual patient data complicated, and 

an alternate solution was identified: meta-analysis 

of summary measures from each registry. As 

summary measures (or effect measures) are 

calculated, the methods used to obtain them are 

decided in advance, including methods used to 

control for confounding and methods used to 

temporarily link exposures and events.

Results

Ten national and local registries at different 

stages of development are associated with the 

registry to date, contributing a total of about 

27,800 patients. While the registry is too new to 

have published results, planned activities and 

analyses include comparative data on treatment 

strategies for psoriasis in Europe, rapid alerts on 

newly recognized unexpected events, regular 

reports on effectiveness and safety data, and 

analyses of risk factors for lack of response as a 

preliminary step to identifying relevant 

biomarkers.

Key Point

Data from multiple registries in different 

countries may be combined to provide larger 

patient populations for study of long-term 

outcomes and surveillance for rare or delayed 

adverse events. Meta-analysis of prospectively 

calculated summary measures can be a useful 

tool. 

For More Information

Psonet: European Registry of Psoriasis. http://

www.psonet.eu/cms/. 

Lecluse LLA, Naldi L, Stern RS, et al. National 

registries of systemic treatment for psoriasis and 

the European ‘Psonet’ initiative. Dermatology. 

2009;218(4):347–56.

Naldi L. The search for effective and safe disease 

control in psoriasis. Lancet. 2008;371:1311–2.
40

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

Case Example 37. Linking a procedure-based 

registry with claims data to study long-term 

outcomes

Description The CathPCI Registry measures 

the quality of care delivered to 

patients receiving diagnostic 

cardiac catheterizations and 

percutaneous coronary 

interventions (PCI) in both 

inpatient and outpatient settings. 

The primary outcomes evaluated 

by the registry include the 

quality of care delivered, 

outcome evaluation, comparative 

effectiveness, and postmarketing 

surveillance.

Sponsor American College of Cardiology 

Foundation through the National 

Cardiovascular Data Registry. 

Funded by participation dues 

from catheterization laboratories.

Year Started 1998

Year Ended Ongoing

No. of Sites 1,450 catheterization laboratories

No. of Patients 12.7 million patient records; 4.5 

million PCI procedures

Challenge

The registry sponsor was interested in studying 

long-term patient outcomes for diagnostic cardiac 

catheterizations and PCI, but longitudinal data 

are not collected as part of the registry. Rather 

than create an additional registry, it was 

determined that the most feasible option was 

linking the registry data with available third-party 

databases such as Medicare.

Before the linkage could occur, however, several 

legal questions needed to be addressed, including 

what identifiers could be used for the linkage and 

whether institutional review board (IRB) approval 

was necessary.

Proposed Solution

The registry developers explored potential issues 

relating to the use of protected health information 

(under the Federal HIPAA [Health Insurance 

Portability and Accountability Act] law) to 

perform the linkage; the applicability of the 

Common Rule (protection of human subjects) to 

the linkage; and the contractual obligations of the 

individual legal agreement with each 

participating hospital with regard to patient 

privacy. The registry gathers existing data, 

including direct patient identifiers collected as 

part of routine health care activities. Informed 

consent is not required. The registry sponsor has 

business associate agreements in place with 

participating catheterization laboratories for 

which the registry conducts the outcomes 

evaluations.

After additional consultation with legal counsel, 

the registry sponsor concluded that the linkage of 

data could occur under two conditions: (1) that 

the data sets used in the merging process must be 

in the form of a limited data set (see Chapter 7), 

and (2) that an IRB must evaluate such linkage. 

The decision to implement the linkage was based 

on two key factors. First, the registry participant 

agreement includes a data use agreement, which 

permits the registry sponsor to perform research 

on a limited data set but also requires that no 

attempt be made to identify the patient. Second, 

since there was uncertainty as to whether the 

proposed data linkage would meet the definition 

of research on human subjects, the registry 

sponsor chose to seek IRB approval, along with a 

waiver of informed consent.

Results

The registry data were linked with Medicare data, 

using probabilistic matching techniques to link 

the limited data sets. A research protocol 

describing the need for linkage, the linking 

techniques, and the research questions to be 

addressed was approved by an IRB. Researchers 

must reapply for IRB approval for any new 

research questions they wish to study in the 

linked data.

Results of the linkage analyses were used to 

develop a new measure, “Readmission following 

PCI,” for the Centers for Medicare & Medicaid 

Services’ hospital inpatient quality pay-forreporting program. 
41

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

Case Example 37. Linking a procedure-based 

registry with claims data to study long-term 

outcomes (continued)

Key Point

There are many possible interpretations of the 

legal requirements for linking registry data with 

other data sources. The interpretation of legal 

requirements should include careful consideration 

of the unique aspects of the registry, its data, and 

its participants. In addition, clear documentation 

of the way the interpretation occurred and the 

reasoning behind it will help to educate others 

about such decisions and may allay anxieties 

among participating institutions.

For More Information

https://www.ncdr.com/webncdr/cathpci/ 

Case Example 38. Linking registry data to 

examine long-term survival

Description The Yorkshire Specialist Register 

of Cancer in Children and Young 

People is a population-based 

registry that collects data on 

children and young adults 

diagnosed with a malignant 

neoplasm or certain benign 

neoplasms, living within the 

Yorkshire and Humber Strategic 

Health Authority (SHA). The 

goals of the registry are (1) to 

serve as a data source for 

research at local, national, and 

international levels on the causes 

of cancer in children, teenagers, 

and young people, and (2) to 

evaluate the delivery of care 

provided by clinical and other 

health service professionals.

Sponsor Primary funding is provided by 

the Candlelighters Trust, Leeds.

Year Started 1974

Year Ended Ongoing

No. of Sites 18 National Health Service 

(NHS) Trusts

No. of Patients 7,728

Challenge

In 2002, approximately 1,500 children in the 

United Kingdom were diagnosed with cancer. 

Previous estimates of malignant bone tumors in 

children have been approximately 5 per million 

person-years in the United Kingdom. The registry 

collects data on individuals younger than 30 

living within the Yorkshire and Humber SHA, and 

diagnosed with a malignant neoplasm or certain 

benign neoplasms by pediatric oncology and 

hematology clinics or teenage and young adult 

cancer clinics. Primary patient outcomes of the 

registry include length of survival, access to 

specialist care, late effects following cancer 

treatment, and hospital activity among long-term 

survivors. While bone cancer is ranked as the 

seventh most common malignancy in the United 

Kingdom, the relative rarity of this type of 

childhood cancer makes it difficult to gather 

sufficient data to evaluate incidence and survival 

trends over time.

Proposed Solution

The registry participated in a collaborative effort 

to combine its data with three other populationbased registries—the Northern Region Young 

Persons’ Malignant Disease Registry, the West 

Midlands Regional Children’s Tumour Registry, 

and the Manchester Children’s Tumour Registry. 

Together, the four population-based registries 

represented approximately 35 percent of the 

children in England.

Results

In a 20-year period from 1981 to 2002, 374 cases 

of malignant bone tumors were identified in 

children ages 0 to 14 years. The age-standardized 

incidence rate for all types of bone cancers (i.e., 

osteosarcoma, chondrosarcoma, Ewing sarcoma, 

and “other”) was reported to be 4.84 per million 

per year. For the two most common types of bone 

cancer, osteosarcoma and Ewing sarcoma, the 

incidence rates were 2.63 cases per million 

person-years (95% confidence interval [CI], 2.27 
42

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

Case Example 38. Linking registry data to 

examine long-term survival (continued)

Results (continued)

to 2.99) and 1.90 cases per million person year 

(95% CI, 1.58 to 2.21), respectively. While an 

improvement in survival was observed in patients 

with Ewing sarcoma, no survival improvement 

was detected in patients with osteosarcoma. The 

5-year survival rate for children with all types of 

diagnoses observed in the study was an estimated 

57.8 percent (95% CI, 52.5 to 63).

Key Point

In the analysis of rare diseases, the number of 

cases and deaths included in the study determines 

the statistical power for examining survival trends 

and significant risk factors, and the precision in 

estimating the incidence rate or other parameters 

of disease. In cases where it is difficult to obtain a 

large enough sample size within a single study, 

considerations should be given to combining 

registry data collected among similar patient 

populations.

For More Information

Eyre R, Feltbower RG, Mubwandarikwa E, et al. 

Incidence and survival of childhood bone 

tumours in Northern England and the West 

Midlands, 1981. Br J Cancer. 

2002;2009(s100):188–93.

Case Example 39. Linking longitudinal 

registry data to Medicaid Analytical Extract 

files

Description The Cystic Fibrosis Foundation 

(CFF) Patient Registry is a 

rare-disease registry that collects 

data from clinical visits, 

hospitalizations, and care 

episodes to track national trends 

in morbidity and mortality, 

assess the effectiveness of 

treatments, and drive quality 

improvement in patients with 

cystic fibrosis (CF).

Sponsor Cystic Fibrosis Foundation

Year Started 1986

Year Ended Ongoing

No. of Sites 110 CFF-accredited care centers 

in the United States

No. of Patients More than 26,000

Challenge

Clinical services and health information 

generated outside of clinic visits and 

hospitalizations at accredited care centers may or 

may not be captured in the CFF Patient Registry. 

Therefore, administrative claims data such as 

Medicaid Analytical Extract (MAX), with 

comprehensive information on reimbursed health 

services, are necessary to completely evaluate 

drug exposure for epidemiological studies. To 

protect patient information, the CFF Patient 

Registry only collects the last four digits of the 

Social Security number, gender, and date of birth 

as direct patient identifiers. Since these identifiers 

are largely non-unique, linkage of the registry 

data to other data sources presents a challenge. 

Proposed Solution

A deterministic patient matching algorithm, or 

linkage rule, between the CFF Patient Registry 

and MAX data using non-unique patient 

identifiers was developed to link the two data 

sources. MAX patients (with at least two in- or 

outpatient claims with diagnosis for CF) and CFF 

registry patients born between January 1, 1981, 

and December 31, 2006, were included. We 

examined the following variables for linking 

plausibility: date of birth, last four digits of the 

Social Security number, Zip Code, gender, date 

of sweat test, date of gene testing, and date of 

hospital admission. Specifically, we determined 

the percentage of unique records for each selected 

variable or combination of variables in the MAX 

data set and the registry data set. Only variable 

combinations with a 99 percent level of 

uniqueness (99 percent of records unique) were 

considered for the deterministic rule definitions. 

We then examined the linkage performance of 
43

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

References for Chapter 16

1. Clayton EW. Ethical, legal, and social implications 

of genomic medicine. N Engl J Med. 2003 Aug 

7;349(6):562-9. PMID: 12904522.

2. Westin, AF. Health Care Information Privacy: A 

Survey of the Public and Leaders. Conducted for 

EQUIFAX, Inc. New York, NY: 1993.

3. Lake Research Partners. Consumers and Health 

Information Technology: A National Survey. 

Conducted for the California Healthcare 

Foundation. April 2010. http://www.chcf.org/

publications/2010/04/consumers-and-healthinformation-technology-a-national-survey. 

Accessed March 11, 2013.

4. Gottlieb S. US employer agrees to stop genetic 

testing - Burlington Northern Santa Fe News. 

BMJ. 2001;322:449.

5. Sterling R, Henderson GE, Corbie-Smith G. 

Public willingness to participate in and public 

opinions about genetic variation research: a review 

of the literature. Am J Public Health. 2006 

Nov;96(11):1971-8. PMID: 17018829. 

PMCID: 1751820.

6. Institute of Medicine. Beyond the HIPAA Privacy 

Rule: Enhancing Privacy, Improving Health 

Through Research. January 27, 2009. http://www.

iom.edu/Reports/2009/Beyond-the-HIPAAPrivacy-Rule-Enhancing-Privacy-ImprovingHealth-Through-Research.aspx. Accessed August 

15, 2012.

7. Beckerman JZ, Pritts J, Goplerud E, et al. Health 

information privacy, patient safety, and health care 

quality: issues and challenges in the context of 

treatment for mental health and substance use. 

BNA’s Health Care Policy Report. 2008 Jan 

14;16(2):3-10. http://ihcrp.georgetown.edu/pdfs/

pritts0208.pdf. Accessed September 30, 2013.

8. Solove D. A taxonomy of privacy. University of 

Pennsylvania Law Review. 2006;154:477-560.

9. Duncan GT, Jabine TB, de Wolf VA. Private lives 

and public policies: confidentiality and 

accessibility of government statistics. Panel on 

Confidentiality and Data Access. Commitee on 

National Statistics. Washington, D.C.: National 

Research Council and the Social Science Research 

Council, National Academy Press; 1993.

Case Example 39. Linking longitudinal 

registry data to Medicaid Analytical Extract 

files (continued)

Proposed Solution (continued)

each rule and the validation parameters (i.e., 

sensitivity, specificity, and positive predictive 

value [PPV]) of these rules were compared 

against the selected gold standard (defined as the 

rule with the highest linkage performance). 

Results

We assessed 14,515 and 15,446 patient records in 

MAX and CF registry data sets, respectively. A 

total of nine linkage rules were established. The 

linkage rule including gender, date of birth, and 

Social Security number had the highest 

performance with 32.04 percent successfully 

linked records and was considered the gold 

standard. Linkage rule performance ranged from 

1.4 percent (95% CI, 1.2 to 1.6) to 32.0 percent 

(95% CI, 31.3 to 32.8). As expected, rules with 

lower linkage performance had fewer or no 

matching records. Compared with the selected 

gold standard, the sensitivity of the other linkage 

rules ranged from 4.3 percent (95% CI, 3.8 to 

4.9) to 73.3 percent (95% CI, 72.0 to 74.6); the 

specificity ranged from 88.2 percent (95% CI, 

87.6 to 88.9) to 99.9 percent (95% CI, 99.8 to 

99.9); and the PPV ranged from 68.2 percent 

(95% CI, 62.6 to 73.4) to 99.0 percent (95% CI, 

96.5 to 99.8).

Key Point

The defined linkage rules exhibited varying 

operational characteristics of sensitivity, 

specificity, and PPV. When using deterministic 

linkage methods to link registry data with 

administrative claims data, relying on multiple 

linkage rules may be necessary to optimize 

linkage performance. Applying probabilistic 

record linkage methods should be considered 

when deterministic linkage methods are likely to 

fail; however, the absence of a set criterion for 

establishing probability weights could pose a 

challenge for its implementation. 
44

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

10. Fienberg SE. Confidentiality and disclosure 

limitation. Encyclopedia of Social Measurement. 

1. San Diego, CA: Academic Press; 2005.

pp. 463-9.

11. Federal Committee on Statistical Methodology: 

Report on statistical disclosure limitation 

methodology. Statistical Policy Working paper 22; 

2005. Publication No. NTIS PB94-165305. 

http://www.fcsm.gov/working-papers/spwp22.

html. Accessed August 17, 2012.

12. Code of Federal Regulations, Title 45, Public 

Welfare, Department of Health and Human 

Services, Administrative Data Standards and 

Related Requirements, General Administrative 

Requirements, General Provisions, Section 103, 

Definitions. http://www.gpo.gov/fdsys/pkg/

CFR-2011-title45-vol1/xml/CFR-2011-title45-

vol1-sec160-103.xml.

13. Fellegi IP, Sunter AB. A theory for record linkage. 

Journal of the American Statistical Association. 

1969;40:1183-210.

14. Bilenko M, Mooney R, Cohen WW, et al. 

Adaptive name matching in information 

integration. IEEE Intelligent Systems. 

2003;18(5):16-23.

15. Herzog TN, Schuren FJ, Winkler WE. Data 

Quality and Record Linkage Techniques. New 

York: Springer-Verlag; 2007.

16. Abowd J, Vilhuber L. The sensitivity of economic 

statistics to coding errors in personal identifiers 

(with discussion). Journal of Business and 

Economic Statistics. 2005;23(2):133-65.

17. Winkler, WE. Overview of the record linkage and 

current research directions. U.S. Census Bureau. 

2006. http://www.census.gov/srd/papers/pdf/

rrs2006-02.pdf. Accessed September 30, 2013.

18. Christen P, Churches T, Hegland M, editors. 

A parallel open source data linkage system. 8th 

Pacific Asia Conference on Knowledge Discovery 

and Data Mining; Sydney, Australia; May 2004.

19. Christen P. Data Matching: Concepts and 

Techniques for Record Linkage, Entity 

Resolution, and Duplicate Detection. Berlin: 

Springer; 2012.

20. Sadinle M, Fienberg SE. A generalized FellegiSunter framework for multiple record linkage with 

application to homicide record-systems. Journal of 

the American Statistical Association. 

2013;108(502):385-97.

21. Lyons RA, Jones KH, John G, et al. The SAIL 

databank: linking multiple health and social care 

datasets. BMC Med Inform Decis Mak. 2009;9:3. 

PMID: 19149883. PMCID: 2648953.

22. Karr AF, Fulp WJ, Lix X, et al. Secure, privacy 

preserving analysis of distributed databases. 

Technometrics. 2007;49(3):335-45.

23. Karr AF, Lin X, Sanil AP, et al. Privacy-preserving 

analysis of vertically partitioned data using secure 

matrix products. Journal of Official Statistics. 

2009;25(1):125-38.

24. Rivest RL, Adleman L, Dertouzos ML. On data 

banks and privacy homomorphisms. In: DeMillo 

R, ed. Foundations of Secure Computation. New 

York: Academic Press; 1978.

25. Hall R, Fienberg SE. Privacy-preserving record 

linkage. In: Domingo-Ferrer J, Magkos E, eds. 

Privacy in Statistical Databases 2010, Lecture 

Notes in Computer Science 6344. Berlin: 

Springer; 2010. pp. 269-83.

26. Golle P. Revisiting the uniqueness of simple 

demographics in the U.S. population. ACM 

Workshop on Privacy in the Electronic Society. 

2006:77-80.

27. Sweeney L. Uniqueness of simple demographics 

in the U.S. population. Pittsburg, PA: Carnegie 

Mellon University Data Privacy Laboratory; 2000.

28. Karr AF, Banks DL, Sanil AP. Data quality: a 

statistical perspective. Statistical Methodology. 

2006;3(2):137-73.

29. Code of Federal Regulations, Title 45, Public 

Welfare, Department of Health and Human 

Services, Administrative Data Standards and 

Related Requirements, Security and Privacy, 

Privacy of Individually Identifiable Health 

Information, Section 164.514(b), Other 

Requirements Related to Uses and Disclosures of 

Protected Health Information, Implementation 

Specifications: Requirements for De-Identification 

of Protected Health Information . http://www.gpo.

gov/fdsys/pkg/CFR-2002-title45-vol1/pdf/

CFR-2002-title45-vol1-sec164-514.pdf.

30. U.S. Department of Health & Human Services. 

Guidance Regarding Methods for Deidentification of Protected Health Information in 

Accordance with the Health Insurance Portability 

and Accountability Act (HIPAA) Privacy Rule. 

http://www.hhs.gov/ocr/privacy/hipaa/

understanding/coveredentities/De-identification/

guidance.html. Accessed March 11, 2013.
45

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

31. Skinner C. Statistical disclosure risk: separating 

potential and harm (with discussion). International 

Statistical Review. 2012;80:349-81.

32. Sweeney L. Weaving technology and policy 

together to maintain confidentiality. J Law Med 

Ethics. 1997 Summer-Fall;25(2-3):98-110, 82. 

PMID: 11066504.

33. 45 CFR 164.514(b)(2)(i).

34. El Emam K, Jonker E, Arbuckle L, et al. A 

systematic review of re-identification attacks on 

health data. PLoS One. 2011;6(12):e28071. 

PMID: 22164229. PMCID: 3229505.

35. Malin B, Karp D, Scheuermann RH. Technical and 

policy approaches to balancing patient privacy and 

data sharing in clinical and translational research. 

J Investig Med. 2010 Jan;58(1):11-8. 

PMID: 20051768. PMCID: 2836827.

36. Rothstein MA. Is deidentification sufficient to 

protect health privacy in research? Am J Bioeth. 

2010 Sep;10(9):3-11. PMID: 20818545. 

PMCID: 3032399.

37. Malin BA. An evaluation of the current state of 

genomic data privacy protection technology and a 

roadmap for the future. J Am Med Inform Assoc. 

2005 Jan-Feb;12(1):28-34. PMID: 15492030. 

PMCID: 543823.

38. Lin Z, Owen AB, Altman RB. Genetics. Genomic 

research and human subject privacy. Science. 2004 

Jul 9;305(5681):183. PMID: 15247459.

39. Homer N, Szelinger S, Redman M, et al. 

Resolving individuals contributing trace amounts 

of DNA to highly complex mixtures using 

high-density SNP genotyping microarrays. PLoS 

Genet. 2008 Aug;4(8):e1000167. 

PMID: 18769715. PMCID: 2516199.

40. Cassa CA, Schmidt B, Kohane IS, et al. My 

sister’s keeper?: genomic research and the 

identifiability of siblings. BMC Med Genomics. 

2008;1:32. PMID: 18655711. PMCID: 2503988.

41. Rothstein MA. Genetic secrets: promoting privacy 

and confidentiality in the genetic era. New Haven: 

Yale University Press; 1997.

42. Kass N, Medley A. Genetic screening and 

disability insurance: what can we learn from the 

health insurance experience? J Law Med Ethics. 

2007 Summer;35(2 Suppl):66-73. 

PMID: 17543060.

43. Phelan JC. Geneticization of deviant behavior and 

consequences for stigma: the case of mental 

illness. J Health Soc Behav. 2005 Dec;46(4):

307-22. PMID: 16433278.

44. Public Law 110-233, Genetic Information 

Non-Discrimination Act of 2008. http://www.gpo.

gov/fdsys/pkg/PLAW-110publ233/pdf/

PLAW-110publ233.pdf.

45. Duncan GT. Confidentiality and statistical 

disclosure limitation. In: Smelser N, Baltes P, eds. 

International Encyclopedia of the Social and 

Behavioral Sciences. 4. New York, NY: Elsevier; 

2001. pp. 2521-5.

46. Gouweleeuw JM, Kooiman P, Willenborg LCRJ, 

et al. Post randomization for statistical disclosure 

control: theory and implementation. Journal of 

Official Statistics. 1998;14:463-78.

47. Rubin DB. Discussion: statistical disclosure 

limitation. Journal of Official Statistics. 

1993;9(2):461-8.

48. Raghunathan TE, Reiter JP, Rubin DB. Mutliple 

imputation for statistical disclosure limitation. 

Journal of Official Statistics. 2003;19:1-16.

49. Doyle P, Lane J, Theeuwes J, et al., eds. 

Confidentiality, Disclosure and Data Access: 

Theory and Practical Applications for Statistical 

Agencies. New York: Elsevier; 2001.

50. Fienberg SE, Makov UI, Steele RJ. Disclosure 

limitation using perturbation and related methods 

for categorical data (with discussion). Journal of 

Official Statistics. 1998;14(4):485-511.

51. Bishop YM, Fienberg SE, Holland PW, et al. 

Discrete Multivariate Analysis: Theory and 

Practice. New York: Springer-Verlag; 1995, 

Reprinted 2007.

52. Dobra A, Fienberg SE. Bounds for cell entries in 

contingency tables given marginal totals and 

decomposable graphs. Proc Natl Acad Sci USA. 

2000 Oct 24;97(22):11885-92. PMID: 11050222. 

PMCID: 17264.

53. Fienberg SE, Slavkovic AB. Preserving the 

confidentiality of categorical data based when 

releasing information for association rules. Data 

Mining and Knowledge Discovery. 2005;11:

155-80.

54. Fienberg SE. Statistical perspectives on 

confidentiality and data access in public health. 

Stat Med. 2001 May 15-30;20(9-10):1347-56. 

PMID: 11343356.
46

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data with Other Data Sources

55. Cox LH, Karr AF, Kinney SK. Risk-utility 

paradigms for statistical disclosure limitation: how 

to think but not how to act (with discussion). 

International Statistical Review. 2011;79:160-99.

56. Malin BA, Sweeney L. A secure protocol to 

distribute unlinkable health data. AMIA Annu 

Symp Proc. 2005:485-9. 

PMID: 16779087. PMCID: 1560734.

57. Boyens C, Krishnan R, Padman R, editors. On 

privacy-preserving access to distributed 

heterogeneous healthcare information. 37th 

Hawaii International Conference on System 

Sciences, Publication No HICSS-37 2004; 2009.

58. Fienberg SE. Privacy and confidentiality in an 

e-commerce world: data mining, data 

warehousing, matching and disclosure limitation. 

Statistical Science. 2006;21:143-54.

59. Dwork C, McSherry F, Nissim K. Calibrating 

noise to sensititvity in private data analysis. In: 

Halevi S, Rabin T, eds. TCC, Lecture Notes in 

Computer Science. 3876. Berlin: Springer-Verlag; 

2006a. pp. 265-84.

60. Dwork C, Kenthapadi K, McSherry F, et al. Our 

data, ourselves: privacy via distributed noise 

generation. EUROCRYPT2006. p. 486-503.

61. Fienberg SE, Rinaldo A, Yang X. Differential 

privacy and the risk utility tradeoff for multidimensional contingency tables. In: DomingoFerrer J, Magkos E, eds. Privacy in Statistical 

Databases 2010, Lecture Notes in Computer 

Sciences 6344. Berlin: Springer; 2010. 

pp. 187-99.
47

Chapter 17. Managing Patient Identity Across Data 

Sources

1. Introduction

Electronic health care data are increasingly being 

generated and linked across multiple systems, 

including electronic health records (EHRs), patient 

registries, and claims databases. In general, every 

system assigns its own identifier to each patient 

whose data it maintains. This makes it difficult to 

track patients across multiple systems and identify 

duplicate patients when different systems are 

linked. Efforts to address this challenge are 

complicated by the need to protect patient privacy 

and security. 

Patient identity management (PIM) has been 

defined as the “ability to ascertain a distinct, 

unique identity for an individual (a patient), as 

expressed by an identifier that is unique within the 

scope of the exchange network, given 

characteristics about that individual such as his or 

her name, date of birth, gender [etc.].”1 For the 

purposes of this chapter, the scope of this 

definition will be expanded to refer to PIM as the 

process of accurately and appropriately 

identifying, tracking, managing, and linking 

individual patients and their digitized health care 

information, often within and across multiple 

electronic systems.2 A related idea is the concept 

of patient identity integrity, which is defined as 

“the accuracy and completeness of data attached to 

or associated with an individual patient.”3 Efficient 

patient identity management leads to high patient 

identity integrity. 

The need for PIM strategies in the realm of health 

care data is rising, primarily because of the 

continued increase in the quantity and linkage of 

electronic health care data. The quantity of 

electronic health care data continues to grow. 

EHRs are increasingly being used to generate 

electronic health care data—72 percent of officebased physicians in the United States now use 

some form of EHR.4 This number is likely to 

increase significantly in response to the EHR 

incentive programs enacted by the Centers for 

Medicare & Medicaid Services (CMS), which 

“provide a financial incentive for the ‘meaningful 

use’ of certified EHR technology to achieve health 

and efficiency goals.”5 In addition to office-based 

EHRs, electronic health care data may be created 

by hospital EHRs, billing systems, insurance 

claims systems, pharmacy record systems, medical 

devices, and even by patients themselves via 

electronic patient health record systems. Large 

amounts of electronic health care data are also 

being generated from clinical research. Patient 

registries, for example, often use electronic data 

capture tools to collect and manage their data. 

This increase in the quantity of electronic health 

care and research data creates new opportunities 

and need for data linkage. Pharmaceutical 

companies conducting clinical trials on specific 

genetic markers are seeking ways to more easily 

identify and recruit potential patients. EHRs and 

patient registries are interfacing with each other to 

minimize the burden of data entry on participating 

centers and practices (see Chapter 15). Data from 

patient registries and other electronic sources are 

being pooled together to form larger, more 

statistically powerful data sets for research and 

analysis (see Chapters 16 and 18 and Case 

Examples 42 and 43). 

As more electronic health care data are generated 

and linked together, PIM has become crucial in 

order to (1) enable health record document 

consumers to obtain trusted views of their patient 

subjects, (2) facilitate data linkage projects, 

(3) abide by the current regulations concerning 

patient information–related transparency, privacy, 

disclosure, handling, and documentation,2 and 

(4) make the most efficient use of limited health 

care resources by reducing redundant data 

collection. To address this growing need, a number 

of standards development organizations are 

involved in the development of PIM strategies and 

standards. Several major organizations currently 

include: Integrating the Healthcare Enterprise6; 

Health Level Seven International7; and The 

Regenstrief Institute, Inc.8 See Appendix C for a 

more complete list.
48

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

2. PIM Strategies

The challenge of PIM is not a new one, and it has 

existed since health care information was first 

digitized. In general, PIM is conducted in one of 

two environments: either shared identifiers are 

present or they are absent. When shared identifiers 

exist, the main PIM strategy that has emerged is to 

assign a unique patient identifier (UPI) to each 

patient. In situations where shared identifiers do 

not exist, the most common PIM strategy is to use 

patient-matching algorithms to determine whether 

two sets of information belong to separate patients 

or the same patient.

2.1 When Shared Identifiers Are 

Present—UPI

2.1.1 Definition and Context

One of the most straightforward PIM strategies is 

the creation of a unique health identifier for 

individuals, or a UPI. Generally, a UPI is defined 

as a “unique, non-changing alphanumeric key for 

each patient”9 in a health care system, which is 

associated with each medical record or instance of 

health care data for that patient. Some proposed 

desirable characteristics of a UPI include that it be 

unique, nondisclosing, invariable, canonical, 

verifiable, and ubiquitous.10 In this context, 

“nondisclosing” means that the UPI does not 

contain any personal information about the patient, 

such as date of birth or Social Security number.

The concept of a universal UPI (i.e., a UPI that is 

assigned to a patient for life and is consistent 

across all electronic health care systems in the 

United States) has been discussed and debated for 

a number of years. The Health Insurance 

Portability and Accountability Act (HIPAA) of 

1996 called for the adoption of “standards 

providing for a standard unique health identifier 

for each individual, employer, health plan, and 

health care provider for use in the health care 

system.”11 Since the passage of HIPAA, the 

concept of a UPI has generally been welcomed by 

the health care industry, which views it as a tool to 

reduce administrative workload and increase 

efficiency in exchanging electronic health data.12

Other groups, including private citizens and 

experts attending a National Committee on Vital 

and Health Statistics hearing in July 1998, have 

expressed serious concerns about the effects that a 

universal UPI might have on patient privacy and 

data security.12 These concerns have halted further 

efforts at creating a UPI in the United States until 

appropriate privacy legislation is in place13, a even 

though recent research has argued that adoption of 

a universal UPI would actually strengthen patient 

privacy and security (by limiting the number of 

access points to patient health care data) and, while 

requiring a significant upfront cost, could pay for 

itself in cost savings from error reduction and 

administrative efficiency.14 The adoption of a 

universal UPI is also viewed by some as the logical 

next step in strengthening and developing the 

national health information network.9

2.1.2 Current Uses of UPIs

UPIs have long been used within individual patient 

registries and data sets, especially those with 

prospective data collection, to track and link a 

particular patient’s data over time. One of the most 

familiar types of UPI is a medical record 

number—a unique number assigned by a hospital 

or physician practice that links a patient with their 

medical record at that institution. Some hospitals 

have multiple electronic health information 

systems (e.g., EHRs, administrative/billing 

systems, lab systems, pharmacy dispensing 

systems) that assign UPIs to the patients within 

their domains, and a patient may not necessarily 

have the same UPI from system to system. Many 

patient registries also assign a UPI to patients upon 

screening or enrollment, and UPIs remain the 

simplest and most straightforward way to uniquely 

identify patients in a controlled data set.

aPrivacy and security concerns did not prevent CMS from developing the National Plan & Provider Enumeration 

System (NPPES) to assign unique identifiers to health plans and health care providers. The National Provider 

Identifier (NPI) has been implemented since 2006, and a standard identifier has not yet been implemented for health 

plans. (https://nppes.cms.hhs.gov/NPPES/Welcome.do. Accessed June 28, 2012.)
49

Chapter 17. Managing Patient Identity Across Data Sources

UPIs have also been used on a slightly larger scale 

in aggregated data sets and to link existing 

databases with administrative data sets. For 

example, the National Database for Autism 

Research aggregates data from many different 

collections of autism data and biospecimens and 

generates a global unique identifier for each 

patient represented in the aggregated data set.15

Similarly, in 2008 the Society of Thoracic 

Surgeons Database began collecting unique 

patient, surgeon, and hospital identifier fields to 

facilitate long-term patient followup via linking to 

the Social Security Death Master File and the 

National Cardiovascular Data Registry.16

Outside the United States, UPIs have been used on 

a wider scale. In Sweden, for example, the 

personal identity number (PIN) is a unique 

administrative identifier assigned to all permanent 

residents in Sweden since 1947. The PIN is used to 

track vital statistics and also link patients between 

several national-scale patient registries, including 

the Patient Register (containing inpatient and 

outpatient data), Cancer Register, Cause of Death 

Register, Medical Birth Register,17 and Knee 

Arthroplasty Register.18 In England, a new health 

identifier was introduced in 1996—the NHS 

number is a 10-digit unique identifier used solely 

for the purpose of patient identification.19

2.1.3 Future Directions for UPIs

Recently, interest has increased in expanding the 

use of existing administrative identifiers (such as 

the Social Security number in the United States) to 

serve as UPIs in the health care arena. In 2009, the 

U.S.–based nonprofit Global Patient Identifiers 

proposed the Voluntary Universal Healthcare 

Identifier project, which aims to make unique 

health care identifiers available to any patient who 

uses the services of a regional health information 

organization or health information exchange 

(HIE).20 In May 2011, production deployment on 

the system began. The voluntary nature of this 

project and its capacity for patients to have both an 

“open” voluntary identifier and a “private” 

voluntary identifier (which can be used to control 

which caregivers have access to clinically sensitive 

information) make it an interesting alternative to a 

mandated universal UPI that would likely be 

assigned and administered by a Federal 

Government agency. In March 2011, the eCitizen 

Foundation began requirements-gathering work on 

the Patient Identity Service Project, an opensource, open standards–based patient identity 

service that will be able to identify and 

authenticate a patient across multiple systems to 

gain access to their health records and services.21

The project is funded by the OpenID Foundation 

of Japan, and future goals include research and 

development, design, implementation, and testing 

of the service.

2.1.4 Registries and UPIs

UPIs offer a straightforward way to identify 

specific patients within a particular registry. 

However, the implementation of a universal UPI in 

the United States has been halted by concerns over 

patient privacy, security, and confidentiality, which 

are unlikely to be resolved soon. 

In Sweden, the ability to link data from separate 

national patient registries using the PIN has 

allowed researchers to pull from a pool of millions 

of Swedish residents to address difficult 

epidemiological questions. Concerns about patient 

privacy and confidentiality have been addressed by 

requiring that an ethical review board review and 

approve the planned study before any data are 

released to researchers. Past precedent has been 

that the review boards allow most PIN-based 

registry linkages, on the condition that the PINs 

are removed from the combined data set and 

replaced with different, unique serial numbers. 

Researchers also sign a legal agreement ensuring 

secure storage of the data and agreeing not to 

attempt to re-identify the patients in the deidentified data set they are given.17

2.2 When Shared Identifiers Are Not 

Present—Patient-Matching Algorithms

2.2.1 Definition and Context

In the absence of a national UPI in the United 

States, most researchers and hospital 

administrators have turned to patient-matching 

algorithms and other statistical matching 

techniques as a way to manage patient identities 

within the confines of a specific patient registry, 

research project, institution, or other grouping of 

health care data. This method of PIM involves 
50

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

comparing identifiable patient attributes (often 

demographics such as date of birth, gender, name, 

and address, but sometimes other individually 

identifiable information) using a logic model that 

then classifies each pair as a match, a non-match, 

or a possible match that may require manual 

review. 

In the realm of patient and record matching, 

algorithms can be either deterministic or 

probabilistic. Deterministic algorithms are more 

straightforward and classify a pair of records as a 

match if they meet a specified threshold of 

agreement. The definition of agreement can vary 

depending on which data elements are available, 

the quality of the data (including the level of 

missing data), and the desired sensitivity and 

specificity of the algorithm. Probabilistic 

algorithms treat the match status of individual data 

elements as observable variables and the match 

status of the record pair as a latent variable, and 

model the observable variables as a pattern 

mixture. This method characterizes the uncertainty 

in the matching process, making it a more 

sophisticated (and less straightforward) method 

than deterministic matching.22

One major consideration in choosing an 

appropriate matching algorithm is the accuracy 

with which it matches patients. Matching accuracy 

is affected by the number of patients being 

compared, the number and type of common data 

elements being compared, and the mathematical 

validity of the algorithm itself. An algorithm that 

returns close to 100-percent matching in a pool of 

few patients with many data elements may perform 

less accurately in a pool of many patients with 

fewer data elements. Importantly, an algorithm that 

does not perform accurately may limit the 

conclusions and results able to be drawn from a 

particular data set. 

2.2.2 Current Uses of Patient Matching 

Algorithms

Patient-matching algorithms are widely used when 

disparate health care data sources are combined 

and no unique, common patient identifier is 

available. The two main options are to use an 

existing record linkage software program or to 

develop a new matching algorithm independently. 

Commercial software options, such as Link Plus 

and The Link King, apply probabilistic algorithms 

that have been found to provide a higher sensitivity 

than matching using a basic deterministic 

algorithm.23 As described in Case Example 40, an 

open-source product (Febrl) was used to combine 

data from 11 different data sources into KIDSNET, 

a computerized registry that gives providers an 

overall view of children’s use of preventive health 

services.24 Case Example 41 describes a different 

approach to patient matching. 

Many patient-matching algorithms have been 

developed to meet the needs of specific projects. 

For example, a group at Partners HealthCare 

developed an algorithm to compare data in the 

Social Security Death Master File with 

demographic data in the Partners EHR system to 

identify patient deaths that may have occurred 

outside of Partners institutions (and therefore were 

not recorded in the patients’ medical record). They 

then developed another algorithm using clinical 

data to identify false-positives resulting from the 

first algorithm (e.g., if clinical data for a 

supposedly deceased patient is recorded as more 

than 30 days after the date of death in the Social 

Security Death Index [SSDI], that patient must 

have been falsely matched to an SSDI entry).25 In 

another example, researchers at the University of 

Alabama Birmingham used matching algorithms 

to link emergency medical services data with 

hospital EHRs and a statewide death index to 

characterize the medical conditions and 

comorbidities of patients who received out-ofhospital endotracheal intubation.26

New and innovative algorithms that are unrelated 

to specific projects also continue to be developed, 

with the goal of advancing patient matching 

algorithm science. Recent examples include 

algorithms proposed by groups at Vanderbilt 

University in Nashville, Tennessee,27 John 

Radcliffe Hospital in the United Kingdom,28 and 

the University of Duisburg-Essen in Germany.29

2.2.3 Future Directions of Patient-Matching 

Algorithms

Any statistical matching approach is dependent on 

three factors, listed below:
51

Chapter 17. Managing Patient Identity Across Data Sources

1. The quality of the data it is comparing: Are the 

data entered correctly, without mistakes? Are 

the data complete, or is there a high level of 

missing data? The quality of data within a 

particular registry will always be a factor of the 

practices employed by that registry. See 

Chapter 11 for recommended best practices. 

2. The comparability of the data it is comparing: 

Are the data from the different sources 

collected in the same format and in the same 

way? There are a number of current initiatives 

to improve the standardization of data elements 

being used in patient registries,30 but the area 

with the most need for future work is the 

testing and standardization of the algorithms 

themselves. 

3. The accuracy of the matching algorithm: What 

is the likelihood of the algorithm returning a 

false positive match or missing true matches? 

While there has been some scientific research 

validating specific matching algorithms,31-33

the Health Information Technology Policy 

Committee recently called for increased 

standards around patient matching, including 

standardized formats for demographic data 

fields; internal evaluation of matching accuracy 

within institutions and projects; accountability 

to acceptable levels of matching accuracy; the 

development, promotion, and dissemination of 

best practices in patient matching; and 

supporting the role of the patient.34

Another emerging trend in patient matching 

algorithms is privacy-preserving record linkage, or 

“finding records that represent the same individual 

in separate databases without revealing the identity 

of the individuals.”29 This concept was expanded 

upon by researchers at University of DuisburgEssen in Germany, mentioned in the previous 

section, who propose a method that encrypts 

patient identifiers while allowing for errors in 

identifiers. Given the concerns about patient 

privacy and confidentiality surrounding patient 

identity management, this method may be 

increasingly used in the future.

2.2.4 Registries and Patient Matching Algorithms 

As mentioned above, patient matching algorithms 

have become the default PIM strategy for registries 

that link with outside data sources, due to the lack 

of a universal UPI in the United States. As a result, 

many different algorithms have been developed—

some commercially available, some open-source, 

some developed for specific projects, and some 

developed with broader applications in mind. The 

performance and effectiveness of matching 

algorithms can impact the results produced by the 

registries that are using them. The type of registry 

also impacts the type of patient-matching 

algorithm needed. Registries used for direct patient 

care may require an algorithm with different 

sensitivity, specificity, and timeliness than those 

used for population-based research efforts. 

Registry owners and operators would benefit from 

standards surrounding patient-matching 

algorithms, which would allow them to more 

confidently and effectively use appropriate 

algorithms for linking projects.

3. Emerging Strategies and 

Related Ideas

In addition to a universal patient identifier and 

patient matching algorithms, other strategies are 

emerging to manage patient identities in disparate 

electronic health care data sources, including 

biometrics and master patient indices. In the 

technical realm of patient-centric document 

exchange, HIEs are becoming increasingly 

important in providing the interoperability 

infrastructure for successful EHR implementations 

within and across affinity domains.

3.1 Biometrics

One new option in the PIM field is the use of 

biometrics—that is, “automated methods of 

recognizing an individual based on measurable 

biological (anatomical and physiological) and 

behavioral characteristics.”35 Some examples of 

biometric measurements are: fingerprint, palm 

print, hand geometry, DNA, handwriting, finger or 

hand vascular pattern, iris/retina, facial shape, 

voice pattern, and gait. 

Biometrics are attractive because of their difficulty 

to fabricate, their resistance to change over time 

(unlike demographic information such as name 

and address), and their high degree of 

uniqueness—making them effectively biological 
52

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

UPIs. For biometrics to be used as UPIs, though, 

there would need to be agreement on which 

biometric to use and the format in which it should 

be collected. Also, some biometric measurements 

are more unique than others. For example, a 

fingerprint is highly unique to an individual, while 

a person’s hand geometry is not as unique. Hand 

geometry therefore is often used to confirm a 

person’s identity (i.e., in combination with another 

identifier) rather than as a sole identifier. 

One drawback to using biometrics is the 

investment in specialized technology and 

equipment required to capture many of these 

measurements. There is also concern about the 

privacy and security implications surrounding the 

use of biometrics, connected with their history of 

use in law enforcement and their potential misuse 

to derive information other than identity 

(e.g., analyzing DNA for genetic diseases).36

Some hospitals have begun using biometrics to 

verify provider identity and restrict access to 

EHRs. Biometrics are also being used in some 

hospitals to verify patient identity upon hospital 

admission37 and identify critically injured, 

unconscious patients presenting to an emergency 

room.38

Many registries, particularly those with biobanks 

associated with them, already collect biometric 

data (e.g., DNA). However, the data are often used 

for purposes other than PIM, including 

investigating genetic components of disease39 and 

risk factors for disease.40

Biometrics remains an attractive option for PIM; 

the largest obstacle to its use in patient registries is 

likely the investment in technology and equipment 

that it requires, although this would vary 

depending on where registry data are collected. A 

multisite, practice-based registry would probably 

be less able to accommodate the collection of 

biometric data, while a registry based out of a 

single hospital that already collects biometric data 

for other purposes would be able to begin 

collecting biometrics for a registry more easily, 

since the initial investment in technology has 

already been made. Registries using biometrics 

would also be subject to the same concerns about 

privacy and security as biometric use in other 

disciplines. 

3.2 Master Patient Index 

A master patient index (MPI) facilitates the 

identification and linkage of patients’ clinical 

information within a particular institution. The 

term “enterprise master patient index” (EMPI) is 

sometimes used to distinguish between an index 

that serves a single institution (i.e., MPI) and one 

that contains data from multiple institutions 

(EMPI). MPIs are not themselves patient identity 

management strategies, but rather informational 

infrastructures within which those strategies are 

applied. Most MPIs use a patient matching 

algorithm to identify matches and then assign a 

UPI that is associated with that patient record 

going forward. MPIs and EMPIs are created for 

the purpose of assigning a UPI to each patient 

treated within a certain health care system—

providers can then use that identifier to have a 

global view of the patient’s care across multiple 

institutions within that system. 

Several leading software companies have released 

commercially available MPI and EMPI products. 

Oracle has published a thorough description of the 

design and functionality of their EMPI product.41

Open-source options are also available, including 

one developed by Project Kenai called 

OpenEMPI.42

EMPIs are used as supplemental tools to apply 

PIM strategies for data sharing efforts such as 

HIEs, described more fully in the next section. For 

example, the Michigan Clinical Research 

Collaboratory at the University of Michigan 

created the “Honest Broker” system, which serves 

three functions: facilitating the actual exchange of 

data between members of the collaboratory for 

research, maintaining an MPI to manage patient 

identities within that data, and de-identifying data 

sets in conformance with HIPAA standards.43

Figure 17–1 is adapted from the Integrating the 

Healthcare Enterprise integration profile44 and 

illustrates the actors that participate in the Patient 

Identifier Cross-referencing profile. The entity 

often called an MPI is represented by the 

combination of the Patient Identity Source 

(“Source”) and the Patient Identity Cross-reference 

Manager (“Manager”). The Source provides 

patient identity information (Patient Identity Feed) 

to the Manager. It is common to have multiple 
53

Chapter 17. Managing Patient Identity Across Data Sources

patient identity sources that provide patient ID 

feeds to the Manager. The Manager is responsible 

for managing patient identities by detecting 

matches and creating and maintaining crossreferences of patient identifiers across these 

various sources. The Patient Identifier Crossreference Consumer (“Consumer”) retrieves 

Patient Identity Cross References or aliases. This 

allows patients to be linked across multiple 

systems or domains that use different patient 

identifiers to represent the same patient.

Figure 17–1. Basic process flow with patient identifier cross-referencing

Illustrating how users may interact with an MPI in 

daily practice may be helpful. In one possible 

scenario, an emergency room physician sees a 

patient presenting at the emergency room with 

vague and poorly defined pain who specifically 

asks to be prescribed narcotics. A new quality 

improvement program being implemented in this 

emergency room requires the physician to check 

the patient’s history of filling prescriptions before 

issuing a prescription for a narcotic drug. The 

emergency room’s EHR system and the hospital 

pharmacy’s electronic dispensing record system 

each assign their own patient IDs to patients within 

their systems, and send patient feeds to the 

hospital’s MPI (the Manager in this scenario) each 

time a new patient ID is assigned. The MPI creates 

and maintains cross-references of all identifiers for 

patients and provides the cross-references to 

consumers who seek that information. The 

consumer in this scenario would be the emergency 

room system, which sends the MPI a patient 

identity cross-reference or demographic query 

with information about the patient in question. The 

MPI notifies the emergency room system that the 

patient identified in the emergency room as 

“ER703” matches the patient whose pharmacy 

records are under the pharmacy system identifier 

“012.” The emergency room system then queries 

the pharmacy system for the identifier “012,” and 

presents the dispensing record data to the 

emergency room physician.

Health care institutions that use MPIs to manage 

patient identities across their multiple data sources 

(e.g., EHRs, pharmacy records, administrative and 

billing records) are desirable partners for data 

linkage projects and for inclusion in patient 

registries, since they are able to draw from a 

broader pool of data than any one of the data 

sources alone. By addressing PIM needs upfront, 

they minimize the work needed for outside sources 

to link to their data for research uses. 

Patient Identity 

Source

Patient Identifier 

Cross-reference 

Consumer

Patient Identifier 

Cross-reference 

Manager

Patient Identity Feed

Patient Identity Feed

Patient Identity 

Cross References 
54

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

Figure 17–2. Data flow through a health information exchangea

In the relational infrastructure shown in Figure 

17–1, registries can act as Patient Identity Sources, 

Patient Identifier Cross-reference Consumers, or 

both. Registries that contain patient identifiers and 

other demographic information can act as Patient 

Identity Sources and send patient identity feeds to 

a Manager. Registries can also act as Patient 

Identifier Cross-reference Consumers, if they 

request and receive patient identity cross 

references from an MPI or other Patient Identity 

Cross-reference Manager. This may be done to add 

new patients to a registry or to augment existing 

data in a registry with additional information on 

the same patients.

3.3 Health Information Exchange

An HIE is an integrated open standards–based 

solution to enable information sharing across 

disparate health care applications. (See Case 

Example 34, which describes the Oakland 

Southfield Physicians HIE.) HIEs are 

interoperability platforms that provide the means 

to share patient data produced by health care 

applications with other applications that consume 

and use the data, such as EHRs. HIEs implement 

standards-based health care messages and provide 

the requisite authentication and auditory services 

for data governance. HIEs are not themselves 

patient identity management strategies, but they 

implement those strategies to manage their data. 

Most HIEs achieve this by incorporating an MPI to 

manage and cross-reference the identity of patients 

within the HIE. See Figure 17–2 for a graphical 

representation of the relationship between HIEs, 

MPI/EMPIs, and data creators and consumers.

EHR = Electronic Health Record; PHR = Personal Health Record; EMPI = Enterprise Master Patient Index; HIE = 

Health Information Exchange; ID = Identifier

aCopyright 2010 Health Information and Management Systems Society. Figure reprinted with permission.
55

Chapter 17. Managing Patient Identity Across Data Sources

Key components of an HIE include:

• Patient Identity Cross-Reference Manager: An 

implementation of an MPI that cross-references 

multiple identifiers and serves the linked 

identifiers, global patient identifier, and unified 

patient demographics to information consumers 

and other HIE components.

• Document Repository: Clinical document 

repository for storing patient records and 

documents.

• Document Registry: Registry of patient’s 

documents located in various document 

repositories.

• Cross Community Gateway: Serves as the entry 

point for communications between HIE 

communities.

Content creators who create new patient identifiers 

provide patient feeds to the Identity Manager, 

which in turn cross-references it to a global patient 

identifier. Content consumers and creators can 

query the Identity Manager for the global identifier 

by providing a subset of patient demographics or 

one of their local identifiers. This global identifier 

is used by the document registry to keep track of 

patient clinical documents. This infrastructure 

facilitates an interoperable environment that 

respects data ownership demands but also provides 

a complete view of the patient’s clinical records 

from multiple sources.

HIEs can be powerful research tools. A group at 

the Swansea University School of Medicine has 

developed the Secure Anonymized Information 

Linkage (SAIL) databank, containing more than 

500 million records from multiple health and 

social care service providers in the United 

Kingdom.45 The SAIL databank has already been 

used to demonstrate the feasibility of identifying 

potential clinical trial participants at the primary 

care level, which may be especially useful for 

disease areas in which recruitment of clinical trial 

participants is historically difficult (e.g., chronic 

conditions such as diabetes).46

Because they contain patient data, HIEs are subject 

to the same privacy and security concerns and 

regulations as patient registries. A white paper 

published in April 2011 by the American Health 

Information Management Association/Healthcare 

Information Management and Systems Society 

(AHIMA/HIMSS) HIE Privacy & Security Joint 

Work Group provides a summary of these 

considerations.47

A patient registry may contribute data to an HIE, 

but registries and HIEs are distinct and separate 

endeavors. Data contained in HIEs are not 

necessarily collected using observational study 

methods, as patient registry data are; rather, they 

are often collected and aggregated by linking to 

existing databases (which may be, for example, 

registries, administrative databases, or public 

health surveillance systems). The purpose of an 

HIE is not just to evaluate specified outcomes in a 

defined patient population or even to serve any one 

predetermined scientific, clinical, or policy 

purpose, but to provide an aggregated database 

that can be used for a variety of purposes (which 

may include identifying patients to recruit for 

clinical trials, or conducting ecological studies, for 

example). 

4. Major Challenges and 

Barriers

The process of patient identity management 

introduces several technical, ethical, and 

operational challenges, including selecting the 

appropriate PIM strategy, discussed earlier in this 

chapter. Additional challenges include the 

obligation to protect the privacy and security of 

patient data and the technical interoperability (or 

lack thereof) of disparate health care data sources.

4.1 Protecting Patient Privacy and 

Security

One of the most pressing challenges in PIM is 

addressing the tension between linking patient data 

in order to manage patients’ identities and 

protecting the privacy and security of those data. 

This challenge has inherent ethical, regulatory, and 

technical considerations.

4.1.1 Ethical and Regulatory Considerations

The concepts of protecting patient privacy and 

security and PIM have always been intertwined. 

Managing patient identities is essential for 
56

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

protecting the privacy and security of those 

patients. Conversely, regulations and ethical 

considerations compel the protection of patients’ 

privacy and security when managing their 

identities (i.e., it is not enough to know who they 

are and which information is theirs; one must also 

protect this information).

Many stakeholders in the health information 

technology field recognize this relationship. The 

Health Information Security and Privacy 

Collaboration counts patient and provider 

identification as one of its nine domains of privacy 

and security.48 The Commission on Systemic 

Interoperability released a report in 2005 in which 

it recommended that Congress authorize the 

Department of Health and Human Services to 

“develop a national standard for determining 

patient authentication and identity,” and “develop a 

uniform federal health information privacy 

standard for the nation, based on HIPAA and 

pre-empting state privacy laws [...].” These 

recommendations were made simultaneously, “to 

advance progress of the connectivity of health 

information technology.”49 Thus, it is widely 

recognized that PIM and patient privacy and 

security are closely related, but there continues to 

be disagreement about how they should relate.

The regulatory framework that guides this 

discussion in the United States is HIPAA, enacted 

in 1996. As mentioned previously in this chapter, 

HIPAA mandated the implementation of a 

nationwide unique patient identifier, but in 1999 

concerns about patient privacy and security 

prompted the barring of any funding for this 

endeavor. While HIPAA has not led to the 

implementation of a standard PIM method, it does 

set forth a framework for the protection of patient 

privacy and health information security. This 

framework is summarized in Table 7–1 in Chapter 

7.

In Europe, recently proposed data protection 

regulations may have a profound impact on the 

regulatory environment in which registries conduct 

PIM activities. The directive proposed by the 

European Commission in January 2012 includes a 

provision for the “right to be forgotten,” essentially 

giving individuals the power to remove their 

personal data from third party data holders at any 

time they choose.50 If adopted by the European 

Parliament and European Union member states, 

the directive will take effect within two years. The 

implications that this may have for health care 

research and registries operating in Europe remain 

to be seen.

4.1.2 Technical Considerations

Data holders employ three main technical methods 

of ensuring the privacy and security of patient 

data: anonymization, encryption, and 

pseudonymization: 

• Anonymization is the practice of removing 

information that is identifiable to an individual 

or that may enable an individual’s identity to be 

deduced. This is a viable option in some data 

use situations (e.g., conducting a research study 

that does not require patient followup), but not 

an option in others (e.g., maintaining 

comprehensive health records for patients in an 

EHR). It is also not a reversible process—once 

identifiers are removed from data, they cannot 

be reinserted.

• Encryption involves applying a mathematical 

calculation or algorithm to transform a patient’s 

original data (plain text) into coded data 

(cypher text). In order to read the cypher text, a 

user or system must have access to a key that 

decrypts the data back into plain text. This is an 

attractive option because it does not involve 

deleting or removing patient data, and because 

the coded data is not in a readable format if it 

falls into the wrong hands. However, 

encryption requires robust data management 

policies and resources to be implemented 

successfully.51

• Pseudonymization is a more sophisticated 

approach to patient privacy protection. It 

involves two steps: depersonalization, in which 

identifiable data are separated from other 

clinical data and stored in a separate location, 

and pseudonymization, in which a unique 

identifier is generated and applied to the 

depersonalized data set. The unique identifier, 

or pseudonym, does not change for a given 

patient over time, and is not derived from any 

identifiable attributes of the patient. 

Pseudonymization can be reversible, if the 
57

Chapter 17. Managing Patient Identity Across Data Sources

relationship between the pseudonym and the 

identifiable data is maintained in a secure way 

and can facilitate re-identification of the patient 

under specific circumstances (e.g., a trusted 

third party maintains the relationship, and only 

discloses that relationship if the requestor has 

knowledge of a particular key or password). 

Pseudonymization can also be irreversible, if a 

situation arises in which the relationship 

between the pseudonym and the identifiable 

data is not maintained, and re-identification is 

not possible.52, 53

4.2 Interoperability 

In the same way that health care enterprises such 

as hospitals, clinics, and physician offices require 

patient identifier cross-referencing, that is the 

linking of patients across different domains, it is 

necessary to consider how registries may fit within 

this model and the challenges that level of 

interoperability may impose. Separate patient 

registries may use the same PIM infrastructure to 

register their patient identifiers within a shared 

patient identifier cross-reference manager, 

allowing the identifiers to be linked back to 

relevant health care and related systems. This 

approach may represent a possible solution 

whereby registries can more easily and securely be 

linked to other systems across known domains 

such as an HIE, but challenges still remain in 

terms of how this approach could successfully be 

used more broadly across nonparticipating health 

care enterprises.

5. Summary

Patient identity management is a fast-growing and 

evolving field, influenced by emerging 

technologies, regulations, and opportunities to use 

electronic health care data. The current status of 

PIM in the United States is primarily a factor of 

the provision in HIPAA for “standards providing 

for a standard unique health identifier for each 

individual [...] for use in the health care system,”11

the debate this provision has generated over 

implications for patient privacy and security, and 

the subsequent blocking of any funding being 

allocated to the pursuit of a national UPI. As a 

result, most PIM endeavors in the United States 

(including attempts to link patient registries with 

other health care data sources) use patientmatching algorithms to identify duplicates and 

manage patient identities. The lack of standards in 

this area means that the accuracy and effectiveness 

of these algorithms can vary widely.

Debate continues around how to best address the 

challenge of PIM, and stakeholders generally hold 

one of two views. Some view a national UPI as the 

best solution, provided the long-standing concerns 

about protecting patient privacy and security can 

be adequately addressed in the future. Others 

believe that resources would be better spent 

developing and standardizing the PIM methods 

that have grown organically in the absence of a 

national UPI; namely, EMPIs and patient matching 

algorithms. These two endeavors are not 

necessarily mutually exclusive, and patient 

registries and data linkage projects would benefit 

from the advancement of either or both. 
58

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

Case Example 40. Integrating data from 

multiple sources with patient ID matching

Description KIDSNET is Rhode Island’s 

computerized registry to track 

children’s use of preventive 

health services. The program 

collects data from multiple 

sources and uses those data to 

help providers and public health 

professionals identify children in 

need of services. The purpose of 

the program is to ensure that all 

children in the State receive 

appropriate preventive care 

measures in a timely manner.

Sponsor State of Rhode Island, Centers 

for Disease Control and 

Prevention, and others

Year Started 1997

Year Ended Ongoing

No. of Sites 216 participating practice sites 

and more than 150 other groups 

of authorized users

No. of Patients 314,211

Challenge

In the 1990s, the Rhode Island Department of 

Health recognized that its data on children’s 

health were fragmented and program specific. 

The State had many children’s health initiatives, 

such as programs for hearing assessment and lead 

poisoning prevention, but these programs 

collected data separately and did not attempt to 

link the information. This type of fragmented 

structure is common in public health agencies, as 

many programs receive funding to fulfill a 

specific need but no funding to link that 

information with other programs. This type of 

linkage would benefit the Department’s activities, 

as children who are at risk for one health issue 

are often at risk for other health issues. By 

integrating the data, the Department would be 

able to better integrate services and provide better 

service.

To integrate the data from these multiple sources 

and to allow new data to be entered directly into 

the program, the Department implemented the 

KIDSNET computerized registry. The registry 

consolidates data from eight electronic data 

sources, in addition to immunization and online 

data entry from four more public health programs 

to provide an overall picture of a child’s use of 

preventive health care services. The sources are 

newborn developmental risk screening; the 

immunization registry; lead screening; hearing 

assessment; the Women, Infants, and Children 

(WIC) program; home visiting; early 

intervention; blood spot screening; foster care; 

birth defects; vital records data; asthma 

environmental inspection referrals, early child 

developmental screening, and audiology results. 

The goals of the registry are to monitor and 

assure the use of preventive health services, 

provide decision support for immunization 

administration, give providers reporting capacity 

to identify children who are behind in services, 

and provide recall services and quality assurance.

After being launched in 1997, the registry began 

accumulating data on children who were born in 

the State or receiving preventive health care 

services in the State. Some of the data sources 

entered data directly into the registry, and some 

of the data sources sent data from another 

database to the registry. The registry then 

consolidated data from these sources into a single 

patient record for each child by matching the 

records using simple deterministic logic. As the 

registry began importing records, the system held 

some records as questionable matches, since it 

could not determine if the record was new or a 

match to an existing record. These records 

required manual review to resolve the issue, 

which was time consuming, at approximately 3 

minutes per record.

Due to lack of resources to devote to the manual 

review, the number of records held as 

questionable matches increased to 48,685 by 

2004. The time to resolve these records manually 

was estimated at 17 months, and the registry did 

not have the resources to devote to that task. 

Case Examples for Chapter 17
59

Chapter 17. Managing Patient Identity Across Data Sources

Case Example 40. Integrating data from 

multiple sources with patient ID matching 

(continued)

Challenge (continued)

However, the incomplete data resulting from so 

many held records made the registry less 

successful at tracking children’s health and less 

used by providers.

Proposed Solution

To resolve the issue of patient matching, the 

sponsor implemented an automated solution to 

the matching problem after evaluating several 

options, including probabilistic and deterministic 

matching strategies and commercial and opensource options for matching software. Since the 

State had limited funds for the project, an opensource product, Febrl, was selected.

A set of rules to process incoming records was 

developed, and an interface was created for the 

manual review of questionable records. Using the 

rules, the software determines the probability of a 

match for each record. The registry then sets 

probability thresholds above which a record is 

considered a certain match and below which a 

record is considered a new record. All of the 

records that fall into the middle ground require 

manual review.

Results

After considerable testing, the new system was 

launched in spring 2004. Immediately upon 

implementation, 95 percent of the held records 

were processed and removed from the holding 

category, resulting in the addition of 

approximately 11,000 new patient records to the 

registry. The new interface for manual review 

reduced the time to resolve an error from 3 

minutes to 40 seconds. With these improvements, 

the registry now imports 95 percent of the data 

sent to the database and is able to process the 

questionable records through the improved 

interface.

Key Point

Many strategies and products exist to deal with 

matching patients from multiple data sources. 

Once a product has been selected, careful 

consideration must be given to the probability 

thresholds for establishing a match. Setting the 

threshold for matches too high may result in an 

unmanageable burden of manual review. 

However, setting the threshold too low could 

affect data quality, as records may be merged 

inappropriately. A careful balance must be found 

between resources and data quality in order for 

matching software to help the registry. In 

addition, matching quality should be monitored 

over time, as matching rules and probability 

thresholds may need to be adjusted if the 

underlying data quality issues change.

For More Information

Wild EL, Hastings TM, Gubernick R. et al. Key 

elements for successful integrated health 

information systems: lessons learned from the 

states. J Public Health Manag Pract. 2004 

Suppl:S36–S47.
60

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

Case Example 41. Using patient identity 

management methods to combine health 

system data

Description The clinical breast program at 

Providence Health & Services—

Oregon provides screening, 

diagnosis, and treatment of 

breast conditions for women in 

seven hospitals within a regional 

health care system. The 

Providence Regional Breast 

Health Registry integrates 

patient data from multiple 

sources to improve patient care 

and outcomes, conduct research, 

and collaborate on national 

quality initiatives.

Sponsor Providence Health & Services—

Oregon; Safeway Foundation

Year Started 2008

Year Ended Ongoing

No. of Sites 7 health system hospitals in 

Oregon

No. of Patients 265,130 encounters as of 

December 2011

Challenge

Leaders of the clinical breast program at 

Providence Health & Services—Oregon are 

interested in collecting patient-level data for 

reporting performance and outcome measures 

related to health care quality (e.g., biopsy rates); 

health services (e.g., screening volumes over 

time); research questions; and accreditation with 

the National Accreditation Program for Breast 

Centers (NAPBC). However, patient data reside 

in numerous information systems, including the 

hospital electronic health record, administrative 

billing systems, imaging systems (e.g., 

mammography, MRI, ultrasound), and the 

pathology system. The health system uses a 

patient corporate number (PCN), assigned to each 

patient in the health system as a patient identifier. 

Each hospital assigns its own medical record 

number (MRN) to each patient and a separate 

encounter number for each visit. 

Meeting the reporting and research needs of the 

breast clinic program requires integrating data 

from all of these multiple systems as well as 

managing the identities of patients whose data 

could be contained in one or all systems.

Proposed Solution

In 2008, the Providence Regional Breast Health 

Registry was created. Registry data are housed in 

a structured query language (SQL) database that 

imports data from the various systems and applies 

matching algorithms to appropriately group data 

from the same patient. To make the match, the 

algorithms take into account the PCN, MRN, and 

encounter numbers for patients with breast health 

encounters based on a breast-specific ICD-9 and 

CPT procedure query. Transformation of data 

from different systems is sometimes necessary to 

allow matching (e.g., changing the patient 

corporate number from 12 to 10 digits).

Results

As of December 2011, the registry contained data 

on 265,130 patient encounters. It continues to 

collect and integrate data, and is expanding 

across the health system to accommodate data 

from affiliated clinics. Registry data are used to 

create quarterly updates on quality and outcomes 

measures identified by program leadership. For 

the two hospitals in the health system that are 

NAPBC-accredited, registry data are used to 

create their required annual reports on outcomes 

and benchmarks. Registry data have also been 

used for research purposes, such as identifying 

factors related to progression from premalignant 

to invasive lesions.

Key Point

Registries can take advantage of patient identity 

management solutions to link data from health 

information systems, regardless of whether a 

common patient identifier is present. Such linked 

data provide opportunities for quality 

improvement, research, and accreditation.
61

Chapter 17. Managing Patient Identity Across Data Sources

References for Chapter 17

1. North Carolina Health & Wellness Trust Fund 

Commission. North Carolina Health Information 

Exchange Strategic Plan. http://www.healthwellnc.

com/HealthIT/Docs/HITC_NCHIE_StrategicPlan.

pdf. Accessed August 17, 2012.

2. NorthPage Research LLC. 5 Tips for Successful 

Patient Identity Management in Government 

Agencies. http://www.govhealthit.com/sites/

govhealthit.com/files/resource-media/pdf/

orthpagereportpatientidentitymanagement

tipsforgovtagencies.pdf. Accessed August 17, 

2012.

3. HIMSS Patient Identity Integrity Work Group. 

Patient Identity Integrity. 2009. http://www.himss.

org/files/HIMSSorg/content/files/PrivacySecurity/

PIIWhitePaper.pdf. Accessed June 28, 2012.

4. Hsiao CJ, Hing E. Use and Characteristics of 

Electronic Health Record Systems Among 

Office-based Physician Practices: United States, 

2001–2012. NCHS Brief No. 111, Hyattsville, 

MD: National Center for Health Statistics. 2012. 

http://www.cdc.gov/nchs/data/databriefs/db111.

pdf. Accessed September 30, 2012.

5. Centers for Medicare and Medicaid Services. 

CMS EHR Meaningful Use Overview. http://

www.cms.gov/ehrincentiveprograms/30_

Meaningful_Use.asp. Accessed August 17, 2012.

6. Integrating the Healthcare Enterprise. http://www.

ihe.net. Accessed September 30, 2013.

7. Health Level Seven. http://www.hl7.org. Accessed 

May 1, 2013.

8. The Regenstrief Institute, Inc. http://www.

regenstrief.org/. Accessed September 30, 2013.

9. Hillestad R, Bigelow JH, Chaudhry B, et al. 

IDENTITY CRISIS: An Examination of the Costs 

and Benefits of a Unique Patient Identifier for the 

U.S. Health Care System. RAND Corporation 

Monograph. October 2008, No. 753. http://www.

rand.org/content/dam/rand/pubs/

monographs/2008/RAND_MG753.pdf. Accessed 

August 17, 2012.

10. American Society for Testing and Materials 

(ASTM). Standard Guide for Properties of a 

Universal Healthcare Identifier (UHID). http://

www.astm.org/Standards/E1714.htm. Accessed 

August 17, 2012.

11. Health Insurance Portability and Accountability 

Act of 1996, Pub. L. No. 104-191 Sec. 1173(b) 

(August 21, 1996). 

12. National Committee on Vital and Health Statistics 

(NCVHS), Subcommittee on Standards and 

Security. Hearing Minutes. July 20-21, 1998, 

Chicago, Il. http://ncvhs.hhs.gov/980720mn.htm. 

Accessed August 17, 2012.

13. Omnibus Consolidated and Emergency 

Supplemental Appropriations Act of 1999, Pub. L. 

No. 105-277 112 Stat. 2681-386.

Case Example 41. Using patient identity 

management methods to combine health 

system data (continued)

For More Information

http://oregon.providence.org/patients/

healthconitionscare/breast-health/Pages/

askanexpertland.aspx?TemplateName=

Providence+Breast+Health+Care+Registry&

Templatetype=FormsandInstructions

Nelson HD, Wang L, Weerasinghe R, et al. Trends 

and Influences on Mammography Screening in a 

Community Health System. Poster presented at: 

Women's Health 2011: The 19th Annual 

Congress. Washington, DC; April 1-3, 2011.

Soot L, Weerasinghe R, Wang L, et al. Core 

Needle vs. Surgical Excision Breast Biopsy in a 

Community-based Health System. Poster 

presented at: 13th Annual Meeting of the 

American Society of Breast Surgeons. Phoenix, 

AZ; May 2–6 2012.

Soot L, Weerasinghe R, Nelson H, et al. How 

often are High-risk Breast Lesions on Initial Core 

Biopsy Upgraded after Subsequent Excisional 

Biopsy? Poster Presentation at American College 

of Surgeons meeting, Chicago; October 2012.
62

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

14. Greenberg MA, Ridgely M. Patient identifiers and 

the National Health Information Network: 

debunking a false front in the privacy wars. 

Journal of Health and Biomedical Law. 

2008;4(1):31-68.

15. Johnson SB, Whitney G, McAuliffe M, et al. 

Using global unique identifiers to link autism 

collections. J Am Med Inform Assoc. 2010 

Nov-Dec;17(6):689-95. PMID: 20962132. 

PMCID: 3000750.

16. Jacobs JP, Haan CK, Edwards FH, et al. The 

rationale for incorporation of HIPAA compliant 

unique patient, surgeon, and hospital identifier 

fields in the STS database. Ann Thorac Surg. 2008 

Sep;86(3):695-8. PMID: 18721549.

17. Ludvigsson JF, Otterblad-Olausson P, Pettersson 

BU, et al. The Swedish personal identity number: 

possibilities and pitfalls in healthcare and medical 

research. Eur J Epidemiol. 2009;24(11):659-67. 

PMID: 19504049. PMCID: 2773709.

18. Robertsson O, Dunbar M, Knutson K, et al. 

Validation of the Swedish Knee Arthroplasty 

Register: a postal survey regarding 30,376 knees 

operated on between 1975 and 1995. Acta Orthop 

Scand. 1999 Oct;70(5):467-72. PMID: 10622479.

19. National Health Service. “Records – The NHS 

Number.” http://www.nhs.uk/NHSEngland/thenhs/

records/Pages/thenhsnumber.aspx. Accessed 

August 17, 2012.

20. Global Patient Identifiers, Inc. VUHID System. 

https://gpii.info/. Accessed August 17, 2012.

21. Kantara Initiative. Kantara Initiative & eCitizen 

Foundation announces kick-off of Patient ID 

Service (PIDS) pilot. March 1, 2011. http://

kantarainitiative.org/kantara-initiative-ecitizenfoundation-announces-kick-off-of-patient-idservice-pids-pilot/. Accessed September 30, 2013.

22. Li X, Shen C. Linkage of patient records from 

disparate sources. Stat Methods Med Res. 2013 

Feb;22(1):31-8. PMID: 21665896.

23. Campbell KM, Deck D, Krupski A. Record 

linkage software in the public domain: a 

comparison of Link Plus, The Link King, and a 

'basic' deterministic algorithm. Health Informatics 

J. 2008 Mar;14(1):5-15. PMID: 18258671.

24. Wild EL, Hastings TM, Gubernick R, et al. Key 

elements for successful integrated health 

information systems: lessons from the States. 

J Public Health Manag Pract. 2004 

Nov;Suppl:S36-47. PMID: 15643357.

25. Turchin A, Shubina M, Murphy SN. I am Not 

Dead Yet: Identification of False-Positive Matches 

to Death Master File. AMIA Annu Symp Proc. 

2010;2010:807-11. http://www.ncbi.nlm.nih.gov/

pmc/articles/PMC3041274/. Accessed August 17, 

2012.

26. Wang HE, Balasubramani GK, Cook LJ, et al. 

Medical conditions associated with out-of-hospital 

endotracheal intubation. Prehosp Emerg Care. 

2011 Jul-Sep;15(3):338-46. PMID: 21612386. 

PMCID: 3103090.

27. Durham E, Xue Y, Kantarcioglu M, et al. Private 

medical record linkage with approximate 

matching. AMIA Annu Symp Proc. 2010 Nov 13; 

2010:182-6. PMID: 21346965. PMCID: 3041434.

28. Finney JM, Walker AS, Peto TE, et al. An efficient 

record linkage scheme using graphical analysis for 

identifier error detection. BMC Med Inform Decis 

Mak. 2011;11:7. PMID: 21284874. PMCID: 

3039555.

29. Schnell R, Bachteler T, Reiher J. Privacypreserving record linkage using Bloom filters. 

BMC Med Inform Decis Mak. 2009;9:41. PMID: 

19706187. PMCID: 2753305.

30. Developing a Registry of Patient Registries 

(RoPR). Project Abstract. Agency for Healthcare 

Research and Quality. http://www.

effectivehealthcare.ahrq.gov/index.cfm/search-forguides-reviews-and-reports/?productid=690&page

action=displayproduct. Accessed August 14, 2012.

31. Pacheco AG, Saraceni V, Tuboi SH, et al. 

Validation of a hierarchical deterministic recordlinkage algorithm using data from 2 different 

cohorts of human immunodeficiency virusinfected persons and mortality databases in Brazil. 

Am J Epidemiol. 2008 Dec 1;168(11):1326-32. 

PMID: 18849301. PMCID: 2638543.

32. Meray N, Reitsma JB, Ravelli AC, et al. 

Probabilistic record linkage is a valid and 

transparent tool to combine databases without a 

patient identification number. J Clin Epidemiol. 

2007 Sep;60(9):883-91. PMID: 17689804.
63

Chapter 17. Managing Patient Identity Across Data Sources

33. Alemi F, Loaiza F, Vang J. Probabilistic master 

lists: integration of patient records from different 

databases when unique patient identifier is 

missing. Health Care Manag Sci. 2007 

Feb;10(1):95-104. PMID: 17323657.

34. U.S. Department of Health & Human Services. 

The Office of the National Coordinator for Health 

Information Technology. Health IT Policy 

Committee: Recommendations to the National 

Coordinator for Health IT. Transmittal Letter. 

Feburary 8, 2011. http://healthit.hhs.gov/portal/

server.pt/community/healthit_hhs_gov__policy_

recommendations/1815. Accessed August 17, 

2012.

35. National Science and Technology Council. 

Biometrics Glossary. http://www.biometrics.gov/

Documents/Glossary.pdf. Accessed August 17, 

2012.

36. Prabhakar S, Pankanti S, Jain A. Biometric 

recognition: security and privacy concerns. IEEE 

Security & Privacy. 2003 March/April;1(2):33-42.

37. Lawrence S. Biometrics bring fingerprint ID to 

hospitals. CIO Insight. 2005 Mar 24. http://www.

cioinsight.com/c/a/Health-Care/Biometrics-BringFingerprint-ID-to-Hospitals/. Accessed August 17, 

2012.

38. Marohn D. Biometrics in healthcare. Biometric 

Technology Today. 2006;14(9):9-11.

39. Rasmussen A, Sevier S, Kelly JA, et al. The lupus 

family registry and repository. Rheumatology 

(Oxford). 2011 Jan;50(1):47-59. 

PMID: 20864496. PMCID: 3307518.

40. Wolf EJ, Miller MW, Krueger RF, et al. 

Posttraumatic stress disorder and the genetic 

structure of comorbidity. J Abnorm Psychol. 2010 

May;119(2):320-30. PMID: 20455605. 

PMCID: 3097423.

41. Ouaguenouni S, Sivaraman K, Braun T. Identity 

Resolution and Data Quality Algorithms for 

Master Person Index: An Oracle White Paper. 

http://www.oracle.com/us/industries/healthcare/

identity-resolution-algorithm-wp-171743.pdf. 

Accessed August 17, 2012.

42. OpenEMPI. An Open Source Enterprise Master 

Patient Index. http://openempi.kenai.com/. 

Accessed August 17, 2012.

43. Boyd AD, Saxman PR, Hunscher DA, et al. The 

University of Michigan Honest Broker: a Webbased service for clinical and translational 

research and practice. J Am Med Inform Assoc. 

2009 Nov-Dec;16(6):784-91. PMID: 19717803. 

PMCID: 3002130.

44. Integrating the Healthcare Enterprise (IHE). IHE 

IT Infrastructure (ITI) Technical Framework. 

Volume 1 (ITI-TF1) Integration Profiles. http://

www.ihe.net/Technical_Framework/upload/IHE_

ITI_TF_Rev8-0_Vol1_FT_2011-08-19.pdf. 

Accessed August 17, 2012.

45. Lyons RA, Jones KH, John G, et al. The SAIL 

databank: linking multiple health and social care 

datasets. BMC Med Inform Decis Mak. 2009;9:3. 

PMID: 19149883. PMCID: 2648953.

46. Brooks CJ, Stephens JW, Price DE, et al. Use of a 

patient linked data warehouse to facilitate diabetes 

trial recruitment from primary care. Prim Care 

Diabetes. 2009 Nov;3(4):245-8. PMID: 19604741.

47. Durkin S, Sullivan C, et al. The Privacy and 

Security Gaps in Health Information Exchanges. 

http://library.ahima.org/xpedio/groups/public/

documents/ahima/bok1_049023.pdf. Accessed 

August 17, 2012.

48. Dimitropolous L, Alakoye A, Anderson H, et al. 

Privacy and Security Solutions for Interoperable 

Health Information Exchange: Nationwide 

Summary. 2007. Rockville, MD: Agency for 

Healthcare Research and Quality. http://healthit.

ahrq.gov/portal/server.pt/community/ahrq-funded_

projects/654/outcomes_from_the_privacy_and_

security_solutions_for_interoperable_health_

information_exchange_project/24069. Accessed 

August 17, 2012.

49. Commission on Systemic Interoperability. Ending 

the Document Game: Connecting and 

Transforming Your Healthcare Through 

Information Technology. Washington, DC: U.S. 

Government Printing Office; 2005. http://

endingthedocumentgame.gov/PDFs/entireReport.

pdf. Accessed August 17, 2012.

50. European Commission. Proposal for a Regulation 

of the European Parliament and of the Council, 

COM (2012) 11 final (Jan. 25, 2012). http://ec.

europa.eu/justice/data-protection/document/

review2012/com_2012_11_en.pdf. Accessed 

August 17, 2012.
64

51. Miller AR, Tucker CE. Encryption and the loss of 

patient data. J Policy Anal Manage. 2011 

Summer;30(3):534-56. PMID: 21774164.

52. Noumeir R, Lemay A, Lina JM. Pseudonymization 

of radiology data for research pu rposes. J Digit 

Imaging. 2007 Sep;20(3):284-95.

53. Neubauer T, Heurix J. A methodology for the 

pseudonymization of medical data. Int J Med 

Inform. 2011 Mar;80(3):190-204. 

PMID: 21075676.

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources
65

Chapter 18. Analysis of Linked Registry Data Sets

1. Introduction

This chapter provides a review and discussion of 

the analytic challenges faced by studies that use 

existing administrative databases and patient 

registries. We provide additional detail and 

examples of the issues raised in Chapter 13. While 

that chapter focused on the analysis of registry 

data in accordance with the registry’s purpose and 

objectives, this chapter tackles the issues and 

opportunities that arise when using registry data, 

often in combination with other data sources, to 

investigate hypotheses or questions that are 

secondary to the original reason for data 

collection. Case Examples 42 and 43 provide 

real-world examples of the analysis of linked 

registry data sets.

The use of administrative databases and medical 

registries to provide data for epidemiologic 

research has blossomed in the last decade,1

fulfilling prophecies that date to the mid-1970s.2 

Studies that use data collected for a primary 

purpose other than research (e.g., administrative 

databases) or collected for research purposes but 

used to support secondary studies (e.g., patient 

registries) have contributed substantial information 

to the understanding of the incidence, prevalence, 

outcomes, and other descriptive characteristics of 

many diseases. For simplicity, this chapter will 

refer to all such studies as “retrospective database 

studies.” Retrospective database studies have also 

contributed information to the understanding of 

disease etiology, patterns of treatment and 

disparities in care, adverse effects and late events 

associated with disease treatments, and the 

comparative effectiveness of some therapies. 

Despite these achievements, retrospective database 

studies sometimes receive criticism because of 

their potential to yield invalid results.1, 3 Weiss, for 

example, points out the potential for retrospective 

database studies to ascertain exposures, outcomes, 

and potential confounding variables with poor 

accuracy, or to provide an invalid reference group 

(the unexposed in a cohort design or controls in a 

case-control design). Ray1 provides a table of 

potential pitfalls in “automated database studies,” 

which includes a similar warning about inaccurate 

measurement of exposure, outcomes, and 

covariates, the potential for unmeasured 

confounding and missing data, and the potential to 

include immortal person-time. 

While these examples and lists of pitfalls provide 

valuable guidance, none of them is unique to 

retrospective database studies. Nonrandomized 

studies of all designs are susceptible to systematic 

errors arising from mismeasurement of analytic 

variables,4 unmeasured confounding,5 and poor 

choice of reference group.6, 7 Also, immortal 

person-time bias is not limited to retrospective 

database studies; it has even plagued a secondary 

analysis of data gathered in a randomized trial.8, 9

Taking a different approach, this chapter begins 

with a review of the fundamentals of sound study 

design and analysis. These fundamentals apply to 

epidemiologic research nested in any study 

population, but the chapter will focus on and 

illustrate the topics with examples that use 

retrospective database studies. In the subsequent 

sections, important considerations in retrospective 

database studies will be discussed, with the 

recognition that studies nested in other study 

populations may have the same considerations, but 

perhaps less often or to a lesser degree than 

retrospective database studies.

2. Fundamentals of Design 

and Analysis in Retrospective 

Database Research

2.1 Statement of Objective

Most productive epidemiologic research begins 

with a clear statement of objective. This objective 

might be descriptive; for example, to measure the 

incidence of a particular disease in some 

population, to characterize the patterns or costs of 

treatment for a particular disease in some 

population, or to measure the occurrence of 

outcomes among patients with a particular disease. 
66

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

The objective might also involve a comparison; for 

example, to compare the incidence of a particular 

disease in two or more subgroups defined by 

common characteristics (e.g., etiologic research), 

to compare the cost or quality of care for a 

particular disease in two or more subgroups (e.g., 

health services research or disparities research), or 

to compare the rate of outcomes among two or 

more subgroups of patients (often defined by 

different types or levels of treatment) with a 

particular disease (e.g., clinical research). In all 

cases, the overarching objective is to obtain an 

accurate (valid and precise) and generalizable 

estimate of the frequency of an outcome’s 

occurrence, or its relative frequency compared 

across groups.10 A valid estimate is one that can be 

interpreted with little influence by systematic 

error. A precise estimate is one that can be 

interpreted with little influence by random error. A 

generalizable estimate is one that provides 

information pertinent to the target population, the 

population for which the study’s information 

provides a basis for potential action, such as a 

public health or medical intervention. Often times 

the objective will be accompanied by a specific 

hypothesis (see Chapter 13), although that is less 

important than the statement of objective.

2.2 Selection of a Study Population 

Once the study’s objectives have been stated, the 

next step in the research plan is to select a study 

population. Selection of a study population 

requires identifying potential participants in time 

and place, including inclusion/exclusion 

(admissibility) criteria related to the study’s 

objectives and feasibility. Admissibility criteria 

related to the study’s objectives include focusing 

on a clinically relevant study population of 

individuals in whom sufficient events will occur to 

provide adequate precision for the estimates of 

disease frequency, and in whom the exposure 

categories will occur with sufficient frequency to 

provide adequate precision for the estimates of 

association. These criteria are also used to exclude 

people with characteristics that can introduce 

significant bias into the estimates of disease 

frequency or estimates of association, and that 

cannot be controlled easily or adequately in the 

analysis. Precision and validity criteria for 

admissibility pertain to all studies, regardless of 

whether they are nested in a health database. 

Admissibility criteria related to feasibility center 

on access to the data. Many ongoing cohort studies 

have established procedures for data sharing. 

Similarly, most publicly funded health databases 

have established procedures for data access. 

Investigators must ordinarily provide a statement 

of the study’s objective, a protocol for data 

collection from the database and for data analysis, 

a list of individuals who will have access to the 

data, and a study timeline. Some databases charge 

a fee for data access, although many do not. 

An advantage of retrospective database studies is 

the potential to study associations between rare 

exposures and rare outcomes in a population large 

enough to provide sufficient precision, with nearly 

complete followup, and with few exclusion criteria 

pertaining to age, comorbidity, or other factors that 

sometimes limit participation in clinical trials.11, 12

For example, surveillance databases that monitor 

adverse events potentially associated with 

pharmaceuticals identified signals suggesting that 

use of HMG CoA-reductase inhibitors (statins) 

might increase the risk of amyotrophic lateral 

sclerosis (ALS).13, 14 The only available 

epidemiologic evidence came from pooling 41 

randomized trials, in which ten ALS cases 

occurred among 56,352 individuals assigned to 

placebo and nine ALS cases occurred among 

64,602 individuals assigned to the statins arm.14

Using Danish databases, a case-control study 

identified 556 cases of ALS or other motor neuron 

syndromes and 5,560 population controls.15 The 

odds ratio associating disease occurrence with 

statins use was 0.96 (95% CI, 0.73 to 1.28), 

thereby rapidly and cost-efficiently providing 

evidence to counter the drug-monitoring studies 

and with far greater precision than provided by the 

pooled clinical trials. 

Selection of a study population inevitably involves 

balancing accuracy and generalizability concerns, 

as well as cost and feasibility considerations. For 

example, restriction is one of the most effective 

strategies for control of confounding through study 

design.16 If one is concerned about confounding 

by sex, a simple and effective strategy to control 

that confounding is to restrict the study population 
67

Chapter 18. Analysis of Linked Registry Data Sets

to a single sex. However, such restriction reduces 

the study’s precision by decreasing the sample 

size, and may also reduce the generalizability of 

the results (only applicable to half of the target 

population). An alternative would be to include 

both sexes and to stratify the analysis by sex. 

While this approach would improve the 

generalizability of the results, and allow an 

evaluation of confounding, the precision of the 

estimated association would be reduced, and 

perhaps substantially reduced, if the estimate of 

effect in men was substantially different from the 

estimate of effect in women. In this circumstance, 

the study becomes effectively two studies. 

2.3 Definition of Analytic Variables

The protocol for an epidemiological study should 

provide a clear, unambiguous definition of the 

outcome being studied, a description of how it will 

be measured, and a discussion of the accuracy of 

that measurement. When sensitivity of a 

dichotomous disease classification is 

nondifferential and independent of any errors in 

classification of exposure categories, and there are 

expected to be few false positives (near perfect 

specificity), there will usually be little bias of a 

ratio measure of association.4 This exception to the 

rule that “nondifferential misclassification biases 

towards the null” has important design 

implications. It suggests that retrospective 

database studies should be designed to optimize 

specificity; in fact, to ideally make the specificity 

perfect so there will be no false positives. Such a 

design might require more stringent criteria 

applied to the outcome definition than are 

ordinarily applied in a clinical setting, and 

therefore more stringent than might be found in a 

disease registry. For example, the estimated 

prevalence of dementia in a cohort of men and 

women aged 65 years or older varied by a factor of 

10 depending on the diagnostic criteria that were 

applied.17 Strategies to reduce inclusion of falsepositive cases can include requiring evidence in 

the patient record of medical procedures (e.g., 

cholecystectomy for gallstone disease or podiatry 

examination for diabetes) or interventions (e.g., 

insulin or glucose lowering medications for 

diabetes) that provide greater confidence in the 

validity of the case-finding definition.18 Such an 

approach often results in fewer included cases and 

reduced precision, but improved validity.19

If the study objective is to compare the frequency 

of outcome across subgroups, then the protocol 

should provide a definition of the exposure 

contrast(s). It is critical that both the index 

condition (i.e., the “exposed” or “treated” group) 

and the reference condition (i.e., the “unexposed” 

or “untreated/placebo” group) are well defined.6, 20

One frequent shortcoming of epidemiologic 

research is to compare the occurrence of disease in 

an index group with the occurrence of disease in 

all others who do not satisfy the index group 

definition. Studies of this design are easily 

constructed with retrospective database research, 

because of the abundance of participants who do 

not meet the index group definition. This “all 

others” reference group is therefore usually a 

poorly defined mixture of individuals.21 For 

example, if one uses a pharmaceutical registry to 

compare the incidence of a disease in statins users 

with the incidence of disease in those who do not 

use statins, the reference group of nonusers will 

contain individuals with indications for statin use 

but who have not been prescribed statins, as well 

as individuals without indications for statins use. 

Nonusers also differ from users in the frequency of 

contact with medical providers, which raises the 

potential for differential accuracy of ascertainment 

of health outcomes. It is therefore preferable to 

first ensure that the reference group of nonusers 

contains individuals who have indications for use 

of the treatment,18 and who, if possible, are 

receiving alternative therapies for the same 

indication.22 If one has a biologic basis to separate 

statins into categories, such as hydrophilic and 

hydrophobic statins, then a comparison of users of 

hydrophilic statins with users of hydrophobic 

statins would often be more valid. With these 

definitions, only individuals with indications for 

statins, and treated with statins, are included in the 

analysis, thereby reducing the potential for 

confounding by indication and differential 

followup.23

Finally, considerable attention should be given to 

identifying and accurately measuring potential 

confounders and effect modifiers.4, 24 The 

opportunity to examine important etiologic 
68

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

questions with considerable precision has 

expanded significantly with the availability of 

large databases, but systematic error due to 

confounding by unmeasured or poorly measured 

confounders remains a central concern. 

Fortunately, databases generally capture inpatient 

and outpatient clinical events and medication use 

that can characterize comorbidities and health 

care resource utilization, which can aid in the 

control of confounding. As discussed further 

below, information on behavioral and lifestyle 

factors (e.g., cigarette smoking, alcohol use, diet) 

is infrequently captured or is poorly measured in 

many databases. Thus, retrospective database 

researchers should carefully consider the available 

information on confounders before initiating 

studies. When data on critical confounders cannot 

be obtained in a database, and cannot be obtained 

by linking to another data source, an alternative 

data set might be better suited to accomplish the 

study’s objectives. Alternatively, in the presence 

of unmeasured confounding, researchers can use 

bias analysis5, 25 to assess the potential impact of 

residual confounding on their observed 

findings.26, 27

2.4 Validation Substudies

The goal of quality study design and analysis is to 

reduce the amount of error in an estimate of 

association. With this goal in mind, investigators 

have an obligation to quantify how far they are 

from this goal, and bias analysis is one method by 

which to achieve this goal.5, 25 Bias analysis 

methods require data to inform the bias model, and 

these data are obtained from internal or external 

validation substudies. Retrospective database 

research is often amenable to collection of internal 

validation data, for example by medical record 

review. In addition, many databases have internal 

protocols that constantly validate at least some 

aspects of the data. The validation data generated 

by these protocols can provide an initial indication 

of the data quality. To facilitate data collection for 

study-specific internal validation studies, 

investigators should consider the important threats 

to the validity of their research while designing 

their study, and should allocate project resources 

accordingly. This consideration should 

immediately suggest the corresponding bias 

analyses, which will then inform the data 

collection required to complete the bias modeling. 

For example, in the study of statin use related to 

ALS and neurodegenerative diseases described 

above,15 the ICD-10 code used to identify cases 

(G12.2) corresponded to diagnoses of ALS or 

other motor neuron syndromes. The investigators 

therefore selected a random sample of 25 

individuals from among all those who satisfied the 

case definition, and a clinician investigator 

reviewed their discharge summaries. The 

proportion of these 25 who did not have ALS (32 

percent) was used to inform a bias analysis to 

model the impact of these false-positive ALS 

diagnoses. Assuming a valid bias model, the bias 

analysis results showed that the null association 

was unlikely to result from the nondifferential 

misclassification of other diseases as ALS.

In this example, there was no effort to validate that 

non-cases of ALS were truly free of the disease. 

Non-cases are seldom validated, because falsenegative cases, especially of rare diseases, occur 

very rarely. Furthermore, validating the absence of 

disease often requires a study-supported medical 

examination of the non-case patients, an 

expensive, time-consuming, and invasive 

procedure. Prevalent diseases with a lengthy 

preclinical period and relatively simple diagnostic 

tests, such as diabetes, are more amenable to 

validation of non-cases. The ALS example also 

illustrates that an internal validation study requires 

protocol planning and allocation of study 

resources to collect the validation data. A protocol 

should be written that specifies how participants in 

the validation sample will be selected from the 

study population. Participation in the validation 

substudy might require informed consent to allow 

medical record review, whereas the database data 

itself might be available without individual 

informed consent. These aspects should be 

resolved in the planning stage, and the analytic 

plan should include a section devoted to bias 

modeling and analysis.5
69

Chapter 18. Analysis of Linked Registry Data Sets

3. Important Considerations

Once an investigator decides to pursue a research 

objective using a retrospective database study, 

there are a number of important considerations to 

evaluate before undertaking the study. These 

considerations mostly pertain to the quality and 

completeness of the database,28, 29 and especially 

to the potential for systematic errors in the 

database to affect the validity of the study’s result.

3.1 Structural Framework for Data 

Collection

Health databases collect data for various primary 

purposes30 and can be categorized as follows: 

(1) data collected for the purpose of reimbursing 

health care providers; (2) data collected for the 

purpose of monitoring care provided to 

beneficiaries of an integrated health care system; 

(3) data collected for the purpose of surveillance 

regarding a particular disease or disease category; 

(4) data collected for the purpose of surveillance 

for individuals with a specific exposure; and (5) 

data collected on individuals with a single 

admission-defining disease or medical procedure. 

Each type has strengths and limitations (presented 

in Table 18–1) to consider when evaluating the 

database for use in studies.

Databases that collect information for 

reimbursement (e.g., Medicare, Medicaid, or 

Ingenix), which are sometimes called “claims” or 

“administrative” databases, are quite useful for 

understanding health care costs and can provide 

important surveillance information on clinical 

practices and outcomes. However, they may be 

susceptible to systematic errors if data entries are 

manipulated by the data generators to affect (likely 

increase) their reimbursement. For instance, 

certain clinical conditions with high 

reimbursement rates may be preferentially reported 

on claims for patients who have those conditions 

but who present in the hospital or outpatient 

setting with other clinical issues, particularly if the 

presenting conditions are reimbursed at lower 

rates. The accuracy of some claims data sets have 

been questioned for diagnoses and procedures 

including dialysis,31 weight management,32

neutropenia,33 heart failure,34 diabetes,35 and 

functional outcomes after prostatectomy,36 as 

examples. On the other hand, the accuracy of 

registered diagnoses can be quite good.37 The 

accuracy of the claims data for its intended 

objective should therefore be considered, and 

preferably estimated quantitatively by an internal 

validation substudy.38, 39 Alternatively, estimates of 

the data’s accuracy may be available from an 

analogous study population from the same or a 

similar claims data set; an example is an external 

validation study. Claims data often lack important 

information on laboratory parameters, diagnostic 

test results, and behavioral and lifestyle 

characteristics, which may limit their utility for 

research in some topic areas.

The second type of database collects information 

on the health care provided to beneficiaries within 

an integrated health care system. This system can 

be a health insurer (e.g., Kaiser Permanente), a 

benefits program provided to selected individuals 

(e.g., Veteran’s Health Administration), or a 

national health care system (e.g., the United 

Kingdom’s Clinical Practice Research Database40). 

These databases typically use an integrated 

electronic health records system to capture health 

care information directly from physicians’ offices, 

hospitals, pharmacies, and other sites where care is 

provided (e.g., infusion centers, surgical centers). 

The granularity and quality of data captured in 

these databases is quite good and includes 

demographic and clinical characteristics, 

medication use, major clinical events including 

death, and importantly, results of diagnostic tests 

and laboratory assays. As with many 

epidemiological studies, some databases are 

limited in their geographic coverage and in the 

demographic characteristics of their patient 

populations. This lack of representativeness may 

affect the generalizability of results from studies 

nested in them.

The intended purpose of a third set of databases is 

surveillance of the incidence and outcomes related 

to a particular disease or disease category. These 

databases, or surveillance registries, often pertain 

to infectious diseases, cancer, and end-stage renal 

disease (ESRD). Surveillance for infectious 

diseases sometimes recognizes that only a 

proportion of cases will be reported, but assumes 

that the sensitivity and specificity of reporting 
70

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

remain constant over time, so that changes in the 

relative frequency of reported incidence provides a 

signal regarding the true incidence in the 

population. Thus, although the data quality is high, 

the completeness may be low. In contrast, both the 

data quality and completeness in most cancer 

registries are quite high, and the motivation for 

manipulation to influence reimbursement does not 

exist because the registry data are not used for that 

purpose. For example, the U.S. Cancer 

Surveillance, Epidemiology, and End Results 

(SEER) registry has a history of quality control 

and improvement dating to its inception in 1973 

and has been linked to the Medicare administrative 

database to provide data on cancer treatments and 

outcomes. In the United States and some other 

countries, patients with ESRD (patients receiving 

chronic dialysis or who are transplant recipients) 

are guaranteed coverage of all dialysis services 

including medications, procedures, and 

hospitalizations. These benefits extend throughout 

the patient’s life and require significant resources. 

Consequently such countries have established 

surveillance programs like the United States Renal 

Data System to monitor the health care provided to 

these patients and the costs associated with their 

health care.

The fourth type of database collects data on 

patients with a common exposure, and is 

commonly used as part of a postmarketing 

pharmacovigilance program related to a biologic 

or pharmaceutical product or a medical device. 

This type of database is typically designed to 

monitor the incidence of adverse events related to 

the exposure. These databases are often patient 

registries. 

A last type of database is a clinical patient registry 

of individuals with a single admission-defining 

disease or medical procedure. In fact, the first 

known health-related registry was the Leprosy 

Registry in Norway, initiated in 1856. In keeping 

with this history, many of the current clinical 

registries are found in Scandinavia. For example, 

the Danish government supports clinical databases 

used for quality assurance and research (e.g., 

breast cancer, colorectal cancer, hip arthroplasty, 

and rheumatologic diseases), as well as disease 

registries (e.g., the multiple sclerosis registry) used 

for monitoring and research.41 In fact, a central 

objective of disease-specific registries may be to 

provide an infrastructure for clinical trials 

pertaining to treatments for the disease. The main 

advantage of these registries and databases is the 

quality of data on disease characteristics, received 

treatments, and outcomes related to the disease. 

The main disadvantage is that they are difficult to 

use for studies of the etiology of the disease that 

initiates membership in the registry, since the 

registry includes only individuals with the disease.
71

Chapter 18. Analysis of Linked Registry Data Sets

Table 18–1. Types of databases used for retrospective database studies, and their typical advantages 

and disadvantages

Database Type Strengths Limitations Examples

Reimbursement 

purposes (“claims” 

data) 

• Population-based

• Captures inpatient and 

outpatient clinical events

• Captures mortality data 

• Captures oral and injectable 

medication use

• All claims are adjudicated 

• Specific patient populations 

(65+ yrs old/disabled; 

employed)

• Limited information on 

subject characteristics (e.g., 

lifestyle factors)

• Does not capture laboratory 

or test results

• Missing medication use in the 

hospital

• Reflects regional practices

• Medicare

• Ingenix

• Marketscan

Monitoring of 

health care provided 

to beneficiaries

• Population-based

• Captures inpatient and 

outpatient clinical events

• Captures oral and injectable 

medication use 

• Captures subject 

characteristics (e.g., body 

mass index, smoking, blood 

pressure)

• Limited racial and ethnic 

diversity

• Specific health care practices 

(e.g., selected formulary)

• Nordic Hospital 

Registries (Denmark, 

Sweden, Finland)

• Kaiser Permanente, 

Group Health 

Cooperative

• Clinical Practice 

Research Database 

(CPRD)

Disease or 

disease category 

surveillance

• Population-based

• Captures granular diseasespecific data (e.g., cancer 

stage)

• Captures outcome events

• Variable amounts of health 

care utilization information

• Limited information on 

subject characteristics (e.g., 

lifestyle factors)

• SEER, SEERMedicare

• USRDS

Exposure 

surveillance 

• Prospectively designed

• Typically collects granular 

information on relevant 

covariates

• Designed to capture all 

potential drug-related 

adverse events

• Limited information on 

comparator treatments.

Quality assurance 

or research 

regarding patients 

with a single 

admission-defining 

disease or procedure

• High-quality data on the 

index disease or procedure

• High quality data on the 

treatment and outcomes 

associated with the index 

disease or procedure

• Potential to link with other 

data sources to obtain more 

complete data

• Absence of an equivalent 

comparison group without the 

index disease or procedure

• Limited data on health 

conditions and treatments not 

related to the index disease or 

procedure

• Limited data on behavioral 

health (tobacco, diet, exercise, 

and alcohol consumption)

• Danish Breast Cancer 

Cooperative Group

• Danish Multiple 

Sclerosis Registry

• Danish Hip 

Arthroplasty registry

SEER = United States Cancer Surveillance, Epidemiology, and End Results Registry; USRDS = United States Renal 

Data System.
72

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

3.2 Changes in Coding Conventions 

Over Time

A common problem with retrospective database 

research is the impact of changes in coding 

conventions over the lifetime of the database. 

These changes can take the form of diagnostic 

drift,42 changes in discharge coding schemes, 

changes in the definition of grading of disease 

severity, or even variations in the medications on 

formulary in one region but not others at different 

points in time. For example, the Danish National 

Registry of Patients (DNRP) is a database of 

patient contacts at Danish hospitals. From 1977 to 

1993, discharge diagnoses were coded according 

to ICD-8, and from 1994 forward discharge 

diagnoses were coded according to ICD-10. 

ICD-10 included a specific code for chronic 

obstructive pulmonary disease (J44), whereas 

ICD-8 did not [ICD-8 496 (COPD not otherwise 

specified) did not appear in the DNRP]. In 

addition, from 1977 to 1994 the DNRP registered 

discharge diagnoses for only inpatient admissions, 

but from 1995 forward discharge diagnoses from 

outpatient admissions and emergency room 

contacts were also registered. COPD patients seen 

in outpatient settings before 1995 were therefore 

not registered; this excluded patients who likely 

had less severe COPD on average. The change in 

ICD coding convention in 1994 and the exclusion 

of outpatient admissions before 1995 presented a 

barrier to estimating the time trend for incidence 

of all admissions for COPD in any period that 

overlapped these two changes to the DNRP.43

The General Practice Research Database (GPRD) 

was a medical records database capturing 

information on approximately 5 percent of patients 

in the United Kingdom44 (as of March 2012, the 

GPRD became the Clinical Practice Research 

Database). Information was directly entered into 

the database by general practitioners trained in 

standardized data entry. When the GPRD was 

initiated in 1987, diagnoses were recorded using 

Oxford Medical Information Systems (OXMIS) 

codes, which were similar to ICD-9 codes. In 

1995, the GPRD adopted the Read coding system, 

a more detailed and comprehensive system that 

groups and defines illnesses using a hierarchical 

system. Without knowledge of this shift in coding 

and how to align codes for specific conditions 

across the different coding schemes, studies using 

multiple years of data could produce spurious 

findings. 

3.3 Other Data Quality Considerations

3.3.1 Selection of Registered Population

An important advantage of some retrospective 

database research is that it is population based, and 

therefore provides good representativeness for the 

target population. However, not all retrospective 

database research provides this advantage. For 

example, the U.S. Veterans Health Administration 

databases provide an important resource 

retrospective database research. A recent analysis 

of individuals receiving Veterans Health 

Administration services in fiscal years 2004 and 

2005 reported a mortality rate due to accidental 

poisoning of about 20 per 100,000 person-years.45

However, this database includes only U.S. military 

veterans, a selected subpopulation of the U.S. 

population, with a higher proportion of men than 

the overall population, and probably an 

unrepresentative proportion of other characteristics 

as well. The rate of accidental poisonings was thus 

almost twice that of the U.S. general population, 

after adjusting for differences in the age and sex 

distributions. Similarly, the Medicare 

administrative database provides an important 

resource for retrospective database research, 

including its links with the SEER cancer registry 

mentioned above. However, the former includes 

only Medicare recipients, almost all of whom are 

65 years of age or older, and many variables are 

unavailable for members of this population who 

participate in managed health care plans. Whether 

the lack of representativeness in these two 

examples, and others like them, affects inference 

made to the target population depends on the 

particular topic. 

3.3.2 Probability of Registration in Relation to 

Disease Severity

A second type of incomplete data arises at the 

level of registered individuals, rather than 

afflicting the whole database. In an earlier 

example, cases of COPD were registered in the 

Danish National Registry of Patients in reference 
73

Chapter 18. Analysis of Linked Registry Data Sets

to ICD-8 before 1994 and in reference to ICD-10 

thereafter. Only inpatient diagnoses of COPD were 

registered in the DNRP before 1995; inpatient, 

outpatient, and emergency department contacts 

were registered thereafter. At no time has the 

DNRP registered COPD cases diagnosed and 

treated only by a Danish General Practitioner. The 

least severe cases of this progressive disease are, 

therefore, missing from the DNRP throughout its 

history,46 and patients treated as outpatients are 

missing from the DNRP before 1995. Similar 

problems occur with hospital databases of other 

progressive diseases such as diabetes, Alzheimer’s 

disease, or Parkinson’s disease. Patients treated by 

their general practitioners will often eventually 

appear in the hospital database with the proper 

discharge diagnosis, since these progressive 

diseases become more severe over time. The less 

severe cases do not appear in hospital discharge 

databases, and their absence presents a barrier to 

studies of population-based incidence or 

prevalence, as well as to the accurate 

determination of whether exposure to a potential 

etiologic agent preceded the disease diagnosis,47

since neither the date of first diagnosis by the 

general practitioner nor the date of symptom onset 

is recorded.

Databases often lack accurate measurements of 

lifestyle and behavioral factors, such as tobacco 

use, alcohol drinking, exercise habits, and diet. 

Some databases can provide proxy measurements 

of these behavioral factors. For example, poor lung 

function or diagnosis of COPD is a proxy marker 

for tobacco smoking history, alcohol-related 

diseases such as cirrhosis or prescriptions for 

disulfiram can be used as proxy markers for 

alcohol abuse, and medically diagnosed obesity 

may be a proxy marker for poor diet and lack of 

exercise. None of these proxies provides a reliable 

measure of the actual concept, however. 

Other methods of estimation may add information. 

For diseases that can be identified by use of 

specific medications, one could compare the 

incidence of that medication use with the 

incidence in the hospitalization database to 

estimate the proportion of total cases that are 

registered. Comparison of the date of onset of the 

medication use with the date of first outpatient or 

inpatient diagnosis of the disease would provide an 

estimate of the typical delay between diagnosis by 

a general practitioner and progression of the 

disease to a severity level treated in the outpatient 

or inpatient setting.

3.3.3 Missing Data

Item nonresponse and missing data at the level of 

an individual record are often less of a problem for 

retrospective database research than for 

comparable cohort studies. Cohort studies that rely 

on participation by study subjects are subject to 

attrition and nonresponse. Attrition occurs when 

participants early in the cohort’s followup stop 

replying to regularly mailed surveys, telephone 

interviews, or emailed data collection instruments. 

These losses to followup are sometimes related to 

exposure characteristics and health outcomes, 

which introduces a form of selection bias,48 even if 

subjects rejoin the study at a later time.49 Item 

nonresponse occurs when a participant answers a 

survey or interview, but does not provide a 

response for one or more of the data fields. Item 

nonresponse can also occur when data on an 

exposure or outcome are collected by other 

methods, such as when a biospecimen is 

unavailable to provide tissue for an assay of a 

genetic or protein biomarker. This missing data 

may also be related to exposure and disease 

characteristics, and can introduce a bias, although 

reliable methods have been developed to resolve 

bias from item non-response (missing data) in 

many circumstances.50 Likewise, inverse 

probability weighting can sometimes be used to 

address selection bias and loss to followup,51

although it has seldom been implemented to date.

Retrospective database research ordinarily uses 

data collected for a primary purpose other than 

research. Item nonresponse (one form of missing 

data) is also often less of a concern, since the 

databases often have inherent quality control 

methods to assure high data completeness. Other 

forms of missing data can, however, plague 

retrospective database research in other ways. For 

example, left truncation is sometimes an important 

problem in retrospective database research, and is 

basically a missing data problem (although it can 

also be conceptualized as an information bias).52
74

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

Left truncation occurs when information required 

to characterize prevalent exposures, covariates, or 

diseases precedes the establishment of the 

database. With left truncation, unexposed 

individuals (e.g., nonusers of a medication) may 

have been users before the database was 

established, and apparently incident cases of a 

disease may have been diagnosed before the 

database was established, which would make them 

prevalent cases. Furthermore, covariate 

information collected at the inception of the 

database might have been affected by the medical 

history before the database was established. For 

example, blood pressure measured soon after a 

database began might be affected by blood 

pressure medications prescribed before the 

database began. Characterizing this initial 

measurement as baseline (i.e., preceding the first 

recorded prescription for blood pressure 

medications) would fail to account for the effect of 

the prevalent prescription for blood pressure 

medications, which was prescribed during the left 

truncation period. 

As a second example, in a study of the association 

between metformin use and the occurrence of 

breast cancer, the prescription database used to 

ascertain use of metformin among diabetic patients 

was not established until after the medication came 

to market.53 Data on use of metformin were 

therefore left-truncated, which can be 

conceptualized as a missing data problem for 

time-varying characterization of metformin use in 

the years preceding the database. (See Ibrahim and 

colleagues54 for a review of methods to model 

time-varying data.) Alternatively, this distortion 

can be conceptualized as the more general problem 

of having poor sensitivity of ever/never 

classification of metformin use. 

Left truncation is a common problem whenever 

prevalent conditions may have preceded the 

establishment of a database. For example, many 

etiologic epidemiology and clinical epidemiology 

studies exclude prevalent cases of the outcome at 

the inception of followup. However, some cases of 

disease may have occurred before followup began 

and even before the database’s inception, and these 

prevalent cases would be impossible to identify 

unless they also appeared in the database after its 

inception but before the followup time began. For 

many prevalent diseases with good survival, 

contact with the medical system is frequent, so 

most prevalent cases should be identifiable after 

the database is 5 to 10 years old. However, the 

potential for left truncation to mask some 

prevalent cases of the disease under study should 

be considered as a question specific to the research 

topic.

Right censoring can also occur in retrospective 

database research. For example, studies that use 

birth registries to ascertain congenital defects 

usually fail to detect defects that are diagnosed 

later in life, such as congenital heart anomalies. 

These defects are usually never recorded in the 

birth registry, so must be ascertained by some 

other method. Without such continued followup, 

the measurement of the outcome is right censored 

at the date of last followup by the birth registry.

Left truncation and right censoring are specific 

examples of the more general problem of data 

gaps. Data gaps occur when databases pertain only 

to a particular subgroup of the larger 

population,and membership in that subgroup is 

dynamic. Examples include individuals covered by 

Medicaid and members enrolled in managed care 

plans. In both examples, the databases pertain to 

participants in a health insurance program, and 

membership in those programs can change 

frequently. Data are collected only while the 

participants are members. If membership is lost 

and restored again later, there will be a data gap. 

Importantly, membership in these plans might be 

related to other characteristics that affect health, 

such as socioeconomic status or employment.55

Similar problems can arise when there are gaps in 

residency and the database is based on national 

health care data, or when individuals have health 

insurance from more than one source.

Data gaps in retrospective database research can 

also arise when medications are dispensed in the 

hospital, since many databases do not capture 

in-hospital medication use, leading to a form of 

information bias. In drug safety studies examining 

mortality risk related to the use of a particular 

medication, missing in-hospital medication use 

can result in spurious estimates of treatment 

effects.56 This bias was illustrated in a case-control 
75

Chapter 18. Analysis of Linked Registry Data Sets

study examining mortality risk related to inhaled 

corticosteroid use from the Saskatchewan, Canada, 

database. Analyses that failed to account for 

missed corticosteroid use during hospitalization 

events preceding death or the matched date for 

controls showed a beneficial effect (RR=0.6; 95% 

CI, 0.5 to 0.73). The RR estimates changed 

markedly once the missing in-hospital 

corticosteroid use was included (RR=0.93; 95% 

CI, 0.76 to 1.14 and RR=1.35; 95% CI, 1.14 to 

1.60).56 This bias has also been observed in 

studies of injectable medications in dialysis 

patients where hospitalization events preceding 

death resulted in spuriously low effect estimates.57 

3.4 Confounding by Indication

Confounding by indication may occur in 

nonrandomized epidemiologic research that 

compares two treatments (or treatment with no 

treatment).58 In the absence of randomization, the 

indications for selecting one treatment in 

preference to another (or in preference to no 

treatment) are often also related to the outcome 

meant to be achieved or prevented by the 

treatment.59 For example, randomized trials in 

younger breast cancer patients have shown that 

chemotherapy prevents breast cancer recurrence.60 

However, in a nonrandomized study of older breast 

cancer patients, those who received chemotherapy 

had a higher rate of recurrence than those who did 

not, probably because chemotherapy was offered 

only to the women with the most aggressive 

cancers.61 This example is a classic illustration of 

confounding by indication. Importantly, this study 

collected complete detailed data on every 

prognostic marker of recurrence and all of the 

other breast cancer treatments, yet adjustment for 

this detailed suite of variables did not resolve the 

confounding by indication, even using more 

advanced methods.23

Retrospective database research is as susceptible to 

confounding by indication as any other design. 

However, strategies to reduce the strength of this 

confounding have been proposed21 and may be 

most successful when used in the large study 

populations often achievable only in databases.62

Explained here is a special class of confounding 

by indication, which might arise especially in 

retrospective database research: time-dependent 

confounding by indication generated by dynamic 

dosing.63 Dynamic dosing refers to the clinical 

situation in which a medication’s dose is titrated 

(increased or decreased) in response to a changing 

biomarker or clinical measurement on which the 

medication acts (i.e., a clinical intermediate).63 

Examples include diabetes medications titrated in 

reaction to hemoglobin A1c (HbA1c) 

measurements, erythropoiesis stimulating agents 

(ESAs) titrated in reaction to hemoglobin levels, 

blood pressure medications titrated in reaction to 

systolic and diastolic blood pressure values, and 

antiretroviral therapy titrated in reaction to CD4 

counts. The clinical intermediate is therefore both 

a consequence of therapy and a predictor of future 

therapy. Time-dependent confounding arises when 

the clinical intermediate is also a prognostic 

indicator.64 For example, hemoglobin 

concentration is a time-dependent confounder of 

the effect of ESA therapy on survival because it is 

a risk factor for mortality, it predicts future ESA 

dose, and past ESA therapy predicts future 

hemoglobin concentration. Dynamic dosing 

therefore introduces time-dependent confounding 

of the treatment’s association with outcomes in the 

presence of this structure of confounding by 

indication.63 

It is important to recognize that the structure 

requires the clinical intermediate to be both a 

causal intermediate and a confounder. If it is only 

a confounder, such as baseline comorbidity or 

time-dependent comorbidity, the confounding can 

be addressed by conventional analytic methods. 

However, when the causal structure indicates that 

the clinical intermediate is both a causal 

intermediate and a confounder, inverse probability 

of treatment weighting (IPTW) with marginal 

structural models (MSMs) has been proposed as 

one method for valid adjustment.65

Pharmacoepidemiological studies that have used 

MSMs to address time-dependent confounding 

have shown significant improvements in 

confounding control relative to traditional timedependent analysis.66-68 In a study of the effect of 

highly active antiretroviral therapy (HAART) on 

time to AIDS, the hazard ratio using standard timedependent Cox regression to adjust for timevarying covariates such as CD4 count and HIV 
76

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

RNA level was 0.81 (95% CI, 0.61 to 1.07). Using 

an MSM, this effect was strengthened substantially 

(HR=0.54, 95% CI, 0.38 to 0.78), providing 

stronger evidence of the benefit of HAART.66 

Studies examining the effect of titrated ESA doses 

on mortality risk in dialysis patients that have used 

MSMs have found hazard ratio estimates at or 

below the null,67, 68 whereas results from 

traditional models found substantially elevated 

hazard ratio estimates.67

3.5 Precision Considerations When 

Standard Errors Are Small (OverPowered)

The large size of the study population that can 

often be included in retrospective database study is 

both a strength and a limitation. The sample size 

allows adjustment for multiple potential 

confounders with little potential for over-fitting or 

sparse data bias,69 and allows design features such 

as comparisons of different treatments for the 

same indication (comparative effectiveness 

research) to reduce the potential for confounding 

by indication.21 Nonetheless, systematic errors 

remain a possibility, and these systematic errors 

dominate the uncertainty when estimates of 

association are measured with high precision as a 

consequence of a large sample size.70 When 

confidence intervals are narrow, systematic errors 

remain, and/or inference or policy action will 

potentially result, investigators have been 

encouraged to employ quantitative bias analysis to 

more fully characterize the total uncertainty.25 Bias 

analysis methods have been used to address 

unmeasured confounding,27 selection bias,71 and 

information bias27, 72 in retrospective database 

research.

A second potential problem is the possibility of 

overweighting results from retrospective database–

based research in a quantitative meta-analysis of 

an entire body of research on a particular topic. In 

such meta-analyses, weights are in proportion to 

the inverse of variance, so large studies carry most 

of the weight. The variance, however, measures 

only sampling error; it does not measure 

systematic error. This problem of large studies 

dominating the weights pertains to any metaanalysis that includes one or two studies much 

larger than the others. However, given the large 

sample sizes often achieved by retrospective 

database research, the high-weight studies may 

often come from studies nested in these databases. 

For example, in a 2004 quantitative meta-analysis 

of 11 prospective studies of the association 

between pregnancy termination and incident breast 

cancer,73 the two retrospective database studies74, 75

accounted for 54 percent of the weight in the 

meta-analysis, but only 18 percent (2 of 11) of the 

studies. Random effects meta-analyses76 and other 

weighting methods77 provide only a partial solution 

to this potential overweighting, and only in some 

circumstances. Meta-analysts should therefore 

consider the potential for retrospective database 

research to be overweighted in their quantitative 

summary estimates. A plot of the inverse-normal of 

rank percentile against the corresponding study’s 

estimate of association and confidence interval 

provides a visual depiction of the distribution of 

study results,78 without undue influence by 

overpowered studies. (See, for example, the 

aforementioned meta-analysis of the association 

between pregnancy termination and breast cancer 

risk.73)

4. Special Opportunities

As noted earlier, retrospective database research 

runs the gamut of research topics. There are, 

however, several research areas to which 

retrospective database research studies are 

particularly well suited.

4.1 Rapid Response to Emerging 

Problems, With Prospective Data

Retrospective database research is ordinarily 

secondary to another primary purpose. While the 

collected data may not be optimized to a particular 

research topic, it is often possible to use the 

collected data for rapid response to emerging 

research problems.79 The study mentioned above 

of the association between statins medication and 

incident ALS is also a suitable example here. Drug 

surveillance databases had identified a higherthan-expected prevalence of statins medications 

associated with reports of ALS. A pooled analysis 

of trials data revealed no association, but was 

limited by the small number of ALS cases, short 

duration of followup, and potential for crossover 
77

Chapter 18. Analysis of Linked Registry Data Sets

from the placebo arm to statins treatment after the 

trial finished.80 Thus, there was little evidence to 

evaluate the potential causal association between 

this highly effective drug class—which prevents 

cardiovascular morbidity and mortality81, 82—and 

the incidence of ALS, a progressive, 

neurodegenerative, terminal disease.83

The precisely measured null association reported 

in the case-control study15 provided a rapid and 

reliable basis to assuage concerns about an 

etiologic association between statin use and ALS 

occurrence. Imagine what would have been 

required for a purposefully designed study to 

evaluate the association. The pooled trials result 

had included nearly 120,000 individuals observed 

over more than 400,000 person-years, yet included 

only 19 cases of this rare disease. Few existing 

cohort studies would have had sufficient persontime to expect substantially more cases, and a 

cohort study designed to evaluate the association 

would have required a substantial investment of 

time and financial support. 

A case-control study might have been feasible, but 

imagine the resources required to enroll and 

interview an equivalent number of ALS cases as 

were included in the database study (~550) and 

their matched controls. Furthermore, a casecontrol study of this design would likely have been 

susceptible to recall bias and selection bias.4, 7 The 

retrospective database research study avoided both 

of these biases.15 Recall bias was avoided by 

ascertaining statins use from a prescription 

database. These prescriptions were recorded before 

the ALS incidence, so could not have been affected 

by the subsequent disease occurrence. Selection 

bias was avoided because all ALS cases in the 

region during the followup period were included, 

and controls were selected from the Civil 

Registration System. Neither case/control status 

nor use of statins was likely to be associated with 

participation. Thus, the retrospective database 

research study on this topic provided a rapid, 

cost-efficient, and precise result on an important 

public health topic, which otherwise would have 

gone unevaluated or would have required a 

substantial investment of time and finances to 

achieve an equivalent, or possibly more biased, 

result. This study provides a good example of the 

value of retrospective database research in such 

circumstances.

4.2 Cost-Efficient Hypotheses-Scanning 

Analyses 

Retrospective database research can sometimes 

evaluate multiple associations with only a marginal 

increase in cost over the evaluation of a single 

association. The U.S. Food and Drug 

Administration’s (FDA) Sentinel Initiative will use 

an active surveillance system within electronic 

data from health care information holders to 

monitor the safety of all FDA-regulated 

products.84 Similarly, the EU-ADR project aims to 

use clinical data from health databases, combined 

with prescription databases, to detect adverse drug 

reactions.85, 86 The project uses text mining, 

epidemiological, and computational techniques to 

analyze electronic health records, with the goal of 

detecting combinations of drugs and adverse 

events that merit further investigation. 

As a second example, Latourelle and colleagues 

used retrospective database research to evaluate 

the association between estrogen-related diseases, 

such as osteoporosis or endometriosis, and the 

occurrence of Parkinson’s disease.87 To be 

categorized as “exposed” to these diseases, cases 

or controls had to have them appear as discharge 

codes in the hospital database before the first 

discharge code for Parkinson’s disease. For 

relatively little additional cost, the investigators 

also evaluated the association between 200 other 

diseases and the subsequent diagnosis of 

Parkinson’s disease as a hypothesis scanning study, 

with the objective of suggesting new ideas 

regarding Parkinson’s disease etiology.87 The 

analysis adjusted for multiple comparisons using 

empirical Bayesian methods designed to reduce 

the emphasis on potentially false-positive 

associations.88 This potential for cost-effective 

hypotheses-scanning studies as an explicit 

objective of retrospective database research should 

be viewed as a strength of such research, not a 

limitation, so long as the objective is appropriately 

labeled as such. Hypotheses suggested by these 

types of studies are often further investigated using 

studies designed specifically for the topic.
78

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

4.3 Hybrid Designs 

Retrospective database research does not 

necessarily have to be limited to data collection 

from secondary data sources. Hybrid designs allow 

the use of database research for some aspects of 

data collection, and primary data collection for 

others. For example, a study of drug-drug and 

gene-drug interactions that might reduce the 

effectiveness of tamoxifen therapy began by 

identifying eligible breast cancer patients using the 

Danish Breast Cancer Cooperative Group’s clinical 

registry.89 This clinical registry also provided data 

on prognostic factors such as tumor diameter and 

lymph node evaluation, and on treatments such as 

chemotherapy and radiation therapy. Linkage with 

the Danish Civil Registration System provided 

data on vital status; linkage with the Danish 

National Patient Registry provided data on 

comorbid diseases; and linkage with the Danish 

National Registry of Medicinal Products provided 

data on use of prescription medications. Thus, for 

relatively low cost, a cohort of breast cancer 

patients with complete medical, prognostic, and 

breast cancer treatment data was assembled. A 

case-control study was then nested in this cohort 

by identifying cases of breast cancer recurrence 

and then matching controls to them by risk-set 

sampling.7 Once cases and controls had been 

identified, their tumor blocks were collected from 

the Danish National Pathology Registry,90 and 

these were used for the necessary bioassays. Thus, 

retrospective database research allowed 

identification of the source population and 

selection of cases and controls, and provided all 

but the bioassay data. These data, which are 

expensive to collect, were only obtained for about 

13 percent of the members of the total cohort. This 

hybrid design demonstrates that retrospective 

database research will remain an important 

contributor, even in the era of personalized 

medicine.

In a second example of a hybrid design, survey 

data collected over the Internet were linked to 

retrospective database research.91, 92 The objectives 

of the study were to assess the feasibility and 

validity of studies that use the Internet to recruit 

and follow participants, evaluate the relationship 

between lifestyle and behavioral factors and 

delayed time to pregnancy among women 

attempting to conceive, and evaluate the 

relationship of several exposures to risk of 

miscarriage and infant birth weight among women 

who conceived. Participants were recruited by 

advertisements on Web sites likely to be visited by 

women who intended to become pregnant. They 

were directed to the study’s Web site, where they 

completed an enrollment screening questionnaire 

followed by an interview covering sociodemographics, reproductive and medical history, 

lifestyle, and other factors. Enrolled participants 

were then contacted every 2 months by email for 

12 months or until they reported that conception 

had occurred. Data obtained from the Web-based 

questionnaires were linked to nationwide 

databases, which allowed collection of additional 

data on confounders and outcomes, as well as an 

assessment of the validity of some of the selfreported data, such as prescription drug use. This 

study again demonstrated that retrospective 

database research, in combination with primary 

data collection, can provide a cost-efficient 

resource for collecting some aspects of the study 

data. In contrast to the previous cancer treatment 

example, the cohort in this pregnancy study was 

enrolled following more typical cohort study 

strategies, and not by using the databases to 

identify a source population. 

Hybrid designs have also been used to collect data 

by medical record review for data fields that are 

available for a subset of participants in a 

database.93 Thus, the database provides a costefficient resource for initial data collection, which 

is then supplemented as necessary by medical 

record review or another primary data collection 

method to complete the data set. Once an 

investigator is open to the potential for hybrid 

designs and there are retrospective database 

resources suitable to the research topic, the 

opportunities for combining the databases with 

primary data collection are limited only by the 

investigator’s creativity.
79

Chapter 18. Analysis of Linked Registry Data Sets

4.4 Ample Data Allows for Novel 

Designs 

As mentioned above, the ample data often 

available from retrospective database research can 

lead to overweighting of such studies in 

quantitative meta-analyses. While this problem 

may be disadvantageous, a compensating 

advantage is the opportunity to use retrospective 

database research to implement novel study 

designs. For example, confounding by indication 

and other biases often plague clinical 

epidemiology,3, 23 even in the era of comparative 

effectiveness research. However, the ample study 

size often provided by retrospective database 

research can overcome these threats to validity in 

some situations. The large sample size might allow 

a design with carefully restricted exposure 

groups,1 for example, new users of a 

pharmaceutical only,94 whereas conventionally 

sized cohort studies would not always have 

sufficient study size to implement such a design. 

The new user design in turn facilitates other 

advanced designs, such as propensity score 

matching and instrumental variable analyses,21

which are intended to further counteract these 

threats to validity. These and other novel designs 

can be implemented in studies of any size, but are 

likely most effective when the study size is large.95

4.5 Data Pooling Methods

Although retrospective database research often 

provides relatively large study size within a 

research topic area, a study’s power may still be 

insufficient if the study must be restricted to rare 

exposure subgroups or if the study outcome is rare. 

In these cases, data pooling across similar 

databases may allow sufficient sample size to 

provide adequate power. Data pooling also 

provides advantages over conventional metaanalyses because it allows simultaneous and 

consistent data analyses. However, such pooling 

projects face substantial challenges. 

First among these challenges is harmonization of 

the data elements. To accommodate a pooled 

analysis, data collected from different databases 

must provide analytic variables (exposure, 

confounders, modifiers, and outcomes) with 

equivalent categorizations and definitions. Such 

data harmonization can be quite challenging. 

Harmonization of data elements categorized 

differently or differentially available in two or 

more databases may pose an insurmountable 

barrier to pooling. For example, one database 

might include data on behaviors like alcohol and 

tobacco use, whereas a second database might not. 

The pooling project would then face the 

unenviable decision of controlling for these 

behaviors for some, but not all, data centers (in 

which case the analysis becomes comparable to a 

conventional meta-analysis), or abandoning 

control for these variables at all centers in order to 

achieve the data harmonization goal. Differences 

in the conceptual underpinnings of data elements 

may be more common. Even a variable as 

conceptually simple as the Charlson comorbidity 

index96 can present surprising challenges when 

subject to harmonization considerations. The 

Charlson index includes 19 comorbid conditions 

(e.g., diabetes). As mentioned above, some 

databases might be able to ascertain diabetes 

diagnosed in all medical settings (e.g., general 

practitioner, outpatient, and inpatient), whereas 

others might be able to ascertain diabetes 

diagnosed in only a subset (e.g., only general 

practitioner or only outpatient specialty clinics). 

Diabetes is defined differently in the different 

databases, which are not strictly harmonious, and 

therefore contribute differently to the Charlson 

index. While the definition of the Charlson 

variable may be harmonious across the pooled 

databases, the underlying conceptualization is 

different, and this difference could result in 

differences in the strength of confounding by the 

comorbidity variable or in the degree to which it 

modifies the association between an exposure 

contrast and outcome. 

Ethical and legal constraints, which are often 

placed on data sharing, present a second important 

challenge to pooling projects. Pooling of deidentified data sets can sometimes be arranged 

through data use agreements, but even these 

arrangements can be quite challenging and timeconsuming. Rassen and colleagues compared four 

methods of pooling de-identified data sets:97

(1) full covariate information, which may violate 

privacy concerns; (2) aggregated data methods, 
80

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

which aggregate patients into mutually stratified 

cells with common characteristics, but usually 

delete cells with low frequency counts that might 

defeat the privacy protections of large frequency 

counts; (3) conventional fixed or random effects 

meta-analysis, which provides only summary 

estimates of association for pooling; and 

(4) propensity score-based pooling, for which a 

propensity score summarizes each individual’s 

covariate information. They reported that the last 

alternative provided reasonable analytic flexibility 

and also strong protection of patient privacy, and 

advocated its use for studies that require pooling 

of databases, multivariate adjustment, and privacy 

protection.97

More recently, Wolfson and colleagues proposed a 

pooling method that requires no transfer of recordlevel data to a central analysis center.98 Rather, the 

central analysis center implements statistical 

computing code over a secure network, accessing 

record-level data maintained on servers at the 

individual study centers. Data aggregation occurs 

through return of anonymous summary statistics 

from these harmonized individual-level databases, 

and even iterative regression modeling can be 

implemented. The advantage is a reduced burden 

to comply with ethical and legal requirements to 

protect privacy, since no record-level data are ever 

transferred. The disadvantages include 

requirements for strong data harmonization, secure 

networks that satisfy regulatory oversight, and 

assurances that no record-level data are 

transmitted. It is possible that some summary 

statistics could violate standards for deidentification, but safeguards can be implemented 

to prevent transmission of such summary statistics.

These new methods for pooling provide exciting 

opportunities for pooled projects. At the time of 

this writing, investigators who choose to undertake 

them should expect delays required to explain 

these methods to regulators responsible for 

oversight of data protection, who are not yet 

familiar with them. In addition, it is likely that 

implementing the methods for the first few 

projects will be challenging. With those caveats in 

mind, the path should be blazed, because once the 

methods are familiar and reliable, new research 

opportunities and efficiencies will inevitably arise. 

Investigator teams without the time, resources, or 

patience to implement these new methods can 

ordinarily rely on conventional meta-analysis 

methods,99 which solve the privacy protection 

concerns but also have some important 

disadvantages by comparison.97, 98

5. Summary

Retrospective database research has made 

important contributions to descriptive 

epidemiology, public health epidemiology targeted 

at disease prevention, and clinical epidemiology 

targeted at improving disease outcomes or 

estimates of disease prognosis. Investigators who 

conduct retrospective database research should 

first focus on the fundamentals of epidemiologic 

design and analysis, with the goal of achieving a 

valid, precise, and generalizable estimate of 

disease frequency or association. Beyond the 

fundamentals, retrospective database research 

presents special challenges for design and analysis, 

and special opportunities as well; researchers 

should be aware of both in order to optimize the 

yield from their work.
81

Chapter 18. Analysis of Linked Registry Data Sets

Case Example 42. Combining de-identified 

data from multiple registries to study longterm outcomes in a rare disease

Description Four independent, prospective, 

observational, and multicenter 

disease registries participate in 

an ongoing systematic review of 

their aggregated data to study 

pediatric pulmonary arterial 

hypertension (PAH). The review 

is intended to describe disease 

course and long-term outcomes 

of pediatric PAH in real-world 

clinical settings.

Sponsor Actelion Pharmaceuticals Ltd.

Year Started 2009

Year Ended Ongoing

No. of Sites 4 multicenter registries

No. of Patients Approximately 500

Challenge

PAH is a rare disease that is poorly described in 

pediatric populations. Newly developed PAH 

therapies used in the pediatric population have 

recently led to improved survival, and patients are 

now likely to reach adulthood. This increased 

attention on pediatric PAH patients presents new 

challenges in both data needs and methodology to 

evaluate disease history and progression, general 

development, and clinical and treatment 

experience.

In 2009, the European Medicines Agency (EMA) 

approved Actelion’s product bosentan for an 

expanded indication of pediatric PAH. The 

sponsor then began working with the EMA to 

determine how best to collect longitudinal 

treatment and outcomes data on this population. 

Proposed Solution

Four existing registries already collected data on 

pediatric PAH patients: one is global and three 

are national in scope (in the United States, 

France, and the Netherlands). The sponsor and 

the EMA recognized that a compilation of results 

from these multiple registries within a common 

systematic review protocol would allow them to 

examine data from a large number of patients 

representing a significant proportion of global 

pediatric PAH patients. After the EMA approved 

the systematic review study design, the individual 

registries reviewed the protocol and agreed to 

participate.

The sponsor contacted the individual registries to 

evaluate their data collection and analysis 

practices. As it was not feasible to pool the data 

due to differences in data collection elements 

used by the registries, analyses were done by the 

respective registry data owners using similar 

methods under the guidance of a common 

statistical analysis plan. The de-identified 

summary tables were sent separately to the 

sponsor to be included in the systematic review 

reports. 

The outcomes of interest are disease course and 

long-term outcome (e.g., clinical worsening, 

hospitalization, death) and general development 

(e.g., height, body mass index, sexual maturation, 

onset of puberty). The protocol and statistical 

analysis plan define the study population (all 

patients enrolled in one of the four registries aged 

≤18 years at the time of diagnosis with PAH), 

observation period, appropriate statistical 

methods, and standardized procedures for data 

extraction (including data quality assurance). 

Results

Analyses are performed on an annual basis and 

the same data cutoff date is applied to all 

registries to define the observation times of 

analysis (i.e., from Oct 2009 to the annual 

report’s data cut-off date). This effectively creates 

a new cohort for each annual report, which is a 

stand-alone document. 

The first annual report was sent to the EMA in 

2010. For this first analysis, the sponsor had to 

address technical challenges related to differences 

between the registries. For example, three of the 

registries used the SAS software package to 

conduct their analysis, and one used SPSS, which 

Case Examples for Chapter 18
82

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

Case Example 42. Combining de-identified 

data from multiple registries to study longterm outcomes in a rare disease (cotinued)

Results (continued)

produces a slightly different output. For 

subsequent reports, the sponsor also spent time in 

dialogue with the registries to clarify the detailed 

requirements, definitions, and analyses of the 

statistical analysis plan to ensure that each 

registry understood and interpreted it the same 

way.

Longitudinal analyses will be examined for 

evidence of improvement or deterioration over 

the followup period. The method of analysis 

respects correlations of within-patient 

measurements and is based on all patients with at 

least two measurements during the followup 

period. 

Key Point

For rare disease populations where registries 

already exist, systematic review of registry data 

sets may be a more feasible way to analyze 

outcomes data rather than creating a new patient 

registry. When planning and conducting such a 

study, close collaboration between the parties is 

important to develop a detailed statistical analysis 

plan and clarify expectations for registry-level 

analyses.

For More Information

Berger RM, Beghetti M, Humpl T, et al. Clinical 

features of paediatric pulmonary hypertension: a 

registry study. Lancet. 2012 Feb 11;379(9815):

537-46.

Humbert M, Sitbon O, Chaouat A, et al. 

Pulmonary arterial hypertension in France: 

results from a national registry. American journal 

of respiratory and critical care medicine. 

2006;173(9):1023-30.

McGoon MD, Krichman A, Farber HW, et al. 

Design of the REVEAL registry for US patients 

with pulmonary arterial hypertension. Mayo 

Clinic proceedings. Mayo Clinic. 2008;83(8):

923-31.

Muros-Le Rouzic E, Brand M, Wheeler J, et al. 

Systematic review methods to assess growth and 

sexual maturation in pediatric population 

suffering from pulmonary arterial hypertension in 

real-world clinical settings. 27th International 

Conference on Pharmacoepidemiology & 

Therapeutic Risk Management, August 14-17, 

2011, Chicago, IL. Abstract 825.

van Loon RL, Roofthooft MT, van Osch-Gevers 

M, et al. Clinical characterization of pediatric 

pulmonary hypertension: complex presentation 

and diagnosis. J Pediatr. 2009 Aug;155(2):

176-82.e1. Epub 2009 Jun 12.
83

Chapter 18. Analysis of Linked Registry Data Sets

Case Example 43. Understanding baseline 

characteristics of combined data sets prior to 

analysis

Description The Kaiser Permanente Anterior 

Cruciate Ligament 

Reconstruction (KP ACLR) 

Registry was established to 

collect standardized data on 

ACLR procedures, techniques, 

graft types, and types of fixation 

and implants. The objectives of 

the registry are to identify risk 

factors that lead to degenerative 

joint disease, graft failure, and 

meniscal failure; determine 

outcomes of various graft types 

and fixation techniques; describe 

the epidemiology of ACLR 

patients; determine and compare 

procedure incidence rate at 

participating sites; and provide a 

framework for future studies 

tracking ACLR outcomes.

Sponsor Kaiser Permanente

Year Started 2005

Year Ended Ongoing

No. of Sites 42 surgical centers and 240 

surgeons

No. of Patients 17,000

Challenge

The KP ACLR Registry aimed to collaborate with 

the Norwegian Ligament Reconstruction Registry 

on a series of studies to proactively identify 

patient risk factors as well as surgical practices 

and techniques associated with poor surgical 

outcomes. The Norwegian registry has been 

operating since 2004 and contains data on 14,232 

patients. Combining data from these two registries 

would allow for faster identification of certain risk 

factors and evaluation of low frequency events. 

Proposed Solution

The first step was to compare the patient cohorts 

of the registries and the surgical practices of the 

two countries. Aggregate data were shared 

between the registries in tabular form. Analysis 

was conducted to identify differences that would 

be important to consider when making inferences 

about a population other than that covered by the 

registry. Commonalities were also identified to 

determine when inferences could be made from 

each other’s analysis and when data do not need to 

be adjusted.

Results

The analysis found that the registries generally 

have similar distributions of age, gender, 

preoperative patient-reported knee function, and 

knee-related quality of life. Differences were 

observed between the two registries in race, sports 

performed at the time of injury, time to surgery, 

graft use, and fixation type. While these 

differences should be accounted for in future 

analyses of combined data sets from both 

registries, the results indicate that analyses of the 

combined data sets are likely to produce findings 

that can be generalized to a wider population of 

ACLR patients.

Since this comparison was conducted, two 

hypothesis-driven analyses have begun, 

investigating questions using the combined 

registry data sets. Future plans include further 

collaboration with ACLR registries in additional 

countries.

Key Point

Combining or pooling registry data can be a 

valuable approach to achieving a larger sample 

size for data analysis. However, it is important to 

identify cohort and practice differences and 

similarities between registries before making 

generalizations of registry findings to other 

populations or sharing data for collaboration 

projects. 

For More Information

http://www.kpimplantregistries.org/Registries/acl.

htm 

Granan LP, Inacio MC, Maletis GB, et al. 

Intraoperative findings and procedures in 

culturally and geographically different patient and 

surgeon populations. ACTA Orthop. 2012;83:

577-82.

Maletis G, Granan LP, Inacio M, et al. 

Comparison of a community based anterior 

cruciate ligament reconstruction registry in the 

United States and Norway. The Journal of Bone 

and Joint Surgery. 2011 December; 93 

(Supplement 3): 31-6. 
84

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

References for Chapter 18

1. Ray WA. Improving automated database studies. 

Epidemiology. 2011 May;22(3):302-4. 

PMID: 21464650.

2. Federspiel CF, Ray WA, Schaffner W. Medicaid 

records as a valid data source: the Tennessee 

experience. Med Care. 1976 Feb;14(2):166-72. 

PMID: 768652.

3. Weiss NS. The new world of data linkages in 

clinical epidemiology: are we being brave or 

foolhardy? Epidemiology. 2011 May;22(3):292-4. 

PMID: 21464647.

4. Rothman K, Greenland S, Lash TL. Validity in 

epidemiologic studies. In: Rothman K, Greenland 

S, Lash TL, eds. Modern Epidemiology. 3rd ed. 

Philadelphia: Lippincott Williams & Wilkins; 

2008. pp. 128-47.

5. Lash TL, Fox MP, Fink AK. Applying quantitative 

bias analysis to epidemiologic data. New York, 

NY: Springer; 2009.

6. Rothman K, Greenland S, Poole C, et al. 

Causation and causal inference. In: Rothman K, 

Greenland S, Lash TL, eds. Modern 

Epidemiology. 3rd ed. Philadelphia: Lippincott 

Williams & Wilkins; 2008. pp. 5-31.

7. Rothman K, Greenland S, Lash TL. Case-control 

studies. In: Rothman K, Greenland S, Lash TL, 

eds. Modern Epidemiology. 3rd ed. Philadelphia: 

Lippincott Williams & Wilkins; 2008. pp. 111-27.

8. Souhami L, Bae K, Pilepich M, et al. Impact of 

the duration of adjuvant hormonal therapy in 

patients with locally advanced prostate cancer 

treated with radiotherapy: a secondary analysis of 

RTOG 85-31. J Clin Oncol. 2009 May 

1;27(13):2137-43. PMID: 19307511. 

PMCID: 2674000.

9. Collette L, Studer UE. Selection bias is not a good 

reason for advising more than 5 years of adjuvant 

hormonal therapy for all patients with locally 

advanced prostate cancer treated with 

radiotherapy. J Clin Oncol. 2009 Nov 

20;27(33):e201-2; author reply e4. 

PMID: 19786659.

10. Greenland S, Rothman K. Measures of 

occurrence. In: Rothman K, Greenland S, Lash 

TL, eds. Modern Epidemiology. 3rd ed. 

Philadelphia: Lippincott Williams & Wilkins; 

2008. pp. 32-50.

11. Murthy VH, Krumholz HM, Gross CP. 

Participation in cancer clinical trials: race-, sex-, 

and age-based disparities. JAMA. 2004 Jun 

9;291(22):2720-6. PMID: 15187053.

12. Heiat A, Gross CP, Krumholz HM. Representation 

of the elderly, women, and minorities in heart 

failure clinical trials. Arch Intern Med. 2002 Aug 

12-26;162(15):1682-8. PMID: 12153370.

13. Edwards IR, Star K, Kiuru A. Statins, 

neuromuscular degenerative disease and an 

amyotrophic lateral sclerosis-like syndrome: an 

analysis of individual case safety reports from 

vigibase. Drug Saf. 2007;30(6):515-25. 

PMID: 17536877.

14. Colman E, Szarfman A, Wyeth J, et al. An 

evaluation of a data mining signal for amyotrophic 

lateral sclerosis and statins detected in FDA's 

spontaneous adverse event reporting system. 

Pharmacoepidemiol Drug Saf. 2008 

Nov;17(11):1068-76. PMID: 18821724.

15. Sorensen HT, Riis AH, Lash TL, et al. Statin use 

and risk of amyotrophic lateral sclerosis and other 

motor neuron disorders. Circ Cardiovasc Qual 

Outcomes. 2010 Jul;3(4):413-7. PMID: 20530788.

16. Rothman K, Greenland S, Lash TL. Design 

strategies to improve study accuracy. In: Rothman 

K, Greenland S, Lash TL, eds. Modern 

Epidemiology. 3rd ed. Philadelphia: Lippincott 

Williams & Wilkins; 2008. pp. 162-82.

17. Erkinjuntti T, Ostbye T, Steenhuis R, et al. The 

effect of different diagnostic criteria on the 

prevalence of dementia. N Engl J Med. 1997 

Dec 4;337(23):1667-74. PMID: 9385127.

18. Jick SS, Bradbury BD. Statins and newly 

diagnosed diabetes. Br J Clin Pharmacol. 2004 

Sep;58(3):303-9. PMID: 15327590. 

PMCID: 1884569.

19. Brenner H, Savitz DA. The effects of sensitivity 

and specificity of case selection on validity, 

sample size, precision, and power in hospitalbased case-control studies. Am J Epidemiol. 1990 

Jul;132(1):181-92. PMID: 2192549.

20. Jick H, Garcia Rodriguez LA, Perez-Gutthann S. 

Principles of epidemiological research on adverse 

and beneficial drug effects. Lancet. 1998 Nov 

28;352(9142):1767-70. PMID: 9848368.
85

Chapter 18. Analysis of Linked Registry Data Sets

21. Sturmer T, Jonsson Funk M, Poole C, et al. 

Nonexperimental comparative effectiveness 

research using linked healthcare databases. 

Epidemiology. 2011 May;22(3):298-301. 

PMID: 21464649.

22. Jick SS, Kaye JA, Russmann S, et al. Risk of 

nonfatal venous thromboembolism with oral 

contraceptives containing norgestimate or 

desogestrel compared with oral contraceptives 

containing levonorgestrel. Contraception. 2006 

Jun;73(6):566-70. PMID: 16730485.

23. Bosco JL, Silliman RA, Thwin SS, et al. A most 

stubborn bias: no adjustment method fully 

resolves confounding by indication in 

observational studies. J Clin Epidemiol. 2010 

Jan;63(1):64-74. PMID: 19457638. 

PMCID: 2789188.

24. Greenland S. The effect of misclassification in the 

presence of covariates. Am J Epidemiol. 1980 

Oct;112(4):564-9. PMID: 7424903.

25. Greenland S, Lash TL. Bias analysis. In: Rothman 

K, Greenland S, Lash TL, eds. Modern 

Epidemiology. 3rd ed. Philadelphia: Lippincott 

Williams & Wilkins; 2008. pp. 345-80.

26. Bradbury BD, Wilk JB, Kaye JA. Obesity and the 

risk of prostate cancer (United States). Cancer 

Causes Control. 2005 Aug;16(6):637-41. 

PMID: 16049801.

27. Lash TL, Schmidt M, Jensen AO, et al. Methods to 

apply probabilistic bias analysis to summary 

estimates of association. Pharmacoepidemiol 

Drug Saf. 2010 Jun;19(6):638-44. 

PMID: 20535760.

28. Goldberg J, Gelfand HM, Levy PS. Registry 

evaluation methods: a review and case study. 

Epidemiol Rev. 1980;2:210-20. PMID: 7000537.

29. Roos LL, Mustard CA, Nicol JP, et al. Registries 

and administrative data: organization and 

accuracy. Med Care. 1993 Mar;31(3):201-12. 

PMID: 8450678.

30. Sorensen HT, Baron JA. Registries and medical 

databases. In: Trichopoulos D, Olsen JH, Saracci 

R, eds. Teaching Epidemiology. New York, NY: 

Oxford University Press; 2010. pp. 455-67.

31. Clement FM, James MT, Chin R, et al. Validation 

of a case definition to define chronic dialysis using 

outpatient administrative data. BMC Med Res 

Methodol. 2011;11:25. PMID: 21362182. 

PMCID: 3055853.

32. Aphramor L. Validity of claims made in weight 

management research: a narrative review of 

dietetic articles. Nutr J. 2010;9:30. 

PMID: 20646282. PMCID: 2916886.

33. Kim SY, Solomon DH, Liu J, et al. Accuracy of 

identifying neutropenia diagnoses in outpatient 

claims data. Pharmacoepidemiol Drug Saf. 2011 

Jul;20(7):709-13. PMID: 21567653. 

PMCID: 3142869.

34. Quach S, Blais C, Quan H. Administrative data 

have high variation in validity for recording heart 

failure. Can J Cardiol. 2010 Oct;26(8):306-12. 

PMID: 20931099. PMCID: 2954539.

35. Chen G, Khan N, Walker R, et al. Validating ICD 

coding algorithms for diabetes mellitus from 

administrative data. Diabetes Res Clin Pract. 2010 

Aug;89(2):189-95. PMID: 20363043.

36. Tollefson MK, Gettman MT, Karnes RJ, et al. 

Administrative data sets are inaccurate for 

assessing functional outcomes after radical 

prostatectomy. J Urol. 2011 May;185(5):1686-90. 

PMID: 21419458.

37. Thygesen SK, Christiansen CF, Christensen S, et 

al. The predictive value of ICD-10 diagnostic 

coding used to assess Charlson comorbidity index 

conditions in the population-based Danish 

National Registry of Patients. BMC Med Res 

Methodol. 2011;11:83. 

PMID: 21619668. PMCID: 3125388.

38. Cai S, Mukamel DB, Veazie P, et al. Validation of 

the Minimum Data Set in identifying 

hospitalization events and payment source. J Am 

Med Dir Assoc. 2011 Jan;12(1):38-43. 

PMID: 21194658. PMCID: 3052878.

39. Tirschwell DL, Longstreth WT, Jr. Validating 

administrative data in stroke research. Stroke. 

2002 Oct;33(10):2465-70. PMID: 12364739.

40. Clinical Practice Research Datalink. http://www.

cprd.com/intro.asp. Accessed September 30, 2013.

41. Sorenson HT, Christensen T, Schlosser HK, et al. 

Use of Medical Databases in Clinical 

Epidemiology. Second ed. Denmark: SUN-TRYK 

Aarhus Universitet; 2009.

42. Anderson IB, Sorensen TI, Prener A. Increase in 

incidence of disease due to diagnostic drift: 

primary liver cancer in Denmark, 1943-85. BMJ. 

1991 Feb 23;302(6774):437-40. PMID: 2004170. 

PMCID: 1669338.
86

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources

43. Lash TL, Johansen MB, Christensen S, et al. 

Hospitalization rates and survival associated with 

COPD: a nationwide Danish cohort study. Lung. 

2011 Feb;189(1):27-35. PMID: 21170722.

44. Rodriguez LA, Perez-Gutthann S, Jick SS. The 

UK General Practice Research Database. In: 

Strom BL, ed. Pharmacopepidemiology. 3rd ed. 

Chichester, UK: John Wiley & Sons, LTD; 2000. 

pp. 375-85.

45. Bohnert AS, Ilgen MA, Galea S, et al. Accidental 

poisoning mortality among patients in the 

Department of Veterans Affairs Health System. 

Med Care. 2011 Apr;49(4):393-6. 

PMID: 21407033.

46. Hansen JG, Pedersen L, Overvad K, et al. The 

Prevalence of chronic obstructive pulmonary 

disease among Danes aged 45-84 years: 

population-based study. COPD. 2008 

Dec;5(6):347-52. PMID: 19353348.

47. Alonso A, Jick SS, Jick H, et al. Antibiotic use 

and risk of multiple sclerosis. Am J Epidemiol. 

2006 Jun 1;163(11):997-1002. PMID: 16597708.

48. Hernan MA, Hernandez-Diaz S, Robins JM. 

A structural approach to selection bias. 

Epidemiology. 2004 Sep;15(5):615-25. 

PMID: 15308962.

49. Clough-Gorr KM, Fink AK, Silliman RA. 

Challenges associated with longitudinal 

survivorship research: attrition and a novel 

approach of reenrollment in a 6-year follow-up 

study of older breast cancer survivors. J Cancer 

Surviv. 2008 Jun;2(2):95-103. PMID: 18648978.

50. Donders AR, van der Heijden GJ, Stijnen T, et al. 

Review: a gentle introduction to imputation of 

missing values. J Clin Epidemiol. 2006 

Oct;59(10):1087-91. PMID: 16980149.

51. Howe CJ, Cole SR, Chmiel JS, et al. Limitation of 

inverse probability-of-censoring weights in 

estimating survival in the presence of strong 

selection bias. Am J Epidemiol. 2011 Mar 

1;173(5):569-77. PMID: 21289029. 

PMCID: 3105434.

52. Cain KC, Harlow SD, Little RJ, et al. Bias due to 

left truncation and left censoring in longitudinal 

studies of developmental and disease processes. 

Am J Epidemiol. 2011 May 1;173(9):1078-84. 

PMID: 21422059. PMCID: 3121224.

53. Bosco JL, Antonsen S, Sorensen HT, et al. 

Metformin and incident breast cancer among 

diabetic women: a population-based case-control 

study in Denmark. Cancer Epidemiol Biomarkers 

Prev. 2011 Jan;20(1):101-11. PMID: 21119073.

54. Ibrahim JG, Chu H, Chen LM. Basic concepts and 

methods for joint models of longitudinal and 

survival data. J Clin Oncol. 2010 Jun 

1;28(16):2796-801. PMID: 20439643.

55. Riley GF. Administrative and claims records as 

sources of health care cost data. Med Care. 2009 

Jul;47(7 Suppl 1):S51-5. PMID: 19536019.

56. Suissa S. Immeasurable time bias in observational 

studies of drug effects on mortality. Am J 

Epidemiol. 2008 Aug 1;168(3):329-35. 

PMID: 18515793.

57. Bradbury BD, Wang O, Critchlow CW, et al. 

Exploring relative mortality and epoetin alfa dose 

among hemodialysis patients. Am J Kidney Dis. 

2008 Jan;51(1):62-70. PMID: 18155534.

58. Walker AM. Confounding by indication. 

Epidemiology. 1996 Jul;7(4):335-6. 

PMID: 8793355.

59. Miettinen OS. The need for randomization in the 

study of intended effects. Stat Med. 1983 AprJun;2(2):267-71. PMID: 6648141.

60. NCCN Practice Guidelines in Oncology, Breast 

Cancer - v.2.2011. Invasive breast cancer, systemic 

adjuvant treatment. [National Comprehensive 

Cancer Network]. 2010. http://www.nccn.org/

professionals/physician_gls/f_guidelines.asp. 

Accessed August 15, 2012.

61. Geiger AM, Thwin SS, Lash TL, et al. 

Recurrences and second primary breast cancers in 

older women with initial early-stage disease. 

Cancer. 2007 Mar 1;109(5):966-74. 

PMID: 17243096.

62. Brookhart MA, Wang PS, Solomon DH, et al. 

Instrumental variable analysis of secondary 

pharmacoepidemiologic data. Epidemiology. 2006 

Jul;17(4):373-4. PMID: 16810095.

63. Bradbury BD, Brookhart MA, Winkelmayer WC, 

et al. Evolving statistical methods to facilitate 

evaluation of the causal association between 

erythropoiesis-stimulating agent dose and 

mortality in nonexperimental research: strengths 

and limitations. Am J Kidney Dis. 2009 

Sep;54(3):554-60. PMID: 19592144.
87

Chapter 18. Analysis of Linked Registry Data Sets

64. Weiss NS, Dublin S. Accounting for timedependent covariates whose levels are influenced 

by exposure status. Epidemiology. 1998 

Jul;9(4):436-40. PMID: 9647909.

65. Hernan MA, Brumback B, Robins JM. Marginal 

structural models to estimate the causal effect of 

zidovudine on the survival of HIV-positive men. 

Epidemiology. 2000 Sep;11(5):561-70. 

PMID: 10955409.

66. Cole SR, Hernan MA, Robins JM, et al. Effect of 

highly active antiretroviral therapy on time to 

acquired immunodeficiency syndrome or death 

using marginal structural models. Am J 

Epidemiol. 2003 Oct 1;158(7):687-94. 

PMID: 14507605.

67. Zhang Y, Thamer M, Cotter D, et al. Estimated 

effect of epoetin dosage on survival among elderly 

hemodialysis patients in the United States. Clin J 

Am Soc Nephrol. 2009 Mar;4(3):638-44. 

PMID: 19261818. PMCID: 2653651.

68. Wang O, Kilpatrick RD, Critchlow CW, et al. 

Relationship between epoetin alfa dose and 

mortality: findings from a marginal structural 

model. Clin J Am Soc Nephrol. 2010 

Feb;5(2):182-8. PMID: 20019122. 

PMCID: 2827587.

69. Greenland S, Schwartzbaum JA, Finkle WD. 

Problems due to small samples and sparse data in 

conditional logistic regression analysis. Am J 

Epidemiol. 2000 Mar 1;151(5):531-9. 

PMID: 10707923.

70. Greenland S. Randomization, statistics, and causal 

inference. Epidemiology. 1990 Nov;1(6):421-9. 

PMID: 2090279.

71. Fink AK, Lash TL. A null association between 

smoking during pregnancy and breast cancer using 

Massachusetts registry data (United States). 

Cancer Causes Control. 2003 Jun;14(5):497-503. 

PMID: 12946045.

72. Lash TL, Fox MP, Thwin SS, et al. Using 

probabilistic corrections to account for abstractor 

agreement in medical record reviews. Am J 

Epidemiol. 2007 Jun 15;165(12):1454-61. 

PMID: 17406006.

73. Lash TL, Fink AK. Null association between 

pregnancy termination and breast cancer in a 

registry-based study of parous women. Int J 

Cancer. 2004 Jun 20;110(3):443-8. 

PMID: 15095312.

74. Melbye M, Wohlfahrt J, Olsen JH, et al. Induced 

abortion and the risk of breast cancer. N Engl J 

Med. 1997 Jan 9;336(2):81-5. PMID: 8988884.

75. Goldacre MJ, Kurina LM, Seagroatt V, et al. 

Abortion and breast cancer: a case-control record 

linkage study. J Epidemiol Community Health. 

2001 May;55(5):336-7. PMID: 11297654. 

PMCID: 1731878.

76. Poole C, Greenland S. Random-effects metaanalyses are not always conservative. Am J 

Epidemiol. 1999 Sep 1;150(5):469-75. 

PMID: 10472946.

77. Shuster JJ. Empirical vs natural weighting in 

random effects meta-analysis. Stat Med. 2010 

May 30;29(12):1259-65. PMID: 19475538.

78. Cunnane C. Unbiased plotting positions - a 

review. Journal of Hydrology. 1978;37:205-22.

79. Grady D, Hearst H. Utilizing existing databases. 

In: Hully SB, Cummings SR, Browner WS, et al., 

eds. Designing Clinical Research. 3rd ed. 

Philadelphia: Lippincott Williams & Wilkins; 

2007. pp. 207-21.

80. Sorensen HT, Lash TL. Statins and amyotrophic 

lateral sclerosis--the level of evidence for an 

association. J Intern Med. 2009 Dec;266(6):520-6. 

PMID: 19930099.

81. Thavendiranathan P, Bagai A, Brookhart MA, et 

al. Primary prevention of cardiovascular diseases 

with statin therapy: a meta-analysis of randomized 

controlled trials. Arch Intern Med. 2006 Nov 

27;166(21):2307-13. PMID: 17130382.

82. Aronow HD, Topol EJ, Roe MT, et al. Effect of 

lipid-lowering therapy on early mortality after 

acute coronary syndromes: an observational study. 

Lancet. 2001 Apr 7;357(9262):1063-8. 

PMID: 11297956.

83. Mitchell JD, Borasio GD. Amyotrophic lateral 

sclerosis. Lancet. 2007 Jun 16;369(9578):

2031-41. PMID: 17574095.

84. Rosati K. Using electronic health information for 

pharmacovigilance: the promise and the pitfalls. 

J Health Life Sci Law. 2009 Jul;2(4):171, 3-239. 

PMID: 19673181.

85. Trifiro G, Pariente A, Coloma PM, et al. Data 

mining on electronic health record databases for 

signal detection in pharmacovigilance: which 

events to monitor? Pharmacoepidemiol Drug Saf. 

2009 Dec;18(12):1176-84. PMID: 19757412.
88

86. Coloma PM, Schuemie MJ, Trifiro G, et al. 

Combining electronic healthcare databases in 

Europe to allow for large-scale drug safety 

monitoring: the EU-ADR Project. 

Pharmacoepidemiol Drug Saf. 2011 Jan;20(1):

1-11. PMID: 21182150.

87. Latourelle JC, Dybdahl M, Destefano AL, et al. 

Estrogen-related and other disease diagnoses 

preceding Parkinson's disease. Clin Epidemiol. 

2010;2:153-70. PMID: 20865113. 

PMCID: 2943181.

88. Greenland S, Robins JM. Empirical-Bayes 

adjustments for multiple comparisons are 

sometimes useful. Epidemiology. 1991 

Jul;2(4):244-51. PMID: 1912039.

89. Lash TL, Cronin-Fenton D, Ahern TP, et al. 

CYP2D6 inhibition and breast cancer recurrence 

in a population-based study in Denmark. J Natl 

Cancer Inst. 2011 Mar 16;103(6):489-500. 

PMID: 21325141. PMCID: 3057982.

90. Erichsen R, Lash TL, Hamilton-Dutoit SJ, et al. 

Existing data sources for clinical epidemiology: 

the Danish National Pathology Registry and Data 

Bank. Clin Epidemiol. 2010;2:51-6. 

PMID: 20865103. PMCID: 2943174.

91. Mikkelsen EM, Hatch EE, Wise LA, et al. Cohort 

profile: the Danish Web-based Pregnancy Planning 

Study—'Snart-Gravid'. Int J Epidemiol. 2009 

Aug;38(4):938-43. PMID: 18782897. 

PMCID: 2734065.

92. Huybrechts KF, Mikkelsen EM, Christensen T, et 

al. A successful implementation of 

e-epidemiology: the Danish pregnancy planning 

study 'Snart-Gravid'. Eur J Epidemiol. 2010 

May;25(5):297-304. PMID: 20148289. 

PMCID: 2945880.

93. Thwin SS, Clough-Gorr KM, McCarty MC, et al. 

Automated inter-rater reliability assessment and 

electronic data collection in a multi-center breast 

cancer study. BMC Med Res Methodol. 

2007;7:23. PMID: 17577410. PMCID: 1919388.

94. Ray WA. Evaluating medication effects outside of 

clinical trials: new-user designs. Am J Epidemiol. 

2003 Nov 1;158(9):915-20. PMID: 14585769.

95. Brookhart MA, Rassen JA, Schneeweiss S. 

Instrumental variable methods in comparative 

safety and effectiveness research. 

Pharmacoepidemiol Drug Saf. 2010 

Jun;19(6):537-54. PMID: 20354968. 

PMCID: 2886161.

96. Charlson ME, Pompei P, Ales KL, et al. A new 

method of classifying prognostic comorbidity in 

longitudinal studies: development and validation. 

J Chronic Dis. 1987;40(5):373-83. 

PMID: 3558716.

97. Rassen JA, Solomon DH, Curtis JR, et al. Privacymaintaining propensity score-based pooling of 

multiple databases applied to a study of biologics. 

Med Care. 2010 Jun;48(6 Suppl):S83-9. 

PMID: 20473213. PMCID: 2933455.

98. Wolfson M, Wallace SE, Masca N, et al. 

DataSHIELD: resolving a conflict in 

contemporary bioscience--performing a pooled 

analysis of individual-level data without sharing 

the data. Int J Epidemiol. 2010 Oct;39(5):1372-82. 

PMID: 20630989. PMCID: 2972441.

99. Greenland S, O'Rourke K. Meta-analysis. In: 

Rothman K, Greenland S, Lash TL, eds. Modern 

Epidemiology. 3rd ed. Philadelphia: Lippincott 

Williams & Wilkins; 2008. pp. 652-82.

Section IV. Technical, Legal, and Analytic Considerations for Combining Registry 

Data With Other Data Sources
Section V 

Special 

Applications in 

Patient Registries

91

Chapter 19. Use of Registries in Product 

Safety Assessment

1. Introduction

Once a drug or device is approved for use by a 

regulatory authority, the product is generally used 

by larger and more diverse populations than are 

typically studied in the clinical trials leading up to 

approval. As a result, the period after approval is 

an important phase for identifying and 

understanding product safety concerns associated 

with both acute and chronic use. The need for 

postapproval (also called postmarketing) safety 

assessment as it exists today was, for the most part, 

born out of well-publicized product safety issues 

that were initially detected by clinicians 

recognizing a pattern of rare serious events, such 

as phocomelia caused by prenatal exposure to 

thalidomide1 and rare vaginal cancers that 

occurred in young women who had in utero 

exposure to diethylstilbestrol.2 The detection of 

serious adverse drug reactions after authorization 

has led to much debate about the adequacy of both 

industry and regulatory approaches to 

preauthorization assessment and testing. However, 

the decision to authorize a medicine is a balance 

between wanting to know as much as possible 

about the safety of a product and the need to make 

new drugs available for patients.3 The implication 

is that authorization cannot mean that a medicine 

is completely safe; rather, it is an assessment that 

at the time of authorization, the known benefits for 

the average patient in the approved indication 

outweigh the known risks. But the degree to which 

the known risks represent the actual safety profile 

of a product will depend upon the size, duration, 

representativeness, and thoroughness of the 

clinical trial program, which, in turn, is related to 

the complexity of the patients and the state of 

knowledge of the disease being targeted. Trials 

conducted as part of clinical development are, by 

necessity, of limited duration and size and 

generally focus on a narrowly defined population 

that represents only a small segment of the 

population with the disease or product use of 

interest. Clinical trial populations tend to be 

restricted to those who have limited concurrent 

disease and who are on few, if any, concomitant 

medications. Typically, trial protocols include 

lengthy lists of inclusion and exclusion criteria that 

further restrict the trial population. Unless a drug 

or a product is intended for a very narrow 

indication or a very rare disease, it is not feasible 

to require clinical trials to be inclusive of all types 

of patients likely to ever be exposed to it. Even in 

the case of a narrow indication, the potential 

long-term and delayed effects of a product are 

unlikely to be established during most clinical trial 

development programs.

To address the acknowledged limitations of what is 

known about the safety profile of a product at the 

time of authorization, postmarketing pharmacoand medical device vigilance is traditionally, and 

by regulation, performed through spontaneous 

adverse event (AE) reporting. The exact 

requirements for spontaneous reporting to the 

regulatory authorities vary internationally and 

depend on the country/region, approval type, and 

product type. It is widely acknowledged, however, 

that spontaneous reporting captures an extremely 

small percentage of the actual events occurring, 

and that, while it is useful for identifying rare and 

potentially significant events,4, 5 it has limited use 

in the detection of other equally important types of 

events, including increases in events with a high 

background rate. This form of postmarketing 

surveillance is reactive in that one waits for AEs/

reactions to be spontaneously reported, assesses 

them for causality, and estimates the importance of 

the information.

As well as collecting only an indeterminate 

fraction of adverse reactions, this method of 

surveillance depends upon someone reporting the 

events of interest. There is some evidence that 

clinicians who report AEs are not typical of 

clinicians in general, and other reporters such as 

patients, lawyers, and consumer groups may have 

unclear motivations for reporting, which 

introduces further bias into the equation.6-8

The current methods available for AE reporting are 

seen by many as burdensome and not amenable to 
92

Section V. Special Applications in Patient Registries

incorporation into a clinician’s normal workflow. 

Waiting for reports to arrive and accumulate may 

also delay the detection of adverse reactions. On 

the other hand, a massive uptake of a new drug or 

device, such as seen with Viagra® (sildenafil 

citrate) or coronary artery stents, may lead to a 

sudden flood of reports of nonserious as well as 

serious AEs that could potentially overwhelm 

established systems.

To overcome some of the difficulties associated 

with managing large databases of spontaneous 

AEs, many investigators employ statistical 

methods to identify signals of disproportionate 

reporting (SDRs). These methods identify AEs that 

are reported more frequently with a drug or device 

than would be expected compared with other 

event/product pairs in the database, and do not 

imply any kind of causal relationship.9 It is 

important to be precise as to what is meant when 

using the term “signal” or “signal detection” since 

the terms are ambiguous; in the context of 

automated methods of detecting statistical 

anomalies, the term “SDR” should be used.9

However, these statistical methods may not be 

reliable in certain situations, such as when there is 

major confounding or when the increased risk is 

small compared with the background incidence of 

the event.9 All these above-mentioned limitations 

mean that there are situations when spontaneous 

reporting may not be adequate as the sole method 

of postmarketing surveillance.

To address problems with traditional pharmaco- or 

medical device vigilance when there are particular 

known limitations of knowledge of the safety 

profile of a product and/or to further address 

unresolved safety concerns, some products are 

approved subject to postmarketing commitments, 

which may be requested for safety purposes as 

well as to address other outstanding questions. In 

Europe, in response to concerns over 

pharmacovigilance, marketing authorization 

applicants are required to submit a European 

Union–risk management plan (EU-RMP) when 

seeking a marketing authorization for the majority 

of new chemical entities and biologics. This 

EU-RMP states what is known and not known 

about the safety profile of a medicinal product, 

how its safety profile will be monitored, 

investigated, and characterized, and what risk 

minimization activities will be undertaken. While 

many products will require only routine 

pharmacovigilance, for others more proactive 

methods of pharmacovigilance will be necessary 

to supplement the use of spontaneous adverse 

reaction reporting and periodic safety update 

reports. Although additional clinical trials may 

occasionally be mandated, it is more common for 

observational pharmacoepidemiological studies to 

be conducted to ascertain the safety profile of a 

product under real-world use.

Other observational methods of tracking and 

evaluating safety data have historically included 

active surveillance systems, such as the 

prescription event monitoring (PEM) systems used 

in the United Kingdom (Drug Safety Research 

Unit),10 New Zealand (NZ Intensive Monitoring 

Programme), Japan (J-PEM), and elsewhere, 

targeting new products; and the retrospective use 

of administrative claims data. In the United 

Kingdom, the requirement that access to most 

secondary care is through a general practitioner 

has led to the use of their electronic health care 

systems for pharmacovigilance purposes; however, 

this type of integrated approach is not yet widely 

accessible elsewhere. In May 2008, the U.S. Food 

and Drug Administration (FDA) launched the 

Sentinel Initiative, an effort to create an integrated 

electronic system in the United States for AE 

monitoring, incorporating multiple existing data 

sources including claims data and electronic 

medical record systems.11

Medical devices in the United States have 

surveillance programs different from those for 

drugs. The Safe Medical Devices Act of 1990 

requires that high-risk medical devices be tracked 

after marketing, and that product corrections and 

removals be reported to FDA if actions were taken 

to reduce health risks. Most medical device safety 

tracking is accomplished through reports 

submitted to FDA from medical facilities when 

devices are implanted or explanted. In addition, 

hospitals, nursing homes, ambulatory surgery 

centers, and outpatient treatment facilities are 

required to report to FDA whenever they believe 

that a device caused or contributed to the death of 

a patient, though this reporting is a voluntary 

requirement and not enforceable or audited.12
93

Chapter 19. Use of Registries in Product Safety Assessment

Whether to comply with a postmarketing 

requirement or out of a desire to supplement 

spontaneous reporting, prospective product and 

disease registries are also increasingly being 

considered as a resource for examining unresolved 

safety issues and/or as a tool for proactive risk 

assessment in the postapproval setting. The 

advantage of registries is that their observational 

and inclusive design may allow for surveillance of 

a diverse patient population that can include 

sensitive subgroups and other groups not typically 

included in initial clinical trials, such as pregnant 

women, minorities, older patients, children, or 

patients with multiple comorbidities, as well as 

those taking concomitant medications. In contrast 

to clinical trials, in which the inclusion criteria are 

generally tightly focused and restrictive by design, 

registry populations are generally more 

representative of the population actually using a 

product or undergoing a procedure, since the 

inclusion criteria are usually broad and may 

potentially include all patients exposed regardless 

of age, comorbidities, or concurrent treatments. 

Data collection may lead to insights about provider 

prescribing practices or off-label use and 

information regarding the potential for studying 

new indications within the expanded patient 

population. Followup duration can be long to 

encompass delayed risks, consequences of longterm use, and/or effects of various combinations 

and sequencing of treatments. Such information 

can be used as a source of publications, to assist 

the medical community with developing 

recommendations for monitoring patient safety 

and product usage, and/or to contribute to the 

understanding of the natural history of the disease.

There are also many challenges to the utility of 

registry data for providing more clarity about 

safety concerns and for prospective risk 

surveillance. These challenges relate largely to 

how products are used and the legal, regulatory, 

and ethical responsibilities of registry sponsors. 

Most registries that follow specific products do so 

through cooperation from physicians who 

prescribe (or implant) these products. Depending 

on the setup and legal constraints of the registry, 

sometimes only a subsection of prescribing 

physicians may be involved in entering patients, a 

situation that raises questions about the 

representativeness of the physicians and their 

patients. However, the registry approach has the 

potential to be very useful for studying products 

that are used according to their labeled indications; 

it also allows for effective surveillance of products 

that are used off label but by the same practitioners 

who would use it for the labeled indication. For 

example, a product might be approved for people 

with moderate to severe asthma and used off label 

in patients with mild asthma, yet the prescribing 

medical providers would already be included in the 

registry and could easily provide information 

about all their product use. Off-label use is much 

more difficult to study when a medical product is 

used by a wide variety of medical care providers; 

for example, drugs that promote wakefulness or 

are thought to increase a patient’s ability to 

concentrate, acting as immunomodulators. The 

legal, regulatory, and ethical aspects of registry 

sponsors also affect whether they are required to 

report any AEs that may be observed, since only 

those legal entities that market (or distribute) a 

medical product are required to report AEs. For all 

other parties, such reporting is ethical and 

desirable, but not enforceable or required.

The purpose of this chapter is to examine the role 

of registries as one of the available tools for 

enhanced understanding of product safety through 

AE detection and evaluation. The examination will 

include both the role of registries created 

specifically for the purposes of safety assessment 

and of those in which the collection of safety data 

is ancillary to the registry’s primary objectives. 

The legal obligations of regulated industries are 

discussed by others and are only mentioned briefly 

here. Similarly, issues to consider in the design and 

analysis of registries are covered in Chapters 3 and 

13, respectively. Chapter 12 discusses practical and 

operational issues with reporting AE data from 

registries. The potential ethical obligations, 

technical limitations, and resource constraints that 

face registries with multiple different purposes in 

considering their role in AE detection and 

reporting are also discussed in this chapter. Case 

Examples 44 and 45 offer descriptions of how 

some registries have provided data for product 

safety assessments.
94

Section V. Special Applications in Patient Registries

2. Registries Specifically 

Designed for Safety 

Assessment

Disease and product registries that systematically 

collect data on all eligible patients are a 

tremendous resource for capturing important 

information on safety. Registries commonly enroll 

patients who are not just different from but more 

complicated than those included in clinical trials, 

in terms of the complexity of their underlying 

disease, their comorbidities, and their concomitant 

medications.

2.1 Design Considerations: Disease 

Registries Versus Product Registries

Product registries, by definition, focus on patients 

treated with a particular medical product. To be 

useful, the registry should record specific 

information about the products of interest, 

including route of administration, dose, duration of 

use, start and stop date, and, ideally, information 

about whether a generic or branded product was 

used (and which brand) and/or specific information 

about the product. Biologic medicines and devices 

have their own challenges, ideally requiring 

information about device identifiers, production 

lots, and batches. Disease registries include 

information not only on products or procedures of 

interest, but also on similar patients who receive 

other treatments, other procedures, or no treatment 

for the same clinical indications. By characterizing 

events in the broad population with conditions of 

interest, disease registries can make a meaningful 

contribution to the understanding of AE rates by 

providing large, systematic data collection for 

target populations of interest. Their generally 

broad enrollment criteria allow systematic capture 

on a diverse group of patients, and, provided that 

they collect information about the potential events 

of interest, they can be used to provide a 

background rate of the occurrence of these events 

in the affected population in the absence of a 

particular treatment, or in association with relevant 

treatment modalities for comparison. The utility of 

this information, of course, depends on these 

registries’ capturing relatively specific and clear 

information about the events of interest among 

“typical” patients, and the ability of readers and 

reviewers to gauge how well the registries cover 

information about the target population of interest. 

Generating this kind of real-world data as part of 

disease registries can be informative either for the 

design of subsequent product registries (e.g., to 

establish appropriate study size estimations) or for 

the incorporation of new treatments into the data 

collection as they become available, since the data 

can provide useful benchmarks against which to 

assess the importance of any signals. Some would 

argue that disease registries, rather than specific 

product registries, are more likely to be successful 

in systematically collecting interpretable long-term 

safety data, thereby allowing legitimate 

comparisons, to the extent possible, across types 

and generations of drugs, devices, and other 

interventions.13

Consideration should be given during the registry 

design phase to inclusion/exclusion criteria, 

appropriate comparator groups, definitions of the 

exposure and relevant risk window(s), and analysis 

planning (see Chapter 3). Registries involving 

products new to the market must be cognizant of 

selection bias, channeling bias, and unmeasured 

confounding by indication. Channeling bias occurs 

when patients prescribed the new product are not 

comparable to the general disease population. For 

example, channeling bias occurs when sicker 

patients receive new treatments because they are 

nonresponsive to existing treatments; conversely, 

patients who are doing well on existing treatments 

are unlikely to be switched to new treatments. 

Unmeasured confounding can also be introduced 

by frailty; for example, vaccine effectiveness 

studies can be misleading if only healthy people 

get vaccinated.

In some countries, cost constraints imposed by 

reimbursement status (whether dictated by 

government agencies or private insurance) mean 

that new therapies are restricted to narrower 

populations than indicated by the approved 

indication. For new devices or procedures, 

provider learning curves and experience are 

additional factors that must be considered in 

analysis planning. Since bias is inherent in 

observational research, the key is to recognize and 

control it to the extent possible. In some cases, the 
95

Chapter 19. Use of Registries in Product Safety Assessment

potential for bias may be reduced through inclusion/

exclusion criteria or other design considerations 

(e.g., enrollment logs) (see Chapter 3). In other 

cases, additional data may be collected and analytic 

techniques used to help assess bias (see Chapter 

13). Any recognized potential for bias should be 

discussed in any publications resulting from the 

registry.

In some settings, registries are used to collect 

specific AEs or events of interest. Once the types 

of AEs and/or other special events of interest have 

been identified, the registry must be designed to 

collect the data efficiently. Without adequate 

training of clinical site staff to recognize and 

report events of interest, the registry will be 

reduced to haphazard and inconsistent reporting of 

AEs.

Upon registry inception, clinicians or other health 

care professionals who may encounter patients 

participating in the registry should be educated 

about what AEs or other special events of interest 

should be noted, and how and within what 

parameters (e.g., time) they should report 

untoward events that may occur while they are 

participating in the registry. They should also be 

reminded about the need to follow up on events 

that may not obviously be of immediate interest. 

For example, if a clinician asked a patient how he 

was feeling and the patient replied that he just 

returned from the hospital, it would be incumbent 

on the clinician to obtain additional information to 

determine whether this hospitalization might be a 

reportable event, regardless of whether the patient 

may have recognized it as such. This is particularly 

important in registries designed to capture all 

suspected adverse reactions as opposed to specific 

AEs. Such an active role by participants as well as 

their treating clinicians can contribute to a robust 

safety database. In addition to identifying events 

known to be of interest, the systematic collection 

of followup data can also capture information 

regarding risks not previously identified, risks 

associated with particular subgroups (e.g., 

pediatric or geriatric patients, patients with liver 

impairment, fast or slow metabolizers), or 

differences in event severity or frequency not 

appreciated during clinical development.

Consideration should also be given to 

implementation of routine followup of all registry 

patients for key AEs, as well as vital status and 

patient contact and enrollment information at 

prespecified visits or intervals, to ensure that 

analyses of the occurrence of AEs among the 

registry population are not hampered by extensive 

missing data. Otherwise, the possibility that 

patients “lost to followup” may differ from those 

with repeat visits, with regard to risk of AEs, 

cannot be excluded.

It is also important to keep in mind that it may be 

necessary to revisit the registry design if it 

becomes apparent that the initial plan will not 

meet expectations. For example, the original 

criteria for defining the target population (patients 

and/or health care providers) may not yield enough 

patients, such as when a treatment of interest is 

only slowly coming into use for the intended 

population.

2.1.1 Health-Care Provider– and PatientReported Outcomes

Registries and other prospective data collection 

approaches have the advantage of incorporating 

both health care provider– and patient-reported 

data. Although patients and their advocates may 

spontaneously report postmarketing AEs to 

manufacturers (e.g., via inquiries directed to 

medical information departments) and directly to 

regulatory bodies, this is relatively uncommon. 

Furthermore, spontaneous reports received directly 

from patients that lack health care provider 

confirmation may fall outside of standard 

aggregating processes by regulatory bodies. In 

Europe, there are schemes in some countries to 

encourage patients to report directly to regulatory 

authorities; throughout Europe, manufacturers 

have an obligation to follow up patient reports with 

their health care provider. However, significant 

events that are not clinically recognized may be 

substantially underreported.

In addition, registries may collect health care 

provider–level data, such as training level, number 

of patients seen annually, and practice type and 

locations, that may contribute to an understanding 

of differences in event rates and reporting. This, 

along with the patient-reported data not routinely 
96

Section V. Special Applications in Patient Registries

or consistently captured in the medical record 

(such as concomitant environmental and lifestyle 

exposures and adherence to prescribed regimes), 

differentiates registries from other electronic data 

sources, and in many cases allows for improved 

assessment of confounding and ability to assess 

the potential of a signal internally, prior to further 

signal evaluation or action.

2.1.2 Effects Observed in a Larger Population 

Over Time

Registries, including those used to follow former 

clinical trial participants, are well suited to the 

identification of effects that can only be observed 

in a large and diverse population over an extended 

period of time. They make it possible to follow 

patients longitudinally, and thereby identify 

long-term device failures or consequences or 

delayed drug safety issues or benefits; for example, 

failures of orthopedic implants increasingly placed 

in more active, younger patients. Similarly, such 

long-term followup facilitates evaluation of 

drug-drug interactions (including interactions with 

new drugs as they come to market and are used) 

and differences in drug metabolism related to 

genetic and other patient characteristics.

One of the most consistent risk factors for AEs is 

the total number of medications taken by a 

patient.14 Polypharmacy is commonplace, 

especially in the elderly, and health care providers 

are often unaware of over-the-counter, herbal, and 

other complementary (alternative) medications 

their patients take. Registries that collect data 

directly from patients can seek information about 

use of these products. In the case of registries used 

solely by health care practitioners, data collection 

forms can be designed specifically to request that 

patients be asked about such use.

When designing a registry for safety, the size of 

the registry, the enrolled population, and the 

duration of followup are all critical to ensure 

applicability of the inferences made from the data. 

If the background rate of the AE in the population 

of interest is not established and the time period 

for induction is not well understood, it is extremely 

difficult to determine an exact meaningful target 

size or observation period for the registry, and the 

registry may be too small and have too brief an 

observation period to detect any, or enough, events 

of interest to provide a meaningful estimate of the 

true AE rate. In addition, the broad inclusion 

criteria typical of registries make it likely that 

subgroups of exposed patients may be identified 

and analyzed separately. Such stratified analyses 

may require larger sample sizes to achieve rate 

estimations with confidence intervals narrow 

enough to allow meaningful interpretation within 

strata.

As is also true for clinical trials, which often do 

not have a sufficient sample size for safety, but 

rather, for efficacy endpoints, describing safety 

outcomes from observational studies in statistical 

terms is not always straightforward. Postmarketing 

data may or may not confirm event rate estimates 

seen in clinical trials, and may also identify events 

not previously observed. During clinical 

development, risk of events not yet seen but 

possibly associated with a product class or the 

product’s mechanism of action is often identified 

as part of ongoing risk assessment, and these 

events usually continue to be events of interest 

after approval. An inferential challenge arises 

when such an event is never observed. The “rule of 

three” is often cited as a means of interpreting the 

significance of the fact that a specific event is not

being observed in a finite population (i.e., that the 

numerator of its rate of occurrence is zero). Using 

asymptotic risk estimation, the rule posits that in a 

large enough study (i.e., >30 patients), if no event 

occurs, and if the study were repeated over and 

over again, there can be 95-percent confidence that 

the event (or events) would not actually occur 

more often than one in n/3 people, where n is the 

number of people studied.15 The rule, originally 

described by Hanley and Lippman-Hand in 1983, 

is probably summarized best as a means for 

“estimating the worst case that is compatible with 

the observed data.”16 For the purposes of registries, 

this rule must be carefully applied, since it 

assumes that reporting of all events occurring in 

the study population is complete and that the study 

population is an accurate representation of the 

intended population. Nonetheless, this rule of 

thumb provides some guidance regarding registry 

size and interpretation of results.
97

Chapter 19. Use of Registries in Product Safety Assessment

2.1.3 Challenges

In planning a registry for safety, it is essential to 

consider how patients will be identified and 

recruited in order to understand which types of 

patients will be included, and equally, if not more 

importantly, what types of patients will likely not

be included in the registry. For example, safety 

registries often seek information about all treated 

patients, regardless of whether the product is 

prescribed for an approved indication. While it is 

conceptually straightforward to design a registry 

that would include information on all product 

users, practical challenges include the difficulty of 

raising awareness about the existence of the 

registry, the desirability and importance of 

collecting information on all treated patients, and 

the challenge of specifying the AEs and other 

events of interest without causing undue concern 

about product safety.

Drawing attention to the registry among health 

care providers who use the treatments off label is 

especially challenging, due to competing concerns 

about being inclusive enough to capture all use 

(on-label or not) versus the need, especially if the 

sponsor of the registry is also a manufacturer, to 

avoid the appearance of promoting off-label use 

when contacting physicians in specialties known to 

use the product off label. In addition, diseases 

targeted for off-label use may be markedly 

different from indicated uses and may pose 

different safety issues. In Europe, when there is 

limited knowledge about the safety of a product 

prior to its authorization and when a registry is 

part of a risk management plan, manufacturers 

may be required, prior to launch of the product, to 

notify all physicians who may possibly prescribe 

the product about the existence of a registry 

(sometimes also called in this context a 

postauthorization safety study or PASS), including 

details of how to register patients.

It is more challenging to evaluate the utility of a 

registry when the entire population at risk has not 

been included; however, this situation merits 

careful consideration, since it is far more common 

than one where a registry captures every single 

treated patient. Registries organized for research 

purposes are typically voluntary by design, a 

situation that does not promote full inclusiveness. 

Two key questions concern the target population 

(in terms of representativeness and the potential to 

generalize the results) and the size of the registry. 

When considering the target population, it is 

important to assess (1) whether the patients in the 

registry are representative of typical patients, and 

(2) what types of patients may be systematically 

excluded or not enrolled in the registry. For 

example, do patients come from a diverse array of 

health care settings or are they recruited only from 

tertiary referral hospitals? In the latter case the 

patients can be expected to be more complicated 

or have more advanced disease than other patients 

with a similar diagnosis. Are there competing 

activities in the target population, such as large 

registration trials or other observational studies, 

that may skew participation of sites or patients? 

Are patients in late stages of the disease or with 

greater disease severity more likely to participate? 

(See Chapters 3 and 13 for more information on 

representativeness.)

The ability to use registries for quantification of 

risk is highly dependent on understanding the 

relationship between the enrolled population and 

the target population. While it is intellectually 

appealing to dismiss the value of any registry that 

does not have complete enrollment of all treated 

patients or a documented approach to sampling the 

entire population, registries that can demonstrate 

that the actual population (the population enrolled) 

is representative of the target population through 

other means (e.g., by comparison to external data 

sources) can nevertheless be tremendously 

informative and may be the only feasible way that 

data can be collected.

Consider, for example, the National Registry for 

Myocardial Infarction (NRMI), one of the first 

cardiac care registries.17 NRMI was originally 

intended to obtain information about time to 

treatment for patients presenting with myocardial 

infarction to acute care hospitals. The program 

ultimately resulted in 70 publications (out of more 

than 500) that provided detailed information on 

both specific AEs for specific products and 

comparative information on safety events. 

Although this registry was quite large in terms of 

hospitals and patients, it included neither all MI 

patients nor all patients using the product for 
98

Section V. Special Applications in Patient Registries

which it described safety information. It was 

nevertheless considered to be broadly 

representative of typical MI patients who presented 

for medical care.

2.1.4 Defining Exposure and Risk Windows

Patients will enter a registry at various stages in 

the course of their disease or its medical 

management. Therefore, it is essential to collect 

information on the timing of events in relation to 

the initial diagnosis and in relation to the timing of 

treatments. It is simplest to collect prespecified 

clinical data recorded on standardized forms at 

scheduled assessments, a practice that leads to 

uniformity within the analysis. However, many 

registry patients present themselves for data 

collection on a more naturalistic schedule (i.e., 

data are collected whenever the patient returns for 

followup care, whether or not the visit corresponds 

to a prespecified data collection schedule). The 

more haphazard schedule is more reflective of 

“real-world” settings, yet results in nonuniform 

data collection for all subjects.

Rather than being discarded, these nonuniform 

data can be analyzed both by categorizing patient 

visits in terms of time windows of treatment 

duration (e.g., considering data from all visits 

occurring within 30 days of first treatment, then 

within 90 days, 180 days, etc.), and also by using 

time in terms of patient days/years of treatment. 

This type of analysis facilitates characterization of 

the type and rate of occurrence for various AEs in 

terms of their induction period and patient time at 

risk. When the collection of AE data is completed 

through an ongoing active process and is expected 

to be continued over the long term, periodic 

analysis and reporting should be structured around 

specified time points (e.g., annually, semiannually, 

or quarterly) and may align with the periodic 

safety update reports. The rigor of prespecified 

reporting schedules requires periodic assessment 

of safety and can support systematic identification 

of delayed effects.

In addition to variability in the timing of followup, 

consideration must be given to other recognized 

aspects of product use in the real world; for 

example, switching of therapies during followup, 

use of multiple products in combination or in 

sequence, dose effects, delayed effects, and 

failures of patient compliance. The current realworld practices for the treatment of many 

conditions, such as chronic pain and many 

autoimmune diseases, include either agent rotation 

schemes or frequent switching until a balance 

between effectiveness and tolerability is reached—

practices that make it difficult to determine 

exposure-outcome relationships. Switching 

between biologics may lead to problems with 

immunogenicity because even products that are 

clinically the same, as in the case of the 

erythropoietins, will have different immunogenic 

potential due to differences in manufacturing 

processes and starting cell lines. In addition, as 

with many clinical studies, patient adherence to 

treatment—or lack thereof—during registry 

followup is an important potential confounder to 

consider. Over time, patients may take drug 

holidays and self-adjust dosages, and these actions 

should be, but are not always, captured via the data 

collected in the registry, especially if the interval 

between followup time points is long or the action 

is not known by the treating physician. Assessing 

the temporality of unanticipated events may then 

be hampered by the inability to fully characterize 

exposure.

Delayed effects may include late onset 

immunogenicity, the development of subclinical 

effects associated with chronic use that are not 

appreciated until years later, and effects that 

develop after stopping treatment, related to 

products with a long half-life or extended retention 

in the body. An example of this can be seen in the 

case of bisphosphonates used for bone resorption 

inhibition in the treatment of osteoporosis, where 

the product is retained in the bone for at least 10 

years after stopping therapy, and there is some 

evidence that long-term bone turnover suppression 

puts patients at increased risk of osteonecrosis and 

nonspinal fractures.18 In addition, many biologics 

aimed at immunomodulation carry an increased 

risk of future malignancy that is not fully 

appreciated, as do novel therapies directed at 

angiogenesis. Although registries are well suited to 

long-term followup, consideration must be given 

to how long is long enough to appreciate these 

effects.
99

Chapter 19. Use of Registries in Product Safety Assessment

Noncompliance can have a substantial effect on 

the assessment of AEs, particularly if dose or 

cumulative dose effects are suspected. Patient 

compliance may be affected by expense, 

complexity of dosing schedule, convenience/mode 

of administration, and misunderstanding of 

appropriate administration, and is not fully 

ascertained by data sources that capture 

prescriptions rather than actual product use. With 

products used to treat chronic diseases it is 

possible to estimate compliance via electronic 

health records, by first estimating when repeat 

prescriptions should be issued, and then measuring 

the observed versus expected frequency. Although 

registries may be directly designed to track 

compliance through patient diaries and other 

methods of direct reporting, capturing compliance 

accurately and minimizing recall bias remain 

challenges.

2.1.5 Special Conditions: Pregnancy Registries

The use of specially designed registries for specific 

safety monitoring has a long history. For example, 

pregnancy registries are commonly used to 

monitor the outcomes of pregnancies during which 

the mother or father was exposed to certain 

medical products. The Antiretroviral Pregnancy 

Registry is an example of a registry that collects 

information on a broad class of products to 

determine the risk of teratogenesis.19 (See Case 

Examples 49 through 52.) Pregnancy registries 

provide in-depth information about the safety of 

one or more products and are particularly useful 

since, unless the product is used for lifethreatening diseases or to treat a pregnancy-related 

illness, pregnant women are generally excluded 

from clinical investigations used for product 

approval. Registries and other observational 

studies, by virtue of being sustainable over longer 

periods of time and more amenable to small 

site-to-patient ratios than registration trials, can 

facilitate the active surveillance of safety in these 

populations. In addition, using computerized 

claims or billing data for pregnancy safety 

monitoring is hampered by the fact that patients 

often do not present early in pregnancy, by a lack 

of relevant data on other exposures (since these are 

often unrelated to reimbursement), and by 

difficulty linking maternal and infant records. 

Therefore, direct prospective data collection 

remains the best source of meaningful safety data 

related to pregnancy. A challenge for pregnancy 

registries is to identify and recruit women early 

enough in pregnancy to obtain reliable information 

on treatments used during the first trimester, which 

is a critical time for organogenesis, and to obtain 

information about early pregnancy loss, since this 

information is not always volunteered by women. 

It is also important to obtain information on 

treatments and other putative exposures before the 

outcome of the pregnancy is known, to avoid 

selective recall of exposures by women 

experiencing bad pregnancy outcomes. For more 

information, see Chapter 21.

2.1.6 Special Conditions: Orphan Drugs

A product may be designated an orphan drug (or 

biologic, or medicine in the E.U.) if it fulfills 

certain conditions, which include being used for 

the diagnosis, prevention, or treatment of lifethreatening or chronically debilitating conditions 

affecting a small number of patients. Often these 

diseases are extremely rare, and dossiers submitted 

for authorization purposes may have only tens of 

patients included in clinical trials. Obviously, the 

safety profile of such products is extremely 

limited, and followup of patients treated with the 

products after authorization is likely to be a 

requirement.

With some orphan drugs, the disease may have 

been usually fatal before therapy was available. 

Determining the safety profile of these products is 

especially difficult, in that the natural history of 

the disease when treated is not known, and trying 

to disentangle the effects of the product from those 

of the ongoing disease may be particularly 

problematic. In many of these diseases, the 

problem may be due to faulty enzymes in 

metabolic pathways, leading to accumulation of 

toxic substrates that cause the known 

manifestations of the disease. Treatment may 

involve blocking another enzyme or pathway, 

leading to the accumulation of different substances 

for which the effects may also not be known but 

are less immediately toxic. In this situation, with a 

fatal disease and a first product with proven 

efficacy, it would not be ethical to randomize 

patients in a trial versus placebo for an extended 
100

Section V. Special Applications in Patient Registries

period of time, and so a registry may be the only 

effective means of obtaining long-term safety data. 

Registries in these situations may make 

meaningful contributions to understanding the 

natural history of the disease and the long-term 

effects of treatment, sometimes largely by virtue of 

the fact that most patients can be included and 

long-term followup obtained for orphan products. 

For more information, see Chapter 20.

2.1.7 Special Conditions: Controlled Distribution/

Performance-Linked Access Systems 

Registries in the United States may also be part of 

risk evaluation and mitigation strategies , such as 

restricted distribution systems, referred to as 

performance-linked access systems (PLAS), which 

may be used to monitor the safety of marketed 

products as one of the Elements to Assure Safe 

Use.20 One of the earliest PLAS was a bloodmonitoring program for clozapine implemented in 

1990 to prevent agranulocytosis; the program 

allowed clozapine to be dispensed only if an 

acceptable blood test had been submitted.21 Other 

examples include the STEPS program for 

thalidomide (System for Thalidomide Education 

and Prescribing Safety), implemented in 1998 to 

prevent fetal exposure;22 the TOUCH controlled 

distribution for natalizumab (Tysabri) for patients 

with multiple sclerosis, to detect the occurrence of 

progressive multifocal leukoencephalopathy 

(PML);23 and the iPLEDGE system implemented 

for isotretinoin in 2006, which tightly links the 

dispensing of isotretinoin for female patients of 

childbearing potential to documentation of a 

negative pregnancy test, to prescriber confirmation 

that contraceptive counseling has occurred, and to 

prescriber and patient identification of 

contraceptive methods chosen.24

In many of these programs, access to the product 

is linked directly to participation in the related 

registry. Therefore, all patients treated with the 

product should be in the related registry because 

they cannot otherwise obtain access to it. The 

related registry is looking for a known AE (for 

example, PML) and collects data specifically 

related to that AE. The registry also collects 

information on other factors that may raise a 

patient’s individual risk for this AE, information 

that helps provide important clinical context that 

would not otherwise be available in a systematic 

fashion on a large population of treated patients.25

While PLAS registries are driven by safety 

concerns, they are primarily focused on 

prescribing or dispensing controls rather than 

signal detection. As a result, they use very limited 

data collection forms to minimize burden, and this 

can limit their utility for certain types of analyses.

In Europe, use of registries for risk minimization 

activities can be more problematic, due to 

differences in national legislation and the 

enactment of the European Union data protection 

directive. In some countries it is possible to 

mandate registration of patients in relation to 

particular products (e.g., clozapine in the United 

Kingdom and Ireland), but in others other methods 

must be found. For these reasons, registries are 

more frequently used on a voluntary basis to 

monitor safety and capture AEs, while risk 

minimization is achieved by controlled distribution 

with compulsory distribution of educational 

material, prescribing algorithms, and treatment 

initiation forms to anyone likely to prescribe the 

product. Despite the fact that patient registration is 

voluntary, high enrollment rates can be achieved, 

particularly when clinicians recognize that 

information on the safety profile of the product is 

limited.26 Obviously, if a product has a high 

potential for off-label use, patients enrolled in a 

registry may not be generalizable to all those 

treated with the product, but this can be factored 

into data analysis and interpretation. A voluntary 

registry coupled with controlled distribution may, 

in fact, be reasonably representative, since offlabel use may be severely limited by difficulties 

obtaining the product.

2.1.8 Special Conditions: Medical Devices

Medical devices pose different analytic and data 

challenges from drugs. On the one hand, it is much 

more straightforward to identify when a device is 

implanted and explanted if those records can be 

obtained; however, since not all medical devices 

are covered by medical insurance, it can be more 

difficult to identify all the appropriate practitioners 

and locate all the records. Medical devices that can 

be attached and detached by the consumer, such as 

hearing aids, are very difficult to study in that, 
101

Chapter 19. Use of Registries in Product Safety Assessment

much like products used on an as-needed basis, 

special procedures are required to document their 

use; these procedures are costly and intrusive, and 

are therefore rarely used. Additionally, the lack of 

unique device identifiers has posed a challenge for 

safety surveillance of devices. Recently, the FDA 

proposed a new system that will establish a unique 

identifier for most medical devices, with the goals 

of supporting more accurate reporting and 

analyzing of medical device AE reports (see 

Chapter 23).

Despite these challenges, the safety of medical 

devices is very important due to their widespread 

use; of particular concern are long-term indwelling 

devices, for which recall in the event of a 

malfunctioning product is inherently complicated. 

For example, in the late 1970s/early 1980s, when a 

particular type of Björk-Shiley prosthetic heart 

valve was found to be defective and prone to 

fracture, leading to sudden cardiac death in the 

majority of cases, detailed studies of explanted 

devices, patient factors, and manufacturing 

procedures led to important information that was 

used to guide decisionmaking about which devices 

should be explanted.27, 28 Identification of the 

characteristics of valves at high risk of failure was 

very important due to the perioperative mortality 

risk from explanting a heart valve regardless of its 

potential to fail. This same logic applies to many 

other medical devices that are implanted and 

intended for long-term use. Some of the challenges 

relating to studying medical devices have to do 

with being able to characterize and evaluate the 

skill of the “operator,” or the medical professional 

who inserts or implants the device. These operator 

characteristics may be as, or more, important in 

terms of understanding risk than the characteristics 

of the medical devices themselves.29 For more 

information, see Chapter 23.

3. Registries Designed for 

Purposes Other Than Safety

Registries may be designed to fulfill any number 

of other purposes, including examining 

comparative effectiveness, studying the natural 

history of a disease, providing evidence in support 

or national coverage decisions, or documenting 

quality improvement efforts. Although these 

registries may gather data on AEs and report those 

data (to regulatory authorities, manufacturers or 

others), not all data may necessarily be reported 

through the registry. Thus, the registry may not 

record all events, which would result in an 

imprecise, and possibly inaccurate, estimation of 

the true risk in the exposed population(s). A 

strength of comparative effectiveness registries, 

however, lies in the systematic collection of data 

for both the product of interest and concomitant, 

internal controls.

As an example of the limitations of assessing 

safety events in registries not designed for safety, a 

registry may be sponsored by a payer to collect 

data on every person receiving a certain 

medication. The purpose of the registry may be to 

assess prescribing practices and determine which 

patients are most likely to receive this product. The 

registry may also contain useful data on events 

experienced by patients exposed to the product, 

but may not be considered a comprehensive 

collection of safety data, or may provide 

information regarding a known risk or outcome 

rather than generating data that could identify a 

previously unappreciated event. Alternatively, a 

registry may be designed to study the effectiveness 

of a new product among a population subset, such 

as the elderly. The registry may be powered to 

analyze certain outcomes, such as 

rehospitalizations for a condition or quality of life, 

but may not be specifically of sufficient sample 

size to reliably assess overall safety in this 

population.

It is more challenging to accurately and precisely 

detect AEs of interest when a registry has not been 

designed for a specific safety purpose. In this 

situation, the registry must collect a wide range of 

data from patients to try to catch any possible 

events, or be adapted later should safety become a 

primary objective. Some events may be missed 

because the registry did not anticipate them and 

did not solicit data to identify them. Also, much 

the same as for registries designed specifically to 

detect AEs, some events may be so rare that they 

do not occur in the population enrolled in the 

registry or do not occur during the registry 

followup period. In these circumstances, registries 
102

Section V. Special Applications in Patient Registries

can be designed to provide useful data on some of 

the events that may occur in the exposed 

population. Such data should not be considered 

complete or reliable for determining event rates, 

but, when the data are combined with safety data 

from other sources, trends or signals may become 

apparent within the data set.

4. Ad Hoc Data Pooling

One way to capitalize on data that, because they 

were collected for another purpose, may be 

insufficient for meaningful standalone analysis and 

interpretation due to study size or lack of 

comparators, is to pool the data with other similar 

data. As with any pooling of disparate data, the use 

of appropriate statistical techniques and the 

creation of a core data set for analysis are critically 

important, and are highly dependent on 

consistency in coding of treatments and events and 

in case identification.

It is essential to have an understanding of how 

every data set that will be used in a pooled analysis 

was created. For example, what is recorded in 

administrative health insurance claims depends 

largely on what benefits are covered and how 

medications are dispensed. Noncovered items 

generally are not recorded. For example, mental 

health services are often contracted for under 

separate coverage (so-called “carve-outs”) and not 

covered under traditional health insurance 

coverage; thus, the mental health consultations are 

not likely to be included in administrative 

databases derived from billing claims data. Also, 

some injectable medications (e.g., certain 

antibiotics) may be administered in the physician’s 

office and thus would not be recorded through 

commonly used pharmacy reporting systems that 

are based on filling and refilling prescriptions. The 

absence of information may lead to false 

conclusions about safety issues. Also, AE data 

coded using the same coding dictionary (e.g., 

Medical Dictionary for Regulatory Activities, or 

MedDRA) may still be plagued by inconsistency 

in the application of coding guidelines and 

standards. Recoding of verbatim event reports may 

be required, if feasible, prior to analysis. 

Depending upon the purpose for which the data 

were collected, data on the treatments of interest 

are not always recorded, or are not recorded with 

the specificity needed to understand risk (e.g., 

branded vs. generic, dosage, route of 

administration, batch).

Another consideration is differential followup, 

including the duration and vigor of followup in the 

registries to be pooled. Particular care is needed 

when combining data sets from different European 

countries, since differences in medical practice and 

reimbursement may mean that superficially similar 

data may actually represent different subgroups of 

an overall disease population. Similar caution is 

also advisable when combining information from 

disparate health systems within a single country, as 

some treatments of interest may be noncovered 

benefits in some systems and consequently not 

recorded in that health system’s records. An 

alternative to pooling data is to conduct metaanalyses of various studies using appropriate 

statistical and epidemiologic methods.

While the types of registries described above may 

not be individually of sufficient sample size to 

detect safety issues, combining data from registries 

for other purposes could significantly enhance the 

ability to identify and analyze safety signals across 

broader populations. Core data sets for AEs have 

been suggested for electronic health records 

systems and as part of national surveillance 

mechanisms (e.g., through distributed research 

networks). In such a network, each participating 

registry or data source collects a standardized core 

data set from which results can be aggregated to 

address specific surveillance questions. For 

example, there is significant national interest in 

understanding the long-term outcomes of 

orthopedic joint implants. Currently, there are 

several prominent registries in the United States 

with varying numbers of types of patients and 

types of implants. Many of these registries collect 

data for quality improvement purposes, but have 

sufficient data elements to potentially report on 

AEs. However, only by aggregating common data 

sets across many of these registries can a broadly 

representative population be evaluated and enough 

data accrued to understand the safety profile of 

specific types of devices in particular populations.
103

Chapter 19. Use of Registries in Product Safety Assessment

As described above, while not every registry is 

designed to evaluate safety, even registries 

designed for other purposes might contribute to 

aggregate information about potential harm from 

health care products or services. Yet many 

registries, especially disease registries, are 

conducted by nonregulated entities such as 

provider associations, academic institutions, and 

nonprofit research groups, whose role in AE 

reporting is unclear. Furthermore, sample sizes 

needed to understand safety signals are generally 

much larger than those needed to achieve useful 

information on quality of care or the natural 

history of certain diseases, and the safety analyses 

can require a high degree of statistical 

sophistication. Enrolling additional patients or 

committing additional resources for specialized 

analyses in order to achieve a general societal 

benefit through safety reporting is not feasible for 

most registries when the primary purpose is not 

safety. However, encouraging registries to 

participate in aggregation of data when such 

participation is at minimal cost and enhances the 

common good may be both reasonable and 

appropriate.

Many efforts are underway to improve the 

feasibility of broader safety reporting from both 

registries and electronic health records that serve 

other purposes. These efforts include 

recommending standardized core data sets for 

safety to enhance the aggregation of information 

in distributed networks, and making registries 

interoperable with facilitated safety reporting 

mechanisms or other registries designed for 

safety.30 As facilitated reporting methodologies 

become more common and easier for registries to 

implement, there will be fewer reasons for 

nonparticipation. In addition, linkage of 

population-based registries, such as the 

Surveillance, Epidemiology and End Results 

(SEER) cancer registry program, with other data 

sources, such as Medicare, have proven invaluable 

for evaluating safety and other outcomes.

5. Signal Detection in 

Registries and Observational 

Studies

Although subject to debate, according to the World 

Health Organization definition, a safety signal is 

defined as “reported information on a possible 

causal relationship between an AE and a drug, the 

relationship being unknown or incompletely 

documented previously.”31 Hauben and Aronson 

(2009) define a signal as “information that arises 

from one or multiple sources (including 

observations and experiments), which suggests a 

new potentially causal association, or a new aspect 

of a known association, between an intervention 

and an event or set of related events, either adverse 

or beneficial, which would command regulatory, 

societal or clinical attention, and is judged to be of 

sufficient likelihood to justify verificatory and, 

when necessary, remedial actions.”32 The authors 

further posit that signals, following assessment, 

could subsequently be categorized as 

indeterminate, verified, or refuted.

Additional attempts at defining or describing a 

safety signal for purposes of guiding product 

sponsors, regulators, and other researchers have 

come from various sources, including the Council 

for International Organizations of Medical 

Sciences (CIOMS), the FDA, and the United 

Kingdom’s Medical and Healthcare products 

Regulatory Agency. Nelson and colleagues 

recently provided FDA with a comprehensive 

evaluation of signal detection methods for use in 

postmarketing surveillance, and included a 

discussion of “conventional Phase IV 

observational safety studies,” which would 

encompass registries, as part of a multipronged 

approach to surveillance.33 They noted that despite 

a focus on automated health care data sources, 

such as large health care claims databases, for 

primary surveillance and as the basis for FDA’s 

Sentinel Network, the need for more detailed data 

regarding exposure and outcome measurement, as 

well as collection of relevant confounder data, will 

require that prospective observation studies be 

conducted to address prespecified safety-related 

hypotheses.
104

Section V. Special Applications in Patient Registries

Establishing a threshold of effect size and 

robustness of data that would justify action, such 

as initiation of additional studies, FDA action, or 

changes in payer coverage, remains an important 

question and is unlikely to be uniformly applicable 

to all products and situations. A report was issued 

in 2010 from the CIOMS Working Group VIII, 

whose main goal is to harmonize the development, 

application, and interpretation of signal detection 

methods for use with drugs, vaccines, and 

biologics and to provide practical advice.34

Once a signal that warrants further evaluation is 

identified, it is typically assessed based on the 

strength of the association between exposure and 

the event; biological plausibility; any evidence 

provided by dechallenge and rechallenge; the 

existence of experimental or animal models; and 

the nature, consistency, and quality of the data 

source.35 Signals may present themselves as 

idiosyncratic events affecting a subset of the 

exposed population who are somehow susceptible, 

events related to the pharmacological action of the 

drug, or increased frequency of events normally 

occurring in the population (such as in the 

example of cardiovascular events and rofecoxib). 

Signals may involve the identification of novel 

risks, or new (or more refined) information 

regarding previously identified risks. If an event 

does appear to be product related, further inquiry 

is required to examine whether the occurrence 

appears to be related to a specific treatment, a 

combination or sequence of treatments, or a 

particular dosage and/or duration of use. Events 

with long induction periods are particularly 

challenging for the ascription of a causal 

relationship, since there are likely to be many 

intervening factors, or confounders, that could 

account for the apparent signal.

The constant challenge is to separate a potential 

safety signal from the “noise,” or, in other words, 

to detect meaningful trends and to have a basis for 

evaluating whether the signal is something 

common to people who have the underlying 

condition for which treatment is being 

administered, or whether it appears to be causally 

related to use of a particular product. All methods 

currently used for signal detection have their 

limitations. Attempts to use quantitative, and in 

some cases, automated signal detection methods as 

part of pharmacovigilance, including data mining 

using Bayesian algorithms or other 

disproportionality analyses, are hampered by 

confounders and other biases inherent to 

spontaneously reported data.36, 37 Other 

methodologies also attempt to identify trends over 

time and include potential patterns associated with 

other patient characteristics, such as concomitant 

drug exposures.

These methods of automated signal detection lack 

clinical context and only draw attention to 

deviations from independence between product 

exposure and events. No conclusions regarding 

causality can be drawn without a further 

qualitative and quantitative assessment of extrinsic 

factors (e.g., an artificial spike in reporting due to 

media attention) and potential confounders; in 

some cases, even with quantitative and qualitative 

assessments, the data may be insufficient to 

establish causality. Depending on the original data 

source, it may be impossible to address these 

issues within the database itself and either 

abstracted medical record data or prospective data 

collection may be required to gather reliable data. 

The long-term followup and longitudinal data 

generated by many registries merits particular 

methodological considerations, including how 

often to perform testing, what threshold is 

meaningful for a given event, and whether that 

threshold changes over time.

While some registries can serve as sources of 

initial safety signaling or hypothesis generation, 

they may also be used for further investigation of a 

signal generated from surveillance and quantitative 

analysis. As an example, existing data from the 

Swedish Coronary Angiography and Angioplasty 

Registry (SCAAR), sponsored by the Swedish 

Health Authorities, was used to look at long-term 

outcomes related to bare-metal and drug-eluting 

stents, once it became clear through FDA-designed 

and other registries in the postmarket setting that 

off-label use was very common and that the risk of 

restenosis and other long-term outcomes in the 

real-world patient population was not fully 

understood. Due to the existence of comprehensive 

national population registries in Sweden, 

researchers were able to reliably combine SCAAR 
105

Chapter 19. Use of Registries in Product Safety Assessment

data, which captured unselected, consecutive 

angiography and percutaneous coronary 

intervention procedure data, with vital status and 

hospitalization data, to examine fatality rates and 

cardiac events on a population level.38 This use of 

procedure and national registries provides an 

example of how a registry that included a welldefined population allowed for safety assessments 

coincident with comparative effectiveness.

6. Potential Obligations for 

Registry Developers in 

Reporting Safety Issues

In considering what actual and potential 

obligations there are, or may be, for registries in 

product safety assessment, it is useful to separate 

the issues into several parts. First, there are two 

key questions that can be asked for each registry: 

(1) What is the role of registries not designed for 

safety purposes with respect to the search for AEs? 

and (2) What are the obligations, especially for 

those registries not sponsored by regulated 

manufacturers, to further investigate and report 

these events when found? As discussed above, 

registries can be classified by whether or not they 

were designed for a safety purpose, and also by 

whether or not they have specified regulatory 

obligations for reporting. Beyond these 

distinctions, several factors need to be considered, 

including the ethical obligations of the registry 

developer, the technical limitations of the signal 

detection, and resource constraints.

Registries designed for safety assessment purposes 

should have a clear and deliberate plan in place, 

not only for detecting the signal of interest, but for 

handling unanticipated events and reporting them 

to appropriate authorities. Only in the case of 

registries supported by the regulated industries are 

rules for reporting drug or device AEs explicit. 

Therefore, it would be helpful if other registries 

would also formulate plans that ensure that 

appropriate information will reach the right 

stakeholders, either through reporting to the 

manufacturer or directly to the regulator, in a 

timely manner similar to those required by the 

regulated industries. There should not be two 

different standards for reporting information 

intended to safeguard the health and well-being of 

all.

Registries that are not designed specifically for 

safety assessment purposes, particularly those that 

are not sponsored by a manufacturer, raise more 

complex issues. While researchers have an 

obligation to the patients enrolled in any research 

activity to alert them should information regarding 

potential safety issues become known, it is less 

clear how far this obligation extends. In the United 

Kingdom, the General Medical Council includes 

in its advice on “Good Medical Practice” the 

requirement to “report suspected adverse drug 

reactions in accordance with the relevant reporting 

scheme.”39 It is therefore clear that in the United 

Kingdom contributing to the safety profile of a 

medicine is regarded as part of the duties of a 

medical practitioner. During its review of research 

registries, an institutional review board (IRB) in 

the United States or ethics committee (EC) in 

Canada or the European Union may specify the 

creation of an explicit incidental findings plan 

prior to approval. Such a plan is often part of 

studies producing or compiling nonclinical 

imaging and genetic data. In addition, some 

investigators will have an obligation to report to an 

IRB or EC any unanticipated problems involving 

risks to subjects or others under the regulations on 

human research protections. In turn, IRBs and ECs 

have an obligation to report such incidents to 

relevant authorities.

At a minimum, all registries should ensure that 

standard reporting mechanisms for AE information 

are described in the registry’s procedural 

documents. These mechanisms should also be 

explained to investigators and, where feasible, their 

reporting efforts should be facilitated. For 

example, all registries in the United States can 

make available to registry participants access to 

the MedWatch forms40 and train them in the 

appropriate use of these forms to report 

spontaneous events. As described in Section 4 

above, in the near future it should be possible for 

registries that collect data electronically to actually 

facilitate the reporting of AEs by linking with 

facilitated safety reporting mechanisms. This 

mechanism is attractive because it reduces the 

work of the investigator in generating the report 
106

Section V. Special Applications in Patient Registries

and ensures that the report will go to a surveillance 

program prepared to investigate and manage both 

events and potential safety signals.

Obligations beyond facilitation are less clear. 

Furthermore, there are both technical and resource 

obstacles to thoroughly investigating potential 

signals, and risks that inaccurate and potentially 

injurious information will be generated. For 

example, publicizing product safety issues can 

result in some patients discontinuing use of 

potentially life-saving products regardless of the 

strength of the scientific evidence. As described 

earlier, registries designed for safety assessment 

should ideally have both adequate sample size and 

signal evaluation expertise in order to assess safety 

issues. Registries not designed for safety purposes 

may not have enough patients or statistical signal 

detection expertise to investigate potential signals, 

or may not have the financial resources to devote 

to unplanned analyses and investigations. It would 

seem that, at a minimum, registries not designed 

for safety purposes should use facilitated reporting 

(via training, providing forms, etc.) of individual 

events through standard channels to meet their 

ethical obligations, and that they should check 

with any institutions with which they are affiliated 

to determine whether they are subject to additional 

reporting requirements. However, should a registry 

identify potential signals through its own analyses, 

obligations arise.

While registries that are approved by IRBs report 

safety issues to those IRBs, incidental analytic 

findings, which may represent true or false signals, 

may need more definition and should best be 

further investigated and reported for the public 

good. One approach would be to report summary 

information to the relevant regulatory authority for 

further evaluation. To avoid doubt, registry 

developers should consider these issues carefully 

during the planning phase of a registry, and should 

explicitly define their practices and procedures for 

AE detection and reporting, their planned analyses 

of AEs, and how incidental analytic findings will 

be managed. Such a plan should lay out the extent 

to which registry owners will analyze their data for 

AEs, the timing of such analyses, what types of 

unanticipated issues will be investigated internally, 

what thresholds would merit action, and when 

information will be provided to regulators or other 

defined government entities, depending on the 

nature of the safety issue.

7. Summary

The ongoing challenge, in the use both of existing 

data and of prospective data collection efforts such 

as registries, is to cast a wide enough net to 

capture not only rare events, but also more 

common events and events that are not anticipated 

(i.e., not part of a preapproval or postapproval 

potential risk assessment). In some cases, existing 

registries may add additional data collection to 

address questions regarding possible AEs that arise 

after registry initiation. In addition, it must be 

considered that all observational data sources are 

only as strong as their ability to measure and 

control for potential biases, including confounding 

and misclassification.

Large registries, linkage and distributed network 

schemes, and sentinel surveillance are all tools 

being actively developed to create an integrated 

approach to medical product safety and, 

specifically, to signal detection and verification.

In contributing to the evidence hierarchy 

surrounding the generation of signals for detection 

and confirmation of potential AEs, registries are 

likely to make their strongest contributions 

through: detection of novel AEs associated with 

product use as reported by treating physicians, 

which, in turn, constitutes a signal necessitating 

further study; gathering information about 

pregnant women and other hard-to-study 

subpopulations of product users; linking with 

additional data sources such as the MedicareSEER data linkage, thereby broadening the range 

of questions that can be addressed beyond the 

constraints of data collected for a registry; and 

confirming or validating signals generated in other 

data, such as from automated signal generation in 

large claims databases. Ideally, a clear and 

prospective understanding among stakeholders is 

needed regarding whether and under what 

circumstances signal monitoring within registries 

is appropriate; the timing or periodicity of any 

such analyses; what should be done with the 

information once it is identified, and what, if any, 
107

Chapter 19. Use of Registries in Product Safety Assessment

are the ethical obligations to collect, analyze, and 

report safety information if doing so is not a 

planned objective of the registry, and if the registry 

sponsor is not directly required by regulation to 

conduct such reporting.

Thoughtfully designed registries can play 

important roles in these newly emerging strategies 

to use multiple available data sources to generate 

and strengthen hypotheses in product safety. 

However, as with all data sources, it is important to 

assess the effects of registry design, the type of 

data, reason for the data collection, how the data 

were collected, and the generalizability to the 

target population, in order to assess the strengths, 

weaknesses, and validity of the results provided 

and their contribution to the knowledge of the 

safety profile of the medicine or device under 

study.

Case Examples for Chapter 19

Case Example 44. Using a registry to assess 

long-term product safety

Description The British Society for 

Rheumatology Biologics 

Register is a prospective 

observational study conducted 

to monitor the routine clinical 

use and long-term safety of 

biologics in patients with severe 

rheumatoid arthritis and other 

rheumatic conditions. The 

United Kingdom-wide national 

project was launched after the 

introduction of the first tumor 

necrosis factors (TNF) alpha 

inhibitors.

Sponsor The British Society for 

Rheumatology (BSR) 

commissioned the registry, 

which receives restricted funding 

from Abbott Laboratories, 

Biovitrum, Schering Plough, 

Roche, and Wyeth 

Pharmaceuticals. The registry is 

managed by the BSR and the 

University of Manchester.

Year Started 2001

Year Ended Ongoing

No. of Sites All consultant rheumatologists in 

the United Kingdom who have 

prescribed anti-TNF therapy 

participate.

No. of Patients More than 17,000

Challenge

Rheumatoid arthritis (RA) is a progressive 

inflammatory disease characterized by joint 

damage, pain, and disability. Among the 

pharmacologic treatments, nonbiologic diseasemodifying antirheumatic drugs (DMARDs) are 

considered the first-line treatment. Novel biologic 

therapies represent a new class of agents that 

prevent inflammation and have demonstrated 

efficacy in RA patients. The most commonly used 

biologics are tumor necrosis factors (TNF) 

inhibitors (etanercept, infliximab, and 

adalimumab). However, results from clinical 

trials and pharmacovigilance studies have raised 

potential safety concerns, and limited long-term 

data on these therapies are available. Of particular 

concern has been an increase of tuberculosis 

observed in patients treated with anti-TNF 

therapy.

Proposed Solution

A prospective observational registry was 

launched in 2001 to monitor the safety of new 

biologic treatments. The registry collects data on 

response to treatment and potential AEs every six 

months, and patients are followed for the life of 

the registry. Over 4,000 patients are enrolled for 

each of the anti-TNF agents (etanercept, 

infliximab, and adalimumab), and the registry 

represents approximately 80 percent of RA 

patients treated with these biologics in the United 

Kingdom. In addition to patients receiving 

anti-TNF therapy, the registry has enrolled a 

control cohort of patients receiving nonbiologic 

DMARDs.
108

Section V. Special Applications in Patient Registries

Case Example 44. Using a registry to assess 

long-term product safety (continued)

Results

Data from the registry were analyzed to 

determine whether an increased risk of 

tuberculosis existed in RA patients treated with 

anti-TNF therapy (Dixon et al., 2010). In more 

than 13,000 RA patients included up to April 

2008, 40 cases of tuberculosis were observed in 

the anti-TNF cohort and no cases in the DMARD 

group. A differential risk was reported among the 

three anti-TNF agents, with the lowest risk 

observed in the etanercept group. The incidence 

rates were 144, 136, and 39 cases per 100,000 

person-years for adalimumab, infliximab, and 

etanercept, respectively. In addition, the incidence 

rate ratio, median time to events, and influence of 

ethnicity were evaluated.

Key Point

As novel drugs and treatments are developed and 

licensed, registries may be useful tools for 

collecting long-term data to assess known and 

emerging safety concerns.

For More Information

Dixon WG, Hyrich KL, Watson KD. et al. 

Drug-specific risk of tuberculosis in patients with 

rheumatoid arthritis treated with anti-TNF 

therapy: Results from the British Society for 

Rheumatology Biologics Register (BSRBR). 

Annal Rheum Dis. 2010;69(3):522–8.

Zink A, Askling J, Dixon WG. et al. European 

biological registers: methodology, selected results 

and perspectives. Annal Rheum Dis. 

2009;68:1240–6.
109

Chapter 19. Use of Registries in Product Safety Assessment

Case Example 45. Using a registry to monitor 

long-term product safety

Description SINCERETM (Safety in 

Idiopathic arthritis: NSAIDs and 

Celebrex Evaluation Registry) 

was a multi-center registry 

designed to monitor the longterm safety of nonsteroidal 

anti-inflammatory drugs 

(NSAIDs) in patients with 

juvenile idiopathic arthritis 

(JIA). The registry included 

patients ages 2 to 17 and 

collected demographic, 

developmental, clinical, and 

safety data. The followup period 

was at least 2 years, and as long 

as 4 years for some patients.

Sponsor Pfizer, Inc.

Year Started 2009

Year Ended 2012 (terminated early)

No. of Sites 16 sites in the United States

No. of Patients Planned enrollment of 200 

patients on celecoxib and 200 

patients on other NSAIDs.

Actual enrollment of 219 

patients on other NSAIDs, 55 on 

celecoxib, for a total of 274.

Challenge

Nonsteroidal anti-inflammatory drugs (NSAIDs) 

have been used for more than 30 years to relieve 

pain and inflammation in juvenile idiopathic 

arthritis (JIA), and it is estimated that 80 to 90 

percent of JIA patients will use an NSAID at 

some point. However, little is known about the 

long-term safety of chronic use of NSAIDs in 

children with JIA. This question is particularly 

important, as many children with JIA will 

continue to use NSAIDs well into adulthood. Due 

to the rarity of JIA and the special ethical issues 

surrounding children’s participation in 

experimental studies, randomized controlled trials 

of NSAIDs in JIA are considerably smaller and of 

shorter duration than adult arthritis trials; the 

pivotal trial for celecoxib in JIA, one of the 

largest NSAID JIA studies, had 100 patient-years 

of exposure. In addition, randomized trials may 

not be generalizable to typical JIA populations. 

Lastly, it is unclear if the emerging safety 

concerns in adult NSAID and celecoxib users 

translate to children, who are much less likely to 

develop serious cardiovascular thromboembolic 

events or gastrointestinal bleeding events.

The development of a long-term observational 

study was necessary to address these knowledge 

gaps, fulfill a postmarketing safety commitment, 

and respond to concerns of regulators, patients, 

physicians, and the sponsor.

Proposed Solution

This multicenter registry was designed to gather 

long-term safety data on NSAIDs use in children 

with JIA, and was intended to enroll a quasiinception cohort of patients aged 2 to 17 years 

and >10 kg who were prescribed (not more than 6 

months prior) either celecoxib (n=200) or other 

NSAIDs (n=200). Pediatric rheumatologists from 

16 sites in the United States entered data 

quarterly for the first 12 months and twice 

annually thereafter. The registry intended to 

follow all patients for at least 2 years and as long 

as 4 years, as all patients were encouraged to 

remain in the registry until the last patient 

completed the minimum followup. Concomitant 

medications and treatment switches were 

permitted, and patients were followed for residual 

effects even if NSAID treatment was 

discontinued.

Targeted events of interest (i.e., cardiovascular, 

gastrointestinal, and hypertension) and general 

safety serious and nonserious AEs were collected 

in a systematic manner. The Common 

Terminology Criteria for Adverse Events 

(CTCAE ver 3.0) criteria were used to both code 

and grade all AEs to minimize variability across 

physicians. In designing the registry, particular 

attention was paid to collecting potential 

covariates relevant to confounding by indication, 

given the expected differential prescribing 

between celecoxib and other NSAIDs. The 

analyses summarized the incidence of the 

targeted events and AEs in general, and 

exploratory analyses may further characterize AE 

rates by other clinical and demographic factors.
110

Section V. Special Applications in Patient Registries

Case Example 45. Using a registry to monitor 

long-term product safety (continued)

Results

The registry was terminated early due to low 

patient recruitment, despite multiple attempts to 

improve site and patient enrollment. The primary 

issue identified as a barrier to enrollment was that 

the treatment paradigm had changed since 

celecoxib was first approved: with the advent and 

increasing use of biologic therapies, NSAIDs 

were no longer being used long-term in JIA very 

often. As a result, the objective of the study, to 

assess the long-term safety of celecoxib as used 

for JIA, could not be met. This change in 

treatment paradigm, coupled with safety 

information from the registry and other sources 

that indicated no new signals nor change to the 

benefit-risk profile of celecoxib as used in 

children with JIA, allowed the FDA to release the 

sponsor from the commitment and terminate the 

study early due to futility. The registry 

nevertheless provided over 410 patient-years of 

observation in this cohort of NSAID and 

celecoxib users, providing additional safety data 

on these drugs as used for JIA in routine clinical 

practice; no new safety issues were identified. 

This information may facilitate appropriate 

therapeutic decisionmaking for doctors and 

patients.

Key Point

Registries may be useful tools for examining 

long-term product safety, particularly in 

populations such as children that are difficult to 

study in randomized controlled trials. Changes in 

the treatment paradigm of a disease may affect 

the utility and feasibility of a long-term product 

safety registry.

For More Information

Beukelman T, Patkar NM, Saag KG, et al. 2011 

American College of Rheumatology 

recommendations for the treatment of juvenile 

idiopathic arthritis: initiation and safety 

monitoring of therapeutic agents for the treatment 

of arthritis and systemic features. Arthritis Care 

Res. 2011 Apr;63(4):465-82.

References for Chapter 19

1. Lecutier MA. Phocomelia and internal defects due 

to thalidomide. Br Med J. 1962 Dec 1;2(5317):

1447-8. PMID: 13929004. PMCID: 1926787.

2. Herbst AL, Ulfelder H, Poskanzer DC. 

Adenocarcinoma of the vagina. Association of 

maternal stilbestrol therapy with tumor appearance 

in young women. N Engl J Med. 1971 Apr 

15;284(15):878-81. PMID: 5549830.

3. Hartzema AG, Tilson HH, Chan KA, eds. 

Pharmacoepidemiology and Therapeutic Risk 

Management. Cincinnati: Harvey Whitney Books; 

2008.

4. Drazen JM, Rainey J, Begg H, et al. Adverse Drug 

Event Reporting: The Roles of Consumers and 

Health-Care Professionals: Workshop Summary. 

Washington, DC: National Academies Press; 

2007.

5. McClellan M. Drug safety reform at the FDA—

pendulum swing or systematic improvement? 

N Engl J Med. 2007 Apr 26;356(17):1700-2. 

PMID: 17435081.

6. Eland IA, Belton KJ, van Grootheest AC, et al. 

Attitudinal survey of voluntary reporting of 

adverse drug reactions. Br J Clin Pharmacol. 

1999 Oct;48(4):623-7. PMID: 10583035. 

PMCID: 2014371.

7. Moore N, Hall G, Sturkenboom M, et al. Biases 

affecting the proportional reporting ratio (PPR) in 

spontaneous reports pharmacovigilance databases: 

the example of sertindole. Pharmacoepidemiol 

Drug Saf. 2003 Jun;12(4):271-81. 

PMID: 12812006.

8. Figueiras A, Herdeiro MT, Polonia J, et al. An 

educational intervention to improve physician 

reporting of adverse drug reactions: a clusterrandomized controlled trial. JAMA. 2006 Sep 

6;296(9):1086-93. PMID: 16954488.
111

Chapter 19. Use of Registries in Product Safety Assessment

9. European Medicines Agency. Guideline on the use 

of statistical signal detection methods in the 

Eudravigilance Data Analysis System. 2008. 

http://eudravigilance.emea.europa.eu/human/

docs/10646406en.pdf. Accessed September 30, 

2013.

10. Finney DJ. The Design and Logic of a Monitor of 

Drug Use. J Chronic Dis. 1965 Jan;18:77-98. 

PMID: 14252273.

11. U.S. Food and Drug Administration. The Sentinel 

Initiative: national strategy for monitoring medical 

product safety. May 2008. www.fda.gov/

downloads/Safety/FDAsSentinelInitiative/

UCM124701.pdf. Accessed August 17, 2012.

12. Samuel FE, Jr. Safe Medical Devices Act of 1990. 

Health Aff (Millwood). 1991 Spring;10(1):192-5. 

PMID: 2045049.

13. Schmitt-Egenolf M. Psoriasis therapy in real life: 

the need for registries. Dermatology. 

2006;213(4):327-30. PMID: 17135739.

14. Tulner LR, Frankfort SV, Gijsen GJ, et al. Drugdrug interactions in a geriatric outpatient cohort: 

prevalence and relevance. Drugs Aging. 

2008;25(4):343-55. PMID: 18361544.

15. Hanley JA, Lippman-Hand A. If nothing goes 

wrong, is everything all right? Interpreting zero 

numerators. JAMA. 1983 Apr 1;249(13):1743-5. 

PMID: 6827763.

16. Eypasch E, Lefering R, Kum CK, et al. 

Probability of adverse events that have not yet 

occurred: a statistical reminder. BMJ. 1995 Sep 

2;311(7005):619-20. PMID: 7663258. 

PMCID: 2550668.

17. Rogers WJ, Bowlby LJ, Chandra NC, et al. 

Treatment of myocardial infarction in the United 

States (1990 to 1993). Observations from the 

National Registry of Myocardial Infarction. 

Circulation. 1994 Oct;90(4):2103-14. 

PMID: 7923698.

18. Odvina CV, Zerwekh JE, Rao DS, et al. Severely 

suppressed bone turnover: a potential 

complication of alendronate therapy. J Clin 

Endocrinol Metab. 2005 Mar;90(3):1294-301. 

PMID: 15598694.

19. Tilson HH, Doi PA, Covington DL, et al. The 

antiretrovirals in pregnancy registry: a fifteenth 

anniversary celebration. Obstet Gynecol Surv. 

2007 Feb;62(2):137-48. PMID: 17229330.

20. Food and Drug Administration Amendments Act 

of 2007, Pub. L. No. 110-85 (2007), Title IX.

21. Peck CC. FDA’s position on the clozaril patient 

management system. Hosp Community 

Psychiatry. 1990;41(8):876-7.

22. Zeldis JB, Williams BA, Thomas SD, et al. 

S.T.E.P.S.: a comprehensive program for 

controlling and monitoring access to thalidomide. 

Clin Ther. 1999 Feb;21(2):319-30.

23. Biogen Idec. The TYSABRI TOUCH® Prescribing 

Program. http://www.tysabri.com/safety-withtysabri.xml. Accessed September 30, 2013.

24. Honein MA, Lindstrom JA, Kweder SL. Can we 

ensure the safe use of known human teratogens?: 

The iPLEDGE test case. Drug Saf. 2007;30(1):

5-15.

25. Kleinschmidt-DeMasters BK, Tyler KL. 

Progressive multifocal leukoencephalopathy 

complicating treatment with natalizumab and 

interferon beta-1a for multiple sclerosis. N Engl 

J Med. 2005 Jul 28;353(4):369-74. 

PMID: 15947079.

26. Humbert M, Segal ES, Kiely DG, et al. Results of 

European post-marketing surveillance of bosentan 

in pulmonary hypertension. Eur Respir J. 2007 

Aug;30(2):338-44. PMID: 17504794.

27. Walker AM, Funch DP, Sulsky SI, et al. Patient 

factors associated with strut fracture in BjorkShiley 60 degrees convexo-concave heart valves. 

Circulation. 1995 Dec 1;92(11):3235-9. 

PMID: 7586309.

28. Walker AM, Funch DP, Sulsky SI, et al. 

Manufacturing characteristics associated with 

strut fracture in Bjork-Shiley 60 degrees ConvexoConcave heart valves. J Heart Valve Dis. 1995 

Nov;4(6):640-8. PMID: 8611980.

29. Curtis JP, Luebbert JJ, Wang Y, et al. Association 

of physician certification and outcomes among 

patients receiving an implantable cardioverterdefibrillator. JAMA. 2009 Apr 22;301(16):

1661-70. PMID: 19383957. PMCID: 2805129.

30. The *ASTER Pilot Project: Improving the 

Reporting of Adverse Events. *ASTER: A 

Collaborative Study to Improve Drug Safety. 

www.asterstudy.com. Accessed August 15, 2012.

31. World Health Organization’s International Drug 

Monitoring Programme. Uppsala Monitoring 

Centre (UMC). http://www.who-umc.org/. 

Accessed August 15, 2012.
112

Section V. Special Applications in Patient Registries

32. Hauben M, Aronson JK. Defining ‘signal’ and its 

subtypes in pharmacovigilance based on a 

systematic review of previous definitions. Drug 

Saf. 2009;32(2):99-110. PMID: 19236117.

33. Nelson J, Cook A, Yu O. Evaluation of signal 

detection methods for us in prospective postlicensure medical product safety surveillance. 

March 2009. http://www.fda.gov/OHRMS/

DOCKETS/98fr/FDA-2009-N-0192-rpt.pdf. 

Accessed August 15, 2012.

34. Council for Interntational Organizations of 

Medical Sciences (CIOMS). Practical aspects of 

signal detection in pharmacovigilance. Report of 

CIOMS Working Groups VIII. . Geneva: 2010.

35. Meyboom RH, Egberts AC, Edwards IR, et al. 

Principles of signal detection in 

pharmacovigilance. Drug Saf. 1997 

Jun;16(6):355-65. PMID: 9241490.

36. Hauben M, Zhou X. Quantitative methods in 

pharmacovigilance: focus on signal detection. 

Drug Saf. 2003;26(3):159-86. PMID: 12580646.

37. Szarfman A, Machado SG, O’Neill RT. Use of 

screening algorithms and computer systems to 

efficiently signal higher-than-expected 

combinations of drugs and events in the US FDA’s 

spontaneous reports database. Drug Saf. 

2002;25(6):381-92. PMID: 12071774.

38. Lagerqvist B, James SK, Stenestrand U, et al. 

Long-term outcomes with drug-eluting stents 

versus bare-metal stents in Sweden. N Engl J 

Med. 2007 Mar 8;356(10):1009-19. 

PMID: 17296822.

39. General Medical Council. Good Medical Practice. 

2006. http://www.gmc-uk.org/guidance/good_

medical_practice.asp. Accessed August 15, 2012.

40. U.S. Food and Drug Administration. Reporting 

Serious Problems to the FDA. http://www.fda.gov/

Safety/MedWatch/HowToReport/default.htm. 

Accessed August 15, 2012.
113

Chapter 20. Rare Disease Registries

1. Introduction

There is no single, unifying definition of a rare 

disease. Rare diseases are defined, from a 

regulatory and policy perspective, as any condition 

or disease affecting fewer than 200,000 individuals 

in the United States, or alternatively, determined to 

be of low prevalence (fewer than 5 individuals per 

10,000) in the European Union.1, 2 In the United 

States, the Orphan Drug Act (P.L. 97-414) was 

adopted in 1983 in an effort to encourage activities 

by industry (and to a lesser extent other funding 

and research bodies) through tax incentives, 

market exclusivity, user fee exemptions and other 

incentives to target development of therapies for 

rare diseases. This legislation, as well as other 

regulations and similar international initiatives, 

resulted in a marked increase in rare disease 

research funding and development efforts for 

related drugs and biologics. Success of these 

compounds in coming to market, however, has 

been hampered by an incomplete understanding of 

the underlying disease mechanisms and relevant 

clinical endpoints, as well as limitations associated 

with identifying a large enough sample of 

comparable patients for clinical trials. 

The scarcity of relevant knowledge and experience 

with most rare diseases creates a unique need for 

cooperation and infrastructure. Support is needed 

for research initiatives that aim to better 

understand the distribution and determinants of 

these diseases and to develop new therapies and 

other interventions. Innovations in genetics, 

molecular and computational biology, and other 

technological advances in basic research are 

rapidly evolving; however, translating this progress 

into clinic research and securing governmental or 

private funding in early stages remains 

challenging. Some of these challenges can be 

addressed efficiently through a systematic 

collection of clinical, genetic, and biologic data in 

the form of longitudinal patient registries and other 

coordinated data sources.

The use of observational data methods, including 

prospective long-term patient registries, is a 

critical tool in building a broad and comprehensive 

knowledge base for these often heterogeneous 

diseases. Important data include the prevalence 

and distribution of these diseases and key patient, 

familial, and disease characteristics, including the 

natural history of the disease. Although many of 

the basic concepts around registry planning, 

design, and implementation are directly applicable 

these disease registries, rare diseases pose some 

unique challenges. The range of stakeholders for 

rare diseases is inherently different, which has a 

direct effect on implementation, governance, 

funding, communication and as well as their level 

of interest and willingness to participate in the 

study of rare diseases. Clinicians with relevant 

expertise and direct exposure to managing these 

patients are limited, necessitating a broad outreach 

to identify and recruit enough patients to 

understand the epidemiology and natural history of 

the disease. In addition, because of knowledge 

gaps, the scope and objectives of rare disease 

registries are often broader than in a typical 

disease registry. The absence of standards of care 

or treatment guidelines in many cases, the 

common use of experimental and adjunctive 

therapies, and the incomplete understanding of 

how these conditions should be monitored in the 

absence of established or widely accessible 

biomarkers provide opportunities for rare disease 

registries to set the agenda for disease research. 

Since amassing a sizable population from which 

any patterns of rare diseases can be discerned is 

more difficult, novel approaches are often required 

to both define rare diseases and their relevant 

outcomes (in other words, scientifically validated 

and accepted criteria may not exist). Lastly, patient 

advocacy and support groups are smaller for these 

often less well-known diseases and may play 

different roles than in a more traditional disease 

registry. 
114

Section V. Special Applications in Patient Registries

This chapter provides an overview of the 

development of patient registries for rare diseases 

and the key stakeholders and challenges that are 

specific to these registries. Case Examples 46, 47, 

and 48 offer some descriptions of rare disease 

registries. The reader is directed to other chapters 

regarding relevant good registry practices.

2. Genesis of a Rare Disease 

Registry

2.1 Rare Disease Registry Objectives 

and Scope

Rare disease registries are initiated by many 

organizations, such as patients and their families, 

patient advocacy groups, clinicians, national health 

systems, and biopharmaceutical product 

manufacturers, for many reasons. Often, rare 

disease patient registries have grown organically. 

In rare diseases where patients are few, research 

agendas do not exist, standard case guidelines are 

absent, and patient communities have not yet 

formed, patient registries are an intuitive first step 

for stakeholders trying to understand the number 

of people affected, their geographical distribution, 

and the basic demographic and clinical 

characteristics of the disease. The scope of these 

registries may evolve over time, maturing from an 

outreach/community-building effort or a means for 

a basic understanding of patient and disease 

characteristics, to a supportive mechanism for 

research funding and attracting health care 

providers. As with all registries, a single rare 

disease registry need not fulfill all goals for all 

potential stakeholders. Ideally, however, a welldesigned registry provides an infrastructure that 

can support different needs in an efficient way and 

eliminate barriers to scientific progress.

It should be noted that rare disease registries 

include not only diseases that are inherently rare, 

but also common diseases that are rare in specific 

populations, especially those defined by 

demographics. Thus, plaque psoriasis—common 

among adults—is rare in children, and breast 

cancer—common among women—is rare among 

men. While some of the objectives specific to rare 

disease registries will apply (e.g., patient 

identification and recruitment), others may not 

(e.g., disease classification, measuring diseasespecific outcomes).

Registries can be developed to serve multiple 

purposes. The design of the registry depends upon 

the maturity of the research plan around the 

disease, the availability and duration of funding, 

and, to some extent, the number of patients 

affected. For rare diseases, the perception of 

relative importance of research often correlates 

with the number of patients affected or the number 

of empowered disease advocates.

The specific objectives of rare disease registries 

typically cluster into the following categories:

1. To connect affected patients, families, and 

clinicians. Patients and families of affected 

individuals are often interested in knowing 

about others who share their disease. Many rare 

diseases have a genetic basis. However, even if 

multiple family members are affected with the 

condition, the motivation to be connected to 

others may be quite strong, driven by their 

personal desire to know more about the 

condition, its natural history, alternative coping 

mechanisms and treatment options, and the 

diversity of clinical courses and outcomes. The 

need to connect is enhanced if the patient or 

family has difficulty in finding an expert to 

provide advice or the doctor or genetic 

counselor points out how little is known about 

the rare condition. 

Registry meetings provide an opportunity to 

talk and to share personal experiences. These 

meetings may include lectures and discussion 

among patients and families and with experts 

in medicine, dentistry, nursing, sociology, and 

many other fields. The advent of social media 

has increased patient involvement in these 

types of activities by encouraging patient-topatient dialogue and assisting with recruitment 

for research and support. Patients and families 

often want to connect to advocate support of 

patients’ services and financial support for 

patient care and research. 

Similarly, physicians and other clinicians may 

want to connect with other clinicians to learn 

more about the disease and possible treatment 
115

Chapter 20. Rare Disease Registries

options. Most clinicians have not seen a wide 

spectrum of rare diseases, and little 

information on some diseases may be available 

in the literature. Registries may offer a 

connection to essential information and to 

experts in the disease area to assist health care 

providers with advising and counseling 

patients. 

2. To learn the natural history, evolution, risk, and 

outcomes of specific diseases. Stakeholders 

often initiate registries to learn the natural 

history of a rare disease. Typically, rare diseases 

are described in a general way based on their 

symptoms at the time of diagnosis. With 

refinement in diagnostic techniques, including 

genetic, biochemical, and physiological testing, 

classical disease descriptions are broadened, 

and diseases are better described in terms of the 

range and likelihood of specific outcomes. 

Unlike more common diseases, for which 

criteria for classification will often evolve, such 

evolution may not be possible for rare diseases, 

but the acceptance of some general criteria that 

derive from these studies will inform and help 

subsequent research. As general and specific 

therapies emerge, the natural history often 

changes and the “classical description” may no 

longer apply. With better therapies for treatment 

and supportive care, new complications may 

also be recognized. For treatments that extend 

life expectancy, what is known about the 

trajectory of disease can change drastically. A 

disease registry incorporating patients with rare 

diseases from many centers allows for 

gathering stronger and more generalizable 

safety, diagnostic, and prognostic information. 

For industry, natural history of disease 

registries are often developed to better 

understand the burden of disease, elements of 

disease progression, disease genotypic and 

phenotypic heterogeneity, and potential 

endpoints (or surrogate endpoints) that may be 

used in therapeutic clinical development. 

Increasingly, these types of disease registries 

are also used to understand patient and 

caregiver quality of life and the economic 

consequences of these diseases, as well as to 

understand the background risk of specific 

outcomes (i.e., to provide a reference 

population) that usually cannot be found or 

inferred from other sources of data when the 

disease is rare.

3. To support research on genetic, molecular, and 

physiological basis of rare diseases. Research 

on features of disease, both clinical and basic, 

is a common objective of a registry. Clinical 

research depends on having a representative 

population for determining the timing and 

frequency of natural events and complications, 

such as development of autoimmune 

complications, unusual infections, and related 

or unrelated malignancies. For this reason, rare 

disease registries benefit from a comprehensive 

database that is sufficient to address critical 

clinical questions, while at the same time not 

being so all-inclusive that the data cannot be 

acquired and maintained with reasonable effort 

on the part of the registry team. 

Patients, researchers, and clinicians share 

interests in understanding diseases at the 

genetic, molecular, and cellular level. Such 

studies usually require a biorepository of 

materials for research, including tissue (fresh 

and frozen), DNA, RNA, cellular proteins, and 

bodily fluids. Creating a meaningful 

repository for the study of rare diseases 

requires collection of materials from a 

sufficient population to permit generalization 

about the fundamental features and diversity 

of the disease at the genetic, molecular, and 

cellular level. A registry is an important 

complement to any biorepository; similarly, 

biorepositories are far stronger if they are 

closely linked to a registry that contains 

relevant longitudinal clinical or phenotype 

data. In cases where multiple small or 

regional registries exist for a specific 

condition, a centralized biorepository can 

serve as a common link and research resource. 

Valid interpretation of biosample research 

depends on understanding the clinical features 

of the patients and the heterogeneity of the 

disease in the study population. In addition, 

the existence of parallel relevant longitudinal 

clinical data allows for assessment of genetic 

and environmental disease modifiers.
116

Section V. Special Applications in Patient Registries

4. To establish a patient base for evaluating 

drugs, medical devices, and orphan products. 

Stakeholders are vitally interested in 

developing drugs, devices and other therapies 

for rare diseases, and many rare disease 

registries have been developed to support the 

drug development process. Patient registries for 

rare diseases may emerge from suggestion, 

pressure, or advocacy of affected patients and/

or families. Direct influence can be seen when 

patients and their caregivers decide they want a 

registry, raise the funds, and push for its 

creation. Indirect influence can be seen when 

patients or special interest groups drive 

government to make research on that disease a 

priority. Researchers and industry recognize 

that a population of patients is essential for 

clinical testing, and industry may provide rare 

disease groups with support to begin or expand 

a fledgling registry so that ultimately a 

potentially useful drug or device can be tested 

in the disease population. 

Often, developing a treatment for a rare 

disease will provide information about 

pathophysiology that informs treatment 

development of a related disease. If the rare 

disease is serious with few or no treatment 

options, regulators may relax some of the 

requirements for drug registration (as is 

indicated by the requirements for orphan drug 

development).

2.2 Rare Disease Registry 

Stakeholders

Any registry endeavor has a number of 

stakeholders, often with both convergent and 

divergent agendas. Stakeholders may include 

patient advocacy groups (often multiple), 

regulatory agencies (especially if the registry is 

being developed to support future drug 

development and approval or to fulfill 

postmarketing commitments or requirements), 

clinicians, scientists, industry, payers, and the 

individuals and families affected by the disease. 

Collaboration between stakeholder groups has 

been critical to the progress made in research and 

product development, the adoption of important 

public policy changes in the United States and 

worldwide, and the promotion of patient access to 

treatments as they become available.3 Table 20–1 

describes potential registry stakeholders and the 

roles they may play in registries. 

The importance of patient registries in rare 

diseases and the need to support many 

organizations has also brought umbrella patient 

organizations (e.g., NORD, the Genetic Alliance, 

EURORDIS) in as stakeholders, as these groups 

are charged with advising and supporting the 

development of registries. As the number of 

registries increases along with the number of 

commercial companies to develop and host them, 

these umbrella organizations are becoming brokers 

for services and are motivated to identify standards 

and shared efficiencies to support patient registries 

for the thousands of rare disorders that need them. 

In addition, the proliferation of patient registries 

for rare diseases brings standards development 

organizations and standards interests into the fold, 

as the need for standards that can facilitate data 

sharing (i.e., common data elements) between 

patient registries and other aspects of health care 

and clinical research has become evident. More 

broadly, the vision of patient registries that can 

share data between electronic health records and 

personal health records, as well as with clinical 

research or national public health efforts, has 

engaged a variety of commercial application 

providers in the field. 

Representatives from any of the groups mentioned 

as stakeholders can function as registry sponsors 

or developers. A distinction is made between 

registry sponsors, as the entities who fund, plan, 

and often select data collection content for a 

registry, and developers, as the technology and 

computing professionals who build the registry.
117

Chapter 20. Rare Disease Registries

Although data from registries are not a substitute 

for controlled trials, rare disease registry data may 

be the only source of information (especially about 

a specific product’s use) available to stakeholders. 

This information may serve to inform industry 

such that a controlled trial can be determined to be 

feasible, designed appropriately, and well informed 

upon inception. Disease registry data complement 

trials, especially those conducted in rare diseases, 

for which other sources of data are rare or 

nonexistent. Industry supports many rare disease 

patient registries, both disease-based and productbased, as sponsors and developers. This is 

particularly common in rare disorders for which 

the clinical development program is often 

abbreviated and inclusive of only a small, 

relatively heterogeneous subpopulation of the 

disease population. These registries are often well 

received by patient groups who do not have funds 

to operate a registry independently, but stakeholder 

objectives are not always aligned. For example, 

industry-sponsored registries are in some cases 

treatment- or product-based registries, where 

patients are included for study based upon 

treatment exposure. However, some product 

exposure registries create a fragmented system that 

does not allow researchers or policymakers to see 

the entire spectrum of disease.4 These different 

product registries have different sponsors and 

collect different data (often at the behest of 

regulators who seek answers to different 

questions), rendering them difficult to combine 

during research. When more than one treatment 

exists for a given condition, the different 

postmarketing treatment registries are often not 

comparable, nor are the full spectrum data (from 

multiple registries hosted by multiple companies) 

easily accessible for academic researchers. 

Additionally, if patients are exposed to multiple 

treatments, their data might be in multiple 

registries, but their full experience across 

treatments is not appreciated. 

Disease registries (rather than exposure or 

treatment registries) create the possibility of 

assessing the long-term safety and benefit of 

different treatments, perhaps leading to treatment 

algorithms that allow more choices for patients 

Stakeholder Role in Registry Motivations for Registry Involvement

Patients and their families/ 

caregivers

Participants Increase knowledge about the disease; create community; 

facilitate development of new treatments.

Patient advocacy groups Advocates, sponsors Increase knowledge about the disease; increase access 

to care; support training and research in disease area; 

raise profile of the disease to encourage funding for more 

research.

Clinicians/Investigators Data contributors Increase knowledge about the disease; learn from the 

registry community; gather data to refine complex or 

undefined diagnoses; develop and inform treatment 

guidelines.

Academia Principal investigators, 

scientific advisors

Improve understanding of disease; create data source for 

research in disease area.

Biopharmaceutical industry Sponsors, developers Understand the natural history of the disease to design 

better clinical trials and evaluate potential relevant clinical 

endpoints; fulfill postmarketing commitment; provide 

patient pool for interventional studies; determine potential 

market share and access patients; generate publications.

Government/regulatory 

agencies/payers

Sponsors, recipients of 

information 

Increase knowledge about the disease; monitor the safety 

of approved products; evaluate cost-effectiveness and 

budget impact; evaluate evidence for reimbursement.

Table 20–1. Role of stakeholders in rare disease registries
118

Section V. Special Applications in Patient Registries

and clinicians. Regulators have increasingly 

recognized the value of disease registries for 

historical comparator data and long-term 

evaluation (especially for drug safety) and as a 

complement to randomized clinical trials to “fill in 

the blanks” about outcomes that were not 

addressed in the limited controlled studies. These 

registries become even more important to 

regulators (and others involved) when the disease 

is rare and registries may be the only means by 

which data can be obtained. The marriage of 

stakeholder interests may create conflicts of 

interest for these registries that require careful 

scrutiny of available resources. If an effective 

partnership can be established and maintained, the 

creation of clinician and patient/caregiver 

communities can be a powerful agent in the 

success of a product in development or evaluation. 

Even more effective in rare disease research is a 

collaborative approach in which multinational and 

multi-institutional stakeholders combine resources. 

As resources are combined, standardization 

becomes more important to allow data to be 

compared across registries. Regulatory 

organizations such as the U.S. Food and Drug 

Administration and the European Medicines 

Agency can guide standardization across multiple 

postmarket registries within specific disease areas 

and promote the creation of multisponsor registries 

where appropriate. Other organizations like 

PARENT (PAtient REgistries iNiTiative) are 

facilitating cross-border collaborations to develop 

comparable and transferrable patient registries. 

Resources and tools for identifying and sharing 

patient registry questions, such as the PRISM 

(Patient Registry Item Specifications and Metadata 

for Rare Diseases) library of patient registry 

questions,5 will support these standardization 

efforts. 

Although the creation of a single global registry 

for each disease (or group of diseases) is 

theoretically a sound idea, in practice it may not 

always be feasible or in the best interest of 

researchers. A viable alternative can be a network 

of registries and resources, such as TREAT-NMD 

(Translational Research in Europe—Assessment & 

Treatment of Neuromuscular Diseases), a network 

of neuromuscular disease researchers that was 

launched in early 2007. TREAT-NMD aims to 

create an infrastructure to promote the 

development of tools (e.g., core outcome sets) that 

industry, clinicians, and scientists need to bring 

novel therapeutic approaches through preclinical 

development and into the clinic and to establish 

best-practice care for neuromuscular patients 

worldwide.6 Similarly, multiple registries could be 

connected via a centralized biorepository or 

biobank to provide larger sample sizes to 

understand disease processes and how they 

correlate with patient outcomes. As with any 

collaborative research, the challenges lie in who 

manages the collaboration, who funds it, and what 

governance infrastructure is required to bring 

together researchers who may be reluctant to share 

their data. The availability of indices of registries 

[e.g., OrphaNet7 in Europe and the new Registry 

of Patient Registries (RoPR) in the United States8] 

is helpful for identification of potential data 

sources and collaborators.

3. Implementation of a Rare 

Disease Registry

3.1 Patient Population

Because patient registries can collect clinical 

information from larger, more heterogeneous 

populations than those included in a clinical trial, 

they are becoming increasingly valuable, 

particularly for diseases affecting very small 

patient populations, such as lysosomal storage 

disorders and for specific populations such as 

children.9 Whereas selection of patients may be 

highly restrictive in general disease registries, rare 

disease registries often have more liberal criteria 

for inclusion. In many cases, a physician 

diagnosis, rather than the more common strict 

classification schema, may be sufficient for 

inclusion in a rare disease registry. Reasons may 

include: (1) no classification criteria exist; 

(2) knowledge of the rare disease is so limited that 

being more inclusive is desirable; and (3) the 

population is so small that being more inclusive is 

desirable. With some exceptions, rare disease 

registries typically do have broad inclusion criteria 

and attempt to enroll most, if not all, eligible 

patients within a targeted geographical area. 
119

Chapter 20. Rare Disease Registries

Although they may not be sufficient for 

population-based estimates of disease, these data 

sources can be used to estimate the numbers of 

affected patients and the number of patients 

potentially available for research, and can enable 

the mobilization of disease-specific communities 

for advocacy. Since a large proportion of 

recognized rare diseases are genetic in origin, 

enrolling family members greatly improves 

understanding of the disease, but may create 

additional complexities around confidentiality, 

logistical issues (e.g., different last names and 

other tracking issues), and considerations for 

enrollment of minors. The issue of “study fatigue” 

should also be considered when developing patient 

enrollment plans. Because of the limited numbers 

of available patients, some patients may be asked 

to participate in multiple studies over time. 

Patients may become overtaxed by frequent 

participation in studies and reluctant to join new 

studies. 

For registries examining treatment-related 

outcomes, the challenges in creating an inclusive 

patient cohort include differences in health care 

delivery systems, local regulations, and budgetary 

considerations that create barriers to care and/or 

specific treatments. For example, if a disease is 

rare, a drug or device manufacturer may choose 

not to go through the rigorous process required to 

have the product approved or priced locally—for 

example, in a small country—as the number of 

patients who might ultimately use the product does 

not support the cost of time and effort. This may 

create difficulties in enrolling a representative 

patient cohort from such regions.

3.2 Data Collection 

Most registries are tempted to (and often do) 

include as many data elements as possible in order 

to glean as much information as possible from 

their study population. This often leads to 

increasing respondent and investigator burden, 

high rates of discontinuation, and substantial 

challenges in data management. In some registries, 

these drawbacks may be offset by the ability to 

continue to recruit additional patients and/or the 

availability of sufficient numbers of patients 

already enrolled despite dropout. This is rarely the 

case, however, with rare disease registries. Thus, 

balancing the need for a broad dataset with the 

burden of data collection is highly important for 

rare disease registries. 

In many respects, data collection for rare disease 

registries is similar to data collection for other 

types of registries. Like other registries, rare 

disease registries aim to collect a uniform set of 

data on each patient. Data elements should be 

clearly defined to ensure consistency in 

interpretation across participating sites, and data 

collection and management procedures should be 

designed to support the collection of high quality 

data. Other chapters in this document discuss these 

concepts in more detail as they apply to registries 

generally. However, while many of the best 

practices described elsewhere in this document are 

applicable to rare disease registries, rare disease 

registries face unique data collection challenges 

not addressed by those best practices. In particular, 

rare disease registries may encounter additional 

hurdles when attempting to use common data 

elements, selecting quality of life or patientreported outcome measures, collecting biomarkers, 

obtaining long-term followup data, and assuring 

data quality.

3.3 Creating Efficiencies in Registry 

Development

A major step in the development of any registry is 

the selection of the data elements. This task can be 

time consuming and resource intensive, 

particularly when multiple stakeholders are 

involved in defining the data set. As noted in 

Chapter 4, the primary goals in selecting data 

elements are to ensure that the necessary data are 

collected to achieve the objectives of the study and 

that the data set is not so overly burdensome as to 

limit participation in the registry. A critical 

component of developing the data set is defining 

the data elements and determining how each piece 

of data will be collected. Many registries develop 

and define their own data elements. This approach 

can be costly, and it limits the ability of data from 

the registry to be linked or compared with data 

from other registries or data sources. It is more 

challenging to standardize the data collection for 

rare diseases, as the understanding of the disease is 
120

Section V. Special Applications in Patient Registries

likely to be limited and, until recently, established 

standardization efforts were limited.

Common data elements (CDEs) may offer a 

potential solution to some of these issues. A CDE 

can be defined as “a data element that can be 

consistently collected across all clinical studies.”10

CDEs include standard definitions, code lists, and 

instructions so that the data are collected and 

stored in the same manner by each participating 

site, in each study. CDEs may be general, meaning 

they can be used across disease or therapeutic 

areas (e.g., demographics, vital signs)11 or diseasespecific, meaning they are designed for research in 

a particular disease area (e.g., congenital muscular 

dystrophy).12 By using CDEs, registries may be 

able to reduce the time and effort involved in 

developing a dataset and to enable the registry data 

to be linked or compared with data from other 

studies using the same CDEs. 

CDEs are particularly important for rare disease 

research. CDEs may lower the cost of developing a 

new registry, making registries more accessible for 

diseases where funding is limited. CDEs may also 

enable data from multiple small registry projects to 

be linked or compared to increase knowledge 

about the disease. The Institute of Medicine noted 

the potential importance of CDEs for rare disease 

research. In the 2010 report, Rare Diseases and 

Orphan Products: Accelerating Research and 

Development, the IOM stated, “The NIH [National 

Institutes of Health] should support a collaborative 

public-private partnership to develop and manage a 

freely available platform for creating or 

restructuring patient registries and biorepositories 

for rare diseases and for sharing de-identified data. 

The platform should include mechanisms to create 

standards for data collection, specimen storage, 

and informed consent by patients or research 

participants.”1 Recognizing the potential value of 

CDEs, NIH recently funded the PRISM project. 

The objective of the PRISM project, which is 

administered through the National Library of 

Medicine and supported by the Office of Rare 

Disease Research (ORDR), is to develop a library 

of standardized questions that will be relevant to a 

broad mix of rare diseases and that can be used to 

develop new registries or to update existing 

registries. Ultimately, the project aims to develop 

tools that will support the rapid implementation of 

new rare disease registries, the revision of existing 

registries, and interoperability between rare 

disease registries and other data sources.5, 13

In January 2010, NIH and ORDR hosted a 

workshop titled “Advancing Rare Disease 

Research: the Intersection of Patient Registries, 

Biospecimen Repositories, and Clinical Data,”14, 15

which launched the development of the Global 

Rare Disease Registry and Data Repository, a 

Web-based data registry that will link existing 

registries, future registries, and biorepositories.16

Two significant work results are expected out of 

this initiative. The first is a 2-year pilot program in 

collaboration with Patient Crossroads, Children’s 

Hospital of Philadelphia, and WebMD that will 

develop a Web-based template to allow any patient 

group to establish its own patient registry. The 

second work result is a preliminary set of 

recommended CDEs17 that has been drafted and 

has undergone revision based on feedback received 

from public comment. The CDEs are generally 

applicable to any rare disease registry. This CDE 

list mixes required and optional elements in the 

following categories: current contact information; 

sociodemographic information; diagnosis; family 

history; birth and reproductive history; 

anthropometric information; patient-reported 

outcome; medications, devices, and health 

services; clinical research participation and 

biospecimen donation; communication and 

preferences. ORDR is working closely with the 

Clinical Data Interchange Standards Consortium 

(CDISC), which has reviewed the ORDR CDEs18

and is in the process of a similar initiative focused 

on CDEs for clinical trials. Planned next steps 

include working with the rare disease community 

(including clinicians, patients, and advocacy 

groups) to develop CDEs for specific rare diseases. 

While CDEs have significant potential for rare 

disease registries, they do have some limitations. 

First, while general CDEs may be relevant for rare 

disease registries, these CDEs can typically only 

cover a small portion of a data set necessary for 

studying a rare disease. The currently available 

disease-specific CDEs tend to focus on prevalent 

diseases, such as cancer, cardiovascular disease, 

and neurological disorders. (Refer to Chapter 4 for 
121

Chapter 20. Rare Disease Registries

a table of currently available CDEs.) Some of 

these CDEs may be relevant for some rare disease 

registries, but many may not be useful. Second, 

CDEs may change over time to reflect changes in 

practice or new trends in clinical research. 

Registries with shorter durations (1 to 2 years) 

may not be affected by changes in CDEs, but rare 

disease registries are often designed to follow 

patients for long periods (e.g., several years or 

until death). To retain the benefits of linkage and 

comparison, registries that use CDEs would need 

to update their data collection tools to reflect the 

changes in CDEs. 

In selecting data elements for a rare disease 

registry, it is useful to consider using CDEs as a 

first step, before developing new data elements 

independently. Available CDEs may be relevant for 

basic information, such as demographics, and for 

some disease-specific information, and the use of 

these CDEs is encouraged, when possible. When 

CDEs are not available, a review of the literature 

and searches of ClinicalTrials.gov and other 

similar databases may identify other registries or 

clinical studies in the disease area. Those studies 

may be willing to share information on their data 

sets, so that the new registry can either align with 

those data elements to support future linkages or 

comparisons or perhaps learn from issues that have 

come up in the other studies and apply that 

knowledge to the development of a new data set. 

As noted in Chapter 4, other considerations in 

selecting data elements include feasibility (Are the 

data elements routinely collected in clinical 

practice?), burden (What scope of data collection 

is desirable; what burden on participants is 

tolerable?), and geographic variations in 

terminology and practice (Do disease definitions 

differ? Are data collected the same way in all 

registry locations? Do terminologies vary by 

country?).

3.4 Including Quality of Life or PatientReported Outcome Measures

Quality of life measures and patient-reported 

outcome measures (PROs) are increasingly being 

used in registries to understand patient experiences 

and preferences. In rare disease research, quality 

of life data and PROs may be particularly 

important when well-defined, widely accepted 

clinical outcomes are not available. The 

progression and mechanisms of rare diseases are 

often not well understood, a situation that inhibits 

the identification of meaningful clinical outcomes, 

the development of new therapies, and the 

assessment of the effectiveness of existing 

therapies or disease management strategies. 

Quality of life measures and PROs may provide 

useful data to show that disease management 

strategies or treatments are effective at improving 

patient (and caregiver) outcomes or quality of life. 

The selection of quality of life measures or PROs 

for a rare disease registry can be challenging. 

Disease-specific measures are often not available, 

and generic measures that were developed with 

consideration to more common diseases (e.g., the 

SF-36) are not detailed enough to capture relevant 

changes in the patient’s (or caregiver’s) quality of 

life. New measures may be difficult and expensive 

to develop, given the small number of patients, 

validation requirements, and the need to have 

measures that can be used in multiple languages. 

As with the selection of data elements, registries 

may seek to identify other existing studies in the 

disease area and use similar measures to allow for 

future comparisons. When selecting measures, 

burden on the participant is a major consideration. 

The inclusion of multiple quality of life measures 

and PROs can be tempting, but they may deter 

patient participation if the burden is excessive. 

Considerations for selecting measures, collecting 

the data, and analyzing and interpreting the data 

are further discussed in Chapter 5. 

In addition to utilizing a PRO tool, a registry may 

be used to validate one or more PRO instruments 

across a large number of centers and in some cases 

countries. Nonvalidated tools, such as patient 

diaries and other electronic or paper-based data 

collection methods (e.g., treatment logs to track 

compliance, symptom checklists), may also be 

integrated across the registry or within a 

subpopulation of interest.

Health economic data may also play an important 

role in a rare disease registry. While the major 

goals of the registry may be to improve 

understanding of the disease or to monitor 

treatments, an underlying objective may be to 
122

Section V. Special Applications in Patient Registries

build the case for new research in the disease area 

and the development of new therapies. Health 

economic data may be useful for more broadly 

demonstrating the global burden of disease. 

Because of the rarity of the disease, the burden 

may not be well understood, and global burden of 

disease data may be used to gain support for 

funding new research in the disease area. In 

addition, if therapies are developed for rare 

diseases that were once only treated with 

supportive care, some aspects of disease burden 

may improve, while other considerations, such as 

long-term disease management, may be 

introduced.

3.5 Biomarkers

Biomarkers, which may describe risk, exposures, 

intermediate effects of treatment, and biologic 

mechanisms, are an important component of rare 

disease research and may serve as important 

surrogate endpoints for health outcomes.19 In their 

report on accelerating rare disease research, the 

IOM identified biomarkers as an important avenue. 

When biomarkers have been identified for a rare 

disease, registries in that disease area should 

consider collecting biomarker data as part of the 

registry. Registries in disease areas for which 

biomarkers have not been identified may also 

consider collecting biological specimens, 

physiological tests, or radiographic studies, in the 

hope of furthering efforts to develop and validate 

biomarkers.1 

When determining whether to collect biomarkers 

or other biological specimens, registries must 

consider several factors, addressing the following 

questions:

• Does the biosample need to be collected once 

or on an ongoing basis? If ongoing, how often?

• Does the biosample need to be examined by a 

central laboratory? Will multiple laboratories 

be needed because of geographic constraints 

(e.g., the fact that samples collected in the 

European Union cannot be sent to a laboratory 

in the United States) or time constraints (e.g., 

that sample processing is required within 24 

hours)?

• What privacy and ethical issues will the 

collection and storage of biosamples introduce? 

• How will the informed consent document 

discuss the collection and storage of 

biosamples? How broad can the language be? 

Can it address unanticipated use of these 

samples?

• How long will the biosample data be stored? 

For what research purposes may it be used?

As more is learned about a rare disease and its 

origins, the ability to perform exploratory analyses 

on existing samples is critical and should be 

considered when a biorepository is established. 

3.6 Collection of Followup Data

Collection of long-term followup data is often an 

important component of debilitating but not fatal 

rare disease registries. Many rare diseases are 

chronic and lifelong, meaning that registries may 

wish to track patients for several years or even 

until the patient’s death. The collection of longterm followup data for rare diseases raises some 

unique challenges, including what type of 

providers should participate (specialist vs. general 

clinician), how to encourage retention, and 

minimizing lost-to-followup patients.

Many patients with rare diseases see a specialist in 

the disease area on a regular but infrequent basis 

(e.g., annually) and see other clinicians on a more 

regular basis. The specialist may see several 

patients with the same disease and may have 

specialized knowledge of the disease; in that sense, 

the specialist may be an ideal candidate for 

registry participation. However, participation by 

the specialist may result in infrequent data 

collection on the individual patients or missing 

data collected by other clinicians. The registry 

may miss events that occur between specialist 

visits and may not obtain an accurate picture of the 

day-to-day impact of the disease. Participation by 

the clinician (or clinicians) that treats the patient 

on a regular basis is another option. These 

clinicians may only see one or two patients with 

the disease and they may not have specialized 

knowledge of the condition; however, they may be 

able to provide more frequent updates on the 

patient’s condition. Both of these approaches have 
123

Chapter 20. Rare Disease Registries

strengths and limitations. The most comprehensive 

approach is to collect data from both the clinician 

and the specialist. This allows the registry to 

gather both the specialist’s overall perspective on 

the patient’s condition and the more granular 

details of the patient’s care. However, this 

approach raises privacy issues, as the registry may 

need to share the data collected from the specialist 

with the clinician and vice versa. It may also 

present recruitment issues, as both providers must 

participate in order to avoid significant missing 

data. The registry will also need to plan for both 

physicians to participate in order to avoid having 

duplicate patients entered into the registry.

Because the collection of long-term data is often 

critical to the registry’s objectives, the registry 

must devote sufficient effort to patient and 

physician retention. Over time, patients and 

physicians may lose interest in the registry and 

stop participating. Patients who enrolled in the 

registry as minors may change physicians and 

locations on multiple occasions and upon reaching 

adulthood. Patients may move and begin seeing a 

new physician, or physicians may retire and stop 

participating. Direct input and access to registry 

data by affected individuals is technically possible 

and would allow for ongoing registry involvement 

and foster retention of patients. It has been 

observed that patient/family-entered data is 

reliable information, but that it may not be as 

in-depth or conveyed in medical language as is the 

information provided by a physician. Depending 

on the objectives of the registry, one approach 

would be to facilitate patient/family entry of data 

to the patient registry with better design of data 

entry forms/screens, appropriate contact and 

followup with participants, and sharing of study 

results and summary data from the patient registry 

with participants.

Chapter 10, Section 3 describes many factors that 

can encourage retention, as well as some potential 

pitfalls that may impede it. Rare disease registries 

with long-term followup components should have 

plans in place to monitor retention rates and 

should have consistent procedures for when to 

consider patients lost to followup. Registries that 

collect sufficient identifiers may also consider 

linking to other data sources, such as the National 

Death Index in the United States, to determine if 

patients have died. Procedures used to locate 

missing registry participants should be articulated 

in policies and procedures documents and 

communicated to participants. In addition, 

processes to retain patients as their status changes 

(e.g., from treating clinician to clinician; from 

minor to adult) should be clearly stated, and 

multiple contact points should be available for 

both the individual and their next of kin/designated 

secondary contact.

3.7 Data Analysis

Patient registries are critical for accruing a 

sufficient sample size for epidemiological and/or 

clinical research for rare diseases. In most cases, 

the registries are not statistically powered for 

hypothesis testing, and the analytic goals should 

focus on the descriptive techniques relevant to 

observational research. The uptake of common 

data elements could facilitate the formation of 

analysis data sets from the combination of multiple 

data sources in situations in which two or more 

disease registries exist, thus increasing sample 

sizes.

Data collected in rare disease registries prior to the 

introduction of therapies that drastically alter 

standard of care and/or treatment guidelines can 

provide useful information regarding the natural 

history of disease. If these data do not exist, 

removing the effects of widespread treatment(s) 

from registry analysis on disease progression, 

particularly in highly heterogeneous disease, is 

challenging. An additional challenge in rare 

disease registries is the fragmentation of the data. 

Patients may contribute data sporadically, but not 

be completely lost to followup. If data are 

combined across registries (or other databases), 

care should be taken to identify potential duplicate 

patients prior to analysis, as this is more likely to 

occur in a limited population. Many of the other 

considerations for analysis (e.g., controlling for 

confounding, handling of missing data, loss to 

followup) are not unique to rare disease registry 

data and are addressed in other chapters, such as 

Chapters 2, 3, and 13. 
124

Section V. Special Applications in Patient Registries

3.8 Data Access and Communication

Because populations with rare diseases are often 

considered vulnerable and under-resourced, and 

because such populations are smaller, it is 

important to plan for registry data access and 

communications. Data ownership, data access, and 

communication are important issues for all 

registries, but rare disease registries often must pay 

special attention to these issues because of the 

broad range of stakeholders involved and the 

potential interest to others in the disease. 

Ownership of the data should be clearly specified 

during the planning phase of the registry and 

should be communicated to stakeholders and 

participants in the registry. In many cases, the data 

owner is the sponsor of the registry. Some 

registries, though, may have multiple sponsors, or 

the sponsor may designate that another group will 

own the data. In addition to ownership of the 

actual data, ownership of the intellectual property 

resulting from the registry (e.g., case report forms, 

patient-reported outcomes tools, reports, analyses, 

and associated biosamples) should be clearly 

specified.

Rare disease registries should also develop and 

adhere to a data access plan. Many data access 

scenarios are possible. For example, the registry 

data may only be accessible to the data owner or 

the sponsor. Alternately, the registry may develop 

data sharing policies that allow other researchers 

to access the data. The registry may limit data 

sharing to investigators participating in the 

registry, or may allow outside investigators to 

access the data. For example, an outside researcher 

may use data from the registry to assess incidence 

of a particular complication for the purposes of 

informing protocol development for a new study. 

As discussed in Chapter 2, data sharing policies 

should address who can access the data, for what 

purposes, and under what circumstances 

(timeframes, access fees, etc.). For example, will 

manufacturers be able to access data to inform the 

design of new clinical trials? Will researchers be 

able to link the data to other data sources for new 

studies? These types of questions should be 

carefully considered and addressed in data sharing 

policies so that all participants (including patients) 

are aware of the policies and plans for the registry 

data. Written registry policies and procedures are 

encouraged and required by many regulatory 

entities (e.g., institutional review boards).

Publication rights and plans for disseminating 

information from the registry should also be 

considered during the planning phase. This is 

particularly important for rare disease registries 

with multiple stakeholders, who may have diverse 

and conflicting interests. There may also be 

considerations about academic and other interested 

parties not included in the registry who may wish 

to use these data at some later point in time. For 

example, a registry with strong patient advocacy 

group support and industry funding may need to 

balance the desire to publish early (in order to 

share information with the patient community) 

with the desire to publish later (in order to protect 

proprietary information related to treatment 

development). Clear publication plans that are 

shared with registry stakeholders can help to avoid 

disagreements once the registry has begun 

collecting data, and can promote registry 

transparency. 

Like all registries, transparency is important for 

rare disease registries, and perhaps especially 

important.1 Because rare diseases have a limited 

pool of patients, they need to maintain a highly 

motivated patient community engaged and actively 

participating in the registry. Transparency in 

registry operations, analyses, and publications can 

help to reassure participants that the registry is 

fulfilling its objectives and continues to be a 

worthwhile endeavor. This may take the form of 

regular updates on registry enrollment and data 

collection, newsletter updates from principal 

investigators, and information sharing about 

abstracts or publications based on registry data. 

Transparency also requires full disclosure to 

participants about the use of their data, the registry 

funding sources, and any underlying goals or 

motivations for the registry. 

3.9 Governance 

The governance of a rare disease registry can be 

extraordinarily simple or, more often, fairly 

complicated. Complexity stems from the variety 

of stakeholders involved and their different 

agendas, as well as the geographical and cultural 
125

Chapter 20. Rare Disease Registries

distances between the interested parties, 

particularly for international registries. Simplicity 

depends upon having clear goals and adept 

leadership. Registry leadership may be in the form 

of an Advisory Board or other leadership 

committee. Advisory boards and general 

governance principles are discussed in Chapter 2, 

Section 2.6. 

Rare disease registries present unique governance 

challenges because they often represent 

collaborations with many stakeholders and may be 

international in scope. Some examples are 

included below.

• Funding: If not centrally funded, who takes 

responsibility for raising money, writing grants, 

and securing the necessary funds for the 

operation of the registry? If fully funded, what 

role in decisionmaking does the financial 

sponsor(s) have? What are the startup costs 

versus maintenance costs? 

• Privacy: Patients with rare diseases are more 

vulnerable than most to being identified by 

their health information. How does the registry 

protect the privacy of the individuals and 

families while at the same time creating a 

database of information and resources for the 

benefit of all persons having the rare disease of 

interest? 

• Outreach: How does the registry identify 

affected and interested persons for participation 

in the registry? 

• Information: What database is needed for the 

registry? What demographic, clinical, and/or 

longitudinal databases are needed? How will 

the registry adapt as new data needs are 

identified? 

• Ownership: Who owns the information 

collected by the registry? For a tissue 

repository, who owns the materials in it?

• Agenda: Who sets priorities and establishes the 

work plan for the registry? What are the 

respective roles of the stakeholders in setting 

the agenda?

• Collaborations: Does the registry governing 

body create collaborations or can individual 

participants make collaborative agreements? 

What approvals are required? How are conflicts 

avoided and handled?

• Publications: Who takes responsibility for 

determining the publication plan, submitting 

abstracts, writing journal articles, and 

otherwise publishing about the activities of the 

registry? Who does the work? Who gets the 

credit? Are all of the professional and 

nonprofessional participants treated equally?

4. The Future of Rare Disease 

Registries

Existing literature suggests that rare diseases occur 

infrequently and there is a scarcity of information; 

however, in reality, difficulty in correct diagnosis 

and appropriate identification of patients with rare 

diseases is a global issue that precludes knowledge 

of these patients. The lack of information reflects 

the uncertainties in diagnostic criteria and perhaps 

even inadequacies in data gathering procedures. As 

the patient community continues to grow 

throughout the world, fostered by electronic 

communication and social media, knowledge of 

the prevalence of rare diseases will increase and 

access to patients will be more readily available. 

Improved access to information on rare diseases 

continues to expand as rare diseases are addressed 

on a global basis and more people are aware of the 

informational needs of the rare diseases 

community.

The increasing interest in rare disease patient 

registries by a range of stakeholders will likely 

lead to the development of many more patient 

registries. In the absence of a central health care 

system with all demographic and clinical data in 

one place, individual registries for different 

diseases are likely, each with a smaller and smaller 

set of patients. Organizational collaboration and 

shared resources, plus engagement of the rare 

disease community, are needed to move research 

and knowledge forward. These registries may be 

used to identify new pathways for treatment, 

develop clinical research tools such as endpoints, 

scales, or outcome measures, and recruit potential 

participants for clinical trials. However, the 

development of each individual registry requires 

significant effort and resources. For some diseases, 
126

Section V. Special Applications in Patient Registries

well-organized private foundations or 

manufacturers with an interest in product 

development or monitoring are capable of 

developing effective registries. For many other 

diseases, there are few resources to support an 

independent registry. Some efforts are underway to 

develop linked networks of registries for rare 

diseases. For example, the NIH has put forth the 

idea of creating a federation of Internet-based 

registries for rare diseases. The goal of this effort 

is to reduce the costs of developing and running a 

registry. The idea was part of the discussion at a 

January 2010 conference on patient registries and 

rare diseases sponsored by the NIH’s ORDR.20 

The proliferation of registries and the need for 

global multidisciplinary cooperation for rare 

disease research creates an urgent need for 

standards and best practices for these types of 

patient registry projects. The large number of 

registries and the various purposes and 

stakeholders for each complicate any attempts to 

inventory, standardize, or prescribe good design 

features for patient registries in general. As 

previously described, ORDR, in collaboration with 

the rare diseases community, is working to 

establish the Global Rare Diseases (Patient) 

Registry and Data Repository to enable analyses of 

data across many rare diseases. ORDR has 

developed and posted for public use a set of 

general CDEs to be used for rare diseases; these 

have gained support at NIH, CDISC, and in the 

international community. At a recent meeting of 

the International Rare Diseases Research 

Consortium (IRDiRC) in Bethesda, the CDEs were 

accepted as a starting point for rare diseases. The 

ORDR also will make available to all patients/

patient advocacy groups a Web-based template to 

establish a patient registry with the ability to link 

patients’ data and medical information to their 

biospecimens. In addition, ORDR will encourage 

all individuals and organizations that elect to 

develop a patient registry to participate in the 

Registry of Patient Registries to increase public 

exposure to these vital research tools. The next 

stage is to develop organ/system–specific and 

disease-specific CDEs developed as collaborative 

efforts of patients, research investigators, industry 

clinicians, and other partners in the rare diseases 

community. ORDR continues to collaborate with 

NIH research institutes to identify existing rare 

disease patient registries and use the common data 

elements.

Because of the increase in the number of registries, 

more efficient ways to implement and maintain 

rare disease registries and maximize utility for all 

stakeholders will be required. Technological 

advances, such as means for integrating data 

sources, should result in processes that are more 

streamlined for the data provider as well as the 

analyst. The growth of Web-based patient 

communities and social media may also be 

increasingly integrated into registry data collection 

and conduct, as community building across 

geographical boundaries continues to become 

simpler. 
127

Chapter 20. Rare Disease Registries

Case Examples for Chapter 20

Case Example 46. Using registries to 

understand rare diseases

Description The International Collaborative 

Gaucher Group (ICGG) Gaucher 

Registry aims to enhance the 

understanding of the variability, 

progression, and natural history 

of Gaucher disease, with the 

ultimate goals of better guiding 

and assessing therapeutic 

intervention, and providing 

recommendations on patient care 

to the medical community that 

will improve the outcomes for 

patients affected by this disease 

around the world. 

Sponsor Genzyme, a Sanofi company, 

Cambridge, MA

Year Started 1991

Year Ended Ongoing

No. of Sites 700+ sites have enrolled patients

No. of Patients More than 6,500 with openended followup

Challenge

Rare diseases pose special and unique research 

challenges. The small number of affected patients 

often results in limited clinical experience within 

individual health care centers. Therefore, the 

clinical description of rare diseases may be 

incomplete or skewed. The medical literature 

often consists of individual case reports or small 

case series, limiting understanding of the natural 

history of rare diseases. Furthermore, randomized 

controlled trials with adequate sample size and 

length of followup to assess treatment outcomes 

may be extremely difficult or not feasible. The 

challenge is even greater for rare diseases that are 

chronic in nature, where long-term followup is 

especially important. As a result, rare diseases are 

often incompletely characterized and lack 

published data on symptomatology, disease 

manifestations, and long-term treatment 

outcomes.

Gaucher disease, a rare enzyme deficiency that 

affects fewer than 10,000 known patients 

worldwide, illustrates many of the challenges 

facing researchers involved in rare diseases. 

Gaucher disease has three clinical presentations: 

Type 1, non-neuronopathic; Type 2, acute 

neuronopathic; and Type 3, subacute 

neuronopathic. Physicians who encounter patients 

with Gaucher disease typically have just one or 

two affected patients in their practices; only a few 

physicians around the world have more than 10 to 

20 patients with Gaucher disease in their care. 

Understanding Gaucher disease is further 

complicated by the fact that it is a highly 

heterogeneous and rare disorder with variable 

progression among patients; a patient cohort from 

a single center may represent a subset of the 

entire spectrum of disease phenotypes.

The rarity and chronic nature of Gaucher disease 

also pose challenges in conducting clinical 

research. The clinical trial that led to U.S. Food 

and Drug Administration approval of enzyme 

replacement therapy (ERT) for Gaucher disease 

(Ceredase®, alglucerase for injection) in 1991 

was a single-arm, open-label study involving only 

12 patients followed for from 9 to 12 months. In 

1994, a recombinant form of ERT was approved 

(Cerezyme®, imiglucerase for injection) based on 

a randomized two-arm clinical trial comparing 

Ceredase and Cerezyme in 30 patients (15 in each 

arm) followed for 9 months.

Proposed Solution

Established in 1991, the registry is an ongoing, 

international, longitudinal disease registry, open 

to voluntary participation by physicians who care 

for patients with all subtypes of Gaucher disease, 

regardless of their treatment status or treatment 

type. Data on patient demographics; clinical 

characteristics; treatment regimen; and 

laboratory, radiologic, and quality-of-life 

outcome measures are entered and analyzed to 

address the research challenges of this rare 

disease. Because of the rarity of Gaucher disease, 

it is important to create and maintain a reliable, 

comprehensive registry that serves as an 
128

Section V. Special Applications in Patient Registries

Case Example 46. Using registries to 

understand rare diseases (continued)

Proposed Solution (continued)

educational resource not only for physicians but 

also for patients and their families and caregivers. 

Responsibility for the use, integrity, and 

objectivity of the data and analyses is invested in 

the ICGG Board of Advisors, which consists of 

physician-investigators worldwide who are not 

employees of the sponsor and who advise on the 

medical and scientific agendas of the registry. 

Results

The registry has longitudinal data on more than 

6,500 patients from more than 700 health care 

centers in more than 60 countries. The followup 

period is open-ended, and the registry currently 

has up to 20 years of followup data from 

individual patients. The registry has collected 

more than 50,000 patient-years of followup 

during the past 21 years. Physician participation 

and patient enrollment have increased 

consistently from year to year since 1991.

Analyses of the extensive body of longitudinal 

data have increased knowledge of the disease in a 

broad range of topics, including the natural 

history of Gaucher disease; phenotypic and 

genotypic variation among patients; diagnosis, 

treatment, and management of the disease; 

disease manifestations in children; long-term 

treatment outcomes for ERT; bone disease and 

complications associated with the disease; and 

neuronopathic Gaucher disease. Data generated 

from the registry have been published in nearly 

30 key articles and have provided much needed 

and important insight into this rare genetic 

disease. 

In 2002, the registry published the clinical 

outcomes of 1,028 patients treated with ERT with 

up to 5 years of followup. As more data have 

been gathered through the registry over the past 

decade, long-term outcomes in patients with Type 

1 Gaucher disease after 10 years of ERT have 

become available, thus providing new reference 

benchmarks for assessing clinical responses to 

ERT for various disease parameters. Other more 

recent publications based on analyses of data 

from the registry have focused on important 

specific aspects of Gaucher disease, such as the 

effects of early intervention with ERT on the 

incidence of bone pathology, demographic and 

clinical characteristics of patients with 

neuronopathic Gaucher disease, ERT doseresponse relationships for disease parameters in 

patients with Gaucher disease type 1, and 

phenotypic heterogenicity and genetic variation 

among patients.

Along with the growth of the registry and the 

availability of data on Gaucher disease, interest in 

special patient populations and specific aspects of 

Gaucher disease continually emerge. As a result, 

research initiatives into disease subpopulations 

have been launched recently: the Neurological 

Outcomes Subregistry, which will begin to 

evaluate the neurologic manifestations of 

Gaucher disease and the effects of treatment on 

these complications; and the Pregnancy 

Subregistry, which will track the management of 

Gaucher disease during pregnancy as well as 

pregnancy outcomes.

The collective clinical experience of the registry 

led to the development of recommendations for 

evaluation and monitoring of patients with 

Gaucher disease. The analysis of registry data on 

treatment outcomes has facilitated the 

establishment of therapeutic goals for patients 

with Type 1 Gaucher disease. Together, these 

publications have formed the foundation for a 

consensus- and evidence-based disease 

management approach, something usually only 

possible for much more common diseases. In 

2008, a benchmark analysis was published that 

documented the achievement of therapeutic goals 

after 4 years of ERT among registry patients.

As disease awareness has increased over time, 

health care providers have sought more direct 

access to general and patient-specific disease 

information. Therefore, when the registry 

changed its technology platform in 2011, it 

established two key objectives: to simplify data 

entry to help keep data complete and accurate, 

and to support the community’s increased interest 

in access to data, aggregate reports, and 

collaborative expertise. To help meet these goals, 
129

Chapter 20. Rare Disease Registries

Case Example 46. Using registries to 

understand rare diseases (continued)

Results (continued)

the registry ensured that the new platform 

included functionality that allows physicians 

direct access to aggregate and patient-specific 

reporting as well as the ability to download their 

own data to support their own research. This 

important application of technology enables the 

registry to “give back” supportive and research 

tools to those who contribute to the overall 

registry data set. This includes the availability of 

data to address clinical and scientific questions; 

useful disease management tools, such as 

interactive patient case reports that a physician 

can share with other health care providers and 

with patients themselves; and a larger, betterconnected, worldwide community of physicians 

and allied health providers who can share 

information, identify trends, improve best 

practices, and build awareness of Gaucher disease 

that will optimize patient outcomes. 

Key Point

For rare or ultra-rare conditions, an international, 

longitudinal disease registry may be the best or 

only feasible way to comprehensively increase 

knowledge about the clinical characteristics and 

natural history of the disease and assess the 

long-term outcomes of treatment. 

For More Information

Weinreb NJ, Charrow J, Andersson HC, et al. 

Effectiveness of enzyme replacement therapy in 

1028 patients with type 1 Gaucher disease after 2 

to 5 years of treatment: a report from the Gaucher 

Registry. Am J Med. 2002;113(2):112-9.

Vom Dahl S, Weinreb N, Charrow J, et al. Longterm Clinical Outcomes in Type 1 Gaucher 

Following 10 Years of Treatment with 

Imiglucerase. Presented at the 2011 Workshop of 

the European Study Group on Lysosomal Disease 

(ESGLD), September 3-6, 2011; Langvik, 

Finland.

Mistry PK, Deegan P, Vellodi A, et al. Timing of 

initiation of enzyme replacement therapy after 

diagnosis of type 1 Gaucher disease: effect on 

incidence of avascular necrosis. Br J Haematol. 

2009;147(4):561-70.

Tylki-Szymanska A, Vellodi A, El-Beshlawy A, et 

al. Neuronopathic Gaucher disease: demographic 

and clinical features of 131 patients enrolled in 

the International Collaborative Gaucher Group 

Neurological Outcomes Subregistry. J Inherit 

Metab Dis. 2010;33(4):339-46.

Fairley C, Zimran A, Phillips M, et al. Phenotypic 

heterogeneity of N370S homozygotes with type I 

Gaucher disease: an analysis of 798 patients from 

the ICGG Gaucher Registry. J Inherit Metab Dis. 

2008;31(6):738-44.
130

Section V. Special Applications in Patient Registries

Case Example 47. Studying rare diseases in 

an existing registry population

Description The National Cooperative 

Growth Study (NCGS) collected 

data on children with growth 

disorders who were treated with 

a specific growth hormone (GH). 

The purpose of the multicenter, 

observational, post-marketing 

surveillance registry was to 

collect long-term safety and 

effectiveness information on the 

GH preparations, with the goals 

of better understanding the 

growth response to GH therapy 

and establishing a safety profile 

in large populations of different 

patient diagnostic groups.

Sponsor Genentech, Inc.

Year Started 1985

Year Ended 2010

No. of Sites More than 550

No. of Patients 65,205

Challenge

The registry was launched following U.S. Food 

and Drug Administration approval in 1985 of 

recombinant human growth hormone (rhGH). 

While the primary purpose of the registry was to 

monitor the safety and effectiveness of rhGH in 

all pediatric patients undergoing this treatment, 

there were insufficient numbers of patients in the 

clinical trials in each subgroup of patients for 

whom the drug was indicated to establish a true 

picture of their medical risks and the interaction 

or impact of GH on their medical safety. There 

was particular interest in studying girls with 

Turner syndrome (TS), a rare chromosomal 

abnormality known to be commonly associated 

with multiple medical conditions. 

Proposed Solution

Pediatric patients with growth disorders were 

voluntarily enrolled in the registry when therapy 

with rhGH was initiated, and were followed until 

discontinuation of therapy. The median length of 

followup for patients in the registry is 3.3 years, 

allowing for longitudinal analyses of the natural 

history of growth disorders and their treatment, 

and addressing physician queries on the longterm safety and effectiveness of rhGH therapy for 

their patients. The broad enrollment criteria of the 

registry enabled capture of a meaningful sample 

of patients with rare syndromes or diseases. For 

example, the registry population included over 

5,000 patients with TS. 

Results

For 25 years, the registry monitored the safety 

and efficacy of rhGH therapy in 65,205 children 

with growth disorders treated in more than 550 

sites in the United States and Canada, with more 

than 800 investigators, and accrued over 220,000 

patient-years of observation. During this time, 

analyses resulted in more than 100 publications 

on safety, dosing, height prediction, outcomes, 

subgroups of patients, and regulatory safety 

assessments, with more than 1,200 citations in 

the research literature. The registry remains the 

largest North American repository for auxological 

and clinical outcome data for rhGH-treated 

children with growth-related disorders. 

Analyses were conducted on 5,220 registry 

patients with TS, resulting in a seminal paper that 

described the safety profile of GH in this 

condition and highlighted the natural history of 

many of the known medical conditions these 

patients have. The safety profile included 

assessment of cardiac risks, development of 

autoimmune disorders, and detected occurrence 

of a disproportionate number of cases of 

pancreatitis, compared with the other patient 

groups in the NCGS. This later finding 

contributed to a recent label change warning of 

the risk of pancreatitis for all GH products, 

including a reference to the published data that 

suggest higher risk in patients with TS. In 

addition, a substudy in the registry assessed the 

degree to which pediatric endocrinologists were 

following recent guidelines for screening of 

concurrent medical conditions in TS. This 

guideline substudy revealed that, in a cohort of 

955 girls, screening for cardiac, renal, and 

hearing abnormalities was not occurring at the 
131

Chapter 20. Rare Disease Registries

Case Example 47. Studying rare diseases in 

an existing registry population (continued)

Results (continued)

expected rate. The clinical implications of these 

studies were a broader picture of the natural 

health history of girls with TS as well as specific 

issues of safety with respect to GH. 

The registry closed enrollment in 2010, but the 

database continues to be a resource for practicing 

physicians facing patient treatment decisions, 

averaging 1–2 queries per week from former 

investigators on the safety and effectiveness of 

GH treatment. Recent examples of query topics 

include patients with TS, safety in patients with 

intractable seizures, medulloblastoma and 

secondary malignancies associated with GH, 

primary pulmonary hypertension, and other 

conditions. One query, on rhGH use in GHdeficient or idiopathic short stature patients 

treated with stimulants for ADHD, led to a 

publication that found no significant differences 

in safety or effectiveness of rhGH treatment for 

these patients in comparison to non-ADHD 

treated patients.

Key Point

A large registry can provide a resource of study 

subjects for focused investigations on specific 

rare diseases. Even after study closure, registry 

data can be a useful resource for continued 

investigations, and for informing treatment in 

clinical practice.

For More Information

Allen DB, Julius JR, Breen TJ, et al. Treatment of 

glucocorticoid-induced growth suppression with 

growth hormone. National Cooperative Growth 

Study. J Clin Endocrinol Metab. Aug 1998;

83(8):2824-9.

Bell J, Parker KL, Swinford RD, et al. Long-term 

safety of recombinant human growth hormone in 

children. J Clin Endocrinol Metab. Jan 

2010;95(1):167-77

Bolar K, Hoffman AR, Maneatis T, et al. Longterm safety of recombinant human growth 

hormone in Turner syndrome. J Clin Endocrinol 

Metab. 2008;93(2):344-51.

Frindik JP, Morales A, Fowlkes J, et al. Stimulant 

medication use and response to growth hormone 

therapy: An NCGS database analysis. Horm Res. 

2009;72:160-6.

Parker KL, Wyatt DT, Blethen SL, et al. 

Screening girls with Turner syndrome: the 

National Cooperative Growth Study experience. 

J Pediatr. Jul 2003;143(1):133-5.

Romano AA, Dana K, Bakker B, et al. Growth 

response, near-adult height, and patterns of 

growth and puberty in patients with Noonan 

syndrome treated with growth hormone. J Clin 

Endocrinol Metab. Jul 2009;94(7):2338-44.
132

Section V. Special Applications in Patient Registries

Case Example 48. Site motivation and 

retention in rare disease registries

Description The Digital Ulcers Outcome 

(DUO) Registry collects data on 

patients with systemic sclerosis 

in an effort to describe digital 

ulcers disease history, clinical 

and patient-reported functional 

assessment status and treatment 

pattern at the time of enrollment, 

disease course, and patient 

management during followup. 

For patients treated with 

bosentan, data are collected on 

physician adherence to labeling 

guidelines and safety events.

Sponsor Actelion Pharmaceuticals

Year Started 2008

Year Ended Ongoing

No. of Sites More than 350

No. of Patients 3,609

Challenge

Systemic sclerosis (scleroderma) is a rare disease 

affecting less than 2 in 10,000 persons 

worldwide. Digital ulcers affect nearly 30 

percent of patients with this disease, resulting in 

substantial morbidity such as gangrene and 

amputation. Despite the severity of digital ulcers, 

very little is known about this complication, due 

to the rarity of the underlying condition. To 

improve understanding of this condition, data are 

needed from specialized participating 

scleroderma centers (sites).

The DUO Registry was mandated by the 

European Medicines Agency (EMA) as a postapproval licensing requirement for the expanded 

indication of bosentan to treat digital ulcers. The 

registry, which operates in 18 European countries, 

is observational and voluntary in nature, and 

participating sites are reimbursed solely for data 

entry time. After 4 years of operation, motivation 

of the participating physicians started to stagnate, 

and the sponsor observed a decrease in followup 

data entry. Because the registry was mandated by 

the EMA and because of the paucity of outcomes 

data available about digital ulcers, the sponsor 

sought to increase participation and, in particular, 

increase the collection of followup data. 

Proposed Solution

The sponsor identified academic and professional 

interest in the registry’s findings around digital 

ulcer management as one of the primary 

motivators for investigators to participate in the 

study. To respond to this interest, the sponsor 

implemented new efforts to engage investigators 

and regularly inform them of the study’s progress. 

Efforts included the use of newsletters to provide 

enrollment updates and tips on using the 

electronic data capture system, presentation of 

abstracts in scientific congresses, and distribution 

of letters from the registry steering committee to 

the investigators encouraging them to enter 

followup data. An in-person investigator meeting 

was held, where the registry’s scientific 

committee discussed registry findings with the 

investigators. The sponsor also established a 

process for investigators to suggest publication 

ideas and determined that all publications of 

registry data would include the phrase “and DUO 

Registry investigators” in the author byline.

Results

From 2011 to 2012, the proportion of patients 

having at least one followup visit increased from 

63 percent to 73 percent, and the mean number of 

visits per patient increased from 1.7 to 2.6. The 

registry has now enrolled more than 3,600 

patients. 

The steering committee and registry investigators 

published the first original article on registry data 

in January 2012, reporting on data from 2,439 

patients and confirming the disease burden of 

digital ulcers in systemic sclerosis patients. The 

sponsor had received feedback that the shared 

authorship is valuable to some investigators and a 

major motivating factor for their participation in 

the registry.

A poster highlighting the current variation in 

treatment of digital ulcers across Europe and 

stressing the need for a concentrated approach to 

establish disease management practices was 

presented at the European League Against 

Rheumatism 2012 Annual Congress.
133

Chapter 20. Rare Disease Registries

References for Chapter 20

1. Institute of Medicine. Rare diseases and orphan 

products: Accelerating research and development. 

Washington, DC: The National Academies Press, 

2010. http://books.nap.edu/openbook.php?record_

id=12953. Accessed August 17, 2012.

2. Moliner AM. Creating a European Union 

framework for actions in the field of rare diseases. 

Adv Exp Med Biol. 2010;686:457-73. 

PMID: 20824460.

3. Dunkle M, Pines W, Saltonstall PL. Advocacy 

groups and their role in rare diseases research. 

Adv Exp Med Biol. 2010;686:515-25. 

PMID: 20824463.

4. Hollak CE, Aerts JM, Ayme S, et al. Limitations 

of drug registries to evaluate orphan medicinal 

products for the treatment of lysosomal storage 

disorders. Orphanet J Rare Dis. 2011;6:16. 

PMID: 21496291. PMCID: 3102605.

5. Richesson R, Shereff D, Andrews J. [RD] PRISM 

library: patient registry item specifications and 

metadata for rare diseases. J Libr Metadata. 2010 

Apr 1;10(2-3):119-35. PMID: 21057650. 

PMCID: 2967796.

6. Bushby K, Lynn S, Straub T, et al. Collaborating 

to bring new therapies to the patient—the TREATNMD model. Acta Myol. 2009 Jul;28(1):12-5. 

PMID: 19772190. PMCID: 2859629.

7. Weinreich SS, Mangon R, Sikkens JJ, et al. 

[Orphanet: a European database for rare diseases]. 

Ned Tijdschr Geneeskd. 2008 Mar 1;152(9):

518-9. PMID: 18389888. Orphanet: een Europese 

database over zeldzame ziekten.

8. Developing a Registry of Patient Registries 

(RoPR). Project Abstract. Agency for Healthcare 

Research and Quality. http://www.

effectivehealthcare.ahrq.gov/index.cfm/search-forguides-reviews-and-reports/?productid=690&page

action=displayproduct. Accessed August 14, 2012.

9. Jones S, James E, Prasad S. Disease registries and 

outcomes research in children: focus on lysosomal 

storage disorders. Paediatr Drugs. 2011 Feb 1;

13(1):33-47.

10. National Institute of Neurological Disorders and 

Stroke. NINDS Common Data Elements – 

Glossary. http://www.commondataelements.ninds.

nih.gov/Glossary.aspx. Accessed August 17, 2012.

11. National Institute of Neurological Disorders and 

Stroke. General Data Standards. http://www.

commondataelements.ninds.nih.gov/General.aspx. 

Accessed August 17, 2012.

12. National Institute of Neurological Disorders and 

Stroke. Congenital Muscular Dystrophy (CMD). 

http://www.commondataelements.ninds.nih.gov/

CMD.aspx#tab=Data_Standards. Accessed August 

20, 2012.

13. PRISM. http://prism.epi.usf.edu/index.htm. 

Accessed August 17, 2012.

Case Example 48. Site motivation and 

retention in rare disease registries (continued)

Key Point

Site engagement is particularly important in rare 

disease registries, because of the limited number 

of patients. Consistent communication 

highlighting study objectives and achievements, 

visibility at scientific meetings, and sharing 

authorship or acknowledgement on publications 

can bolster investigators’ motivation. 

For More Information

Denton CP, Krieg T, Guillevain L, et al. 

Demographic, clinical and antibody 

characteristics of patients with digital ulcers in 

systemlic sclerosis: data from the DUO registry. 

Ann Rheum Dis. 2012;71:718-21.

Matucci-Cerinic M, Guillevin L, Denton CP, et al. 

Management of digital ulcer disease varies across 

Europe: findings from the DUO registry. Berlin, 

Germany. Poster presented at The European 

League Against Rheumatism 2012 Annual 

Congress, 6-9 June 2012. https://www.duoregistry.com/ACT7001/

(S(pgk5tifal0a3yw45f3wrde55))/DUO/

documents/EULAR%202012%20DUO%20

poster%2029052012.pdf Accessed August 8, 

2012.
134

Section V. Special Applications in Patient Registries

14. Forrest CB, Bartek RJ, Rubinstein Y, et al. The 

case for a global rare-diseases registry. Lancet. 

2011 Mar 26;377(9771):1057-9. 

PMID: 20674966.

15. Rubinstein YR, Groft SC, Bartek R, et al. Creating 

a global rare disease patient registry linked to a 

rare diseases biorepository database: Rare 

Disease-HUB (RD-HUB). Contemp Clin Trials. 

2010 Sep;31(5):394-404. 

PMID: 20609392. PMCID: 2930109.

16. National Institutes of Health. Office of Rare 

Diseases Research. Global Rare Diseases Patient 

Registry and Data Repository-GRDR. Common 

Data Elements (CDEs). http://rarediseases.info.

nih.gov/files/Common_Data_Elements.pdf. 

Accessed September 30, 2013.

17. National Institutes of Health. Office of Rare 

Disease Research. List of CDEs. http://

rarediseases.info.nih.gov/files/List_CDEs.pdf. 

Accessed August 17, 2012.

18. Clinical Data Interchange Standards Consortium 

(CDISC). Related Standards and Innovations. 

http://www.cdisc.org/content2897. Accessed 

August 17, 2012.

19. Institute of Medicine. Evaluation of Biomarkers 

and Surrogate Endpoints in Chronic Disease. 

Washington, DC: The National Academies Press. 

2010. http://www.iom.edu/Reports/2010/

Evaluation-of-Biomarkers-and-SurrogateEndpoints-in-Chronic-Disease.aspx. Accessed 

August 17, 2012.

20. National Institutes of Health. Advancing Rare 

Disease Research: The Intersection of Patient 

Registries, Biospecimen Repositories and Clinical 

Data. ORDR co-sponsored Scientific Conferences. 

http://rarediseases.info.nih.gov/

ScientificConferences.aspx?ID=1021. Accessed 

August 17, 2012.
135

Chapter 21. Pregnancy Registries

1. Introduction

A pregnancy exposure registry is an observational 

prospective cohort of women receiving a 

biopharmaceutical product(s) of interest as part of 

their routine clinical care who are enrolled 

voluntarily during gestation, before outcomes can 

be known. Participants are followed until the end 

of pregnancy or longer to systematically collect 

information on specific pregnancy outcomes and 

evaluate their frequency relative to a scientifically 

valid reference population(s).1-3 Specific examples 

of pregnancy registries can be found on the Food 

and Drug Administration (FDA) Web site.4

This chapter reviews the “why, how, and who” of 

conducting pregnancy registries. In it we first 

discuss why pregnancy registries are needed to 

assess risks and benefits of medications during 

pregnancy. Second, we describe the distinctive 

methodological aspects of these registries, 

including design, study population, enrollment and 

followup of pregnant women, ascertainment and 

definition of exposures and outcomes, reference 

groups, statistical power, and validity issues. Third, 

we take a more pragmatic approach and present 

key operational aspects such as protocol structure, 

recruitment and retention of participants, methods 

of data collection, timing of the release of findings, 

role of advisory boards, and challenges of global 

designs. Finally, we describe characteristics to 

consider when evaluating pregnancy registries. 

Case Examples 49, 50, 51, and 52 offer some 

descriptions of pregnancy registries.

2. Justification

All patients in need of treatment should have 

access to medications that have been adequately 

studied, and they should be given information to 

assess risks versus benefits of using the 

medication. Collecting postmarketing data on the 

safety of medications during pregnancy is 

commonly done through the use of pregnancy 

exposure registries or pregnancy disease registries 

that collect treatment information. Pregnancy 

registries are prospective observational studies 

specifically designed to collect clinically relevant 

data and provide information for treating or 

counseling not only women who are pregnant but 

also women of childbearing potential. In 2002, 

FDA published its guidance for pregnancy 

registries2 with a goal of encouraging the regular 

use of more formal, prospective study designs to 

obtain clinically relevant human data that can be 

used in product labeling. Similar guidelines were 

published by the European Medicines Agency in 

2005.5 In 2007, the Food and Drug Amendments 

Act provided the authority under Title IX6 to 

require pregnancy registries as a postmarketing 

requirement. Pregnancy registries are now required 

at the time of a new drug approval when there is a 

safety concern or when there is a need to gather 

data on the use of the product in pregnancy based 

on the following circumstances: (1) prior 

knowledge of the product suggests a safety 

concern based on the pharmacologic or chemical 

class, or on data from animal studies or clinical 

trials; (2) the product will be indicated for use 

during pregnancy (e.g., vaccines and medications 

for chronic illness); or (3) there is a high 

likelihood of use in females of reproductive age 

such that inadvertent exposure during pregnancy 

may be expected.

For nonpregnant individuals, safety and efficacy 

data that yield such information are derived from 

well-controlled clinical trials conducted prior to a 

drug’s approval. When it comes to pregnant 

women, however, the situation is different. Clinical 

trials rarely include pregnant women because there 

is a lack of safety information on the drug’s use in 

pregnancy.7, 8 As a consequence, most information 

regarding the safety/risk profile of drugs during 

pregnancy is collected after the drug has been 

approved and used by pregnant women 

intentionally or unintentionally: intentionally 

because some conditions require treatment during 
136

Section V. Special Applications in Patient Registries

pregnancy, or unintentionally because 

approximately half of all pregnancies in the United 

States are unplanned,9 so embryo/fetal exposure to 

medications can occur before pregnancy is 

detected. 

Tests in animal models are a regulatory 

requirement for new drugs and biologics prior to 

approval. In some cases, these animal toxicology 

studies can provide a means to detect teratogenic 

effects. Often, however, results are not easily 

translated into human risk because of variations in 

teratogenic response among species.10 In addition, 

animal toxicology studies are designed so that at 

least one dose tested will provoke an adverse toxic 

response. The results at those dose levels may not 

predict those that might be observed at the 

intended therapeutic doses used in humans.11

In humans, passive data collection such as FDA’s 

Adverse Event Reporting System (AERS) can 

suggest potential drug safety issues for further 

study. The system includes any patient population 

that may have been exposed to the drug. Reporting 

to the database is voluntary (although required for 

manufacturers) and underreporting is a significant 

issue, with extent of reporting thought to vary 

substantially depending on the drug, the indication 

for use, and the nature of the adverse event.12, 13

There is no reference group and no information is 

available on the number of individuals taking the 

drug who did not have an adverse event. It is often 

not clear whether the adverse event reported to 

AERS is a medication-related event, an event 

resulting from the underlying illness, or a 

coincidence. Since adverse pregnancy outcomes 

such as pregnancy losses and congenital 

malformations are relatively common, they will 

inevitably occur among exposed individuals; 

selected reporting of exposed cases can lead to 

false alarms. Consequently, although data from 

AERS can be useful for identifying initial signals 

of adverse events, it cannot be used to quantify 

risks for a particular product or to compare risks 

between drugs.14 Similarly, case series published 

in scientific journals cannot distinguish chance 

from causation or be used to quantify and assign 

teratogenic risks. 

Information on human teratogens must come from 

adequately controlled epidemiological studies, 

which include case-control and cohort designs. 

Case-control studies identify births with the 

outcome of interest (e.g., a specific birth defect) 

and compare their frequency of exposures to that 

in a control group without this outcome. This 

design offers advantages in detection and 

confirmation of associations between prenatal 

exposure to the medication and the risk for rare 

events.15 However, case-control studies have some 

limitations. They collect information on exposure 

retrospectively, rarely have enough sample size to 

evaluate infrequently used medications, and can 

estimate the relative risks but not the absolute risks 

associated with the drug. 

Followup studies of pregnant women have the 

advantage of identifying drug exposure before the 

adverse outcomes are recognized. In nonpregnant 

populations, health care utilization databases such 

as Medicaid claims files or records in large health 

maintenance organizations have become a standard 

source of information for drug safety studies.16

These databases are a resource for large-scale 

observational postmarketing studies because they 

offer the ability to study rare consequences of drug 

use. Some of these databases have limitations for 

the study of pregnancy outcomes because they do 

not routinely record evidence of pregnancy 

(e.g., estimated date of conception) or provide 

child-mother linkages. Important reproductive 

information, such as gestational age at birth, birth 

weight, and maternal reproductive history, is rarely 

available.17 Moreover, when exposure to the 

specific drug of interest involves a small fraction 

of the pregnant population, even these large 

cohorts are constrained in their statistical power. In 

this scenario, concentrating on women exposed to 

selected drugs through a pregnancy registry can 

increase efficiency. However, the FDA-funded 

Medication Exposure in Pregnancy Risk 

Evaluation Program has established collaboration 

among a selected group of large administrative 

databases with the ability to link mothers and 

babies and with linkage to birth certificates, which 

contain additional information.18 This resource 

will serve as an important tool to study outpatient 
137

Chapter 21. Pregnancy Registries

dispensing of medications during pregnancy and a 

number of validated pregnancy outcomes.

3. Pregnancy Registry 

Objectives

The overall purpose of pregnancy exposure 

registries is to provide human data on the safety of 

biopharmaceutical products during pregnancy.1, 19

Pregnancy registries should have specific primary 

and secondary objectives defined a priori in a 

scientifically sound study protocol.2 Many 

exposure registries have as their primary objective 

to “assess the risk of major congenital 

malformations” in the offspring of women exposed 

to a given drug just before or during pregnancy. 

Implicit in this objective is to determine whether 

that risk is higher or lower than expected. 

Registries can evaluate multiple maternal, 

obstetrical, fetal, and infant outcomes, from 

pregnancy complications to developmental delays. 

Moreover, they may provide an opportunity to 

evaluate not only the safety, but also the 

effectiveness of drugs, as well as the risks 

associated with untreated diseases during 

pregnancy. They can also evaluate the effects of 

dose and gestational timing of exposure, as well as 

effect modification by maternal characteristics.1

Since the ultimate goal is to inform the decisions 

of medical care providers and patients, it is in the 

common interest of all the parties to initiate the 

registry as soon as possible after marketing 

authorization, use proactive enrollment strategies 

(i.e., if possible, broaden the source population to 

obtain, for example, 1,000 exposed women in 1 

year rather than 100 per year for 10 years), and 

analyze the data and report findings on a regular 

basis.2 As more data accumulate over time, the 

registry can provide narrower boundaries of 

uncertainty around the point estimates, which 

leads to increasing assurance of relative safety or 

more precise quantification of relative risks. These 

issues are discussed more extensively in Sections 

17 and 17.9 below, respectively.

4. Design

Whether stated or not, one scientific question in 

the evaluation of drugs during pregnancy is often 

“What would have been the outcome of this 

pregnancy had the woman not been exposed?” 

Since the counterfactual outcome of a given 

pregnancy is unknown, the closest strategy to 

respond to this question would be to randomize a 

group of women periconceptionally either to the 

drug of interest or to a reference group and follow 

them in a blinded manner until the outcome of 

interest is fully assessed. This design would ensure 

that the groups differ only in their drug exposure 

and that the same methodology and clinical 

judgment are used to assess the outcomes. The 

reference group could be randomized to placebo, if 

not treating is clinically acceptable, or to an 

alternative therapy, and the comparative efficacy 

and safety of two or more therapeutic options 

could be evaluated. However, since for ethical 

reasons pregnant women are rarely included in 

sufficient numbers in randomized controlled trials 

(RCTs) during the process of drug development, 

the safety and effectiveness of drugs during 

pregnancy needs to be studied in postmarketing 

nonrandomized studies.3, 19

The specific design of a pregnancy registry is 

determined by its purpose.1 To rule out strong 

teratogenic effects (e.g., more than a 20 percent 

risk of malformations after prenatal exposure to 

thalidomide), enrollment of 100 exposed 

pregnancies in a simple uncontrolled cohort might 

suffice. The effect of major teratogens is so large 

as to overwhelm the potential impact of common 

methodological biases on relative risks. However, 

most known teratogens are associated with a more 

moderate increase in the risk of relatively rare 

malformations.20 To detect moderate teratogens, 

registries need to enroll a larger number of 

gestations and be carefully designed.19

Although the same rigor and most principles of 

RCTs can be applied to any observational study, 

the lack of randomization calls for additional 

epidemiological methods.21 The following 

sections discuss how pregnancy registries differ 

from RCTs; how deviation from the standards of 
138

Section V. Special Applications in Patient Registries

RCTs can compromise the validity of results; and 

how biases can be minimized through collection of 

detailed data on exposure and other maternal 

characteristics, close followup, accurate 

assessment of outcomes, and inclusion of 

comparable reference groups. 

5. Study Population: Who and 

When

Through clear inclusion and exclusion criteria, 

registries target a well-defined study population 

that, ideally, should be closer to real clinical 

practice than the selected populations of a RCT. 

However, to the extent that pregnancy registries 

rely on patients’ or their health care providers’ 

hearing about the registry, contacting the registry, 

and agreeing to participate, there is the potential 

for selective enrollment. Women who do 

participate may differ systematically from the 

population of exposed pregnant women who are 

not part of the registry, with respect to factors 

related to the pregnancy outcome. Therefore, 

baseline risks in the registry population may differ 

from those in the general population of women 

using the drug of interest.

In assessing the relative risks associated with the 

drug, there is often a tradeoff between 

generalizability and validity. For example, 

generalizability would be enhanced by use of a 

population-based sampling strategy in situations 

where the underlying population can be 

enumerated. However, inclusion of non-motivated 

individuals might increase losses to followup, 

misclassification of information, and the variability 

and impact of confounders, thus reducing validity. 

In addition, population-based sampling strategies 

can be logistically complicated and would arguably 

estimate a similar relative risk, since a teratogenic 

effect found in participants would probably apply 

to “non-volunteers” as well. 

A peculiarity of pregnancy registries is that the 

population can be defined based on women, 

pregnancies, or fetuses. A woman might have more 

than one pregnancy, and she might enroll in the 

same registry more than once. Clustered analyses 

are often used in this situation. In addition, 

multifetal gestations result in more than one fetus 

“enrolled” within the same pregnancy. Although 

there may be several ways of dealing with multiple 

gestations, it is prudent to collect information 

about all the fetuses. When reporting risks, 

whether using fetuses or pregnancies as the unit of 

analysis, both the numerator and denominator 

should be consistent with the choice.22

6. Enrollment and Followup

An ideal pregnancy cohort would enroll women at 

conception and follow them for months beyond 

delivery. However, this sequence rarely happens 

for logistical reasons, and consequently pregnancy 

cohorts have some degree of unintended truncation 

on both sides of the ideal followup. Left truncation 

occurs because followup can only start after 

women realize they are pregnant (in patientinitiated enrollment in registries) or health care 

providers identify the pregnancy in a patient (in 

clinician-initiated enrollment in registries), and the 

process of enrollment itself can further delay the 

inclusion. Right truncation occurs because 

followup would end with unknown outcomes when 

there are losses to followup or pregnancy 

terminations without fetal autopsy. 

As a result, time from enrollment to end of 

followup can range from 1 month to over 1 year. In 

any study, longer followup periods naturally lead 

to higher opportunities for diagnosis and therefore 

both larger cumulative risk estimates and greater 

statistical power. We will examine later how 

inclusion of either prenatal diagnoses or outcomes 

identified during infancy would result in higher 

risks than restriction to delivery hospital discharge 

diagnoses. More worrisome in pregnancy registries 

is that selection bias can be introduced if the 

outcome explicitly or implicitly affects enrollment 

(e.g., a known outcome affects eligibility or 

influences self-selection) or retention in the cohort 

(e.g., through exclusion of study subjects after an 

abortion or neonatal death). 

6.1 Enrollment

Registries should include women as soon as 

possible after conception, or even earlier at 

pregnancy planning stages, to allow the evaluation 

of early pregnancy events. For instance, 
139

Chapter 21. Pregnancy Registries

pregnancies enrolled earlier in gestation would 

result in higher risk estimates for spontaneous 

abortions and terminations than those enrolled 

later. Still, unless periodic pregnancy tests are 

conducted, studies will never pick up fetal losses 

that occur before pregnancy is known.

Women should be enrolled before the pregnancy 

outcome is known to avoid a selection into the 

study affected by the outcome. Retrospective 

enrollment of women after prenatal screening 

(i.e., nuchal translucency, chorionic villous 

sampling, amniocentesis, alpha fetoprotein 

measurements, and second-trimester ultrasound—

first-trimester dating ultrasounds do not assess 

malformations), can introduce bias towards a lower 

or higher risk of malformations. Bias may be 

introduced regardless of whether the prenatal 

screening test is normal or abnormal.1

Underestimation of the risk may occur if 

enrollment after informative screening tests are 

conducted selects a survivor cohort of women with 

uneventful pregnancies. For example, women 

might be less willing to contact a “pregnancy 

registry” after a major malformation diagnosis, 

clinicians enrolling patients might miss women 

who had a therapeutic abortion, and some 

registries do not allow enrollment of women with 

abnormal prenatal tests or pregnancy losses when 

an abnormality has been identified. On the other 

hand, overestimation of the risk might occur if 

participation is allowed after an abnormal test and 

there is a preferential enrollment of women with a 

diagnosis; for example, if the diagnosis prompts 

the exposed woman to look for information, find 

the registry, and enroll. These two scenarios can 

coexist and even occur differentially in unexposed 

and exposed women, leading to spurious 

associations. For example, unexposed women 

might be more willing to volunteer to be in a 

reference group if they are enjoying an uneventful 

gestation, while women with chronic conditions 

(e.g., rheumatoid arthritis) might contact a 

pregnancy registry after receiving an adverse 

pregnancy diagnosis, seeking both personal 

support and contact with peers with the same 

medical condition. Because this bias is difficult to 

identify and correct in the analytic phase, it needs 

to be prevented in the design by enrolling subjects 

prospectively before the outcome is known. 

6.2 Followup

Another peculiarity of pregnancy studies is that 

followup needs to go beyond the onset of the 

outcome under study. For most major structural 

congenital anomalies, the theoretical followup 

would be from conception to the end of fetal 

organogenesis (i.e., on average 3 gestational 

months). However, one often learns about the fetal 

outcomes only after birth. Longer followups will 

identify more congenital problems, since some 

structural and many functional malformations 

might become clinically apparent only months or 

years later.19, 23 In pregnancy registries, followup 

typically stops at 3 to 12 months after the end of 

pregnancy. Some registries restrict the cohort to 

infants with a minimal standard followup time, 

such as at least 3 months after birth. (Of note, fetal 

deaths and infant deaths before the minimal 

followup should also be included in the assessment 

to avoid excluding lethal malformations). Efforts 

should be made to minimize losses to followup 

and to obtain outcome information for all 

participants. Information needs to be collected on 

the number of losses and, if possible, on their 

reasons, in order to assess whether they are 

similarly distributed among the exposed and 

unexposed populations. 

7. Exposure Ascertainment

How registries collect information can affect the 

accuracy of the data. Some registries obtain 

information from the women themselves. Women 

generally know more about their habits, 

occupations, medical and obstetrical history, and 

compliance with the drug than individual health 

care providers. Other registries rely on reporting 

by the clinicians and have no contact with the 

patients. Clinicians can provide more complete 

and accurate information regarding diagnoses and 

indications.24 However, there is a risk of exposure 

misclassification if women stop or incompletely 

comply with the prescription drug regimen during 

pregnancy. A woman might be more willing to tell 

an interviewer not related to her care than to tell 
140

Section V. Special Applications in Patient Registries

her doctor that she decided to reduce her dose or 

quit taking a medication. Including unexposed 

subjects in the exposed group (i.e., false positives) 

can dilute any potential association. 

In order to maximize the quality of data, the 

combination of several sources of information is 

an optimal strategy. With adequate help, women 

can recall exposure during pregnancy. As noted 

previously, they should be enrolled before the 

pregnancy outcome is known to reduce selection 

bias, as well as to obtain reliable prospective 

information on exposure and other characteristics 

not affected by the outcome. One interview should 

take place at enrollment (i.e., during the first 

trimester), and at least one interview should take 

place postpartum (e.g., 2 months after end of 

pregnancy). Additional interviews (e.g., 

midpregnancy in order to update exposure 

information; 12 months after delivery in order to 

evaluate development) might be useful depending 

on the objectives of the pregnancy registry. Data 

from treating physicians can document the medical 

condition and confirm the prescription. 

Confirmation of exposure should be blinded to the 

outcome. 

8. Exposure Definition

If one is concerned about structural 

malformations, the etiologically relevant period of 

exposure is the first trimester of pregnancy. To 

identify this period, one needs to establish 

gestational timing. Obstetricians typically time 

pregnancies from the first day of the last menstrual 

period (LMP), determined by maternal recall or, 

preferentially, by a more accurate early pregnancy 

ultrasound. Depending on the pharmacological 

characteristics of the drug, the specific defect, and 

the accuracy of timing information, the window of 

interest might be the second and/or third month, 

any time during the first 4 months after the LMP, 

or even weeks before LMP for drugs with long 

half-lives or unspecified period of effect 

(e.g., vaccines). Exposures later in pregnancy can 

adversely affect other outcomes.25-27 

Treatment strategies change during pregnancy, and 

doses are commonly adjusted. Therefore, it is 

recommended that detailed information be 

collected on start and stop dates, dose, frequency, 

duration, and indication.28 Some medications 

(e.g., anticonvulsants) are prescribed in 

combination with other drugs. The effect of 

polytherapy resulting from concomitant 

treatments, or from switching drugs, within the 

first trimester should be explored in the analyses. 

Not only the number of drugs, but which drugs are 

combined, might affect the outcome of interest. 

Although the power is usually limited, analyses of 

dose response can inform recommendations, 

analyses of specific timing within the first 

trimester can assess biological plausibility, and 

analyses of indications might help explore 

confounding. If possible, maternal body mass 

index should be considered when evaluating dose 

effects.

9. Covariates: What Else To 

Collect? 

There is always a tension between simplicity and a 

desire to be comprehensive. Although there is no 

general rule, one widely accepted principle is that 

quality is more important than quantity—if the 

information is not trustworthy, do not collect it. 

Minimizing the effort and time from participants 

can increase both the willingness and the quality 

of participation. Necessary information on 

exposure, outcome, and key confounders 

(e.g., history, status, severity, and management of 

the indication) should take preference over 

desirable but less useful information. Most 

registries collect information on demographics, 

concomitant illnesses and medications, and 

reproductive history. A list of variables commonly 

collected in pregnancy studies is provided in 

Table 21–1.
141

Chapter 21. Pregnancy Registries

Table 21–1. Variables commonly collected in exposure pregnancy registries

Identification Exposure Outcome Covariates

Study ID

Date of enrollment

LMP date or EDD

Gestational age at 

enrollment

Date of first data 

collection

Date(s) of followup data 

collection(s)

Date of end of followup

Followup status (e.g., 

complete, loss to 

followup, withdrawal)

Contact information:

Woman

Alternative contact(s)

Obstetrician/prenatal 

health care provider:

Specialist

Pediatrician

Drug

For each drug, each 

episode of use, and each 

dose if changed:

Start date

Stop date

Indication and measure 

of disease severity 

(e.g., CD4 count for 

HIV patients, type of 

epilepsy and convulsions 

during pregnancy for 

anticonvulsants)

Dose

Route

Frequency

Duration

Sex

Status:

Live birth

Elective termination

Spontaneous abortion

Late fetal death

Number of fetuses 

(singleton or multiples)

For fetal loss: 

Date end of pregnancy

Reason for termination

For live birth:

Date of birth

Birth weight

Birth length

Head circumference

Gestational age at birth

Conditions at birth 

(e.g. admission to 

intensive care unit, drug 

withdrawal syndrome)

Congenital anomalies:

Specific defects

Date of diagnosis

Methods of diagnosis

Date and results of any 

prenatal testing

Obstetric outcomes:

Delivery (vaginal, 

C-section type)

Preeclampsia

Premature labor

Preterm delivery

Gestational diabetes

Demographics: 

Maternal age

Race

Occupation

Education level

Pre-pregnancy weight and 

height

Reproductive history: Number 

of previous completed 

pregnancies and miscarriages

Fertility interventions for past 

and current pregnancy

Family history of defects 

(specific defect and degree of 

relationship)

Habits:

Cigarette smoking

Alcohol intake

Use of illicit drugs

Chronic medical conditions: 

Diabetes

Pregravid obesity 

Hypertension

Epilepsy

Depression

Other psychiatric disorders

Hepatitis

Thyroid disease Autoimmune 

disease Asthma

Sexually transmitted disorders

AIDS

Concomitant medications, 

including folic acid 

supplementation and potential 

teratogens.
142

Section V. Special Applications in Patient Registries

10. Outcome Ascertainment

The source of information for outcomes in 

pregnancy registries is critical. Although registry 

designs that incorporate interviews with pregnant 

women can provide an initial source of data 

regarding the results of prenatal diagnosis, 

postnatal events, and malformations that are 

recognized during longer term followup periods, in 

most registries, validation by the health care 

provider of any maternally reported diagnosis is an 

important criterion for inclusion of specific 

outcomes such as major congenital anomalies (see 

Outcomes Definitions below). In registries in 

which the outcome is routinely collected from the 

woman and validated by health care providers, the 

mother’s report can correct potential false negative 

reports from one clinician; in comparison, 

registries in which the outcomes are reported only 

by the provider rarely include maternal validation. 

In some circumstances, a more stringent level of 

validation might be required to confirm an 

outcome (e.g., an echocardiogram to validate the 

presence of specific heart defects). Requiring 

commitment from registrants at enrollment to 

provide consent and medical release of 

information from obstetrician and specialists can 

select a motivated patient population and minimize 

loss to followup and maximize access to medical 

data.2

Whether the exposure is ascertained from the 

patient or from the clinician, it is important to 

obtain delivery data from the obstetrician or 

hospitalization records to ascertain the outcome 

accurately. It is also important to follow up with 

subsequent providers, such as the infant’s 

pediatrician and other specialists, because those 

treating a woman for a non–pregnancy-related 

condition often know little about obstetric or 

pediatric outcomes (e.g., a woman’s neurologist 

might not know about the patient’s preeclampsia), 

and obstetricians often know little about the infant 

after delivery.1, 19 Obtaining this information may 

require the woman’s consent and therefore contact 

with her. In addition, treating physicians might 

have a legal or ethical conflict of interest if they 

are asked to report on a pregnancy outcome when 

they were responsible for the exposure. 

It is important to recognize that maternal report 

alone can result in misclassification of the 

presence or absence of any defect and/or the 

presence or absence of a specific defect. Although 

specificity is most relevant for the evaluation of 

infrequent outcomes, to maximize sensitivity 

medical validation may be required for all births 

and not just those in which the mother reports an 

abnormality. 

Other primary or supplemental methods of 

ascertainment of birth defect outcomes can be 

employed. These include linkage to claims data, 

public birth registries, or birth defects surveillance 

systems. In any case, comparable methods for 

ascertainment of outcomes must be used in the 

exposed and reference groups.

11. Outcome(s) Definition

As part of the study design, pregnancy registries 

must set a priori criteria for defining outcomes. 

Outcomes are typically ranked in order of 

importance to the registry’s objectives. For 

example, most pregnancy registries identify major 

structural birth defects as the “primary” outcome 

of interest. This outcome is frequently defined as 

primary because identification of an increase in 

major malformations, particularly specific major 

malformations, in association with a given 

gestational exposure may indicate a teratogenic 

effect. In addition, some pregnancy registries 

incorporate a measure of “minor” structural 

defects as an outcome representing a broader 

spectrum of potential structural differences that 

may be attributable to an exposure.

Other “secondary” endpoints frequently include 

measures of fetal growth deficiency, preterm 

delivery, spontaneous abortion or stillbirth, and 

elective terminations. Some pregnancy registries 

incorporate longer term measures of outcomes that 

can include, for example, postnatal growth 

deficiency, cognitive and behavioral development, 

or measures of immune function, depending on the 

characteristics of the exposure under study in the 

registry. However, the cost and logistics of 

following children over time usually are 

prohibitive.
143

Chapter 21. Pregnancy Registries

Using the primary outcome of major structural 

defects as an example, but relevant to all outcomes 

in any pregnancy registry, the following definitions 

must be determined a priori. 

11.1 Inclusion/Exclusion Criteria for a 

Defect To Be Defined as “Major”

Criteria for defining defects as “major” must be 

established for a pregnancy registry. For example, 

major structural defects might be defined as 

abnormalities in structural development that are 

medically or cosmetically significant, are present 

at birth, and persist in postnatal life unless or until 

repaired. Similarly, criteria must be established for 

defects that will be excluded. For example, those 

that are transient and maturational and do not 

represent an abnormality in embryonic or fetal 

development, such as a patent ductus arteriosus or 

an inguinal hernia that might occur in a preterm 

infant simply as an artifact of shortened 

gestational age, might be excluded. Another 

example of an excluded defect might be a small 

muscular ventricular septal defect that may 

spontaneously close with no consequences for the 

infant. Attention must be paid to the comparability 

of definitions for inclusion and exclusion between 

the exposed and the reference groups.

Frequently, pregnancy registries employ an 

existing standard coding system for inclusion and 

exclusion of structural defects, such as that 

developed by the U.S. Centers for Disease Control 

and Prevention’s (CDC) Metropolitan Atlanta 

Congenital Defects program.29 This system was 

created and is maintained for use in an ongoing 

population-based surveillance program for birth 

defects that are identifiable up to 1 year of age. For 

some pregnancy registries, additional definitions or 

modified inclusionary/exclusionary criteria may 

need to be employed, depending on the length of 

followup or the specific outcomes of interest. Even 

with standard coding criteria, the information 

available for some reported defects may be 

insufficient or ambiguous for classification, such 

that an additional level of review is required to 

classify the defect appropriately. A method for 

expert adjudication of defect classification, blinded 

to exposure status, is an important component of a 

pregnancy registry.

Minor malformations are defined as those 

uncommon structural differences in the infant that 

have no serious medical or cosmetic consequences 

(e.g., an extra hair whorl on the head). Although 

these minor structural defects are of potential 

interest as more subtle measures of outcome, they 

may not be reliably assessed and therefore are 

frequently excluded unless they are uniformly 

evaluated in all patients. Similarly, positional 

deformities are often excluded (e.g., abnormal 

head shape or plagiocephaly that spontaneously 

resolves shortly after birth). 

The source of information regarding a major 

structural defect must also be defined as meeting 

the criteria for inclusion. For example, maternal 

report of a malformation with no validation by a 

physician or postnatal diagnostic test may be 

defined as insufficient for inclusion of a major 

defect. Another example of a situation that might 

be defined as exclusionary is a defect that is 

suggested through prenatal diagnostic tests but for 

which no postnatal validation is available. 

11.2 Timeframe of Diagnosis

A specified period for followup, during which 

standardized efforts will be made to collect 

outcome information on major birth defects, 

should be determined as part of the pregnancy 

registry design. As length of followup increases, 

the baseline risk of major structural defects is 

expected to increase because not all structural 

defects are reliably recognized at birth. Ideally, the 

longer the followup, the more complete an 

assessment of major birth defects could be. 

Specific outcomes of interest for the exposure 

under study may require longer followup to be 

appropriately assessed. However, the length of 

followup selected for the registry may be 

influenced by the availability of resources and the 

registry’s ability to maintain contact with registry 

participants and/or health care providers over a 

longer term. For comparability reasons, it is 

essential that the timeframe for diagnosis be 

identical in the exposed and the reference 

population or group.
144

Section V. Special Applications in Patient Registries

In addition, the case of major birth defects 

occurring in pregnancies that end in embryonic or 

fetal demise must be considered in the registry 

design. Major defects might be identified in 

spontaneous pregnancy losses by postnatal 

pathology, and criteria for inclusion of those 

defects must be established, as not all 

spontaneously aborted pregnancies or stillbirths 

will be uniformly evaluated for the presence or 

absence of defects. In the special case of elective 

terminations, criteria for ascertainment of 

malformations are critical because terminations for 

defects that are identified on prenatal diagnosis 

may represent an important subset of outcomes 

within the registry. Elective terminations for social 

reasons may not be uniformly evaluated by 

prenatal diagnosis for major structural defects. For 

these reasons, many pregnancy registries treat 

malformations identified in pregnancies ending in 

live birth separately from malformations included 

in the overall sample of registry pregnancies 

including terminations, spontaneous abortions, and 

stillbirths. Failure to include defects detected 

among terminations can decrease power and 

introduce bias, particularly for defects for which 

termination is often chosen after prenatal diagnosis 

(e.g., neural tube defects).

11.3 Analytical Approach

No known teratogen increases the risk of all major 

birth defects. Typically, a specific defect or pattern 

of defects occurs with increasing frequency 

following a teratogenic exposure in the critical 

gestational window for susceptibility. However, 

specific major defects are rare events in the general 

population—the most common occur no more 

frequently than approximately 1 in 1,000 live 

births. Pregnancy registries usually do not have 

sufficient sample size/power to evaluate increased 

risks for specific defects unless the relative risks 

are quite large. Therefore, most registries compare 

the overall proportion of all major defects 

combined in the exposed group to the overall 

proportion in the reference group. The rationale 

for this approach is that if specific defects are 

increased following exposure, these specific 

defects will incrementally inflate the overall 

proportion of malformations in the exposed 

sample and therefore reflect the excess risk 

associated with that exposure. For example, if the 

baseline risk of major malformations is 3 percent 

and the risk of neural tube defects is 0.1 percent, a 

fivefold increased risk of neural tube defects would 

inflate the overall risk to about 3.4 percent. 

Although the analytic approach for a registry may 

be based on a comparison of all malformations 

combined, it is important for pregnancy registries 

to evaluate any potential excess of specific defects 

in the exposed group, even if this evaluation is 

descriptive. 

In some registry designs, it is argued that 

malformations of known etiology should be 

excluded from the overall proportion of major 

structural defects, as they do not have the potential 

to have been “caused” by the exposure. Examples 

might be chromosomal anomalies, those defects 

that have a known single gene cause, or defects 

that occur in families with a positive history for 

that defect. The rationale for these exclusions is 

that inclusion of defects not thought to be caused 

by teratogens can decrease power to detect an 

overall difference in risk between exposed and 

unexposed fetuses. Arguments against exclusion of 

such defects in the analysis are that the true cause 

of the defect may not be known and that it is 

possible that exposure to a medication modifies 

other risks for that defect in a multifactorial 

manner. Therefore, by excluding such defects one 

could miss the effect.30 Inclusion or exclusion of 

chromosomal defects or those of other known 

etiology may also be driven by the inclusion or 

exclusion of those defects in the reference 

population that is selected for comparison.

12. Reference Group(s): 

Internal or External, Exposed 

or Unexposed?

A critical element for pregnancy exposure 

registries is the choice of comparator groups. The 

most valid reference group will have comparable 

(1) outcome definition (e.g., exclusion of minor 

anomalies); (2) outcome assessment (e.g., intensity 

of screening, frequency of terminations, inclusion 

of prenatal diagnoses, availability of diagnostic 

tests, start and stop of followup);23 (3) selection of 

subjects into the study (e.g., gestational age at 
145

Chapter 21. Pregnancy Registries

enrollment); and (4) baseline risk (e.g., distribution 

of risk factors, including indication). Ideally, each 

registry is constructed to include one or more 

internal reference groups. When this is not 

possible, an external reference group must be 

selected with care. Each comparison group has its 

advantages and disadvantages. For example, an 

external population-based reference group is 

generally larger and can provide more stable 

estimates for specific malformations, while an 

internal comparison group, which may be too 

small to support assessment of specific 

malformations, may be able to provide more 

comparable estimates for malformations overall. 

More than one comparison group can be used to 

increase the study generalizability. In anticipation 

of potential conflicting results, however, a primary 

comparison group or groups should be identified 

and justified a priori.

External comparators are usually considered less 

valid than internal comparators; for example, an 

RCT or observational study on the relative risk of 

stroke in an elderly population exposed to a drug 

would never use an external reference. However, 

pregnancy registries often compare their estimates 

with the background risk in “standard populations” 

such as the CDC’s Metropolitan Atlanta 

Congenital Defects Program, where the frequency 

of malformations among pregnancies of 20 weeks 

or greater is 2.1 percent when diagnosed prenatally 

or within the first week of life, and 2.6 percent if 

infants are followed until the first birthday;19, 23 or 

European Congenital Anomalies and Twins 

Registers (EUROCAT) data in Europe, where the 

prevalence of malformations at birth is 2.0 

percent.31 Two advantages of the use of available 

data from large external reference populations are 

that it avoids the costs in time and money of 

enrolling unexposed subjects, and that it provides 

stable risk estimates for common specific 

malformations in the general population. 

External reference groups must be used with 

caution, since the estimated risk of “major 

malformations” can vary widely depending on the 

population, the definition, and the ascertainment 

methodology. When external references are the 

only alternative, they must be appropriate to the 

population being studied. Analyses should at least 

take into account the characteristics of the 

surveillance program and use the same 

methodology in their exposed group. For example, 

characteristics to be considered are whether 

prenatal diagnoses and terminations were counted, 

whether malformations were identified during the 

delivery hospitalization or also through followup 

over a number of months, and whether 

chromosomal malformations and minor 

malformations were included. Many registries use 

external references as a necessity. Information 

from these sources can be helpful, particularly 

when there are no other data available, as long as 

findings are interpreted with caution. For example, 

external references can identify major teratogens 

(like thalidomide), generate hypotheses when 

unusual patterns of malformations are identified, 

and inform the need for additional, targeted, 

epidemiological studies.24

Some registries enroll an internal reference group 

of unexposed women who undergo the same 

processes as the exposed; these registries 

subsequently adjust for potential confounding by 

matching or adjusting for key covariates. There is 

still some risk of differential gestational time at 

enrollment (exposed women may tend to enroll 

earlier) and lack of comparability (unexposed 

women rarely have the underlying condition for 

which the drug was indicated). To make groups 

more comparable, some registries use women 

exposed to other non-teratogenic drugs as the 

reference (e.g., OTIS registries)32 or enroll 

pregnant women with a common condition or 

indication treated with various drugs or untreated 

(e.g., the multi-sponsor North American 

Antiepileptic Drug [NAAED] Pregnancy 

Registry,33 and the HIV Antiretroviral Pregnancy 

Registry34). More recently, the scientific 

community has moved toward the evaluation of the 

comparative safety and efficacy of different 

treatments for similar indications, in similar 

populations whenever possible. This approach 

enhances the comparability of groups, although 

sometimes more severe conditions are channeled 

to specific treatment while milder ones can remain 

untreated; thus, confounding by severity or type of 

disease is still possible. In addition to improving 
146

Section V. Special Applications in Patient Registries

validity, using alternative treatments as a reference 

would answer the clinically relevant question of 

“how to treat” rather than whether to treat. 

Moreover, safety data is often needed for multiple 

drugs with the same indication, evaluation of a 

variety of drugs used to treat the same condition 

could be most efficient. The feasibility of multipledrug registries depends in part on the sponsorship.

Taking advantage of the etiologically relevant 

periods of exposure during pregnancy, some 

studies compare first-trimester use of the drug with 

second- or third-trimester use. These comparisons 

are only possible for nonchronic treatments, and 

researchers should consider that the outcome can 

affect the opportunity for exposure after the first 

trimester. For example, if the pregnancy is 

terminated because of a malformation, there would 

not be second- or third-trimester exposures; 

therefore, later exposures might be artificially 

associated with lower risks.

13. Analysis of Registry Data

Pregnancy registries frequently include multiple 

outcomes as endpoints and may have more than 

one comparison group; in addition, as stated 

above, major birth defects among live births may 

be evaluated separately from major birth defects 

among all pregnancies. To address the problem of 

multiple comparisons in analysis and interpretation 

of registry data, it is essential to establish an 

analysis plan that identifies the primary hypothesis 

being tested—typically the proportion of 

pregnancies involving a major birth defect—and to 

specify which are the primary groups being 

compared. 

Similarly, the analysis plan should attend to the 

design of the registry and the expected sample 

size. For example, registries that involve multiple 

centers/multiple countries should use appropriate 

conditional methods of analysis that account for 

center. For outcomes with low frequencies 

(e.g., major structural defects), sample size 

projections should indicate when exact methods 

must be used for analysis. To evaluate the 

robustness of findings in the registry, the analysis 

plan can incorporate a plan for post hoc sensitivity 

analyses under various scenarios.

14. Statistical Power, Registry 

Size, and Duration

The projected sample size for a specific pregnancy 

registry is affected by the frequency with which 

the medication is used by women of reproductive 

age, the proportion of exposed pregnancies it is 

estimated are possible to identify and recruit into 

the registry, and the scope of the registry (local, 

national, international). The power of the study to 

detect an effect at or above a certain level is 

affected by the sample size and the baseline risk 

for the outcome in the population.19 The estimated 

losses to followup will affect the useful sample 

size. The duration of followup for outcome 

assessment will affect the cumulative risk estimate. 

Missing therapeutic abortions will affect both.

If the medication under study typically is taken 

only for a few days or intermittently, this fact 

should be considered in calculations of power and 

sample size. The effective sample size of 

pregnancies exposed only in specific gestational 

windows of time (any one of which may be a risk 

period while others are not) may be much smaller 

than the registry’s overall projected sample size of 

pregnancies exposed. These factors must be 

balanced against the amount of time needed to 

accumulate a sample size that is sufficient to 

produce a clinically relevant result.

For example, in a pregnancy registry that compares 

the overall proportion of pregnancies resulting in 

an infant with a major birth defect among exposed 

women with the proportion in an external 

reference population with a baseline prevalence of 

major defects of 3 percent, a sample size of 200 

exposed live born infants would be sufficient to 

detect a 2.2-fold relative risk with 80 percent 

power at an alpha of 0.05. However, the same 

sample size of exposed live births would be 

sufficient to detect only a 10.4 or greater relative 

risk for cleft lip with or without cleft palate.34

Because of the limits of power associated with 

typical sample sizes for pregnancy registries, 

especially in detecting risks for specific birth 

defects, an approach that has been used in some 

registries is the “rule of three”: An alert is 

triggered when three or more occurrences of 
147

Chapter 21. Pregnancy Registries

specific defects are reported for a specific 

exposure. This rule is based on the less than 5 

percent likelihood of a chance finding of three or 

more occurrences of the same specific, relatively 

rare, defect in a cohort of 600 or fewer subjects. 

Although some defects or defect groups occur 

frequently enough that the “rule of three” would 

not apply, this method can be used to flag an 

unusual finding for further review.34

15. Biases 

15.1 Selection Biases 

Because pregnancy registries typically enroll 

women only after recognition of pregnancy and in 

some cases much later in pregnancy, the group of 

enrollees is a selective group of pregnancies that 

have survived to that point in gestation and may 

have a shortened remaining period at risk of 

incurring the outcome of interest. There is “left 

truncation” of the registry cohort such that it is 

devoid of women who have already had a 

spontaneous abortion, an elective termination, or a 

stillbirth, depending on the gestational age of 

enrollment. Although statistical methods can be 

used to address left-truncation, survivor bias 

threatens ability to evaluate risk for pregnancy 

outcomes including birth defects, and calls for a 

registry design that encourages recruitment of 

participants as early in gestation as possible. In the 

extreme, one cannot study infertility in a cohort of 

pregnancies because, by definition, the women 

have conceived. Nor would one estimate the 

incidence of spontaneous abortions in women 

enrolled after 20 weeks of pregnancy. Spontaneous 

abortions can still be evaluated when women are 

enrolled during the first trimester, by assessing the 

rate of miscarriages per gestational month, thus 

taking into account both the decreasing trend as 

gestation progresses and the gestational time at 

enrollment.

Early enrollment is also of benefit with respect to 

biases potentially introduced by prenatal diagnosis. 

Prospective registry enrollment before any prenatal 

diagnostic test for major birth defect avoids bias in 

the direction of an increased risk for defects so 

identified, or in the other direction if women enroll 

preferentially after prenatal diagnosis that has 

shown no defect. Pregnancy registry guidelines 

recommend that only participants enrolled prior to 

prenatal screening be included.2 However, as 

prenatal diagnosis becomes feasible earlier in 

gestation, this becomes more difficult to achieve. 

Many registries also collect data retrospectively, 

but these data should be analyzed separately. 

Women enrolled after an abnormal pregnancy test 

can be analyzed as case series in passive 

surveillance systems of spontaneous adverse event 

reports.2 Malformations can be evaluated for 

biological plausibility, and specific patterns of 

malformations or distinct congenital abnormalities 

can generate hypotheses. On the other hand, 

retrospectively enrolled subjects without 

malformations would offer limited information, 

therefore, the benefit (and ethics) of including 

these women is questionable.

Registries should report the gestational age at 

enrollment for their exposed and reference groups. 

If enrollment time differs, methods should be 

applied that adjust for left censoring, such as 

restriction to prospective pretests enrollees. 

Another bias might be associated with right 

truncation of the registry cohort. This occurs when 

pregnancies with unknown outcomes are 

considered ineligible for analyses. By excluding 

terminations, spontaneous abortions, and losses to 

followup, one assumes that the exposure had the 

same effect in these pregnancies as in those that 

remain under observation. Such an assumption is 

less plausible if the exposed group has a higher 

frequency of these outcomes than the reference 

group. The frequency of spontaneous and 

therapeutic abortions, losses to followup, and 

withdrawals should be reported for the exposed 

and reference groups. Of note, a higher frequency 

of terminations among exposed women might 

reflect a higher risk of malformations, as well as 

more fear of malformations with consequent 

abortion if the drug is suspected of being 

teratogenic. 

Bias may also occur with events that shorten the 

followup (e.g., preterm delivery that cuts the 
148

Section V. Special Applications in Patient Registries

possible number of weeks of exposure to the drug 

of interest). The assessment of transient exposures 

(e.g., vaccines) during pregnancy in relation to 

outcomes associated with shorter gestations 

(e.g., preeclampsia, prematurity, pregnancy weight 

gain) needs to take into account the smaller 

opportunity for exposure in shorter pregnancies.

Biases can also be introduced into the analysis by 

stratification or adjustment for covariates that are 

themselves affected by the exposure of interest and 

that are affected by the outcome or share common 

causes with the outcome (e.g., adjusting for 

gestational age at birth when studying the effect of 

a pharmaceutical on structural malformations).35-37

15.2 Information Bias

As noted above, pregnancy registries are 

preferentially “prospective” in design. With 

prospective design, the outcome cannot directly 

affect the accuracy of exposure information, and 

any misclassification of drug exposure tends to be 

nondifferential with respect to the outcome. 

However, nondifferential misclassification of 

exposure is still problematic for safety evaluation, 

since it tends to bias any potential effect toward the 

null. It is therefore crucial to maximize the quality 

of drug exposure information. 

When the women provide information on 

exposure, the accuracy of the recall can be 

maximized by using structured questionnaires, 

detailed questions, and calendars to help establish 

gestational timing and enhance recall of dates. 

Maternal reports on drug utilization are sometimes 

cross-validated with medical records. When the 

health care provider is the only source of data, 

information on prescriptions may not reflect the 

real use of the drug during pregnancy if patient 

compliance is incomplete. Since pregnancy 

registries typically focus on uncommonly used 

medications, the impact of false positives among 

the exposed group is much greater than the 

occasional inclusion of false negatives among the 

unexposed reference group. 

At the time of enrollment in the registry, women 

are reporting their medical history retrospectively, 

knowing that they are enrolled in the registry 

because there is lack of information on the safety 

of the drug under study. This situation can 

potentially influence the accuracy of recall for 

baseline covariates (i.e., recall may be more 

accurate for exposed than for unexposed groups 

because there is a different motivation to recall a 

medical history) and affect the ability to control 

for confounding. For this reason, methods for 

validation of key covariates are of benefit.

Diagnostic bias, or outcome misclassification, also 

can occur in pregnancy registries. These biases can 

be either nondifferential or differential between the 

exposed and the reference group, and can bias the 

estimate of an effect toward or away from the null. 

For example, participation in the registry itself or 

concern that a drug exposure might pose a risk 

could lead to more access to or uptake of prenatal 

diagnostic measures such as ultrasound and to 

more careful examination of infants for defects 

postnatally, potentially leading to differential 

accuracy in detection and classification of defects 

among exposed and unexposed groups. The risk of 

differential outcome classification among the 

exposed and the reference group is greatest when 

external control reference groups are used. 

As mentioned above, reported major 

malformations must be validated. Although neither 

treating physicians nor women are blinded to the 

treatment, registries can do blinded validation and 

adjudication of outcomes. To detect malformations 

not reported by the patients, records for all 

pregnancies would have to be reviewed. For 

example, women might be less likely to volunteer 

information regarding male genital malformations 

in their infants, which could result in underascertainment of these malformations. To detect 

malformations identified by screening and 

frequently terminated (e.g., anencephaly), 

therapeutic abortions need to be included. To 

detect an increase in abnormalities incompatible 

with life, it is important to collect information on 

autopsy results at stillbirth and, if possible, on 

examinations of the fetus after spontaneous or 

induced abortion. The study of spontaneous 

abortions and pregnancy terminations itself 

presents an additional methodological challenge, 

especially in countries where abortions are illegal 
149

Chapter 21. Pregnancy Registries

or when studies include pediatric populations, 

because induced fetal losses are sometimes 

categorized as spontaneous in medical records, and 

questions about terminations might be considered 

sensitive.

15.3 Confounding

Several sources of confounding may affect 

pregnancy registries, as in any other observational 

study. Socioeconomic status, maternal age, 

tobacco and alcohol use, illegal drug use, maternal 

body mass index, and vitamin use are examples of 

potential confounders that might be related to, or 

impact, the exposure under study and are also risk 

factors for some pregnancy outcomes. 

With medications, there is the concern of 

confounding by indication (e.g., the association of 

a weight loss drug with birth defects might be 

confounded by the maternal obesity the drug is 

used to treat). Confounding by indication is 

difficult to address in nonrandomized studies for 

common chronic conditions such as depression, 

asthma, epilepsy, HIV, and autoimmune diseases, 

where separating the effect, if any, of the drug 

from the underlying disease can be challenging. 

A related form of confounding that may be of 

concern is channeling bias (i.e., women with more 

severe underlying disease may be most likely to be 

selected to receive the drug of interest). 

Each of these potential confounding concerns 

provides a strong rationale for inclusion of actively 

recruited comparison groups that are matched to 

the exposed registry participants on the maternal 

underlying disease and, to the extent possible, 

similar in distribution of maternal disease severity. 

16. External Validity or 

Generalizability 

As mentioned above, pregnancy registries typically 

rely on volunteers to participate. With appropriate 

selection of comparison groups and control for 

sources of confounding, pregnancy registries can 

make assertions about internal validity. However, 

there is usually little known about the 

characteristics of the entire population of exposed 

pregnant women from whom the sample is drawn. 

For this reason, it is difficult to make conclusions 

about external validity or generalizability for any 

pregnancy registry. If the participants represent a 

select group, it may be difficult to generalize the 

findings from the registry. However, the registry 

volunteers would have to differ from 

nonparticipants with respect to some characteristic 

that modifies the effect of the drug on the 

pregnancy outcome (e.g., a teratogenic effect 

might vary by race or by baseline folate levels in 

the population).

17. Operations

17.1 Study Protocol

Pregnancy registries are scientifically rigorous 

studies designed to monitor safety of product use 

in pregnancy. As such, they have formal written 

protocols based on epidemiologic principles and 

on regulatory authority guidance documents for 

pregnancy registries.2, 5 A pregnancy registry 

protocol should include a brief and cogent review 

of the literature, registry objectives, study design, 

detailed data collection procedures including 

sources of data, inclusion and exclusion criteria for 

patient enrollment, operational aspects of 

enrollment and retention, definitions of relevant 

endpoints, analytic considerations, statistical plan, 

regulatory and ethical considerations, reporting 

and publication plans, governance, and criteria for 

registry termination. While RCT protocols are 

registered (e.g., on ClinicalTrials.gov), it is still 

controversial whether those for observational 

studies should also be registered.38

Pregnancy registries have procedural differences 

distinct from RCTs and from other observational 

study designs. Thus, they often require specific 

standard operating procedures that clearly document 

their unique processes. Each pregnancy registry 

should also have its own registry management plan 

that serves as the registry team’s roadmap. For more 

details see Chapter 2, Section 2.10. 
150

Section V. Special Applications in Patient Registries

17.2 Human Subjects, Informed 

Consent, and Medical Records Release 

Forms

Pregnancy registries involve human subjects 

research and thus institutional review board 

approval; informed consent of participants; and 

protection of confidentiality in data collection, data 

storage, and publication. Pregnant women are 

considered a “vulnerable” population in legal/

ethical terms, but, importantly, unlike other 

populations considered in this section of 

regulations (mentally challenged patients, inmates 

in penal institutions, and children), they are able to 

give consent. Therefore, registries must adhere to 

guidelines for human research protection in 

pregnant women. Pregnancy registries are typically 

considered minimal risk protocols. As with 

participation in any research of this type, there is 

the potential for loss of the confidentiality of the 

information provided. Methods to manage this risk 

must be in place and communicated to the 

potential participant. There is also the 

psychological risk of participation in a pregnancy 

registry that may occur when a woman becomes 

more aware of the potential effects of exposure on 

the fetus. Resources for expert counseling on fetal 

risks are available in the United States and Canada 

through the Organization of Teratology 

Information Specialists ( https://www.

mothertobaby.org/).

Informed consent is typically required for 

participation in a pregnancy registry. Some 

registries collect data directly from health care 

providers without collecting patient identifiers. 

These types of registries may qualify for a waiver 

of informed consent. For pregnancy registries in 

which the patient is identified and data are 

collected from multiple sources, informed consent 

is required. However, under several conditions, 

signed informed consent may be waived and 

verbal consent allowed. Additionally, in order for 

pregnancy registries to collect data from a 

woman’s health care provider(s) and/or her 

newborn’s pediatric health care provider, the 

woman must complete medical release forms for 

each clinician who will report data or provide 

medical records to the registry. Chapter 7 

thoroughly discusses the application of ethical 

principles for registries. 

17.3 Recruitment and Enrollment

In some registries, pregnant women self-identify 

and self-enroll by calling a toll-free number that 

can be found in key Web sites, printed materials in 

specialists’ offices, or drug inserts. In other 

registries, women are referred by general 

practitioners, specialists treating the condition of 

interest, or obstetricians. Direct enrollment of 

women may allow inclusion of participants earlier 

in gestation, since they are usually the first to know 

about their pregnancy. Enrolling women directly 

may also facilitate pregnancy and postnatal 

followup. However, for some drugs, it might be 

unrealistic to expect self-enrollment and, for some 

conditions, it might be more efficient to identify 

eligible women through specialists. Some 

countries do not allow self-enrollment. Who 

initiates enrollment has implications for data 

collection and informed consent processes. 

Pregnancy registries can be conducted by 

regulatory or other government agencies, academic 

centers, contract research organizations (CROs), or 

drug manufacturers. Who sponsors and who 

conducts the registry can affect the participation of 

health care centers, some of which may have 

barriers for collaborations with industry. 

Biopharmaceutical companies sometimes identify 

exposed pregnancies as outcomes reported to their 

surveillance system for drugs not recommended 

for pregnant women. If identified and enrolled 

before prenatal tests, these pregnancies can be 

followed, yielding a pregnancy registry nested in a 

passive surveillance system.

To maximize enrollment and maintain efficiency, 

pregnancy registries typically do not use a 

traditional “site-based” approach. Rather they 

employ a single coordinating center that recruits 

and enrolls all eligible pregnant women as soon as 

possible after conception. The number of 

coordinating centers that would make the study 

feasible varies among countries and among 
151

Chapter 21. Pregnancy Registries

registries. Therefore, awareness campaigns must 

reach out to pregnant women and their health care 

providers in a broad variety of settings. A carefully 

constructed awareness plan should be designed 

specifically for each pregnancy registry, 

accounting for the aims of the registry, its target 

population, and its geographic scope. In general, 

the plan should incorporate a variety of persistent 

awareness strategies to ensure broad coverage, 

including announcements of the registry with 

contact information posted in the following:

• Product label

• Registry and/or product Internet sites

• Personal mailings to health care specialists, 

particularly high prescribers and high-risk 

obstetricians

• Professional journals

• Exhibits or scientific presentations at 

professional meetings

• Lay magazines

• Advocacy group newsletters and/or Internet 

sites

• Social media

For pregnancy registries sponsored by individual 

biopharmaceutical companies, promotional 

materials are subject to 21 CFR 314.81(b)(3) or 

601.12(f)(4), and should be submitted to the FDA 

Division of Drug Marketing, Advertising, and 

Communications. In general, registry recruitment 

and awareness materials should not promote use of 

the product in pregnancy nor imply that the 

product is safe and effective in pregnancy unless 

sufficient scientific evidence exists to support 

these claims.

Enrollment may also be affected by the lag time 

between approval of the pharmaceutical and 

launching of the registry. Awareness and interest 

may peak right after approval, which is also the 

ideal time to collect safety data in the 

postmarketing setting. Streamlined procedures for 

informed consent and data privacy/HIPAA 

authorization have been demonstrated to increase 

enrollment in pregnancy registries.39

17.4 Retention of Participants During 

Followup 

Participant retention is crucial to achieving the 

registry goals. Retention can be encouraged by 

engaging registry participants, including pregnant 

women and their health care providers, in the 

reporting process, and making it as easy as 

possible to report data to the registry. Registry 

staff should be trained to collect data for 

observational studies from both patients and health 

care providers who do not usually participate in 

research activities. They should develop a rapport 

with the reporters that facilitates data collection 

and promotes retention through relationship 

building.

Streamlined data collection processes and simple, 

concise data collection forms are essential for 

reducing the burden of reporting. For registries 

sponsored by pharmaceutical companies, 

duplication of work because of safety reporting 

should be avoided as much as possible. A small 

monetary stipend may also encourage retention but 

may not be universally accepted by registry 

reporters. For health care providers, especially 

obstetric care providers, registry data are often 

more valuable than a stipend. Giving interim data 

reports to health care providers who report data to 

the registry is a powerful recruitment and retention 

incentive.

17.5 Data Collection

Like any other epidemiological study, registries 

can benefit from the technologic advances in 

communications (e.g., ability to enroll subjects 

through social networks and collect data through 

confidential Web sites). Most registries enter the 

data directly into an electronic database, either 

from records or questionnaires or by means of 

computer-assisted telephone interviews. To ensure 

confidentiality, identifiers should not be included 

in the database containing clinical information. 

The goal is to make reporting data to the 

pregnancy registry as easy and unobtrusive as 

possible. Providing a variety of data-reporting 

mechanisms that are simple to use and fit the 

reporters’ preferred communication practices 

facilitates recruitment and retention. If feasible, a 
152

Section V. Special Applications in Patient Registries

pregnancy registry should be designed to allow 

participants to report data through telephone 

interviews; paper data collection forms that can be 

mailed, scanned, or faxed to the registry; simple 

Internet-based electronic data capture systems; and 

mobile applications. For details on these data 

capture systems, see Chapter 11, Section 2.3. 

Pregnant women are ideally enrolled into a 

pregnancy registry before or soon after the 

exposure of interest and then followed throughout 

pregnancy. If a live infant is born, the infant may 

be followed for a period typically ranging from 3 

to 12 months. Data may be collected at several 

different time points:

• At enrollment, information may be collected on 

product exposure(s), maternal characteristics, 

prenatal testing, and other baseline data. 

• During the second and/or third trimester of 

pregnancy, a brief update on the status of the 

pregnancy and exposures may be obtained.

• At the end of pregnancy, information may be 

collected on additional exposures during 

pregnancy, risk factors, and details regarding 

the pregnancy outcome, including any 

congenital anomalies, gestational age and birth 

weight, and perinatal complications. 

• For live births, data may be collected at several 

points in time (typically 3, 6, and/or 12 months) 

on infant characteristics and health outcomes 

including congenital anomalies, functional or 

developmental deficits, and other outcomes 

pertinent for the drug of interest. 

• Targeted followup may be needed to collect 

additional detailed data on specific outcomes of 

interest. 

The source of baseline and followup data is an 

important consideration. It is best to evaluate all 

options for obtaining data to determine the most 

appropriate source for the specific data requested. 

The pregnant woman can provide detailed data on 

drug use including exposures to the drug of 

interest as well as other drugs, on relevant risk 

factors, and on the pregnancy outcome. Her health 

care providers, such as the prescriber, specialists, 

and/or obstetrician, should be able to provide or 

verify this information. The collection of data from 

a variety of sources throughout pregnancy and 

during infancy contributes to the accuracy and 

comprehensiveness of pregnancy registry data. 

Requiring information from multiple health care 

providers, as well as from the patient, is a distinct 

challenge of pregnancy registries.

Data quality is of the utmost importance in a 

pregnancy registry, and multiple levels of quality 

assurance should be employed, beginning with the 

design of data collection instruments. Instruments 

should be designed with care, thoroughly vetted, 

and pilot tested to ensure ease of reporting valid, 

reliable data. Instructions for self-reporting should 

be clear and succinct. Electronic data capture 

systems should include validity checks. If 

interviews are to be used, the interviewers should 

be thoroughly trained to conduct neutral, unbiased 

interviews using detailed interview scripts. Once 

captured, the data should go through a rigorous 

cleaning and quality assurance process to reduce 

errors, missing data, and misclassifications. When 

possible, patient-reported data should be verified 

by health care providers or medical records.

17.6 Adjudication of Outcomes

Many pregnancy registries enlist the services of a 

clinical geneticist or dysmorphologist to review 

and classify all congenital anomalies. A 

standardized classification system should be used, 

such as the CDC’s Metropolitan Atlanta 

Congenital Defects program.29 The assessor and 

method of assessment should be the same for both 

the exposed group and comparison group, and the 

assessor should be blinded to the exposure of 

interest. The assessor may also evaluate likely 

causes for the particular birth defect, such as 

family history, genetic factors, and/or exposure to 

known teratogens.28 If the exposure of interest is 

made available, the assessor may examine the 

timing of the exposure relative to the origin of the 

birth defect (to attempt to determine if the timing 

of the exposure is relevant to the formation of the 

birth defect).28 Registries can engage the scientific 

advisory board and/or a subgroup of the board to 

review each case and the assessor’s classification 

and reach consensus on the classification.
153

Chapter 21. Pregnancy Registries

17.7 Process of Releasing Findings

17.7.1 Reporting Findings to Regulatory 

Agencies

FDA considers individual case reports from a 

pregnancy registry as derived from active 

solicitation of patient information (FDA 1997), 

including reports from participants enrolled both 

prospectively, where the exposure is reported prior 

to knowledge of the pregnancy outcome, and 

retrospectively, where the pregnancy outcome is 

already known at the time the exposure is 

reported.2 A company sponsor holding marketing 

authorization for an approved drug or licensed 

biological product must submit to FDA, within 15 

calendar days, reports of pregnancy registry 

adverse events that are both serious and 

unexpected by regulatory definition and where a 

reasonable possibility exists that the drug or 

biological product caused the adverse event. 

Current reporting requirements in the regulations 

consider any congenital anomaly a serious adverse 

event. 

Pregnancy registries conducted independently of 

any sponsors holding marketing authorizations are 

not subject to postmarketing regulatory reporting 

requirements. However, investigators conducting 

such registries may forward reports of any serious 

adverse events including congenital anomalies to 

the sponsor of the medical product or report 

directly to FDA’s MedWatch office (1-800-FDA1088 or http://www.fda.gov/Safety/MedWatch/

default.htm).

Any company conducting a pregnancy registry 

required by FDA must submit an annual status 

report to the agency. Companies conducting 

pregnancy registries not subject to annual 

reporting requirements are encouraged to include a 

status report in the periodic safety report. The 

status/interim report should describe the study 

design and summarize the status of the planned, 

initiated, in progress, or completed pregnancy 

registry conducted by or otherwise obtained by the 

sponsor during the reporting period. The status 

report should also provide a descriptive summary 

of progress to date, interpretation of findings, and 

appropriate analyses with comments on the clinical 

significance of the findings. Copies of full reports 

may be appended, if appropriate. Any publications 

based on data from the registry should be included.

The registry status report should include the 

following, presented separately for prospective and 

retrospective reports:

• Number of pregnant women enrolled to date

• Number of pregnancies with outcome known 

(stratified by live birth, spontaneous abortions, 

elective terminations, fetal deaths/stillbirths)

• Number of pregnancies with outcome pending, 

and number of pregnancies lost to followup 

(p 16, FDA guidance)2

For pregnancies with known outcomes, line 

listings (which are tables that organize key 

information about cases with each row 

representing one case and each column 

representing a variable of interest40) and 

summaries of the following: 

• Demographics, obstetrical and medical history 

of mothers

• Weeks of gestational age at exposure

• Dose and duration of exposure

• Whether multiple gestation

• Weeks of gestational age at completion or 

termination of pregnancy

• For live births and deaths/stillbirths:

• Small for gestational age

• Preterm delivery

• Congenital anomalies or other fetal 

abnormalities

Finally, for spontaneous abortions and elective 

terminations, abnormalities in products of 

conception, if known.

17.7.2 Reporting to Sponsors

There are no specific rules for reporting events to 

the sponsor company. The sponsor company 

negotiates this with the academic center or CRO. 

Some pregnancy registries have regular 

(semiannual or annual) data cutoff points and issue 

corresponding periodic interim reports 

summarizing the aggregate data. The sponsor can 

then use this interim report for any regulatory 

reporting requirements.1 Others, such as the 
154

Section V. Special Applications in Patient Registries

NAAED, have prespecified criteria for release of 

results for a positive association.33

Most pregnancy registries administered by CROs 

or academic institutions report all adverse 

pregnancy outcomes (birth defects, spontaneous 

fetal losses, induced abortions), and maternal and 

nondefect fetal events, regardless of attribution or 

seriousness, to the sponsor within a few business 

days. Some sponsors only request specific 

outcomes, or serious and attributable events, 

reported to them. The sponsor reports these events 

to regulatory agencies. Pregnancy is not 

considered an adverse event (AE). The registry 

might also report spontaneous AEs (not registryrelated outcomes) to the sponsor, although these 

may be received infrequently. If the sponsor 

requires further information on a pregnancyrelated AE, it can contact the registry, which would 

contact the health care provider. The sponsor 

follows up on any non-registry cases as needed. 

17.7.3 Publication Policy

Some pregnancy registries have formal publication 

committees.34 In others, such as NAAED, the 

investigators review plans to publish registry data 

and conclusions with the independent scientific 

advisory committee, and a consensus to publish is 

developed.33 Others simply publish the findings 

when they have them. Registry results should be 

published as soon as the number of women in the 

registry permits, in order to allow dissemination of 

the results to the scientific and clinical 

communities.

17.8 Role of (Scientific) Advisory Board

An independent scientific advisory committee 

usually oversees the scientific conduct and analysis 

of the registry. This committee can advise and 

participate in the design and establishment of the 

registry, as well as assist in the review of data, the 

classification of any birth defects, and the 

dissemination of information to ensure that results 

are interpreted and reported accurately. The role 

and duties of the committee should be specified in 

the protocol. Members of the committee could 

include experts in obstetrics, embryology, 

teratology, pharmacology, epidemiology, 

pediatrics, clinical genetics, and any relevant 

therapeutic areas; and consumers representing the 

disease state being treated; and may include 

members from the CDC, the National Institutes of 

Health, academia, and the private sector. The 

advisory committee might review the registry data, 

develop consensus statements, provide 

recommendations on modifications or 

enhancements to the registry, and/or assist in the 

dissemination of information and the formulation 

of strategies to encourage enrollment. In addition 

to the scientific advisory committee, 

multicompany-sponsored pregnancy registries may 

also include a steering committee composed of 

representatives of the sponsoring pharmaceutical 

companies.

17.9 Stopping Rules

The criteria to determine when to end the study 

may be predetermined and specified in the 

protocol. If the registry is conducted by a drug 

company as a regulatory requirement, the decision 

as to when to actually end the study is made jointly 

by the company and the regulatory authority. In 

other pregnancy registries, it is the scientific 

advisory committee that decides or contributes to 

the decision about when to stop. Sometimes, 

findings from the registry might affect the decision 

as to whether it should continue.

Criteria for possible discontinuation of a 

pregnancy registry include the following: 

1. Sufficient information has accumulated to meet 

the scientific objectives of the registry (i.e., 

numeric targets or predetermined effect size, 

such as the “no evidence of risk” or “evidence 

of relative safety” thresholds defined in the 

newly created Vaccines and Medications in 

Pregnancy Surveillance System:20 “Estimates 

of safety cannot be absolute; rather, they reflect 

the degree of confidence that is consistent with 

an observation of no increased risk between a 

given exposure and outcome. As more data are 

collected over time, power increases; for a null 

observation, increasing power leads to 

increasingly narrower confidence bounds and 

increasing assurance of relative safety.”20 For 

example, evidence of relative safety might be 
155

Chapter 21. Pregnancy Registries

reached when the upper bound of the 95 

percent confidence interval excludes a twofold 

increased risk. 

2. The feasibility of collecting sufficient 

information diminishes to unacceptable levels 

because of low exposure rates, poor enrollment, 

and/or high rates of loss to followup. 

3. Other methods of gathering appropriate 

information, such as case-control surveillance 

or large health care databases, become 

achievable or are deemed preferable.2

17.10 Multidrug Pregnancy Registries

A multidrug pregnancy registry actively collects 

information on exposure to various drug therapies 

for specific diseases, such as the Antiretroviral 

Pregnancy Registry34 and NAAED.33 In some 

cases, a general multidrug registry, such as that 

conducted by a teratogen information service, 

collects information on drugs for either unrelated 

or related indications. Multidrug registries have 

advantages over single-drug registries with respect 

to both efficiency and economy; they also allow 

the examination of polytherapy. They may also 

have the advantage of having readily available 

comparison groups of pregnant women unexposed 

to the specific medical product(s) of interest but 

with the same indication (e.g., disease registries) 

or exposed to other drugs for the same indication. 

Disease pregnancy registries for common 

conditions in women of childbearing age 

(e.g., multiple sclerosis) treated with any available 

drug would make most sense from logistical 

(avoidance of duplicating efforts), methodological 

(validity and power), and clinical (comparative 

safety) points of view. However, their 

establishment would require collaboration among 

companies competing for the same therapeutic 

area, which may be difficult.

17.11 Multicenter and Global 

Registries

Like RCTs, registries can be multicenter, national, 

or international. Currently, registries are 

centralized by sponsors, CROs, government 

agencies, or academic centers; they typically focus 

on a single drug, multiple drugs within a class, 

multiple drugs for different indications, or are 

disease based and evaluate drugs used to treat a 

particular condition.1 In the future, rather than 

conducting a new registry for each drug, a global 

centralized mega pregnancy-exposure registry may 

exist, guided and coordinated by collaborations 

among regulatory agencies, pharmaceutical 

companies, contractor organizations, and academic 

centers.19 Although appealing, in practice it will be 

a challenge for this approach to accommodate the 

case-specific needs for each drug.

In terms of setup, management, and analysis, a 

locally run registry is very different in scope from 

a global registry. The latter’s geographic scope 

includes many challenges (e.g., language, culture, 

time zone, regulatory differences) that must be 

taken into consideration in the planning process. 

A distinct feature of a country-specific registry is 

that the patient population tends to be more 

homogeneous with respect to demographic 

characteristics, exposures, length of followup, and 

diagnosis of outcomes than international registries.

18. Advantages of Pregnancy 

Registries

Pregnancy registries are often the initial proactive 

step in assessing the safety of use during 

pregnancy of new drugs after they are first 

marketed, because they provide a number of 

advantages over other approaches:1 While many 

pregnant women use medications, their use of 

individual drugs can be quite rare.41 By enrolling 

an exposure group made up only of women who 

took the medication(s) of interest, pregnancy 

registries are efficient for evaluating the effects of 

infrequently used drugs in the population. This is a 

distinct advantage over other study designs, such 

as case-control studies and small health care 

utilization databases, which usually do not have 

sufficient power to evaluate outcomes following 

rare exposures. 

The longitudinal nature of pregnancy registries 

allows the estimation of absolute risks of 

pregnancy outcomes. For example, registries that 

enroll women before any prenatal testing has been 

performed can estimate the risk of malformations 
156

Section V. Special Applications in Patient Registries

among infants whose mothers used a drug of 

interest. This is in contrast to case-control studies 

that estimate risk relative to that for a reference 

group. Information about the absolute risk of 

outcomes is particularly helpful when counseling 

women who are planning a pregnancy or who have 

already become pregnant while taking a drug. 

Prospective enrollment facilitates ascertainment of 

drug exposures close to the time a medication is 

actually used and before information about the 

pregnancy outcome is known. When registries 

interview pregnant women directly, they can obtain 

accurate information about the timing in relation 

to gestational age, dose, frequency, and duration of 

medication use, as well as covariates, and can 

therefore reduce exposure misclassification, recall 

bias, and confounding. 

Pregnancy registries can compare the risk of 

outcomes among women who have used a variety 

of treatments for a single condition, including 

different monotherapies, different polytherapy 

combinations, or no treatment at all. This 

information is useful both to women and health 

care providers in making decisions about whether 

to treat a condition during pregnancy and which of 

multiple alternate therapeutic strategies to use. An 

additional advantage is that a single registry can 

monitor a variety of pregnancy and infant 

outcomes after medication exposure, including 

postnatal outcomes. 

19. Limitations of Pregnancy 

Registries

The pregnancy registry approach also has a 

number of limitations. While pregnancy registries 

are an efficient means to assess rare exposures, 

they lack the statistical power to evaluate rare 

outcomes. Most teratogenic exposures do not 

increase the prevalence of all malformations, but 

have a more selective effect on individual defects 

or distinct patterns of defects. Pregnancy registries 

are powered to detect common outcomes such as 

the total prevalence of all malformations, and can 

detect only very large increases in these rarer 

individual defects or patterns. However, many 

drugs associated with adverse effects in pregnancy 

result in only moderate increases in these rarer 

outcomes. Therefore, pregnancy registries are 

limited in their ability to detect teratogenic effects 

on specific malformations with statistical certainty. 

Nevertheless, registries can generate hypotheses 

that form the basis of further investigation using 

complementary approaches, study designs, and 

data sources.20

Another important limitation of some pregnancy 

registries is the lack of a comparable reference 

group. Ideally, a comparison group should be 

drawn from the same population as women with 

the exposure of interest, using the same methods 

for recruitment, enrollment, and ascertainment of 

outcomes so that both groups have the same 

baseline risk for adverse pregnancy outcomes. 

Comparison of registry findings with data from 

other studies, such as population-based 

surveillance programs or hospital deliveries, can 

lead to biased results if the subjects in the 

reference group have characteristics different from 

those of the registry participants, or it the 

methodology for case ascertainment is different. 

Identifying an appropriate reference group can be 

particularly difficult for global registries that 

recruit exposed women from multiple countries 

with potentially different populations and 

backgrounds. For registries such as these, a 

comparable unexposed group may not exist. Even 

when an internal reference group is recruited, 

differences between the exposed and unexposed 

groups with respect to factors such as the 

indication for the drug or the proportion of 

subjects lost to followup can affect the validity of 

the results. 

An additional consideration is that findings from a 

pregnancy registry may not be generalizable to the 

broader population of all women who use a drug. 

Enrollment of women in pregnancy registries 

typically is voluntary and self-selected, and 

registry participants represent a small proportion 

of all women who have taken a drug. For these 

reasons, the characteristics and experience of 

women who participate in a registry may differ 

from those of nonparticipants, and these 

characteristics may modify the effect of the drug.

A final limitation of pregnancy registries is the 

length of time typically required to enroll 

sufficient numbers of exposed women to generate 
157

Chapter 21. Pregnancy Registries

Table 21–2. Issues to consider when evaluating reports from pregnancy registries

stable estimates of pregnancy outcomes. This 

timeframe can be affected by the frequency of 

exposure in the general population, and by the 

methods and extent of recruitment efforts by the 

registry. Most registries continue for years before 

publishing final results. This extended period of 

evaluation before reaching conclusions regarding 

adverse pregnancy outcomes can be a disadvantage 

when there are pregnancy outcomes of concern 

that need to be evaluated quickly or when new 

therapeutic agents become available.

20. Evaluation of Reports 

From Pregnancy Registries

It is important to critically assess the results and 

conclusions of reports from pregnancy registries. 

Key issues to consider are summarized in 

Table 21–2.

Area Issues

Objectives What question(s) is the registry attempting to answer? Are the design and methods 

appropriate to do so? 

Background What condition(s) is the drug used to treat (e.g., chronic vs. episodic)?

In what settings is the drug likely to be used (e.g., as primary treatment or as 

adjuvant therapy with other drugs)?

What is the recommended therapeutic dose and duration of use?

Is the drug likely to be used off label for conditions other than the stated indication? 

Study population What is the target population from which pregnant women exposed to the drug are 

drawn?

Does the report describe the characteristics of women enrolled in the registry? 

Could the study subjects differ from women in the target population in ways that 

would affect the generalizability of the results?

Exposure ascertainment What are the sources of information about drug exposure (e.g., maternal interviews, 

physician reports, pharmacy records)?

Are these sources likely to provide valid information about how women actually 

used the drug? 

Are the exposures ascertained in sufficient detail (e.g., dose, frequency, duration, 

timing during gestation) to accurately assess the drug’s potential effects on the 

outcome(s) of interests? 

Outcome ascertainment What are the sources of information about pregnancy outcomes, infant and fetal 

health (e.g., maternal interviews, obstetricians’ reports, pediatric records)?

Are these sources likely to be knowledgeable about the occurrence of the outcomes 

being studied?

Are outcomes among stillbirths, spontaneous abortions, and elective terminations 

included? 

Are the outcomes documented in sufficient detail? 

Reference group(s) What comparison group(s) does the registry use? Is an internal comparison group 

recruited?

Are there potential differences between the exposed and comparison groups that 

could affect the validity of the findings?

Statistical power Does the sample size provide sufficient statistical power to meet the objective(s)?
158

Section V. Special Applications in Patient Registries

Area Issues

Biases When did subjects enroll (i.e., gestational age at enrollment for exposed and 

reference groups)?

What proportion of registry enrollees did not complete the study (i.e., were lost to 

followup)? Do their characteristics differ from those who completed the study in 

ways that could affect the validity of the results?

Are exposed subjects comparable to the reference group? Were the same methods 

for data collection and outcome definition used in the exposed and reference 

groups?

Are there other possible sources of bias in the results? Are these biases addressed 

sufficiently in the analyses or in sensitivity analyses? 

Results Does the report provide estimates of the absolute risk of the adverse pregnancy 

outcomes being studied?

Are the results generalizable to the broader population of pregnant women who will 

use the drug? 

Conclusions Does the report explore possible alternative explanations for the findings? 

Does the report review and compare findings from other studies that assess the 

drug’s effects during pregnancy, or findings for other drugs used to treat the same 

condition(s)?

Do the registry findings provide information that will be useful to health care 

providers and women in making clinical decisions about use of the medication and 

pregnancy management?

Table 21–2. Issues to consider when evaluating reports from pregnancy registries (continued)

21. Summary

Well-designed and -executed pregnancy registries 

are an efficient initial approach to assess the safety 

of biopharmaceuticals during pregnancy, and can 

provide data that health care providers can use in 

treating and counseling patients who are pregnant 

or wish to become pregnant. Although pregnancy 

registries are more appropriate to identify or rule 

out large increases in the risk for malformations 

than to identify more modest teratogenic risks, 

they are a valuable tool to establish safety 

boundaries around risk estimates as data 

accumulate. Pregnancy registries have some 

unique characteristics that distinguish them from 

other registries or types of surveillance. Critical 

methodological issues to consider in their design 

include the prospective enrollment of women 

before the pregnancy outcome is known, inclusion 

of a comparable reference group, thoughtful 

assessment of drug exposure, ascertainment of 

prenatal and postnatal diagnosis, and validation of 

outcomes. Chance and potential biases should be 

considered when interpreting results from any 

observational study. A surveillance program 

should consider a combination of different sources 

of data so that associations detected in one study 

can be replicated or refuted by others.20
159

Chapter 21. Pregnancy Registries

Case Example 49. Expanding an ongoing 

pregnancy registry

Description The Antiretroviral Pregnancy 

Registry is the oldest ongoing 

pregnancy exposure registry. 

This multisponsor, international, 

voluntary, collaborative registry 

monitors prenatal exposures to 

all marketed antiretroviral drugs 

for potential risk of birth defects.

Sponsors Abbott Laboratories, Apotex 

Inc, Aurobindo Pharma Ltd., 

Boehringer Ingelheim 

Pharmaceuticals Inc., BristolMyers Squibb Company, Cipla 

Ltd., Gilead Sciences Inc., HEC 

Pharm, Hetero USA, Janssen 

Infectious Diseases BVBA, 

Merck & Co. Inc., Mylan 

Laboratories, Novartis 

Pharmaceuticals, Pfizer Inc., 

Ranbaxy Inc., Roche, Teva 

Pharmaceuticals, and ViiV 

Healthcare (represented by 

GlaxoSmithKline).

Year Started 1989

Year Ended Ongoing

No. of Sites Not site-based; open to all health 

care providers. Nearly 2,000 

health care providers have 

contributed data to the registry.

No. of Patients 16,732

Challenge

Antiretroviral treatments represent an area of 

particular concern for monitoring safety in 

pregnancy. Women may need to take the drugs 

during pregnancy to manage their own HIV 

infection and to reduce the risk of transmitting 

HIV to the infant, but these benefits must be 

weighed against the risk of teratogenic effects. 

Because of these factors, it is extremely 

important for clinicians and patients to 

understand the risks of using antiretroviral drugs 

during pregnancy in order to make an informed 

decision. However, ethical and practical concerns 

make a randomized trial to gather these data 

difficult, if not impossible.

In 1989, the first manufacturer of an antiretroviral 

drug voluntarily initiated a pregnancy exposure 

registry to track the outcomes of women who had 

used its product during pregnancy. The purpose 

of the registry is to collect information on any 

teratogenic effects of the product by prospectively 

enrolling women during the course of their 

pregnancy and following up with them to 

determine the outcome of the pregnancy. 

Physicians enroll a patient by providing 

information on the pregnancy dates, 

characteristics of the HIV infection, drug dosage, 

length of therapy, and trimester of exposure to the 

antiretroviral drug. Information on the pregnancy 

outcome is gathered through a followup form sent 

to the physician after the expected delivery date.

In 1993, the registry was expanded to include all 

antiretroviral drugs, as other manufacturers 

voluntarily joined the registry once their drugs 

were on the market. The registry is international 

in scope and allows any health care provider to 

enroll a patient who intentionally or 

unintentionally has used an antiretroviral drug 

during pregnancy. The U.S. Food and Drug 

Administration, which has used this registry as a 

model for new pregnancy registries, now requires 

participation in the registry for all new and 

generic antiretroviral drugs.

The year 2012 marks the 20th anniversary of the 

registry. Since 2006, the registry has more than 

doubled the enrollment of the first 15 years, 

increasing enrollment from 6,893 pregnancies in 

2006 to 16,732 in 2012. This increase was partly 

due to the increased number of new antiretroviral 

medications on the market. In 2006, the registry 

monitored 28 medications from 8 companies; by 

2012, it monitored 36 medications from 18 

companies (including manufacturers of both 

branded and generic products). The registry has 

also increased enrollment as well as its 

geographic representation by incorporating the 

Case Examples for Chapter 21
160

Section V. Special Applications in Patient Registries

Case Example 49. Expanding an ongoing 

pregnancy registry (continued)

Challenge (continued)

datasets of comparable, completed 

epidemiological studies. For example, the registry 

added data on nearly 1,000 women from a study 

conducted in Brazil and Argentina of 

antiretroviral-exposed pregnant women who 

delivered between the years 2002 and 2007. 

In addition to this large increase in enrollment, 

electronic data capture (EDC) was introduced in 

2010 as a data collection method for the registry. 

In summary, early challenges for the registry 

included establishing standard processes for 

monitoring and assessing the safety of drugs 

during pregnancy. Key challenges in recent years 

have included managing the methodological and 

analytic implications of a rapid growth in size and 

the operational implications of adding EDC.

Proposed Solution

To ensure both rigor and consistency early on, the 

registry put in place predefined analytic methods 

and criteria for recognizing a potential teratogenic 

signal. Tools for coding and classifying birth 

defects were developed for the registry to 

maximize the likelihood of identifying a 

teratogenic signal. This unique system groups 

birth defects by etiology or embryology rather 

than by general location or category, as does the 

Medical Dictionary for Regulatory Activities 

(MedDRA). Grouping like defects together 

increases the likelihood of detecting a potential 

signal. The registry also codes the temporal 

association between timing of exposure and 

formation of the birth defect, aiding in signal 

detection.

Specific monitoring criteria were developed for 

evaluating signals at various levels, including the 

Rule of Three (the rule that three exposurespecific cases with the same birth defect require 

immediate evaluation). This rule is based on the 

statistical principle that the likelihood of finding 

at least three of any specific defect in a cohort of 

600 or fewer by chance alone is less than 5 

percent.

In the last few years of the registry’s operation, 

large increases in enrollment required reevaluation of the adequacy of existing signal 

detection rules. The Rule of Three continues to 

serve an important role; however, understanding 

weak signals is methodologically challenging. 

Incorporating enrollments from comparable 

epidemiological studies into the registry 

population has boosted enrollment, increased 

cultural diversity, and enhanced signal detection 

capabilities. Each merger of external data 

prompts the need to re-examine the potential for 

selection and ascertainment bias. 

Operationally, each new participating 

manufacturer undergoes a series of trainings and 

is required to obtain institutional review board 

approval before participation in the registry. 

Registry trainings and standard operating 

procedures are reviewed at biannual steering 

committee meetings and revised as appropriate. 

In expanding the options for data entry into the 

registry, a hybrid EDC-paper approach was 

deemed operationally feasible in lieu of an 

EDC-only approach. This allowed a subset of 

established reporters to use EDC, while limiting 

disruption for reporters who preferred to report 

data on paper CRFs.

Results

The registry now contains data on 16,732 

prospective pregnancies with exposure to 36 

medications from 18 companies. Approximately 

40 percent of new enrollments in the registry are 

made using EDC technology.

Registry data have been used in 13 publications, 

9 abstracts, and 25 presentations, and the registry 

design and operation have been the subject of 

many publications and presentations. The registry 

findings can help provide clinicians and patients 

with information to make informed decisions 

regarding use of antiretroviral drugs during 

pregnancy.

Key Point

A pregnancy exposure registry can employ 

continuous quality improvement practices to 

identify and define key quality processes and 
161

Chapter 21. Pregnancy Registries

Case Example 50. Using a pregnancy registry 

to detect major teratogenicity

Description The International Lamotrigine 

Pregnancy Registry was 

established to monitor for the 

signal of major teratogenicity 

following in utero exposure to 

lamotrigine.

Sponsor GlaxoSmithKline

Year Started 1992

Year Ended 2010

No. of Sites Not applicable; health care 

providers reported lamotrigine 

exposure during pregnancy and 

subsequent pregnancy outcomes 

on a voluntary basis.

No. of Patients 1,558

Challenge

Lamotrigine is a second-generation 

anticonvulsant therapy, widely indicated for the 

treatment of epilepsy. Lamotrigine was approved 

in the United States for the treatment of epilepsy 

in 1994 and for the treatment of bipolar disorder 

in 2003. In 1992, following the approvals of 

lamotrigine in several different European 

countries, the International Lamotrigine 

Pregnancy Registry was established to monitor 

the frequency of major teratogenicity following in 

utero exposure to lamotrigine. Major congenital 

malformation (MCMs), identified after birth and 

before hospital discharge, was the primary 

outcome evaluated by the registry. The U.S. 

Centers for Disease Control and Prevention’s 

Metropolitan Atlanta Congenital Defects Program 

case definition was used to classify MCMs. Due 

to the rarity of the outcome, the registry needed 

to enroll enough limotrigine-exposed patients in 

order to have adequate statistical power to detect 

changes in MCM frequencies.

Proposed Solution

The registry targeted an enrollment of 1,000 

limotrigine-exposed pregnant women. 

Prospective reporting early in pregnancy was 

encouraged. The registry also received and 

reviewed retrospective reports, defined as those 

for which the pregnancy outcome was known at 

the time of reporting. Due to successful patient 

enrollment, the registry closed to new prospective 

enrollments in June 2009, and continued to 

follow up with existing enrollments through 

Case Example 49. Expanding an ongoing 

pregnancy registry (continued)

Key Point (continued)

keep the registry current and innovative 

throughout its life cycle. The fact that the registry 

had established, standard policies and procedures 

for coding, monitoring, and analysis was critical 

in incorporating new partners and data sources 

quickly and easily. Regular review of these 

policies and procedures is essential to respond to 

the changing registry environment.

For More Information

Antiretroviral Pregnancy Registry Steering 

Committee. Antiretroviral Pregnancy Registry 

International Interim Report for 1 January 1989 

through 31 January 2012. Wilmington, NC: 

Registry Coordinating Center; 2012. 

http://www.apregistry.com/.

Tilson H, Roberts S, Watts H, et al. The 

Antiretroviral Pregnancy Registry: A 20th 

anniversary celebration. Pharmacoepidemiology 

and Drug Safety. 2011;20(S1):S190.

Tilson H, Doi PA, Covington DL, et al. The 

antiretrovirals in pregnancy registry: A fifteenth 

anniversary celebration. Obstet Gynecol Surv. 

2007;62:137–48.

Covington D, Tilson H, Elder J, et al. Assessing 

teratogenicity of antiretroviral drugs: monitoring 

and analysis plan of the Antiretroviral Pregnancy 

Registry. Pharmacoepidemiol Drug Saf. 

2004;13:537–45.

Scheuerle A, Covington D. Clinical review 

procedures for the Antiretroviral Pregnancy 

Registry. Pharmacoepidemiol Drug Saf. 

2004;13:529–36.
162

Section V. Special Applications in Patient Registries

Case Example 50. Using a pregnancy registry 

to detect major teratogenicity (continued)

Proposed Solution (continued)

March 2010. The results of the International 

Lamotrigine Pregnancy Registry were compared 

descriptively against the results of other ongoing 

anti-epileptic drug (AED) pregnancy registries. 

While major teratogenicity was evaluated, the 

registry was not powered to determine the 

frequency of specific malformation types; 

surveillance for specific types using the European 

Congenital Anomalies and Twins Registers 

(EUROCAT) network is planned.

Results

At registry closure, over 1,500 birth outcomes 

involving first-trimester monotherapy exposure 

had been evaluated during the 18-year registry 

period. The registry was thus adequately powered 

to meet its primary objective, to determine 

whether the overall rate of major malformations 

was increased among the offspring of exposed 

women. The registry did not detect an appreciable 

increase in the outcome of MCMs overall. Over 

an 18-year period, 35 infants with MCMs were 

observed among 1,558 first-trimester 

monotherapy exposures: 2.2% (95% CI, 1.6 to 

3.1). This was similar to estimates from general 

population-based cohorts and no pattern of 

malformation frequency by dose was observed. 

However, the registry was not powered to exclude 

increases in the rates of specific defects.

First-trimester monotherapy results from the 

registry were consistent with several other 

ongoing AED pregnancy registries, such as the 

North American Antiepileptic Drug Pregnancy 

Registry (2.3%, 95% CI, 1.3 to3.8) and the 

EURAP international pregnancy registry (2.9%, 

95% CI 2.1 to 4.1). Monitoring of specific 

malformations among lamotrigine-exposed 

pregnancies will continue through case-control 

surveillance in the EUROCAT network.

Key Point

A drug-specific pregnancy registry can provide 

valuable information about risks of major 

congenital malformations following in utero 

exposure; however, it may take several years to 

collect enough exposed patients to detect a signal 

of teratogenicity with sufficient statistical power. 

For such rare outcomes, accumulating data over 

long periods of time and from multiple registries 

is advantageous to monitoring the safety of 

medical treatments used in pregnancy.

For More Information

Cunnington MC, Weil JG, Messenheimer JA, et 

al. Final results from 18 years of the International 

Lamotrigine Pregnancy Registry. Neurology. 

2011 May 24;76(21):1817-23.
163

Chapter 21. Pregnancy Registries

Case Example 51. Implementing a nonmandated pregnancy registry

Description The Global Gleevec®/Glivec®

& Tasigna® Pregnancy Exposure 

Registry is an international, 

prospective, observational 

registry of women and their 

offspring exposed to Gleevec/

Glivec (imatinib) and/or Tasigna 

(nilotinib) during pregnancy or 

within 6 months prior to 

pregnancy. The primary 

objective of the registry is to 

monitor pregnancies exposed to 

Gleevec/Glivec or Tasigna 

(tyrosine kinase inhibitors to 

treat some cancers) to assess the 

prevalence of birth defects. 

Secondary objectives include 

determining the impact of 

interrupted treatment on 

maternal disease status and 

assessing infant development 

around 12 months postdelivery.

Sponsor Novartis

Year Started 2011

Year Ended Ongoing

No. of Sites 4

No. of Patients 5 (expected enrollment of 150)

Challenge

Chronic myelogenous leukemia and 

gastrointestinal stromal tumor are rare oncologic 

diseases. Since the approval of Gleevec/Glivec in 

2001, life expectancy for these diseases has 

substantially improved and treatment is now 

considered chronic. Tasigna, approved for chronic 

myelogenous leukemia treatment in 2007, is 

shown to have superior efficacy compared with 

Gleevec/Glivec. Patients of child-bearing age are 

now contemplating reproductive opportunities 

that would not have been possible previously, and 

are requesting information about the safety and 

effectiveness of these treatments during 

pregnancy. The sponsor sought to address this 

unmet medical need by collecting data on 

pregnancy outcomes and on the disease status of 

pregnant women who were exposed to Gleevec/

Glivec or Tasigna during pregnancy. 

Pharmaceutical companies typically establish 

pregnancy registries when they are mandated to 

do so by regulatory authorities. This registry is 

voluntary and was not mandated by regulatory 

authorities, and consequently the sponsor 

experienced challenges in study startup and in 

enrollment. Regulatory requirements in countries 

outside the United States evolved during startup, 

and these changes required revisions to study 

design. Additionally, both Gleevec/Glivec and 

Tasigna are classified by the U.S. Food and Drug 

Administration (FDA) as pregnancy category D, 

and the prescribing information carries a warning 

that these drugs should not be administered to 

pregnant women. Thus, the sponsor was careful to 

design an awareness campaign that did not 

promote exposure during pregnancy. 

Most existing pregnancy registry models focus 

primarily on collecting birth outcomes and do not 

collect maternal disease status or postpartum 

data. Since this registry was not mandated, and 

information on maternal health as well as birth 

outcomes was desired, a novel model was needed.

Proposed Solution

The registry was launched in the United States in 

2011 and in Russia, the Netherlands, and 

Denmark in 2012. The sponsor adopted a national 

coordinator (NC) model for the registry. The NC 

is a domestic entity responsible for submitting the 

protocol to the appropriate regulatory authority 

for approval and for facilitating the collection of 

registry data from multiple sources 

(e.g., oncologists, hematologists, obstetricians, 

pediatricians). In the United States, the NC is a 

contract research organization; outside the United 

States, the NC is a participating physician who 

has agreed to assume the above responsibilities. 

Data are collected on maternal disease status at 3 

time points: enrollment, pregnancy outcome, and 

about 12 months postdelivery according to local 

standard of care. Data are also collected on the 

fetus/infant at pregnancy outcome and about 12 

months of age.
164

Section V. Special Applications in Patient Registries

Case Example 51. Implementing a nonmandated pregnancy registry (continued)

Proposed Solution (continued)

A Web site was constructed that provides 

information about the registry and instructs 

patients (only in the United States) and providers 

(globally) on how to enroll into the registry. The 

sponsor worked closely with regulatory 

authorities to ensure that the information 

presented on the Web site is educational and not 

promotional in nature. Patient recruitment is 

facilitated through the existing pregnancy 

reporting infrastructure in the sponsor’s safety 

department; U.S. health care providers calling to 

report pregnancies are invited to enroll eligible 

patients into the registry. Additionally, 

collaborating with patient advocacy groups and 

preparing durable materials facilitates registry 

awareness.

Results

The sponsor encountered unanticipated delays 

and challenges from regulatory authorities, 

possibly because non-mandated pregnancy 

registries are unprecedented. Recent European 

Medicines Agency legislation expanding the 

safety reporting requirement for 

noninterventional studies has necessitated further 

revisions to the protocol and case report form. 

Work is continuing on expanding the registry to 

other countries.

Five patients have been enrolled to date, and this 

number is expected to increase when the 

registry’s Web site and awareness materials are 

approved by FDA for use in the United States. 

Key Point

Pregnancy registries not mandated by regulatory 

authorities present unique operational challenges. 

Beginning a study in locations with a favorable 

regulatory environment may help minimize 

delays in startup and allow sponsors to apply 

lessons learned before expanding the registry to 

other locations. Staying current with regulatory 

requirements in long-term pregnancy registries is 

critical to remaining compliant. When collecting 

data from several different sources (e.g., patients 

and multiple providers), consider an operational 

model that centralizes the responsibility for data 

collection and for health authority and ethics 

committee submission and approval.

For More Information

Juma M, Ericson S, Eng D, et al. Prospective, 

observational registry of branded imatinib and 

nilotinib exposure in pregnant women: voluntary 

post-authorization safety study. Poster presented 

at 2012 annual meeting of the American Society 

of Clinical Oncology (ASCO). Abstract 

# TPS6638.

Juma M, Ericson S, Eng DF, et al. Prospective, 

observational registry of branded imatinib and 

nilotinib exposure in pregnant women: Voluntary 

post-authorization safety study. Poster presented 

at the 2011 ASCO annual meeting. Abstract 

#82968.
165

Chapter 21. Pregnancy Registries

Case Example 52. Using proactive awareness 

activities to recruit patients for a pregnancy 

exposure registry

Description The Ribavirin Pregnancy 

Registry is a component of the 

Ribavirin Risk Management 

Program. It was designed to 

evaluate the association between 

ribavirin and birth defects 

occurring in the offspring of 

female patients exposed to 

ribavirin during pregnancy or the 

6 months prior to conception, as 

well as female partners of male 

patients exposed to ribavirin 

during the same time period. 

The registry collects prospective, 

observational data on 

pregnancies and outcomes 

following pregnancy exposure to 

ribavirin.

Sponsor Aurobindo Phama USA; 

Genentech, Inc.; Sandoz 

Pharmaceuticals Inc.; Schering 

Corporation, a subsidiary of 

Merck & Co. Inc.; Teva 

Pharmaceuticals USA, Inc.; 

Three Rivers Pharmaceuticals, 

LLC; Zydus Pharmaceuticals 

(USA) Inc.

Year Started 2003

Year Ended Ongoing

No. of Sites Not applicable (populationbased)

No. of Patients Approximately 230 evaluable 

pregnancies

Challenge

Ribavirin is used in combination with interferon 

alfa or pegylated interferon alfa for the treatment 

of hepatitis C. Chronic hepatitis C presents a 

serious health concern for approximately three 

million Americans, as the infection, if left 

untreated, can lead to end-stage liver disease, 

primary liver cancer, and death. When used as 

part of a combination therapy, ribavirin can 

significantly increase both viral clearance and 

liver biopsy improvement for hepatitis C patients.

However, ribavirin showed teratogenic properties 

in all animal models tested, making pregnancy 

exposure a concern. There are minimal data on 

ribavirin exposure in human pregnancies. Thus, 

the U.S. Food and Drug Administration (FDA) 

designated ribavirin as a Pregnancy Category X 

product based on the animal data, and ribavirin 

carries product label warnings against becoming 

pregnant.

Despite the product warnings, pregnancies 

continue to occur. Health care professionals have 

insufficient data on the teratogenic properties of 

ribavirin in humans to counsel pregnant women 

exposed to ribavirin either during pregnancy or in 

the 6 months prior to conception. The registry 

was established to gather prospective data on 

ribavirin exposure in pregnancy and pregnancy 

outcomes to better understand the actual risk.

The registry collects data on direct exposures 

through the pregnant female and indirect 

exposures through her male sexual partner. 

Health care providers, pregnant patients, or 

pregnant patients’ male sexual partners may 

submit data to the registry. The registry collects 

minimal, targeted data at each trimester and at the 

outcome of the pregnancy through the obstetric 

health care providers. For live births, the registry 

collects data at 6 months and 12 months after the 

birth by contacting the pediatric health care 

provider.

To gather data on these patients, the registry 

needed to develop proactive awareness activities 

to make patients and providers aware of the 

program and encourage enrollment without 

promoting ribavirin use during pregnancy.

Proposed Solution

The registry team developed a multipronged 

approach to recruiting patients. First, the team 

developed a comprehensive Web site with 

information for patients and providers. The Web 

site contains fact sheets, data forms, information 

on how to participate, and contact information. 
166

Section V. Special Applications in Patient Registries

Case Example 52. Using proactive awareness 

activities to recruit patients for a pregnancy 

exposure registry (continued)

Proposed Solution (continued)

The site also contains a complete slide set that 

health care providers can use for teaching 

activities.

While the site contains detailed information on 

the scientific reasons for the registry, the tone and 

content of the Web site are patient friendly, 

making it a good resource for both potential 

patients and providers.

Next, the team began targeting professional 

service groups whose members might treat 

patients with ribavirin exposure during 

pregnancy. The groups included hepatologists, 

gastroenterologists, obstetricians, and 

pediatricians. By contacting the groups’ 

leadership and sending individualized mailings to 

members, the team hoped to raise awareness 

across a broad spectrum of providers. The team 

communicated with nursing groups, including 

publishing an article in a nursing journal targeted 

to gastroenterology nurses, with the goal of 

utilizing the nurse’s role as a patient educator. As 

a result of these efforts, the American 

Gastroenterological Association placed a link for 

the registry Web site on its Web site, and the 

American Association for the Study of Liver 

Diseases posted on its Web site an expert opinion 

piece written by the former registry advisory 

board chair.

The registry team also raised awareness among 

professional groups by attending conferences. In 

2005, the team presented a poster about the 

registry, including some information on 

demographics and program objectives, at the 

Centers for Disease Control and Prevention 

National Viral Hepatitis Prevention Conference. 

In 2009, the team presented a poster at the 

International Society for Pharmacoepidemiology 

and in 2010 at the conference of the Teratology 

Society. 

In 2010, to expand awareness efforts to health 

care providers, the registry published results after 

five years of enrollment, even though the targeted 

sample size had not been reached. In 2011, the 

registry developed an article for Peri-FACTS, a 

continuing education eJournal for OB/GYNs, 

nurses, and other health care providers, sponsored 

by the University of Rochester. Beginning in 

2012, the registry began providing the executive 

summary of its annual interim report to health 

care providers upon request; this summary 

provides the most up-to-date snapshot of registry 

activity.

To raise awareness among patients, the team 

talked to hepatitis C patient advocacy groups. The 

registry gained exposure with patients when one 

patient group wrote an article about the registry 

for its newsletter and included the registry phone 

number on its fact sheet. This effort led to many 

patient-initiated enrollments, despite the lack of 

patient incentives. In working with patients, the 

registry has found that emphasizing the goal, 

which is to gather information to help future 

patients make better decisions, resonates with 

patients. Most patients submit data to the registry 

over the phone, and the rapport that the 

interviewers have developed with patients has 

helped to reduce the number of patients who are 

lost to followup.

In addition to targeting providers and patients 

directly, the team enlisted the help of public 

health agencies, since the registry has a strong 

public health purpose. Registry Web links are 

posted on the Web sites of the FDA’s Office of 

Women’s Health and the Department of Veterans 

Affairs. A description of the registry is posted on 

ClinicalTrials.gov.

The team also reviewed the registry process to 

identify potential barriers to enrollment. Under 

the initial rules for giving informed consent, the 

registry call center contacted patients and asked 

them if they were interested in participating. If 

patients agreed to participate over the phone, the 

call center sent a package of information through 

the mail, including an informed consent 

document, which the patients needed to sign and 

return before they could enroll. While many 

patients agreed to participate over the phone, a 

much smaller number actually returned the 

informed consent document. The team identified 
167

Chapter 21. Pregnancy Registries

Case Example 52. Using proactive awareness 

activities to recruit patients for a pregnancy 

exposure registry (continued)

Proposed Solution (continued)

the process of obtaining written informed consent 

as a key barrier to enrollment.

After discussions with FDA, the registry team 

and FDA approached the study institutional 

review board about receiving a waiver of written 

informed consent because of the public health 

importance of the registry. The board agreed that 

oral consent over the phone would be sufficient 

for this study. Now, the call center can complete 

the enrollment process in a single step, as they 

can obtain oral consent over the phone and then 

proceed with the interview. This change improved 

and streamlined the enrollment process and 

significantly increased the number of participants 

in the registry.

Throughout all of these recruitment activities, the 

registry team has emphasized that the purpose of 

the registry is to answer important safety 

questions for the benefit of future patients and 

providers. By focusing on the public health 

purpose of the registry, the team has been able to 

encourage participation from both patients and 

providers. The team has also found that a key 

element of their recruitment strategy is their 

detailed awareness plan, which calls for 

completing awareness activities monthly. Because 

the leadership and membership of professional 

groups change and new patients begin taking 

ribavirin, the team has found that continual 

awareness activities are important for keeping 

patients and providers aware of the registry.

Results

Through proactive awareness activities, the 

registry team has generated interest in the project 

and enrolled approximately 230 exposed 

pregnancies with outcome information to date. 

The streamlined oral consent process led to 

increased enrollment.

Key Point

Recruitment activities may include working with 

professional groups, contacting patient groups, 

targeting public health agencies, producing 

publications, and using a Web site to share 

information. Once recruitment and enrollment 

have begun, the registry team may need to 

re-evaluate the process to identify any potential 

barriers to enrollment if enrollment is not 

proceeding as planned. If a registry has an 

ongoing enrollment process, a plan to continually 

raise awareness about the registry is an important 

part of the recruitment plan.

For More Information

Roberts S. Assessing ribavirin exposure during 

pregnancy: the Ribavirin Pregnancy Registry. 

Gastroenterol Nurs. 2008;31(6):413–7.

Roberts SS and the Scientific Advisory Board of 

the Ribavirin Pregnancy Registry. The Ribavirin 

Pregnancy Registry is Established. Poster 

presented at the CDC’s National Viral Hepatitis 

Prevention Conference, Washington, DC, 

December 5-9, 2005. 

Roberts SS, McKain LF, Covington DL, et al. 

Paternal exposures and birth defects: overlooked 

or unnecessary? Pharmacoepidemiology and 

Drug Safety. 2009;18:S77.

Roberts SS, Miller RK, Jones JK, et al. The 

Ribavirin Pregnancy Registry: Findings after 5 

Years of Enrollment, 2003-2009. Birth Defects 

Res A Clin Mol Teratol. 2010,88:551-9. 

Miller RK, Roberts S, Chambers C, et al. Safety 

of medications during pregnancy: the importance 

of prospective studies, pregnancy registries, and 

health care provider collaboration. Peri-Facts, 

Case 942. University of Rochester, May 2011. 
168

Section V. Special Applications in Patient Registries

References for Chapter 21

1. Kennedy DL, Uhl K, Kweder SL. Pregnancy 

Exposure Registries. Drug Safety. 2004;

27:215-28.

2. Guidance for industry: establishing pregnancy 

exposure registries. 2002. http://www.fda.gov/

downloads/Drugs/GuidanceComplianceRegulatory

Information/Guidances/ucm071639.pdf. Accessed 

June 2011. 

3. Wyszynski D. Pregnancy Exposure Registries: 

Academic Opportunities and Industry 

Responsibility. Birth Defects Research Part A: 

Clinical and Molecular Teratology. 2009;

85:93-101.

4. List of pregnancy registries. http://www.fda.gov/

ScienceResearch/SpecialTopics/WomensHealth

Research/ucm134848.htm.

5. Guideline On The Exposure To Medicinal 

Products During Pregnancy: Need For PostAuthorisation Data, (Committee for Medicinal 

Products for Human Use, 2005). 2005. 

http://www.tga.gov.au/pdf/euguide/

phvwp31366605en.pdf. Accessed June 2011. .

6. FDAAA. US Food and Drug Administration 

Amendments Act (FDAAA) of 2007, Pub. L. No. 

75-711, 52 Stat. 1040 (1938) as amended. 2007.

7. Merkatz R, Temple R, Sobel S, et al. Women in 

clinical trials of new drugs. A change in Food and 

Drug Administration Policy. New England Journal 

of Medicine. 1993;329:292-6.

8. Howard T, Tassinari M, Feibus K, et al. 

Monitoring for teratogenic signals: Pregnancy 

registries and surveillance methods. Am J Med 

Genet C Semin Med Genet. 2011;157:209-14.

9. Finer L, Henshaw S. Disparities in Rates of 

Unintended Pregnancy In the United States, 1994 

and 2001. Perspectives on Sexual and 

Reproductive Health. 2006;38:90-6.

10. Wilson JG. Evaluation of human teratologic risk 

in animals. In: Lee DH, Hewson EW, Okun D, eds. 

Environment and birth defects. First ed. New York 

and London: Academic Press; 1973. pp. 146-60.

11. Holmes L. Human teratogens: update 2010. Birth 

Defects Research Part A: Clinical and Molecular 

Teratology. 2011;91(1):1-7.

12. McAdams M, Staffa J, Dal Pan G. Estimating the 

extent of reporting to FDA: a case study of 

statin-associated rhabdomyolysis. 

Pharmacoepidemiol & Drug Safety. 2008;17(3):

229-39.

13. Wysowski D, Swartz L. Adverse drug event 

surveillance and drug withdrawals in the United 

States, 1969–2002: the importance of reporting 

suspected reactions. Archives of Internal 

Medicine. 2005;165(12):1362-9.

14. Mathis L, Iyasu S. Safety monitoring of drugs 

granted exclusivity under the Best 

Pharmaceuticals for Children Act: what the FDA 

has learned. Clinical Pharmacology and 

Therapeutics. 2007;82(2):133-4.

15. Werler M, Louik C, Mitchell A. Case-control 

studies for identifying novel teratogens. Am J Med 

Genet C Semin Med Genet. 2011 Aug 15;

157C(3):201-8.

16. Schneeweiss S, Avorn J. A review of uses of health 

care utilization databases for epidemiologic 

research on therapeutics. Journal of Clinical 

Epidemiology. 2005;58(4):323-37.

17. Charlton RA, Cunnington M, de Vries CS, et al. 

Data Resources for Investigating Drug Exposure 

during Pregnancy and Associated Outcomes. The 

General Practice Research Database (GPRD) as 

an Alternative to Pregnancy Registries. Drug 

Safety. 2008;31:39-51.

18. Andrade S, Davis R, Cheetham TC, et al. 

Medication Exposure in Pregnancy Risk 

Evaluation Program. Matern Child Health. 

2012 Oct;16(7):1349-54.

19. Honein MA, Paulozzi LJ, Cragan JD, et al. 

Evaluation of selected characteristics of pregnancy 

drug registries. Teratology. 1999;60:356-64.

20. Mitchell AA. Systematic Identification of Drugs 

That Cause Birth Defects—A New Opportunity. 

New England Journal of Medicine. 

2003;349:2556-9.

21. ISPE. Guidelines for good pharmacoepidemiology 

practices (GPP). Pharmacoepidemiol & Drug 

Safety. 2008;17:200-8.

22. deMoor C, Covington D, Golembesky A. Differing 

Strategies for Analyzing Multiple Gestation 

Pregnancies in Pregnancy Registries: Impact on 

Birth Defect Rates. Pharmacoepidemiol & Drug 

Safety. 2010;19 (Supplement 1):S124.
169

Chapter 21. Pregnancy Registries

23. Cragan JD, Gilboa SM. Including Prenatal 

Diagnoses in Birth Defects Monitoring: 

Experience of the Metropolitan Atlanta Congenital 

Defects Program. Birth Defects Research Part A: 

Clinical and Molecular Teratology. 2009;85:20-9.

24. Reiff_Eldridge R, Heffner CR, Ephross SA, et al. 

Monitoring pregnancy outcomes after prenatal 

drug exposure through prospective pregnancy 

registries: A pharmaceutical company 

commitment. American Journal of Obstetrics and 

Gynecology. 2000;182:159-63.

25. Tabacova S, Little R, Tsong Y, et al. Adverse 

pregnancy outcomes associated with maternal 

enalapril antihypertensive treatment. 

Pharmacoepidemiol & Drug Safety. 2003;

12:633-46.

26. Chambers CD, Johnson KA, Dick LM, et al. Birth 

outcomes in pregnant women taking fluoxetine. 

New England Journal of Medicine. 

1996;335:1010-5.

27. Chambers C, Hernández-Díaz S, Van Marter L, et 

al. Selective Serotonin-Reuptake Inhibitors and 

Risk of Persistent Pulmonary Hypertension of the 

Newborn. New England Journal of Medicine. 

2006;354:579-87.

28. Briggs GG, Polifka J, Committee OR. Better Data 

Needed from Pregnancy Registries. Birth Defects 

Research Part A: Clinical and Molecular 

Teratology. 2009;85:109-11.

29. Correa A, Cragan J, Kucik J, et al. Reporting birth 

defects surveillance data 1968-2003 Birth Defects 

Research Part A: Clinical and Molecular 

Teratology. 2007;79:65-86.

30. Rothman KJ, Ray W. Should cases with a known 

cause of their disease be excluded from study? 

Pharmacoepidemiology & Drug Safety. 

2002;11:11-4.

31. Dolk H, Loane M, Garne E. The prevalence of 

congenital anomalies in Europe. Adv Exp Med 

Biol. 2010;686:349-64.

32. Chambers C, Braddock SR, Briggs GG, et al. 

Postmarketing surveillance for human 

teratogenicity: A model approach. Teratology. 

2001;64:252-61.

33. Holmes L, Wyszynski D, Lieberman E. The 

Antiepileptic Drug Pregnancy Registry: a six year 

experience. Archives of Neurology. 2004;61:673-8.

34. Covington D, Tilson H, Elder J, et al. Assessing 

teratogenicity of antiretroviral drugs: monitoring 

and analysis plan of the Antiretroviral Pregnancy 

Registry. Pharmacoepidemiol & Drug Safety. 

2004;13:537-45.

35. Hernán MA, Hernández-Díaz S, Werler MM, et 

al. Causal knowledge as a prerequisite for 

confounding evaluation. An application to birth 

defects epidemiology. American Journal of 

Epidemiology. 2002;155:176-84.

36. Hernán MA, Hernández-Díaz S, Robins JM. 

A structural approach to selection bias. 

Epidemiology. 2004;15(5):615-25.

37. Hernández-Díaz S, Schisterman EF, Hernán MA. 

The birth weight “paradox” uncovered? American 

Journal of Epidemiology. 2006;164:1115-20.

38. Editors. The registration of observational studieswhen metaphors go bad. Epidemiology. 

2010;21:607-20.

39. Roberts SS, Covington D, Doi P. Pregnancy 

exposure registries: streamlining regulatory 

requirements to increase enrollment. 

Pharmacoepidemiol Drug Saf. 2005;15(Suppl 2).

40. MacDonald PDM. Methods in Field 

Epidemiology. Burlington, MA: Jones & Bartlett 

Learning; 2012. p. 63.

41. Mitchell AA, Gilboa SM, Werler MM, et al. 

Medication use during pregnancy, with particular 

focus on prescription drugs: 1976-2008. Am J 

Obstet Gynecol. 2011 Jul;205(1):51 e1-8. 

PMID: 21514558.

171

Chapter 22. Quality Improvement Registries

1. Introduction

Quality assessment/improvement registries 

(QI registries) seek to use systematic data 

collection and other tools to improve quality of 

care. While much of the information contained in 

the other chapters of this document applies to QI 

registries, these types of registries face unique 

challenges in the planning, design, and operation 

phases. The purpose of this chapter is to describe 

the unique considerations related to QI registries. 

Case Examples 53, 54, 55, 56, and 57 offer some 

descriptions of quality improvement registries.

While QI registries may have many purposes, at 

least one purpose is quality improvement. These 

registries generally fall into two categories: 

registries of patients exposed to particular health 

services (e.g., procedure registry, hospitalization 

registry) around a relatively short period of time 

(i.e., an event); and those with a disease/condition 

tracked over time through multiple provider 

encounters and/or multiple health services. An 

important commonality is that one exposure of 

interest is to health care providers/health care 

systems. These registries exist at the local, 

regional, national, and international levels. 

QI registries are further distinguished from other 

types of registries by the tools that are used in 

conjunction with the systematic collection of data 

to improve quality at the population and individual 

patient levels. QI registries leverage the data about 

the individual patient or population to improve 

care in a variety of ways. Examples of tools that 

facilitate data use for care improvement include 

patient lists, decision support tools (typically based 

on clinical practice guidelines), automated 

notifications, communication tools (e.g., patient 

educational materials), and patient- and 

population-level reporting systems. For example, a 

diabetes registry managed by a single institution 

might provide a listing of all patients in a 

provider’s practice who have diabetes and who are 

due for a clinical examination or other 

assessments. Decision support tools exist that 

assess the structured patient data provided to the 

registry and display recommendations for care 

based on evidence-based guidelines. This is a 

well-reported feature of the American Heart 

Association’s Get With The Guidelines® registries.1

Certain registry tools will automatically notify a 

provider if the patient is due for a test, exam, or 

other milestone. Some tools will even send 

notifications directly to patients indicating that 

they are due for a treatment such as a flu 

vaccination. Reports are a key part of quality 

improvement. These range from reports on 

individual patients, such as a longitudinal report 

tracking a key patient outcome, to reports on the 

population under care by a provider or group of 

providers, either alone or in comparison to others 

(at the local, regional, or national level). Examples 

of the latter reports include those that measure 

processes of care (e.g., whether specific care was 

delivered to appropriate patients at the appropriate 

time) and those that measure outcomes of care 

(e.g., average Oswestry score results for patients 

undergoing particular spine procedures, compared 

with similar providers).

QI registries can further support improved quality 

of care by giving providers and their patients more 

detailed information based on the aggregate 

experience of other patients in the registry. This 

can include both general information on the 

natural history of the disease process from the 

accumulated experience of other patients in the 

registry and more individual-patient–level 

information on specific risk calculators that might 

help guide treatment decisions. Registries that 

produce patient-specific predictors of short- and 

long-term outcomes (which can inform patients 

about themselves) as well as provider-specific 

outcomes benchmarked against national data 

(which can inform patients about the experience 

and outcomes of their providers) can be the basis 

of both transparent and shared decisionmaking 

between patients and their providers.
172

Section V. Special Applications in Patient Registries

In addition to these examples, there are tools that 

are neither electronic nor necessarily provided 

through the registry systems. Non-electronic 

examples range from internal rounds to review 

registry results and make action plans, to qualityfocused national or regional meetings that review 

treatment gaps identified from the registry data and 

teach solutions, to printed posters and cards or 

other reminders that display the key evidencebased recommendations that are measured in the 

registry. Further, even electronic tools need not be 

delivered through the registry systems themselves. 

While in many cases the registries do provide the 

functionality described above, the same purpose is 

served when an electronic health record (EHR) 

provides access to decision support relevant to the 

goals of the patient registry. In other words, what 

characterizes QI registries is not the embedding of 

the tools in the registry but the use of the tools by 

the providers who participate in the registry to 

improve the care they provide, and the use of the 

registry to measure that improvement.

2. Planning

As described in Chapter 2, developing a registry 

starts with thoughtful planning and goal setting. 

Planning for a QI registry follows most of the 

steps outlined in Chapter 2, with some noteworthy 

differences and additions. A first step in planning 

is identifying key stakeholders. Similar to other 

types of registries, regional and national QI 

registries benefit from broad stakeholder 

representation, which is necessary but not 

sufficient for success. In QI registries, the provider 

needs to be engaged and active, as the program is 

not simply supporting a surveillance function or 

providing a descriptive or analytic function, but is 

often focused on patient and/or provider behavior 

change. In many QI registries, these active 

providers are termed “champions” and are vital for 

success, particularly early in development.2 At the 

local level, the champions are typically the ones 

asking for the registry and almost by definition are 

engaged. Selecting stakeholders locally is 

generally focused on involving individuals with 

direct impact on care or those that can support the 

registry with information, systems, or labor. Yet, 

the common theme for both local and national QI 

registries is that the local champions must be 

successful in actively engaging their colleagues in 

order for the program to go beyond an “early 

adopter” stage and be sustainable within any local 

organization. Once a registry matures, other 

incentives may drive participation (e.g., 

recognition, competition, financial rewards, 

regulatory requirements), but the role of the 

champion in the early phases cannot be overstated. 

A second major difference between planning a QI 

registry and planning other types of registries is 

the funding model. QI registries use a wide variety 

of funding models. For example, a regional or 

national registry may be funded entirely by fees 

paid by participating providers or hospitals. 

Alternately, the registry may supplement 

participation fees with funding from professional 

associations, specialty societies, industry, 

foundations, or government agencies. Some QI 

registries may not charge a participation fee and 

may receive all of their funding from other 

organizations. Local QI registries that operate 

within a single institution may receive all of their 

funding from the institution or from research 

grants. The funding model used by a QI registry 

largely depends on the goals of the registry and the 

stakeholders in the specific disease area. 

Next, in order for a QI registry to meet its goal of 

improving care, it must provide actionable 

information for providers and/or participants to be 

able to modify their behaviors, processes, or 

systems of care. Actionable information can be 

provided in the form of patient outcomes measures 

(e.g., mortality, functional outcomes post 

discharge) or process of care or quality measures 

(e.g., compliance with clinical guidelines). While 

the ultimate goal of a QI registry is to improve 

patient outcomes by improving quality of care, it is 

not always possible for a QI registry to focus on 

patient outcome measures. In some cases, outcome 

measures may not exist in the disease area of 

interest, or the measures may require data 

collection over a longer period than is feasible in 

the registry. As a result, QI registries have often 

focused on process of care measures or quality 

measures. While this has been criticized as less 

important than focusing on measures of patient 

outcomes, it should be noted that quality measures 
173

Chapter 22. Quality Improvement Registries

are generally developed from evidence-based 

guidelines and emphasize interventions that have 

been shown to improve long-term outcomes, 

increasingly recognized through standardized 

processes (e.g., National Quality Forum), and are 

inherently actionable. Patient outcome measures, 

on the other hand, do not yet have consensus 

across many conditions, may be influenced by 

systematic loss to followup, and may be expensive 

and difficult to collect. Furthermore, long-term 

outcomes are generally not readily available for 

rapid-cycle initiatives and may be too distant from 

the time when the care is delivered to support 

effective behavior change. Nonetheless, there has 

been an increasing focus in recent years on 

including outcome measures instead of or in 

addition to process of care measures in QI 

registries. This shift is driven in part by research 

documenting the lack of correlation between 

process measures and patient outcomes3-5 and by 

arguments that health care value is best defined by 

patient outcomes, not processes of care.6

Selecting measures for QI registries typically 

requires balancing the intent to be relevant and 

actionable with the desire to meet other needs for 

providers, for example by reporting quality 

measures to different parties (e.g., accreditation 

organizations, payers). Frequently, this is further 

complicated by the lack of harmonization between 

those measure requirements even in the same 

patient populations.7 Even when there is 

agreement on the type of intervention to be 

measured and how the intervention is defined, 

there still may be variability in how the cases that 

populate the denominator are selected (e.g., by 

clinical diagnosis, by ICD-9 classification, by CPT 

codes). In the planning stages of a QI registry, it is 

useful to consider key parameters for selecting 

measures. The National Quality Forum offers the 

following four criteria for measure endorsement, 

which also apply to measure selection:

1. Important to measure and report, to keep 

our focus on priority areas, where the 

evidence is highest that measurement can 

have a positive impact on health care 

quality.

2. Scientifically acceptable, so that the 

measure when implemented will produce 

consistent (reliable) and credible (valid) 

results about the quality of care.

3. Useable and relevant, to ensure that 

intended users—consumers, purchasers, 

providers, and policymakers—can 

understand the results of the measure and 

are likely to find them useful for quality 

improvement and decisionmaking.

4. Feasible to collect with data that can be 

readily available for measurement and 

retrievable without undue burden.8

The National Priorities Partnership9 and the 

Measure Applications Partnership,10 both of which 

grew out of the National Quality Forum and 

provide support to the U.S. Department of Health 

and Human Services on issues related to quality 

initiatives and performance measurement, also 

offer useful criteria to consider when selecting 

measures.

One approach to consider in selecting measures is 

to perform a cross-sectional assessment using the 

proposed panel of measures to identify the largest 

gaps between what is recommended in evidencebased guidelines or expected from the literature 

and what is actually done (“treatment gaps”). The 

early phase of the registry can then focus on those 

measures with the most significant gaps and for 

which there is a clear agreement among practicing 

physicians that the measure reflects appropriate 

care. The planning and development process 

should move from selecting measures to 

determining which data elements are needed to 

produce those measures (see Section 4 below). 

Measures should ideally be introduced with 

idealized populations of patients in the 

denominator for whom there is no debate about the 

appropriateness of the intervention. This may help 

reduce barriers to implementation that are due to 

physician resistance based on concerns about 

appropriateness for individual patients.

Once the measures and related data elements have 

been selected, pilot testing may be useful to assess 

the feasibility and burden of participation. Pilot 

testing may identify issues with the availability of 

some data elements, inconsistency in definition of 

data elements across sites, or barriers to 
174

Section V. Special Applications in Patient Registries

participation, such as burden of collecting the data 

or disagreement about how exclusion criteria are 

constructed when put into practice. In order for the 

registry to be successful, participants must find the 

information provided by the registry useful for 

measuring and then modifying their behaviors, 

processes, or systems of care. Pilot testing may 

enable the registry to improve the content or 

delivery of reports or other tools prior to the 

large-scale launch of the program. If pilot testing 

is included in the plans for a QI registry, the 

timeline should allow for subsequent revisions to 

the registry based on the results of the pilot testing.

Change management is also an important 

consideration in planning a QI registry. QI 

registries need to be nimble in order to adapt to 

two continual sources of change. First, new 

evidence comes forward that changes the way care 

should be managed, and it is incumbent on the 

registry owner to make changes so that the registry 

is both current and relevant. In many registries, 

such as American Heart Association’s Get With 

The Guidelines Stroke program and the American 

Society of Clinical Oncologists’ QOPI registry, 

this process occurs more than once a year. Second, 

providers participating in registries manage what 

they measure, and, over time, measures can be 

rotated in or out of the panel so that attention is 

focused where it is most critical to overcome a 

continuing treatment gap or performance 

deficiency. This requires that the registry have a 

standing governance body to make changes over 

time, a system of data collection and reporting 

flexible enough to rapidly incorporate changes 

with minimal or no disruption to participants, and 

sufficient resources to communicate with and train 

participants on the changes. The governance 

structure should include experts in the area of 

measurement science as well as in the scientific 

content. The registry system also needs to 

continuously respond to additional (not necessarily 

harmonized) demands for transmitting quality 

measures to other parties (e.g., Physician Quality 

Reporting System, Meaningful Use reporting, 

Bridges to Excellence, State department of public 

health requirements). From a planning standpoint, 

QI registries should expect ongoing changes to the 

registry and plan for the resources required to 

support the changes. While this complicates the 

use of registry data for research purposes, it is vital 

that the registry always be perceived first as a tool 

for improving outcomes. Therefore, whenever 

changes are made to definitions, elements, or 

measures, these need to be carefully tracked so 

that analyses or external reporting of adherence 

may take these into account if they span time 

periods in which changes occurred. 

3. Legal and Institutional 

Review Board Issues

As discussed in Chapters 7, 8, and 9, registries 

navigate a complex sea of legal and regulatory 

requirements depending on the status of the 

developer, the purpose of the registry, whether or 

not identifiable information is collected, the 

geographic locations in which the data are 

collected, and the geographic locations in which 

the data are stored (State laws, international laws, 

etc.). QI registries face unique challenges in that 

many institutions’ legal departments and 

institutional review boards (IRBs) may have less 

familiarity with registries for quality improvement, 

and, even for experts, the distinction between a 

quality improvement activity and research may be 

unclear.11-14 Some research has shown that IRBs 

differ widely in how they differentiate research and 

quality improvement activities.15 What is clear is 

that IRB review and, in particular, informed 

consent requirements, may not only add burden to 

the registry but may create biased enrollment that 

may in turn affect the veracity of the measures 

being reported.16 Potential limitations of the IRB 

process have been identified in other reports, 

including potential problems for comparative 

effectiveness research. These issues will not be 

reviewed here. 

For QI registries, which generally fit under the 

HIPAA (Health Insurance Portability and 

Accountability Act) definition of health care 

operations, the issues that lead to complexity 

include whether or not the registry includes 

research as a primary purpose or any purpose, 

whether the institutions or practices fall under the 

Common Rule, and whether informed consent is 

needed. The Common Rule is discussed in the 

Chapter 7, and informed consent and quality 
175

Chapter 22. Quality Improvement Registries

improvement activities are discussed in Chapter 8. 

To assist in determining whether a quality 

improvement activity qualifies as research, the 

Office for Human Research Protections (OHRP) 

provides information in the form of a “Frequently 

Asked Questions” Web page.17 OHRP notes that 

most quality improvement activities are not 

considered research and therefore are not subject 

to the regulations for the protection of human 

subjects. However, some quality improvement 

activities are considered research, and the 

regulations do apply in those cases. To help 

determine if a quality improvement activity 

constitutes research, OHRP suggests addressing 

the following four questions, in order:

1. Does the activity involve research? 

(45 CFR 46.102(d)) 

2. Does the research activity involve human 

subjects? (45 CFR 46.102(f))

3. Does the human subjects research qualify 

for an exemption? (45 CFR 46.101(b)) 

Is the nonexempt human subjects research 

conducted or supported by HHS or otherwise 

covered by an applicable FWA [Federalwide 

Assurance] approved by OHRP?18

In addressing these questions, it is important to 

note the definition of “research” under 45 CFR 

46.102(d). “Research” is defined as “...a 

systematic investigation, including research 

development, testing and evaluation, designed to 

develop or contribute to generalizable 

knowledge....” OHRP may not view quality 

improvement activities as “research” under this 

definition, and provides some examples of the 

types of activities that are not considered 

research.19 It is also important to note the 

definition of “human subjects” under 45 CFR 

46.101(b). “Human subject” is defined as “a living 

individual about whom an investigator (whether 

professional or student) conducting research 

obtains (1) Data through intervention or 

interaction with the individual, or (2) Identifiable 

private information.” Again, OHRP may not view 

quality improvement activities as human subjects 

research if the data are not considered identifiable 

private information or were not collected through 

interaction or intervention with the individual 

patient (e.g., if the data were abstracted from a 

medical record).20

These questions provide some information helpful 

in determining whether a quality improvement 

registry is subject to the protection of human 

subjects regulations, but some researchers and 

IRBs have still reported difficulty in this area.11, 12

Remaining questions include, for example, if the 

registry includes multiple sites, is separate IRB 

approval from every institution required? If the 

registry is considered human subjects research, in 

what circumstances is informed consent required? 

There have been several recent calls to refine and 

streamline the IRB process for QI registries,11 and 

some of this work is advancing. Recently, OHRP 

has proposed revisions to the Common Rule that 

would address some of these issues; the proposed 

changes were posted for a public comment period, 

which closed in October 2011.21 Without some 

changes and greater clarity around existing 

regulations as they relate to QI registries, it will be 

difficult for some registries to be successful.

4. Design

Designing a QI registry presents several 

challenges, particularly when multiple 

stakeholders are involved. Staying focused on the 

registry’s key purposes, limiting respondent 

burden, and being able to make use of all of the 

data collected are practical considerations in 

developing programs. First, the type of QI registry 

needs to be determined. Is the goal to improve the 

quality of patients with a disease or patients 

presenting for a singular event in the course of 

their disease? For example, a QI registry in 

cardiovascular disease will be different (i.e., with 

respect to sampling, endpoints, and measures) if it 

focuses on patients with coronary artery disease, 

versus patients with a hospitalization for acute 

coronary syndrome. In the first example, the 

registry may need to track patients over time and 

across different providers; reminder tools may be 

needed to prompt followup visits or laboratory 

tests. In the second example, the registry may need 

to collect detailed data at a single point in time on 

a large volume of patients.
176

Section V. Special Applications in Patient Registries

Second, QI registries that collect data within a 

single institution differ from those that collect data 

at multiple institutions regionally or nationally. 

Single-institution registries, for example, may be 

designed to fit within specific workflows at the 

institution or to integrate with one EHR system. 

They may reflect the specific needs of that 

institution in terms of addressing treatment gaps, 

and they may be able to obtain participant buy-in 

for reporting plans (e.g., for unblinded reporting). 

Regional or national level registries, on the other 

hand, must be developed to fit seamlessly into 

multiple different workflows. These registries must 

address common treatment gaps that will be 

relevant to many institutions, and they must 

develop approaches to reporting that are 

acceptable to all participants. 

The appropriate level of analysis and reporting is 

an important consideration for designers of QI 

registries. Reports may provide data at the 

individual patient, provider, or institution level, or 

they may provide aggregate data on groups of 

patients, providers, and institutions. The aggregate 

groups may be based on similar characteristics 

(e.g., disease state, hospitals of a similar size), 

geography, or other factors. The registry may also 

provide reports to the registry participants, to 

patients, or to the public. Reports may be 

unblinded (e.g., the provider is identifiable) or 

blinded, and they may be provided through the 

registry or through other means. In designing the 

registry, consideration should be given to what 

types of reports will be most relevant for achieving 

the registry’s goals, what types of reports will be 

acceptable to participants, and how those reports 

should be presented and delivered. Reporting 

considerations are discussed further in Section 9.

As described above, there are many challenges in 

selecting existing measures or designing and 

testing new measures. Once measures have been 

selected, the “core data” can be determined. Since 

QI registries are part of health care operations, it is 

critical that they do not overly interfere with the 

efficiency of those operations, and therefore the 

data collection must be limited to those data 

elements that are essential for achieving the 

registry’s purpose. One approach to establishing 

the core data set is to first identify the outcomes or 

measures of interest and then work backwards to 

the minimal data set, adding those elements 

required for risk adjustment or relevant subgroup 

analyses. For example, the inclusion and exclusion 

criteria for a measure, as well as information used 

to group patients into numerator and denominator 

groups, can be translated into data elements for the 

registry. Case Example 53 describes this process 

for the Get With The Guidelines Stroke program. 

Depending on the goals of the registry, the core 

data set may also need to align with data collection 

requirements for other quality reporting programs.

Many QI registries have gone further by 

establishing a core data set and an enhanced data 

set for participating groups that are ready to extend 

the range of their measurements. This tiered model 

can be very effective in appealing to a broad range 

of practices or institutions. Examples include the 

Get With The Guidelines program, which allows 

hospitals to select performance measures or both 

performance and quality measures, and the 

American College of Surgeons’ National Surgical 

Quality Improvement Program, which has a core 

data set and the ability to add targeted procedure 

modules.

QI registries also may need to develop sampling 

strategies during the design phase. The goal of 

sampling in QI registries is to provide 

representativeness (i.e., to ensure that the registry 

is reflective of the patients treated by the physician 

or practice) and precision (i.e., to enroll a 

sufficient sample size to provide reasonable 

intervals around the metrics generated from each 

practitioner/practice to be useful in before/after or 

benchmarking comparisons). Sampling frames 

need to balance simplicity with sustainability. For 

example, an all-comers model is easy to 

implement but can be difficult to sustain, 

particularly if the registry uses longitudinal 

followup. For example, an orthopedic registry 

maintained by a major U.S. center sought to enroll 

all patients presenting for hip and knee procedures. 

Since the center performed several thousand 

procedures each year, within a few short years the 

numbers of followups being performed climbed to 

the tens of thousands. This was both expensive and 

likely unsustainable. On the other hand, a sampling 

frame can be difficult and confusing. While a 
177

Chapter 22. Quality Improvement Registries

sampling frame can be readily administered in a 

retrospective chart review, it is much more difficult 

to do so in a prospective registry. Some approaches 

to this issue have included selecting specific days 

or weeks in a month for patient enrollment. But, if 

these frames are known to the practitioners, they 

can be “gamed,” and auditing may be necessary to 

determine if there are sampling inconsistencies. 

Pilot testing can be useful for assessing the pace of 

patient enrollment and the feasibility of the 

sampling frame. Ongoing assessments may also be 

needed to ensure that the sampling frame is 

yielding a representative population. 

An additional implication when considering how 

to implement a sampling strategy is that for QI 

registries in which concurrent case ascertainment 

and intervention is involved, only those patients 

who are sampled may benefit from real-time QI 

intervention and decision support. In these 

circumstances, patients who are not sampled are 

also less likely to receive the best care. This 

disparity may only increase as EHR-enabled 

decision support becomes increasingly 

sophisticated and commonplace. 

5. Operational Considerations

As with most registries, the major cost for 

participants in a QI registry is data collection and 

entry rather than the cost of the data entry platform 

or participation fees. Because QI registries are 

designed to fit within existing health care 

operations, many of the data elements collected in 

these registries are already being collected for 

other purposes (e.g., claims, medical records, other 

quality reporting programs). QI registries are often 

managed by clinical staff who are less familiar 

with clinical research and who must fit registry 

data collection into their daily routines. Both of 

these factors make integration with existing health 

information technology systems or other data 

collection programs attractive options for some QI 

registries. Integration may take many forms. For 

example, data from billing systems may be 

extracted to assist with identifying patients or to 

pull in basic information about them. EHRs may 

contain a large amount of the data needed for the 

registry, and integration with the EHR system 

could substantially reduce the data collection 

burden on sites. However, integration with EHRs 

can be complex, particularly for registries at the 

regional or national level that need to extract data 

from multiple systems. A critical challenge is that 

the attribution of clinical diagnoses in the context 

of routine patient care is often not consistent with 

the strict coding criteria for registries, making 

integration with EHR systems more complex. 

Chapter 15 discusses integration of registries with 

EHR systems. Another alternative for some 

disease areas is to integrate data collection for the 

registry with data collection for other quality 

initiatives (e.g., Joint Commission, Centers for 

Medicare & Medicaid Services). Typically, these 

types of integration can only provide some of the 

necessary data; participants must collect and enter 

additional data to complete the CRFs. 

The burden of data collection is an important 

factor in participant recruitment and retention. 

Much of the recruitment and retention discussion 

in Chapter 10 applies to QI registries. However, 

one area in which QI registries differ from other 

types of registries is in the motivations for 

participation. Sites may participate in other 

registries because of interest in the research 

question or as part of mandated participation for 

State or Federal payment or regulatory 

requirements. When participation is for research 

purposes, they may hope to connect with other 

providers treating similar patients or contribute to 

knowledge in this area. In contrast to registries 

designed for other purposes, participants in QI 

registries expect to use the registry data and tools 

to effect change within their organizations. 

Participation in a QI registry and related 

improvement activities can require significant time 

and resources, and incentives for participation 

must be tailored to the needs of the participants. 

For example, recognition programs, support for QI 

activities, QI tools, and benchmarking reports may 

all be attractive incentives for participants. In 

addition, tiered programs, as noted above, can be 

an effective approach to encouraging participation 

from a wide variety of practice or institution types. 

Understanding the clinical background of the 

stakeholders (e.g., nurses, physicians, allied health 

practitioners, and quality improvement 
178

Section V. Special Applications in Patient Registries

professionals) and their interest in the program is 

critical to designing appropriate incentives for 

participation. 

6. Quality Improvement Tools

As described above, QI tools are a unique and 

central component of QI registries Generally, QI 

tools are designed to meet one of two goals: care 

delivery and coordination or population 

measurement. Care delivery and coordination tools 

aim to improve care at the individual patient level, 

while population measurement tools track activity 

at the population level, with the goal of assessing 

overall quality improvement and identifying areas 

for future improvement activities. For example, a 

report may be used to track an institution’s 

performance on key measures over time and 

compared with other similar institutions. These 

types of reports can be used to demonstrate both 

initial and sustained improvements. Table 22–1 

summarizes some common types of QI tools in 

these two categories and describes their uses.

QI registries may incorporate various tools, 

depending on the needs of their participants and 

the goals of the registry. Table 22–2 below 

describes the types of functionalities that have 

been implemented in three different registries—

two at the national level and one at the regional 

level.

Table 22–1. Common quality improvement tools

Major Goal QI Tool Description

Care 

delivery and 

coordination

Patient lists Lists of patients with a particular condition who may be due for an exam, 

procedure, etc.

Patient-level reports Summaries of data on an individual patient (e.g., longitudinal data on 

blood pressure readings).

Automated 

notifications

To prompt provider or patient when an exam or other action is needed.

Automated 

communications

Summaries of patient information in a format that can be shared with the 

patient or other providers.

Decision support 

tools

Recommendations for care for an individual patient using evidencebased guidelines.

Population 

measurement

Population level 

standardized 

reports

Analysis of population-level compliance with quality improvement 

(QI) measures or other summaries (e.g., patient outcomes across the 

population).

Benchmarking 

reports

Comparisons of population-level data for various types of providers.

Ad hoc reports Participants can analyze registry data to explore their own questions and 

to support continuous quality improvement activities.

Population-level 

dashboards

Snapshot look at QI progress and areas for continued improvement.

Third-party quality 

reporting

Registry data leveraged for reporting to third-party quality reporting 

initiatives.
179

Chapter 22. Quality Improvement Registries

Table 22–2. Quality improvement tools implemented in three registries

Registry Disease/Condition Functionalities Implemented

AHA Get With The 

Guidelines

Heart failure

Stroke

• Decision support (guidelines)

• Communication tools

• Patient education materials

• Real-time quality reports with benchmarks

• Transmission to third parties

MaineHealth Clinical 

Improvement Registry

Diabetes • Patient-care “gap” reports

• Decision support

• Transmission to third parties

National Comprehensive 

Cancer Network

Cancer • Patient-care “gap” reports

• Center-level reports

• Educational materials

7. Quality Assurance

In addition to developing data elements and QI 

tools, QI registries must pay careful attention to 

quality assurance issues. Quality assurance, which 

is covered in Chapter 11, Section 3, is important 

for any registry to ensure that appropriate patients 

are being enrolled and that the data being collected 

are accurate. Data quality issues in registries may 

result from inadequate training, incomplete case 

identification or sampling, misunderstanding or 

misapplication of inclusion/exclusion criteria, or 

misinterpretation of data elements. Quality 

assurance activities can help to identify these types 

of issues and improve the overall quality of the 

registry data. QI registries can use quality 

assurance activities to address these common 

issues, but they must also be alert to data quality 

issues that are unique to QI registries. Unlike other 

registries, many QI registries are linked to 

economic incentives, such as licensure or access to 

patients, incentive payments, or recognition or 

certification. These are strong motivators for 

participation in the registry, but they may also lead 

to issues with data quality. In particular, “cherry 

picking,” which refers to the nonrandom selection 

of patients so that those patients with the best 

outcomes are enrolled in the registry, is a concern 

for QI registries. In addition, whenever data are 

being abstracted from source documents by hand 

and then entered manually into electronic data 

entry systems, there is a risk of typographical 

errors, errors in unit conversions (e.g., 12-hour to 

military time, milligrams to grams). Automated 

systems for error checking can reduce the risk of 

errors being entered into the registry when range 

checks and valid data formats are built into the 

data capture platform. 

Auditing is one approach to quality assurance for 

QI registries. Auditing may involve onsite audits, 

in which a trained individual reviews registry data 

against source documents, or remote audits, in 

which the source documents are sent to a central 

location for review against the registry data. 

Because auditing all sites and all patients is 

cost-prohibitive, registries may audit a percentage 

of sites and/or a percentage of patients. QI 

registries should determine if they will audit data, 

and, if so, how they will conduct the audits. A 

risk-based approach may be useful for developing 

an auditing plan. In a risk-based approach, the 

registry assesses the risk for intentional error in 

data entry or patient selection. Registries that may 

have an increased risk of intentional error include 

mandatory registries, registries with public 

reporting, or registries linked to economic 

incentives. Registries with an increased risk may 

decide to pursue more rigorous auditing programs 

than registries with a lower risk. For example, a 

voluntary registry with confidential reporting may 
180

Section V. Special Applications in Patient Registries

elect to do a remote audit of a small percentage of 

sites and patients each year. A registry with public 

reporting linked to patient access, on the other 

hand, may audit a larger number of sites and 

patients each year, with a particular focus on key 

outcomes included in the publicly reported 

measures. 

Questions to consider when developing a quality 

assurance plan involving auditing include— 

• What percentage of sites should be audited 

each year?

• What percentage of data should be audited (all 

data elements for a sample of patients or only 

key data elements for performance measures)?

• How should sites be selected for auditing 

(random, targeted, etc.)?

• Should audits be conducted on site or 

remotely?

• What constitutes passing an audit?

Depending on the purpose of the registry, quality 

assurance plans may also address issues with 

missing data, for example—

• What percentage of missing data is expected?

• Are data missing at random?

• What lost-to-followup rate is anticipated?

• Are certain subgroups of patients more likely to 

be lost to followup? 

Lastly, quality assurance plans must consider how 

to address data quality issues. Audits and other 

quality assurance activities may identify problem 

areas in the registry data set. In some cases, such 

as when the problem is isolated to one or two sites, 

additional training may resolve the issue. In other 

cases, such as when the issue is occurring at 

multiple sites, data elements, documentation, or 

study procedures may need to be modified. In rare 

instances, quality assurance activities may identify 

significant performance issues at an individual site. 

The issues could be intentional (e.g., cherry 

picking) or unintentional (e.g., data entry errors). 

The registry should have a plan in place for 

addressing these types of issues. 

8. Analytical Considerations

While registries are powerful tools for 

understanding and improving quality of care, 

several analytical issues need to be considered. In 

general, the observational design of registries 

requires careful consideration of potential sources 

of bias and confounding that exist due to the 

nonrandomization of treatments or other sources. 

These sources of bias and confounding can 

threaten the validity of findings. Fortunately, the 

problems associated with observational study 

designs are well known, and a number of analytical 

strategies are available for producing robust 

analyses. Despite the many tools to handle 

analytical problems, limitations due to 

observational design, structure of data, measured 

and unmeasured confounding, and missing data 

should be readily acknowledged. Below are brief 

descriptions of several problems to consider when 

analyzing QI registry data, along with indications 

of how investigators commonly address these 

problems.

Observational designs used in registries offer the 

ability to study large cohorts of patients, and allow 

for careful description of patterns of care or 

variations in practice compared with what is 

considered appropriate or best care. While not an 

explicit intention, registries are often used to 

evaluate an effect of a treatment or intervention. 

The lack of randomization in registries, which 

limits causal inferences, is an important 

consideration. For example, in a randomized trial, 

a treatment or intervention can be evaluated for 

efficacy because different treatment options have 

an equal chance of being assigned. Another 

important characteristic observational studies may 

lack is an even chance of a patient actually 

receiving a treatment. In a randomized trial, 

subjects meet a set of inclusion criteria and 

therefore have an equal chance of receiving a 

given treatment. However, a registry likely has 

some patients with no chance of receiving a 

treatment. As a result, some inferences cannot be 

generalized across all patients in the registry.
181

Chapter 22. Quality Improvement Registries

An inherent but commonly ignored issue is the 

structure of health or registry data. Namely, 

physicians manage patients with routine processes, 

and physicians practice within hospitals or other 

settings that also share directly or indirectly 

common approaches. These clusters or 

“hierarchical” relationships within the data may 

influence results if ignored. For example, for a 

given hospital, a type of procedure may be 

preferred due to similar training experiences from 

surgeons. Common processes or patient selections 

are also more likely within one hospital compared 

with another hospital. These observations form a 

cluster and cannot be assumed to be independent. 

Without accounting for the clustering of care, 

incorrect conclusions could be made. Models that 

deal with these types of clustered data, often 

referred to as hierarchical models, can address this 

problem. These models may also be described as 

multilevel, mixed, or random-effects models. The 

exact approach depends on the main goal of an 

analysis, but typically includes fixed effects, which 

have a limited number of possible values, and 

random effects, which represent a sample of 

elements drawn from a larger population of effects. 

Thus, a multilevel analysis allows incorporation of 

variables measured at different levels of the 

hierarchy, and accounts for the attribute that 

outcomes of different patients under the care of a 

single physician or within the same hospital are 

correlated.

Adequate sample size for research questions is 

also an important consideration. In general, 

registries allow large cohorts of patients to be 

enrolled, but, depending on the question, sample 

sizes may be highly restricted (e.g., in the case of 

extremely rare exposures or outcomes). For 

example, a comparative effectiveness research 

question may address anticoagulation in patients 

with atrial fibrillation. As the analysis population 

is defined based on eligibility criteria, including 

whether patients are naïve to the therapy of 

interest, sample sizes with the exposure may 

become extremely small. Likewise, an outcome of 

angioedema may be extremely rare, and, if being 

evaluated with a new therapeutic, both the 

exposure and outcome may be too small of sample 

to fully evaluate. Thus, careful attention to the 

likely exposure population after establishing 

eligibility criteria as well as the likely number of 

events or outcomes of interest is extremely 

important. In cases where sample sizes become 

small, it is important to determine whether 

adequate power exists to reject the null hypothesis.

Confounding is a frequent challenge for 

observational studies, and a variety of analytical 

techniques can be employed to account for this 

problem. When a characteristic correlates with 

both the exposure of interest and the outcome of 

interest, it is important to account for the 

relationship. For example, age is often related to 

mortality and may also be related to use of a given 

process. In a sufficiently large clinical trial, age 

generally is balanced between those with and 

without the exposure or intervention. However, in 

an observational study, the confounding factor of 

age needs to be addressed through risk adjustment. 

Most studies will use regression models to account 

for observed confounders and adjust for outcome 

comparisons. Others may use matching or 

stratification techniques to adjust for the imbalance 

in important characteristics associated with the 

outcome. Finally, another approach being used 

more frequently is the use of propensity scores that 

take a set of confounders and reduce them into a 

single balancing score that can be used to compare 

outcomes within different groups.

As QI registries have evolved, an important 

attribute is defining eligibility for a process 

measure. The denominator for patients eligible for 

a process measure should be carefully defined 

based on clinical criteria, with those with a 

contraindication for a process excluded. The 

definition of eligibility for a process measure is 

critical for accurate profiling of hospitals and 

health care providers. Without such careful, clear 

definitions, it would be challenging to benchmark 

sites by performance. 

With any registry or research study, data 

completeness needs to be considered when 

assessing the quality of the study. Reasons for 

missing data vary depending on the study or data 

collection efforts. For many registries, data 

completeness depends on what is routinely 

available in the medical record. Missing data may 

be considered ignorable if the characteristics 

associated with the missingness are already 
182

Section V. Special Applications in Patient Registries

observable and therefore included in analysis. 

Other missing data may not ignorable, either 

because of their importance or because the 

missingness cannot be explained by other 

characteristics. In these cases, methods for 

addressing the missingness need to be considered. 

Various options for handling the degree of missing 

data including discarding data, using data 

conveniently available, or imputing data with 

either simple methods (i.e., mean) or through 

multiple imputation methods.

9. Reporting to Providers and 

the Public

An important component of quality improvement 

registries is the reporting of information to 

participants, and, in some cases, to the public. 

The relatively recent origin of clinical data 

registries was directly related to early public 

reporting initiatives by the Federal Government. 

Shortly after the 1986 publication of unadjusted 

mortality rates by the Health Care Financing 

Administration (HCFA), the predecessor of 

Centers for Medicare & Medicaid Services, a 

number of states (e.g., the New York Cardiac 

Surgery Reporting System),22, 23 regions (e.g., 

Northern New England Cardiovascular Disease 

Study Group, or NNE),24, 25 government agencies 

(e.g., the Veteran’s Administration),26-28 and 

professional organizations (e.g., Society of 

Thoracic Surgeons)29-31 developed clinical data 

registries. Many of these focused on cardiac 

surgery. The surgery’s index procedure, coronary 

artery bypass grafting (CABG), is the most 

frequently performed of all major operations; it is 

expensive; and it has well-defined adverse 

endpoints. 

Registry developers recognized that the HCFA 

initiative had ushered in a new era of health care 

transparency and accountability. However, its 

methodology did not accurately characterize 

provider performance because it used claims data 

and failed to adjust for preoperative patient 

severity.32 Clinical registries, and the risk-adjusted 

analyses derived from them, were designed to 

address these deficiencies. States such as New 

York, Pennsylvania, New Jersey, California, and 

Massachusetts developed public report cards for 

consumers, while professional organizations and 

regional collaborations used registry data to 

confidentially feed results back to providers and 

to develop evidence-based best practice 

initiatives.33, 34

The impact of public reporting on health care 

quality remains uncertain. One randomized trial 

demonstrated that heart attack survival improved 

with public reporting,35 and there is evidence that 

low-performing hospitals are more likely to initiate 

quality improvement initiatives in a public 

reporting environment.36 However, a 

comprehensive review37 found generally weak 

evidence for the association between public 

reporting and quality improvement, with the 

possible exception of cardiac surgery, where 

results improved significantly after the initial 

publication of report cards in New York two 

decades ago.23, 38, 39 Some studies have questioned 

whether this improvement was the direct result of 

public reporting, as contiguous areas without 

public reporting also experienced declining 

mortality rates.40 Similar improvements have been 

achieved with completely confidential feedback or 

regional collaboration in northern New England41

and in Ontario.42 Thus, there appear to be many 

effective ways to improve health care quality—

public reporting, confidential provider feedback, 

professional collaborations, state regulatory 

oversight—but the common denominator among 

them is a formal system for collecting and 

analyzing accurate, credible data,43 such as 

registries provide.

Public reporting should theoretically affect 

consumer choice of providers and redirect market 

share to higher performers. However, empirical 

data failed to demonstrate this following the HCFA 

hospital mortality rate publications,44 and CABG 

report cards had no substantial effect on referral 

patterns or market share of high and low 

performing hospitals in New York45, 46 or 

Pennsylvania.47, 48 Studies suggest numerous 

explanations for these findings, including lack of 

consumer awareness of and access to report cards; 

the multiplicity of report cards; difficulty in 

interpreting performance reports; credibility 

concerns; small differences among providers; lack 
183

Chapter 22. Quality Improvement Registries

of “newsworthiness”; the difficulty of using report 

cards for urgent or emergent situations; and the 

finite ability of highly ranked providers to accept 

increased demand.49-51 Professor Judith Hibbard 

and colleagues have suggested report card formats 

that enhance the ability of consumers to accurately 

interpret accurate report cards, including visual 

aids (e.g., star ratings) that synthesize complex 

information into easily understandable signals.52, 53

A recent Kaiser Family Foundation survey54

suggests that, particularly among more educated 

patients, the use of objective ratings to choose 

providers has steadily increased over the past 

decade, and health reform is likely to accelerate 

this trend.

The potential benefits of public reporting must be 

weighed against the unintended negative 

consequences, such as “gaming” of the reporting 

system.55, 56 The most concerning negative 

consequence is risk aversion, the reluctance of 

physicians and surgeons to accept high-risk 

patients because of their anticipated negative effect 

on their report card ratings. Because these highest 

risk patients may derive the greatest benefit from 

aggressive intervention, risk aversion may produce 

a net decrement in public health and a net increase 

in long-term costs because the best treatments 

were not initially used.57-59 Risk aversion 

unquestionably exists, but its extent and overall 

population impact are difficult to quantify. CABG 

risk aversion may have occurred in New York60, 61

and Pennsylvania,48 but studies in California62 and 

England63 have not demonstrated similar findings. 

Numerous studies document probable risk aversion 

in percutaneous coronary interventions.64-66

Possible approaches to mitigate risk aversion 

include demonstrating to providers the adequacy 

of risk adjustment and modifying those models 

when appropriate; excluding highest risk patients 

from reporting; separate reporting of highest risk 

patients; and careful clinical review of patients 

turned down for interventions.

Irrespective of its end results, many believe that 

public reporting is a fundamental ethical obligation 

of physicians.67, 68 It addresses the patient right of 

autonomy or self-determination in decisionmaking. 

Whether or not they choose to exercise this right, 

patients making a choice about treatments should 

be fully informed, which arguably includes their 

right to know the comparative performance of 

potential providers. 

When a decision has been made to publicly report 

outcomes, such measures must meet strict criteria. 

Professional organizations have emphasized the 

need to use high quality, audited clinical data 

whenever possible, and to employ the most 

appropriate statistical methodologies.69, 70

Professional society guidelines provide 

recommendations of varying strength and evidence 

strength, whereas performance measures should be 

a select subset of these guidelines that have the 

highest level of evidence and strongest class of 

recommendation (e.g., ACC/AHA [American 

College of Cardiology/American Heart 

Association] class 1[recommended] or 3 [not 

indicated, or harmful], level A evidence). National 

Quality Forum (NQF) requirements for 

performance measure endorsement have recently 

been updated. In addition to its four basic 

requirements of Importance, Scientific 

Acceptability, Usability, and Feasibility, NQF 

emphasizes the need for robust, systematic 

evaluation of the evidence base and comprehensive 

testing of reliability and validity.8, 71, 72

The unit of analysis in public reporting may be 

controversial. Many states report results for some 

procedures at the physician or surgeon level, but in 

many health care areas sample sizes and the small 

amount of variation attributable to the physician 

make it difficult to reliably discriminate 

performance.73-75 Compiling data from a variety of 

process and outcome endpoints may help to 

mitigate sample size issues, as may aggregation of 

results over multiple years. 

Report cards at the individual physician level may 

be more likely to cause risk aversion compared 

with group- or hospital-level reports. Changes in 

health care delivery models must also be 

considered. As patient care is increasingly 

provided by teams of providers that may even cross 

traditional specialty boundaries, individual 

physician reporting may become less relevant and 

feasible. Reimbursement will increasingly be 

based on the overall care provided to a patient or 

population, and leaders will have a direct financial 

incentive to assess the performance of individual 
184

Section V. Special Applications in Patient Registries

physicians in such care groups (e.g., Accountable 

Care Organizations or ACOs), whether or not such 

results are publicly reported. 

10. Use of QI Registry Data 

for Research Studies

An emerging trend is the use of data from QI 

registries to support additional studies. QI 

registries may collect large volumes of clinical 

data that can be used to support research studies. 

Studies using data from QI registries generally are 

developed in one of two ways.

First, the registry may be modified to collect 

additional data for a substudy. For example, a 

registry may collect in-hospital data on patients 

admitted to the hospital for a specific procedure. 

To study long-term outcomes of the procedure, the 

registry protocol may be modified to collect 

followup data for a subset of patients. An example 

of this approach was the OPTIMIZE-HF registry, 

which collected in-hospital data on patients 

admitted with heart failure. A subset of patients 

provided consent to be contacted after 6 months to 

collect additional data.3 QI registries can also be 

modified to support other types of studies, such as 

studies where a subset of participating sites are 

randomized (cluster randomization) or a subset of 

patients are randomized (experimental trial). When 

modifying the registry protocol to support a 

substudy, the impact on the primary purpose of the 

registry must be considered, as well as any 

additional ethical or regulatory requirements 

introduced by the new data collection effort. 

A second approach to using QI registries to 

support additional studies is to use the registry 

data, either alone or linked to another data set. For 

example, a registry that collects in-hospital data 

may be linked to a claims database to obtain 

information on long-term outcomes or to examine 

other questions.76 In these cases, the technical, 

legal, and ethical considerations related to linking 

registry data sets discussed in Chapter 16. 

Regardless of which approach is used, researchers 

using data from a QI registry for additional 

research studies must understand how the data are 

collected and how patients are enrolled in the 

primary registry in order to draw appropriate 

conclusions from the new study.

11. Limitations of Current QI 

Registries

To summarize some of the key points above, the 

ideal QI registry collects uniform data on risk 

factors, treatments, and outcomes at key points for 

a particular disease or treatment. It obtains the data 

from multiple sources and across care settings, and 

leverages existing health information technology 

systems through interoperability and other data 

sets (from registries, claims, national indices, etc.) 

through linkage. Such a registry uses standardized 

methods to ensure that the patients sampled are 

representative, that data are of high quality and 

that it is comparable across providers. Such 

registries provide feedback at the patient and 

population level, and, in addition to facilitating 

quality improvement, they perform quality 

reporting to third parties. Importantly, they 

maintain high levels of participation by providers 

and patients and have a long term, sustainable 

business model.

Clearly, most QI registries do not achieve this 

ideal. The term “QI registries” is currently used to 

refer to a broad spectrum of registries, from local 

or regional registries aimed at improving care for a 

specific patient population to large, national 

registries with sophisticated benchmarking data. 

Many current QI registries focus on isolated 

conditions or procedures (e.g., the ACC NCDR 

Cath/PCI Registry77; the STS Adult Cardiac 

Surgery Database78). Health reform will require 

the acquisition of data about the overall, 

comprehensive care of conditions such as coronary 

artery disease, or of populations.6 This may be 

facilitated by linkages among related data 

registries, which might include outpatient 

preventive care, inpatient acute care and 

procedures, rehabilitation, and chronic disease 

management. 
185

Chapter 22. Quality Improvement Registries

Current QI registries also have temporal 

limitations. They characteristically collect data 

only in-hospital or for 30 days after admission or a 

procedure. However, patients, payers, and 

regulators are also interested in longer term, 

longitudinal outcomes such as survival, 

readmission, reintervention, and cumulative 

resource use. Such information is useful for shared 

decisionmaking and for comparative effectiveness 

research. By linking together robust clinical data 

registries and administrative databases such as 

MEDPAR or the Social Security Death Master 

File79, 80 that provide long-term data, many of 

these current limitations of clinical registries 

would be mitigated. 

In order for such linkages to be implemented, a 

number of challenges would need to be overcome. 

These include a lack of standardized data sets; 

difficulties collecting data across care settings; 

inability to leverage existing health information 

technology systems to reduce duplication of 

clinician effort; inability to link to other data 

sources that might reduce data collection burden 

or enrich outcomes; significant variation in the 

quality of methods used to collect and report data; 

and quite different levels of participation and 

business models. Even registries in related 

conditions may not be fully compatible. 

Potential solutions to such issues have been 

identified.81 These include, for example, conditionspecific and cross-condition efforts to standardize 

common or core data element specifications, data 

quality and audit standards, and methodological 

considerations such as risk adjustment. Collecting 

data across care settings will be improved by 

solving the patient identity management issues 

(discussed in Chapter 17), which will require 

clarification and perhaps revision of HIPAA and 

Common Rule regulations. Overcoming 

interoperability issues through the promulgation of 

open standards (e.g., Healthcare Information 

Technology Standards Panel TP-50) (as described 

in Chapter 15) could have dramatic impact if 

adopted widely by EHR systems and registries. 

Significant hospital data collection costs are 

additional limitations of clinical registries. Some 

data elements such as laboratory values may be 

automatically extracted from EHRs, but detailed 

clinical data may still require manual extraction. 

Existing national registries must develop 

sustainable business models, and there must be 

incentives and assistance for the development of 

new registries where none currently exist.

12. Summary

QI registries have documented success at 

improving quality of care at the local, regional, 

and national levels. While QI registries differ in 

their area of focus, choice of measures, and level 

of reporting, their consistent features are the use of 

systematic data collection and other tools to 

improve quality of care. QI registries also differ 

from other types of registries in many ways, such 

as in their use of provider “champions,” the 

inclusion of actionable measures, the frequency of 

major changes to the registry data collection, the 

motivations for participation, and the use of 

blinded or unblinded quality reports to providers, 

and, in some cases, the public. Because of these 

differences, QI registries must address unique 

challenges, particularly in the planning, design, 

and operations phases. 
186

Section V. Special Applications in Patient Registries

Case Example 53. Using recognition measures 

to develop a data set

Description Get With The Guidelines® is the 

flagship program for in-hospital 

quality improvement of the 

American Heart Association and 

American Stroke Association . 

The Get With The Guidelines—

Stroke program supports point 

of care data collection and 

real-time reports aligned with 

the latest evidence-based 

guidelines. The reports include 

achievement, quality, reporting, 

and descriptive measures that 

allow hospitals to trend their 

performance related to clinical 

and process outcomes.

Sponsor American Heart Association/

American Stroke Association

Year Started 2003

Year Ended Ongoing

No. of Sites 1,664

No. of Patients 2,063,439

Challenge

The primary purpose of the Get With The 

Guidelines—Stroke program is to improve the 

quality of in-hospital care for stroke patients. The 

program uses the PDSA (plan, do, study, act) 

quality improvement cycle, in which hospitals 

plan quality improvement initiatives, implement 

them, study the results, and then make 

adjustments to the initiatives. To help hospitals 

implement this cycle, the program uses a registry 

to collect data on stroke patients and generate 

real-time reports showing compliance with a set 

of standardized stroke recognition and quality 

measures. The reports also include benchmarking 

capabilities, enabling hospitals to compare 

themselves with other hospitals at a national and 

regional level, as well as with similar hospitals 

based on size or type of institution.

In developing the registry, the team faced the 

challenge of creating a data set that would be 

comprehensive enough to satisfy evidence-based 

medicine but manageable by hospitals 

participating in the program. The program does 

not provide reimbursements to hospitals entering 

data, so it needed to keep the data set as small as 

possible while still maintaining the ability to 

measure quality improvement.

Proposed Solution

The team began developing the data set by 

working backward from the recognition 

measures. Recognition measures, based on the 

sponsor’s guidelines for stroke care, contain 

detailed inclusion and exclusion criteria to 

determine the measure population, and they 

group patients into denominator and numerator 

groups. Using these criteria, the team developed a 

data set that framed the questions necessary to 

determine compliance with each of the 

guidelines. The team then added questions to 

gather information on the patient population 

characteristics. Since the inception of the 

program, data elements and measure reports have 

been added or updated to maintain alignment 

with the current stroke guidelines. Over time, 

certain measures have also been promoted to or 

demoted from the higher tiers of recognition 

measures, depending on current science and 

changes in quality improvement focus.

Results

By using this approach, the registry team was 

able to create the necessary data set for 

measuring compliance with stroke guidelines. 

The program was launched in 2003 and now has 

1,664 hospitals and 2,063,439 stroke patient 

records. The data from the program have been 

used in several abstracts and have resulted in 38 

manuscripts since 2007.

Key Point

Registry teams should focus on the outcomes or 

endpoints of interest when selecting data 

elements. In cases where compliance with 

Case Examples for Chapter 22
187

Chapter 22. Quality Improvement Registries

Case Example 53. Using recognition measures 

to develop a data set (continued)

Key Point (continued)

guidelines or quality measures is the outcome of 

interest, teams can work backward from the 

guidelines or measures to develop the minimum 

necessary data set for their registry.

For More Information

http://www.heart.org

Schwamm L, Fonarow G, Reeves M., et al. Get 

With the Guidelines—Stroke is associated with 

sustained improvement in care for patients 

hospitalized with acute stroke or transient 

ischemic attack. Circulation. 2009;119:107–11.

Schwamm LH, LaBresh KA, Albright D., et al. 

Does Get With The Guidelines improve 

secondary prevention in patients hospitalized 

with ischemic stroke or TIA? Stroke. 

2005;36(2):416–P84.

LaBresh KA, Schwamm LH, Pan W., et al. 

Healthcare disparities in acute intervention for 

patients hospitalized with ischemic stroke or TIA 

in Get With The Guidelines—Stroke. Stroke. 

2005;36(2):416–P275.

Case Example 54. Managing care and quality 

improvement for chronic diseases

Description The Tri State Child Health 

Services Web-based asthma 

registry is part of an asthma 

improvement collaborative 

aimed at improving evidencebased care and outcomes while 

strengthening improvement 

capacity of primary care 

practices.

Sponsor Tri State Child Health Services, 

Inc., a pediatric physicianhospital organization (PHO) 

affiliated with Cincinnati 

Children’s Hospital Medical 

Center

Year Started 2003

Year Ended Ongoing

No. of Sites 39 community-based pediatric 

practices

No. of Patients 12,365 children with asthma

Challenge

Asthma, a highly prevalent chronic disease 

managed in the primary care setting, has proven 

to be amenable to quality improvement initiatives. 

This collaborative effort between the PHO and 

Cincinnati Children’s Hospital Medical Center 

was initiated in 2003 with goals of improving 

evidence-based care, reducing adverse outcomes 

such as asthma-related emergency room visits 

and missed schooldays, and strengthening the 

quality of knowledge and capacity within primary 

care practices. As the asthma initiative spans 39 

primary care practices and encompasses 

approximately 35 percent of the region’s pediatric 

asthma population, the PHO needed to implement 

strategies for improving network-level, 

population-based process and outcome measures.

Proposed Solution

To address the project’s focus on improving 

process and outcome measures across a large 

network, the asthma collaborative decided to 

implement a centralized, Web-based asthma 

registry. Key measures of effective control and 

management of asthma (based on the National 

Heart, Lung, and Blood Institute’s guidelines) are 

captured via a self-reported clinical assessment 

form and decision support tool completed by 

parents and physicians at the point of care. The 

questions address missed schooldays and 

workdays, parent’s confidence in managing 

asthma, health resource utilization (e.g., 

emergency room visits), parent and physician 

rating of disease control, and other topics. In 
188

Section V. Special Applications in Patient Registries

Case Example 54. Managing care and quality 

improvement for chronic diseases (continued)

Proposed Solution (continued)

addition, the clinical assessment form facilitates 

interactive dialogue between the physician and 

family during office visits.

The Web-based registry allows real-time 

reporting at the patient, practice, and network 

level. Reporting is transparent, with comparative 

practice data that support the identification of 

best practices and shared learning. In addition, 

reporting functionalities support tracking of 

longitudinal data and the identification of highrisk patients. The Web-based registry also 

provides access to real-time utilization reports 

with emergency room visit and admission dates. 

All reports are available to participating practices 

and physicians at any time.

Results

The registry provides essential data for 

identifying best practices and tracking 

improvement. The network has documented 

improvement against standard process and 

outcome measures.

Key Point

Registries can be useful tools for quality 

improvement initiatives in chronic disease areas. 

By collecting standardized data and sharing the 

data in patient-, practice-, and network-level 

reports, registries can track adherence to 

guidelines and evidence-based practices, and 

provide information to support ongoing quality 

improvement.

For More Information

Mandel KE, Kotagal UR. Pay for performance 

alone cannot drive quality. Arch Pediatr Adolesc 

Med. 2007;161(7):650–5.

Case Example 55. Use of reporting tools to 

promote quality improvement

Description The Quality Oncology Practice 

Initiative (QOPI®) is a quality 

assessment and improvement 

program for oncology practices.

Sponsor American Society of Clinical 

Oncology (ASCO)

Year Started Pilot program started in 2002; 

registry launched for full ASCO 

membership in 2006

Year Ended Ongoing

No. of Sites 801 registered practices

No. of Patients Approximately 50,000 patient 

charts per year

Challenge

The 1999 Institute of Medicine report “Ensuring 

Quality Cancer Care” identified the opportunity 

for quality improvement initiatives in oncology. 

A clear path to nationwide impact was identified, 

starting with individual practices. The report set 

forth recommendations including to “measure 

and monitor the quality of care using a core set of 

quality measures.” In order to promote this 

endeavor, a methodology and a registry were 

needed. 

Proposed Solution

In 2002, ASCO, in conjunction with a community 

of oncologists, developed QOPI, a voluntary pilot 

program to allow participants to assess and 

improve cancer care within their own practices. 

The oncologist-led program created quality 

measures, developed methodology for data 

collection and analysis, and tested the feasibility 

of the pilot program before offering access to the 

registry to all Society members in 2006. The 

registry provides comparison data to practices on 

more than 100 quality metrics that practices and 

practitioners can use to compare their 

performance to that of their peers, at both the 

practice and practitioner level. A team of 

oncologists, researchers, and staff select, adapt, 

and develop metrics based on clinical guidelines 

and expert consensus opinion. Practices and 

institutions register and manually submit 

abstracted patient chart data through a Web-based 
189

Chapter 22. Quality Improvement Registries

Case Example 55. Use of reporting tools to 

promote quality improvement (continued)

Proposed Solution (continued)

interface during twice-per-year data collection 

periods. Once the data collection periods close, 

the data are analyzed and practices can view 

reports showing their performance and scores 

based on quality measures for that round.

Results

Approximately 600 practices representing nearly 

15 percent of U.S. practitioners have now 

contributed data to the registry. Changes in 

performance rates have been compared among 

metrics surrounding the following domains: core, 

end of life, symptom management, breast cancer, 

colorectal cancer, and Non-Hodgkin lymphoma. 

For example, in a 2010 analysis of registry data, 

practices completing multiple data collection 

cycles with the registry had better performance 

on care of pain for end-of-life care (63%) when 

compared with practices participating in the 

registry for the first time (47%). Registry 

participants who participated in multiple data 

collection cycles also demonstrated better 

performance in the rate of documenting 

discussions of hospice and palliative care, and 

higher rates of hospice enrollment, when 

compared with participants who participated in 

just a single cycle.

Key Point

Access to performance reports can inform 

physician behavior or be used to demonstrate the 

need for process improvements within a practice. 

A registry can provide a systematic approach to 

data collection to support the ongoing use of 

self-assessment and benchmark performance 

reports to facilitate quality improvement. 

For More Information

http://qopi.asco.org/

Blayney DW, Severson J, Martin CJ, et al. 

Michigan oncology practices showed varying 

adherence rates to practice guidelines, but quality 

interventions improved care. Health Aff. 2012 

April; 31(4):718-8.

Campion FX, Larson LR, Kadlubek PJ, et al. 

Advancing performance measurement in 

oncology: Quality Oncology Practice Initiative 

participation and quality outcomes. J Oncol Pract. 

2011 May 1:31s-35s.

Jacobson JO, Neuss MN, McNiff KK, et al. 

Improvement in oncology practice performance 

through voluntary participation in the Quality 

Oncology Practice Initiative. J Clin Oncol. 2008; 

26:1893-8.

Neuss MN, Gilmore TR, Kadlubek PJ. Tools for 

measuring and improving the quality of oncology 

care: The Quality Oncology Practice Initiative 

(QOPI®) and the QOPI Certification Program. 

Oncology (Williston Park). 2011 Sep;25(10):880, 

883, 886-7.
190

Section V. Special Applications in Patient Registries

Case Example 56. Using registries to drive 

quality improvement in chronic conditions

Description The National Parkinson 

Foundation Quality 

Improvement Initiative is a 

registry-based quality care 

program that captures 

longitudinal data on clinical 

interventions and patientreported outcomes to identify, 

implement, and disseminate best 

practices for the treatment and 

management of Parkinson’s 

disease.

Sponsor National Parkinson Foundation

Year Started 2009

Year Ended Ongoing

No. of Sites 20 centers across United States, 

Canada, and internationally

No. of Patients 5,000 patients as of May 2012; 

20,000 targeted enrollment

Challenge

Parkinson’s disease (PD), an incurable, 

progressive neurogenerative disorder associated 

with a high burden of disease, presents unique 

challenges for quality improvement initiatives. 

Treatments for PD generally focus on reducing 

patients’ symptoms and improving quality of life. 

Unlike other chronic conditions where 

improvement can be measured in terms of 

well-defined outcomes such as survival or 

cardiovascular events, quality improvement in PD 

can best be measured using patient-based 

outcomes. However, identifying appropriate 

patient-based outcomes for this disease can be a 

challenge. In addition, variability exists in the 

clinical diagnosis, management, and treatment of 

PD. Studies have shown that PD patients treated 

by a neurologist experience better outcomes, such 

as a decrease in hip fractures or nursing home 

placement. However, the specific management 

and treatment strategies used by these specialists 

have not been studied or well-described. The lack 

of evidence-based treatment standards warranted 

a data-driven approach to identify and understand 

best practices that improve the quality of care and 

quality of life for PD patients. 

Proposed Solution

In 2009, the National Parkinson Foundation 

launched an initiative to improve the quality of 

care in PD. To support an evidence-based 

approach, the foundation initiated a PD registry 

to capture clinical interventions and patientreported outcomes over time from multiple 

centers across the United States, Canada, and 

internationally. The initiative, led by a steering 

committee of movement disorders neurologists, is 

a unique effort in PD research because of its 

ability to collect long-term, longitudinal data 

from multiple centers and its focus on patientbased outcomes data, rather than process of care 

measures. The aims of the registry are to 

accelerate clinical discovery, promote 

collaborative science, and drive advancements in 

clinical practice toward patient-centered care.

Results

As of May 2012, the registry included more than 

5,000 patients from 20 centers; second- and 

third-year data were available for 3,000 and 500 

patients, respectively. Patients’ encounter-based 

data, including demographics, comorbidities, 

hospitalizations, falls, medications, treatments, 

and outcomes, are collected annually on brief 

data collection forms. The registry database 

includes a diverse population of PD patients, and 

analyses have confirmed variation in practice 

patterns across centers. The registry data have 

yielded important findings, including enhanced 

understanding of factors and predictors of 

patients’ quality of life and caregiver burden. 

Additional cross-sectional and longitudinal 

analyses are planned, using physician care and 

patient outcome data to describe practice patterns 

across the registry, identify and improve 

understanding of best practices, and support the 

development of guidelines. 

Many neurologists were initially doubtful about 

the value of a registry in this disease area. For the 

most part, their past experience was with 

mortality-based registries based around 

interventions or fatal illnesses; these failed to 
191

Chapter 22. Quality Improvement Registries

Case Example 57. Clarifying the Federal 

regulatory requirements for quality 

improvement registries

Description The National Neurosurgery 

Quality and Outcomes Database 

(N2QOD) is a prospective, 

longitudinal registry designed to 

measure and improve 

neurosurgical and spine surgical 

care as it exists in the real-world 

health care setting.

Sponsor American Association of 

Neurological Surgeons (AANS)

Year Started 2011

Year Ended Ongoing

No. of Sites 30 U.S. neuropractice groups 

expected in the first year

No. of Patients 7,000 patients expected in the 

first year

Challenge

N2QOD was formed with the aim of measuring 

the quality of real-world neurosurgical and spine 

surgery care, and the registry defined that 

“quality” as safety and effectiveness. Given this 

definition, a patient outcome–centered approach 

to data collection was necessary. This patientcenteredness is aligned with the priorities of 

groups such as the Patient-Centered Outcomes 

Research Institute and the Agency for Healthcare 

Research and Quality, and reflects a wider trend 

in quality improvement (QI) science, moving 

away from processes and process-based measures 

to patient outcomes and outcome measures. 

This move towards patient outcomes necessitates 

a shift in the way QI registries interact with 

patients. It also presents challenges for 

institutional review boards (IRBs) reviewing 

these projects. IRBs can determine that these 

projects are either “health care operations” or 

“human subjects research,” as defined by the 

Health Insurance Portability and Accountability 

Act of 1996 (HIPAA) and the Common Rule. If 

an IRB determines that a registry constitutes 

“health care operations” (i.e., data collection used 

for issues such as clinical care, administrative 

use, or quality assessment), then neither IRB 

approval nor informed consent is required. If an 

IRB determines that a registry constitutes “human 

subjects research,” the registry falls under IRB 

purview, and the IRB may determine that 

informed consent is required of registry 

participants, or it may grant a waiver of informed 

consent. 

Case Example 56. Using registries to drive 

quality improvement in chronic conditions 

(continued)

Results (continued)

model a disease with complex, heterogeneous 

symptomology, where the pathology could not be 

directly measured. Increasingly, providers have 

recognized the value of the statistical power and 

nuanced insight that can be leveraged in this large 

and detailed registry of expert care.

Key Point

Registry-based quality improvement programs 

can be useful in many clinical settings, from 

in-hospital care (e.g., heart failure) to chronic 

progressive diseases (e.g., PD). The design of the 

registry and the quality improvement initiative 

must reflect the nature of the disease and the state 

of existing evidence. For chronic, progressive 

diseases, registries can be useful tools for 

identifying, developing, and disseminating 

guidelines for best practices to improve quality of 

care. 

For More Information

http://www.parkinson.org/Improving-Care/

Research/Quality-Improvement-Initiative

Okun MS, Siderowf A, Nutt JG, et al. Piloting the 

NPF data-driven quality improvement initiative. 

Parkinsonism and Related Disorders 201;16: 

517-21. 
192

Section V. Special Applications in Patient Registries

Case Example 57. Clarifying the Federal 

regulatory requirements for quality 

improvement registries (continued)

Challenge (continued)

Whether an IRB determines a registry to be 

“health care operations” or “human subjects 

research” can have profound operational and 

analytic impacts on the registry. In particular, QI 

registries designated by an IRB as research and 

required to collect informed consent from 

participants can experience a reduction in 

enrollment numbers, and are exposed to the risk 

of selection bias being introduced into the 

registry population.

The registry was introduced to neurosurgical 

practice sites in January 2011, and was initially 

reviewed by 11 IRBs over a 4-month period. Six 

of those evaluations resulted in classifications of 

the registry as quality improvement (QI). The 

remaining five IRBs classified the same project 

description as human subjects research, and 

insisted on full IRB oversight and the requirement 

for informed consent.

Proposed Solution

Given this mixed interpretation of Federal 

regulations from local IRBs, the AANS 

approached the Department of Health and Human 

Services’ (HHS) Office of Human Research 

Protections (OHRP) in May 2011 to request a 

formal review of the registry. AANS and OHRP 

engaged in regular communication over the 

course of several months, and convened a 

multistakeholder meeting at the White House that 

included representatives from OHRP, the Office 

of the President, Centers for Medicare and 

Medicaid Services, U.S. Food and Drug 

Administration, Department of Veterans Affairs, 

HHS Office of Civil Rights, and three clinical 

specialty societies, including neurosurgery.

Results

In August 2011, OHRP clarified that, based on 

these communications and an examination of the 

registry, the sites participating in N2QOD were 

not engaged in human subjects research, and 

therefore the regulations requiring IRB oversight 

did not apply. This communication from OHRP is 

now provided to sites enrolling in N2QOD to 

support their IRB review process.

At the time of this writing, 28 IRBs have formally 

reviewed or re-reviewed the registry. To date, 27 

of the IRBs have classified it as health care 

operations and have waived the requirement for 

IRB review. The remaining IRB has classified the 

same project description as research, and has 

issued a waiver of consent for the project. 

Approximately 30 additional sites are still in 

various stages of institutional review. In 

summary, the OHRP opinion strongly influenced 

local IRB analyses of the registry.

In July 2011, OHRP released an Advanced Notice 

of Proposed Rulemaking (ANPRM) for Revision 

to the Common Rule. These revisions are 

intended to improve human subject research 

while also reducing burdens, delays, and 

ambiguity for investigators and research subjects.

Key Point

QI registries that are focused on patient outcomes 

should be aware of the complexities around 

varied interpretation by multiple IRBs and should 

plan sufficient time and resources to address 

these complexities.

For More Information

http://www.neuropoint.org 

Department of Health and Human Services. 

Office of Human Research Protections. “ANPRM 

for Revision to Common Rule.” http://www.hhs.

gov/ohrp/humansubjects/anprm2011page.html. 

Accessed 20 June 20, 2012.

Neuropoint Alliance, Inc. “The National 

Neurosurgery Quality and Outcomes Database 

(N2QOD): A Prospective Registry for Quality 

Reporting. Background, Project Description, 

Application of Relevant Federal Regulations and 

Project Implementation.” http://www.neuropoint.

org/pdf/N2QOD%20Project%20Description%20

V5%20(25APR2012).pdf. 
193

Chapter 22. Quality Improvement Registries

References for Chapter 22

1. LaBresh KA, Gliklich R, Liljestrand J, et al. Using 

“get with the guidelines” to improve 

cardiovascular secondary prevention. Jt Comm 

J Qual Saf. 2003 Oct;29(10):539-50. 

PMID: 14567263.

2. Raval MV, Bentrem DJ, Eskandari MK, et al. 

The role of Surgical Champions in the American 

College of Surgeons National Surgical Quality 

Improvement Program—a national survey. J Surg 

Res. 2011 Mar;166(1):e15-25. PMID: 21176914.

3. Fonarow GC, Abraham WT, Albert NM, et al. 

Association between performance measures and 

clinical outcomes for patients hospitalized with 

heart failure. JAMA. 2007 Jan 3;297(1):61-70. 

PMID: 17200476.

4. Lee JS, Primack BA, Mor MK, et al. Processes of 

care and outcomes for community-acquired 

pneumonia. Am J Med. 2011 Dec;124(12):

1175 e9-17. PMID: 22000624. PMCID: 3578284.

5. Morse RB, Hall M, Fieldston ES, et al. Hospitallevel compliance with asthma care quality 

measures at children’s hospitals and subsequent 

asthma-related outcomes. JAMA. 2011 Oct 

5;306(13):1454-60. PMID: 21972307.

6. Porter ME. What is value in health care? N Engl 

J Med. 2010 Dec 23;363(26):2477-81. 

PMID: 21142528.

7. Institute of Medicine. Performance Measurement: 

Accelerating Improvement. Committee on 

Redesigning Health Insurance Performance 

Measures, Payment, and Performance 

Improvement Programs. http://iom.edu/Reports/

2005/Performance-Measurement-AcceleratingImprovement.aspx. Accessed August 20, 2012.

8. National Quality Forum. What NQF Endorsement 

Means. http://www.qualityforum.org/Measuring_

Performance/ABCs/What_NQF_Endorsement_

Means.aspx. Accessed August 20, 2012.

9. National Priorities Partnership. 

http://www.qualityforum.org/Setting_Priorities/

NPP/National_Priorities_Partnership.aspx. 

Accessed December 20, 2013.

10. Measure Applications Partnership. National 

Quality Forum. http://www.qualityforum.org/

Setting_Priorities/Partnership/Measure_

Applications_Partnership.aspx. Accessed August 

20, 2012.

11. Casarett D, Karlawish JH, Sugarman J. 

Determining when quality improvement initiatives 

should be considered research: proposed criteria 

and potential implications. JAMA. 2000 May 

3;283(17):2275-80. PMID: 10807388.

12. Dokholyan RS, Muhlbaier LH, Falletta JM, et al. 

Regulatory and ethical considerations for linking 

clinical and administrative databases. Am Heart J. 

2009 Jun;157(6):971-82. PMID: 19464406.

13. Lynn J, Baily MA, Bottrell M, et al. The ethics of 

using quality improvement methods in health care. 

Ann Intern Med. 2007 May 1;146(9):666-73. 

PMID: 17438310.

14. Nerenz DR. Ethical issues in using data from 

quality management programs. Eur Spine J. 2009 

Aug;18 Suppl 3:321-30. PMID: 19365642. 

PMCID: 2899322.

15. Johnson N, Vermeulen L, Smith KM. A survey of 

academic medical centers to distinguish between 

quality improvement and research activities. Qual 

Manag Health Care. 2006 Oct-Dec;15(4):215-20. 

PMID: 17047495.

16. Tu JV, Willison DJ, Silver FL, et al. 

Impracticability of informed consent in the 

Registry of the Canadian Stroke Network. N Engl 

J Med. 2004 Apr 1;350(14):1414-21. 

PMID: 15070791.

17. U.S. Department of Health and Human Services. 

Office for Human Research Protections. Quality 

Improvement Activities – FAQs. http://answers.

hhs.gov/ohrp/categories/1569. Accessed August 

15, 2012.

18. U.S. Department of Health and Human Services. 

Office for Human Research Protections. How does 

HHS view quality improvement activities in 

relation to the regulations for human research 

subject protections? “Quality Improvement 

Activities – FAQs.” http://answers.hhs.gov/ohrp/

questions/7281. Accessed December 20, 2013.

19. U.S. Department of Health and Human Services. 

Office for Human Research Protections. Do the 

HHS regulations for the protection of human 

subjects in research (45 CFR part 46) apply to 

quality improvement activities conducted by one 

or more institutions whose purposes are limited 

to: (a) implementing a practice to improve the 

quality of patient care, and (b) collecting patient 

or provider data regarding the implementation of 

the practice for clinical, practical, or 
194

Section V. Special Applications in Patient Registries

administrative purposes? “Quality Improvement 

Activities – FAQs.” http://answers.hhs.gov/ohrp/

questions/7282. Accessed December 20, 2013.

20. U.S. Department of Health and Human Services. 

Office for Human Research Protections. Can I 

analyze data that are not individually identifiable, 

such as medication databases stripped of 

individual patient identifiers, for research purposes 

without having to apply the HHS protection of 

human subjects regulations? “Quality 

Improvement Activities – FAQs.” http://answers.

hhs.gov/ohrp/questions/7284. Accessed December 

20, 2013.

21. Emanuel EJ, Menikoff J. Reforming the 

regulations governing research with human 

subjects. N Engl J Med. 2011 Sep 

22;365(12):1145-50. PMID: 21787202.

22. Hannan EL, Kilburn H, Jr., O’Donnell JF, et al. 

Adult open heart surgery in New York State. An 

analysis of risk factors and hospital mortality 

rates. JAMA. 1990 Dec 5;264(21):2768-74. 

PMID: 2232064.

23. Hannan EL, Kumar D, Racz M, et al. New York 

State’s Cardiac Surgery Reporting System: four 

years later. Ann Thorac Surg. 1994 

Dec;58(6):1852-7. PMID: 7979781.

24. O’Connor GT, Plume SK, Olmstead EM, et al. 

A regional prospective study of in-hospital 

mortality associated with coronary artery bypass 

grafting. The Northern New England 

Cardiovascular Disease Study Group. JAMA. 

1991 Aug 14;266(6):803-9. PMID: 1907669.

25. O’Connor GT, Plume SK, Olmstead EM, et al. 

Multivariate prediction of in-hospital mortality 

associated with coronary artery bypass graft 

surgery. Northern New England Cardiovascular 

Disease Study Group. Circulation. 1992 

Jun;85(6):2110-8. PMID: 1591830.

26. Grover FL, Hammermeister KE, Shroyer AL. 

Quality initiatives and the power of the database: 

what they are and how they run. Ann Thorac Surg. 

1995 Nov;60(5):1514-21. PMID: 8526678.

27. Grover FL, Johnson RR, Marshall G, et al. Factors 

predictive of operative mortality among coronary 

artery bypass subsets. Ann Thorac Surg. 1993 

Dec;56(6):1296-306; discussion 306-7. 

PMID: 8267428.

28. Grover FL, Johnson RR, Shroyer AL, et al. The 

Veterans Affairs Continuous Improvement in 

Cardiac Surgery Study. Ann Thorac Surg. 1994 

Dec;58(6):1845-51. PMID: 7979780.

29. Edwards FH, Grover FL, Shroyer AL, et al. The 

Society of Thoracic Surgeons National Cardiac 

Surgery Database: current risk assessment. Ann 

Thorac Surg. 1997 Mar;63(3):903-8. 

PMID: 9066436.

30. Edwards FH, Clark RE, Schwartz M. Coronary 

artery bypass grafting: the Society of Thoracic 

Surgeons National Database experience. Ann 

Thorac Surg. 1994 Jan;57(1):12-9. 

PMID: 8279877.

31. Grover FL, Shroyer AL, Hammermeister K, et al. 

A decade’s experience with quality improvement 

in cardiac surgery using the Veterans Affairs and 

Society of Thoracic Surgeons national databases. 

Ann Surg. 2001 Oct;234(4):464-72; discussion 

72-4. PMID: 11573040. PMCID: 1422070.

32. Blumberg MS. Comments on HCFA hospital 

death rate statistical outliers. Health Care 

Financing Administration. Health Serv Res. 1987 

Feb;21(6):715-39. PMID: 3106265. 

PMCID: 1068986.

33. O’Connor GT, Plume SK, Olmstead EM, et al. 

A regional intervention to improve the hospital 

mortality associated with coronary artery bypass 

graft surgery. The Northern New England 

Cardiovascular Disease Study Group. JAMA. 

1996 Mar 20;275(11):841-6. PMID: 8596221.

34. Ferguson TB, Jr., Peterson ED, Coombs LP, et al. 

Use of continuous quality improvement to 

increase use of process measures in patients 

undergoing coronary artery bypass graft surgery: 

a randomized controlled trial. JAMA. 2003 Jul 

2;290(1):49-56. PMID: 12837711.

35. Tu JV, Donovan LR, Lee DS, et al. Effectiveness 

of public report cards for improving the quality of 

cardiac care: the EFFECT study: a randomized 

trial. JAMA. 2009 Dec 2;302(21):2330-7. 

PMID: 19923205.

36. Hibbard JH, Stockard J, Tusler M. Does 

publicizing hospital performance stimulate quality 

improvement efforts? Health Aff (Millwood). 

2003 Mar-Apr;22(2):84-94. PMID: 12674410.
195

Chapter 22. Quality Improvement Registries

37. Fung CH, Lim YW, Mattke S, et al. Systematic 

review: the evidence that publishing patient care 

performance data improves quality of care. Ann 

Intern Med. 2008 Jan 15;148(2):111-23. 

PMID: 18195336.

38. Hannan EL, Siu AL, Kumar D, et al. The decline 

in coronary artery bypass graft surgery mortality 

in New York State. The role of surgeon volume. 

JAMA. 1995 Jan 18;273(3):209-13. 

PMID: 7807659.

39. Hannan EL, Kilburn H, Jr., Racz M, et al. 

Improving the outcomes of coronary artery bypass 

surgery in New York State. JAMA. 1994 Mar 

9;271(10):761-6. PMID: 8114213.

40. Ghali WA, Ash AS, Hall RE, et al. Statewide 

quality improvement initiatives and mortality after 

cardiac surgery. JAMA. 1997 Feb 5;277(5):

379-82. PMID: 9010169.

41. Peterson ED, DeLong ER, Jollis JG, et al. The 

effects of New York’s bypass surgery provider 

profiling on access to care and patient outcomes in 

the elderly. J Am Coll Cardiol. 1998 Oct;32(4):

993-9. PMID: 9768723.

42. Guru V, Fremes SE, Naylor CD, et al. Public 

versus private institutional performance reporting: 

what is mandatory for quality improvement? Am 

Heart J. 2006 Sep;152(3):573-8. 

PMID: 16923433.

43. Hannan EL, Sarrazin MS, Doran DR, et al. 

Provider profiling and quality improvement efforts 

in coronary artery bypass graft surgery: the effect 

on short-term mortality among Medicare 

beneficiaries. Med Care. 2003 Oct;41(10):

1164-72. PMID: 14515112.

44. Vladeck BC, Goodwin EJ, Myers LP, et al. 

Consumers and hospital use: the HCFA “death 

list.” Health Aff (Millwood). 1988 

Spring;7(1):122-5. PMID: 3360387.

45. Hannan EL, Stone CC, Biddle TL, et al. Public 

release of cardiac surgery outcomes data in New 

York: what do New York state cardiologists think 

of it? Am Heart J. 1997 Jul;134(1):55-61. 

PMID: 9266783.

46. Chassin MR. Achieving and sustaining improved 

quality: lessons from New York State and cardiac 

surgery. Health Aff (Millwood). 2002 JulAug;21(4):40-51. PMID: 12117152.

47. Schneider EC, Epstein AM. Use of public 

performance reports: a survey of patients 

undergoing cardiac surgery. JAMA. 1998 May 

27;279(20):1638-42. PMID: 9613914.

48. Schneider EC, Epstein AM. Influence of cardiacsurgery performance reports on referral practices 

and access to care. A survey of cardiovascular 

specialists. N Engl J Med. 1996 Jul 25;335(4):

251-6. PMID: 8657242.

49. Mukamel DB, Weimer DL, Mushlin AI. 

Interpreting market share changes as evidence for 

effectiveness of quality report cards. Med Care. 

2007 Dec;45(12):1227-32. PMID: 18007175.

50. Mukamel DB, Mushlin AI. The impact of quality 

report cards on choice of physicians, hospitals, 

and HMOs: a midcourse evaluation. Jt Comm 

J Qual Improv. 2001 Jan;27(1):20-7. 

PMID: 11147237.

51. Romano PS, Zhou H. Do well-publicized riskadjusted outcomes reports affect hospital volume? 

Med Care. 2004 Apr;42(4):367-77. 

PMID: 15076814.

52. Hibbard JH, Peters E. Supporting informed 

consumer health care decisions: data presentation 

approaches that facilitate the use of information in 

choice. Annu Rev Public Health. 2003;24:413-33. 

PMID: 12428034.

53. Hibbard JH, Peters E, Slovic P, et al. Making 

health care quality reports easier to use. Jt Comm 

J Qual Improv. 2001 Nov;27(11):591-604. 

PMID: 11708039.

54. Kaiser Family Foundation 2008 Update on 

Consumers’ Views of Patient Safety and Quality 

Information. 2008. Henry J. Kaiser Family 

Foundation. http://search.kff.org/gsaresults/search

?site=KFForgnopdfs&filter=0&output=xml_no_dt

d&client=kff&sp=kff&getfields=*&q=7819&

no_pdf=1. Accessed August 20, 2012.

55. Shahian DM, Normand SL, Torchiana DF, et al. 

Cardiac surgery report cards: comprehensive 

review and statistical critique. Ann Thorac Surg. 

2001 Dec;72(6):2155-68. PMID: 11789828.

56. Green J, Wintfeld N. Report cards on cardiac 

surgeons. Assessing New York State’s approach. 

N Engl J Med. 1995 May 4;332(18):1229-32. 

PMID: 7700321.
196

Section V. Special Applications in Patient Registries

57. Jones RH. In search of the optimal surgical 

mortality. Circulation. 1989 Jun;79(6 Pt 2):I132-6. 

PMID: 2785874.

58. Lee TH, Torchiana DF, Lock JE. Is zero the ideal 

death rate? N Engl J Med. 2007 Jul 

12;357(2):111-3. PMID: 17625122.

59. Dranove D, Kessler DA, McClellan M, et al. Is 

more information better? The effects of “report 

cards” on health care providers. Journal of 

Political Economy. 2003;111:555-88.

60. Omoigui NA, Miller DP, Brown KJ, et al. 

Outmigration for coronary bypass surgery in an 

era of public dissemination of clinical outcomes. 

Circulation. 1996 Jan 1;93(1):27-33. 

PMID: 8616936.

61. Burack JH, Impellizzeri P, Homel P, et al. Public 

reporting of surgical mortality: a survey of New 

York State cardiothoracic surgeons. Ann Thorac 

Surg. 1999 Oct;68(4):1195-200; discussion 201-2. 

PMID: 10543479.

62. Li Z, Carlisle DM, Marcin JP, et al. Impact of 

public reporting on access to coronary artery 

bypass surgery: the California Outcomes 

Reporting Program. Ann Thorac Surg. 2010 

Apr;89(4):1131-8. PMID: 20338320.

63. Bridgewater B, Grayson AD, Brooks N, et al. Has 

the publication of cardiac surgery outcome data 

been associated with changes in practice in 

northwest England: an analysis of 25,730 patients 

undergoing CABG surgery under 30 surgeons 

over eight years. Heart. 2007 Jun;93(6):744-8. 

PMID: 17237128. PMCID: 1955202.

64. Moscucci M, Eagle KA, Share D, et al. Public 

reporting and case selection for percutaneous 

coronary interventions: an analysis from two large 

multicenter percutaneous coronary intervention 

databases. J Am Coll Cardiol. 2005 Jun 

7;45(11):1759-65. PMID: 15936602.

65. Apolito RA, Greenberg MA, Menegus MA, et al. 

Impact of the New York State Cardiac Surgery and 

Percutaneous Coronary Intervention Reporting 

System on the management of patients with acute 

myocardial infarction complicated by cardiogenic 

shock. Am Heart J. 2008 Feb;155(2):267-73. 

PMID: 18215596.

66. Resnic FS, Welt FG. The public health hazards of 

risk avoidance associated with public reporting of 

risk-adjusted outcomes in coronary intervention. 

J Am Coll Cardiol. 2009 Mar 10;53(10):825-30. 

PMID: 19264236. PMCID: 2673987.

67. Clarke S, Oakley J. Informed consent and 

surgeons’ performance. J Med Philos. 2004 

Feb;29(1):11-35. PMID: 15449811.

68. Clarke S, Oakley J. Informed consent and 

clinician accountability: the ethics of report cards 

on surgeon performance. Cambridge, UK: 

Cambridge University Press; 2007.

69. Drozda JP, Jr., Hagan EP, Mirro MJ, et al. ACCF 

2008 health policy statement on principles for 

public reporting of physician performance data: 

A Report of the American College of Cardiology 

Foundation Writing Committee to develop 

principles for public reporting of physician 

performance data. J Am Coll Cardiol. 2008 May 

20;51(20):1993-2001. PMID: 18482675.

70. Krumholz HM, Brindis RG, Brush JE, et al. 

Standards for statistical models used for public 

reporting of health outcomes: an American Heart 

Association Scientific Statement from the Quality 

of Care and Outcomes Research Interdisciplinary 

Writing Group: cosponsored by the Council on 

Epidemiology and Prevention and the Stroke 

Council. Endorsed by the American College of 

Cardiology Foundation. Circulation. 2006 Jan 

24;113(3):456-62. PMID: 16365198.

71. National Quality Forum. Evidence Task Force 

Final Report. Availablet at: http://www.

qualityforum.org/Measuring_Performance/

Improving_NQF_Process/Evidence_Task_Force.

aspx. Accessed August 20, 2012.

72. National Quality Forum. Guidance for Measure 

Testing and Evaluating Scientific Acceptability of 

Measure Properties, Final Report. January 2011. 

http://www.qualityforum.org/Measuring_

Performance/Improving_NQF_Process/Measure_

Testing_Task_Force_Final_Report.aspx. Accessed 

August 20, 2012.

73. Dimick JB, Welch HG, Birkmeyer JD. Surgical 

mortality as an indicator of hospital quality: the 

problem with small sample size. JAMA. 2004 Aug 

18;292(7):847-51. PMID: 15315999.
197

Chapter 22. Quality Improvement Registries

74. Hofer TP, Hayward RA, Greenfield S, et al. The 

unreliability of individual physician “report cards” 

for assessing the costs and quality of care of a 

chronic disease. JAMA. 1999 Jun 9;281(22):

2098-105. PMID: 10367820.

75. Fung V, Schmittdiel JA, Fireman B, et al. 

Meaningful variation in performance: a systematic 

literature review. Med Care. 2010 Feb;48(2):

140-8. PMID: 20057334.

76. Li Q, Glynn RJ, Dreyer NA, et al. Validity of 

claims-based definitions of left ventricular systolic 

dysfunction in Medicare patients. 

Pharmacoepidemiol Drug Saf. 2011 

Jul;20(7):700-8. PMID: 21608070.

77. National Cardiovascular Data Registry. 

https://www.ncdr.com/webncdr/. Accessed 

August 7, 2013.

78. Society for Thoracic Surgeons National Database. 

http://www.sts.org/national-database. Accessed 

August 15, 2012.

79. Jacobs JP, Edwards FH, Shahian DM, et al. 

Successful linking of the Society of Thoracic 

Surgeons database to social security data to 

examine survival after cardiac operations. Ann 

Thorac Surg. 2011 Jul;92(1):32-7; discussion 8-9. 

PMID: 21718828.

80. Jacobs JP, Edwards FH, Shahian DM, et al. 

Successful linking of the Society of Thoracic 

Surgeons adult cardiac surgery database to 

Centers for Medicare and Medicaid Services 

Medicare data. Ann Thorac Surg. 2010 

Oct;90(4):1150-6; discussion 6-7. 

PMID: 20868806.

81. Bufalino VJ, Masoudi FA, Stranne SK, et al. The 

American Heart Association’s recommendations 

for expanding the applications of existing and 

future clinical registries: a policy statement from 

the American Heart Association. Circulation. 2011 

May 17;123(19):2167-79. PMID: 21482960.

199

Chapter 23. Registries for Medical Devices

1. Introduction

Medical device registries are critical for the 

identification and study of medical devices 

outcomes. Device registries are used for many 

purposes, including short- and long-term 

surveillance, fulfillment of postmarket 

observational study commitments for regulatory 

bodies, and comparative safety and effectiveness 

assessments, including those in under-studied 

subpopulations. (See Chapter 1 for extensive 

discussion on the definition and purposes of 

registries in general.) Medical device registries 

play an increasingly important role in bridging the 

gap between device performance in clinical trials 

and their use in routine practice over time. Unlike 

clinical trials, device registries allow assessment of 

medical device performance in a real-world 

setting. Registries contain data on large numbers 

of patients receiving care in diverse clinical 

settings and include clinical outcomes over time, 

thus providing a critical platform for capturing the 

experience with a medical device throughout the 

device and patient lifecycle. Moreover, by linking 

device exposures and long-term outcomes, 

registries permit followup that can span decades. 

While devices share some similarities with drugs, 

several major issues unique to devices require 

special consideration in order to construct and use 

a medical device registry successfully. Outcomes 

associated with medical devices can be affected 

not only by underlying patient factors and device 

factors (such as biomaterials), but also by user 

interface (e.g., surgical technique or surgical 

preference and experience) and ancillary 

technologies (e.g., choice of imaging). Adverse 

effects of devices can be localized (e.g., stent 

thrombosis), but may be more systemic (e.g., 

toxic, allergic, autoimmune effects). Furthermore, 

additional hazards may be related to human factor 

errors such as poor design or adverse interactions 

(e.g., drug-device or electromagnetic interference). 

Finally, reasons for device malfunctions may be 

very diverse, ranging from manufacturing 

problems and design-induced errors to 

environmental factors (e.g., humidity) and poor 

maintenance. Also, although certain malfunctions 

or device performance issues may appear to be 

similar, their root cause may vary. 

Special challenges related to registry design, data 

collection, and analysis include the need for 

unique identification of devices, including device 

modifications and device components; information 

on user interface (e.g., surgical technique); 

information on ancillary technology and therapies 

(e.g., drug exposures); detection of device 

performance issues; the need for followup; and the 

impact of health care provider experience and 

learning.

This chapter will address two topics related to 

medical device registries: challenges in design and 

data collection and potential uses of emerging 

technology. The chapter begins with a discussion 

of the major considerations that influence the 

design of device registries and the data that must 

be collected. Potential approaches to address a 

variety of design challenges are described. The 

chapter then discusses emerging technologies that 

will potentially allow integration of automated 

device data capture into registry data sets. Case 

Examples 58, 59, 60, and 61 offer descriptions of 

medical device registries.

2. Differences Between Drugs 

and Devices

Regulatory oversight of the approximately 1,700 

marketed device types in the United States is 

achieved through the use of regulatory controls 

and a classification process to assure reasonable 

device safety and effectiveness. The U.S. Food and 

Drug Administration (FDA) Center for Devices 

and Radiological Health regulates a wide range of 

products. These are identified into classes I, II, and 

III, based on the level of control necessary to 

ensure the safety and effectiveness of the device 

and the intended use and indications for use of the 

device. The Total Product Life Cycle1 follows 
200

Section V. Special Applications in Patient Registries

products from concept to obsolescence and 

incorporates post-approval and surveillance 

information, such as rare adverse events or issues 

within subpopulations, back into these new 

product generations. This useful framework 

establishes a solid baseline for understanding 

effectiveness and safety in devices. 

Differences between drugs and devices persist 

throughout the Total Product Life Cycle. Drug 

modifications occur slowly, if at all, whereas 

device technologies often experience rapid and 

continuous changes over time, potentially driven 

by user feedback and involving the use of new 

materials and approaches.2, 3 In addition, device 

performance is affected by factors beyond the 

device itself, such as operator skill and 

experience.4

3. Design and Data Collection 

Considerations 

As with all registries, the primary purpose of a 

medical device registry will guide design options. 

Many factors related to registry design are similar 

to those discussed in Chapter 3, such as selection 

of a study design and sample size considerations. 

However, some distinctive features of a medical 

device registry require additional planning. It is 

critical that the registry is adaptable to various 

needs that arise during the lifecycle of device 

innovation. Some challenges include the lack of 

unique device identifiers, including model/version 

control and component identification; the need to 

adequately capture device malfunctions and 

failures; the need for longer followup; and the 

impact of provider experience, training, and choice 

of device. It is informative that some device 

registries have been developed from procedure 

registries, with the addition of device identification 

modules. In these cases, likely risks of device 

failures may be identified prior to consequential 

clinical signs and symptoms, allowing for device 

fixes before harms are experienced by the patient. 

This chapter will provide general suggestions for 

addressing these challenges, and although each 

device registry is unique and will require a 

solution appropriate for its specific purpose, 

general principles apply in many cases.

3.1 Device Identification 

Currently, although Unique Device Identifiers 

(UDI) are available for some medical devices (in 

the form of GTIN or HIBC identifiers, which are 

numbers located under bar codes), they are not 

routinely captured in observational data sources 

like billing claims data or registries, as is the case 

with National Drug Codes, which permit universal 

drug identification. The inability to identify 

specific devices affects registry design and data 

collection and poses challenges for researchers and 

regulators. Compounding the problem is the fact 

that device modifications are frequent and part of 

the business model for manufacturing and 

innovation. Researchers may connect safety and 

effectiveness to a class/subset of devices, rather 

than to the device generally. Hence, it is critically 

important to identify an individual device 

accurately. Related challenges include the lack of 

standardized definitions and of attribute 

(descriptor) creation based on specific device 

product codes; difficulties in data collection, such 

as transmitting information from electronic 

medical records (EMR) to registries or automated 

data capture, such as those related to barcode 

scanning accuracy;5 and hurdles in maintaining 

master product lists.

Based on a congressional mandate (Section 226 of 

the FDA Amendments Act of 2007 and Section 

614 of the FDA Safety and Innovations Act), FDA 

recently issued a draft rule detailing the 

requirements of manufacturers to have UDIs for 

their products, and to have this requirement phased 

in over several years. This FDA initiative 

eventually will assist with many of the challenges 

posed by the current lack of standard identifiers for 

medical devices. In the meantime, several 

approaches may be used to capture identity in the 

absence of a UDI that is unique across all devices. 

For example, some devices have identifiers, such 

as catalogue, model, serial, and lot numbers, that 

are unique to a particular manufacturer’s device. 

While these are not standardized and there may be 

several components from different manufacturers 

with similar catalogue or model numbers, these 

numbers can facilitate device identification and 

tracking when combined. 
201

Chapter 23. Registries for Medical Devices

Thus, prior to full UDI adoption and 

implementation, researchers must be creative in 

collecting device information and taking advantage 

of UDI-like data to fill the gap created by the lack 

of identifiers. For example, the Society of Thoracic 

Surgeons Adult Cardiac Database developed a data 

collection form with an exhaustive list of various 

heart valve devices. This checklist enables registry 

participants to collect any information that could 

be relevant to their practice. Orthopedic registries 

worldwide are taking advantage of catalogue 

numbers and lot or serial numbers in order to 

classify and uniquely identify products. An 

inefficient but at times appropriate solution is to 

include device photos in the registry. This strategy 

is most applicable in settings with few devices on 

the market that have marked differences in design 

that can be captured with photographs.

3.2 Device Performance

Devices exhibit numerous types of performance 

issues, so that it is difficult to capture all potential 

performance issues, failure modes, and adverse 

events in a single device registry. The performance 

issues may be related to software, hardware, 

biomaterials, sterility, or other issues. Additionally, 

similar performance problems (e.g., pacemaker 

oversensing) could have various root causes, some 

of which may manifest clinically (e.g., breast 

implant rupture) while others may not. 

Importantly, although clinical trials may provide 

some knowledge of failure rates and timing, 

knowledge of the propensity of these failures to 

develop is limited during the registry design phase. 

Adverse-event reporting for device registries 

should follow the regulatory requirements for 

reporting.6 Researchers should consider methods 

of adjudication and verification of issues with 

device performance during the design phase in 

order to ensure collection of all data elements 

needed to inform those discussions. It is also 

important to consider how potential performance 

issues will be detected for the particular device. 

Automated surveillance within the registry is an 

advanced approach to identifying select 

performance issues with a device (i.e., those that 

manifest uniquely and clinically). When 

implemented correctly, it can permit real-time 

evaluation of performance issues within a large 

sample or population. Surveillance, however, is a 

complex endeavor, and standardized data elements 

and collection procedures are required, likely 

across multiple institutions or registries. There are 

several examples of successful registry 

implementation for surveillance in cardiovascular 

disease. The Data Extraction and Longitudinal 

Trend Analysis (DELTA) network study was the 

first computerized safety surveillance proof-ofconcept study for cardiovascular medical devices. 

This multicenter prospective observational study 

was designed for safety evaluation of drug-eluting 

coronary stents, embolic protection devices, and 

vascular closure devices used during percutaneous 

coronary intervention.7 The study facilitated 

aggregation of safety events across institutions by 

using standard data elements from the American 

College of Cardiology National Cardiovascular 

Data Registry (NCDR®). 

3.3 Device Systems and Components 

In many cases, the device of interest for a registry 

is either part of a larger system of devices or 

contains multiple components that are considered 

devices themselves. Issues around the lack of 

unique identifiers persist and are accompanied by 

the additional challenge of determining which 

component is responsible for the performance 

issues. Sometimes, FDA approves or clears device 

components separately. When a registry is 

designed to understand effectiveness and safety, 

and the device of interest is dependent on 

accompanying devices included in the same 

system, information on all components must 

ideally be captured in enough detail to assess how 

well the device of interest is functioning. 

In some instances, FDA approves devices as full 

systems rather than singular components. 

Examples of this include implantable pacemakers, 

implantable cardioverter defibrillators, and hip and 

knee implants. In these cases, surgeons may “mix 

and match” multiple manufacturers or multiple 

brands into one system. Such mixing presents a 

data collection issue as well as an analytical 

challenge. Heterogeneous devices may need to be 

grouped together in order to perform analysis.
202

Section V. Special Applications in Patient Registries

In addition to the actual device, some implantable 

devices require assistance from procedural 

devices, including other commodity devices or 

operative instruments, or ancillary devices, such as 

imaging equipment. In these cases, clinically 

relevant additional information should be 

collected. For example, in hernia repair, 

information on the method of mesh adhesion, such 

as staples, glue, or sutures, may need to be 

collected, as these adhesives could interact with a 

specific type of mesh and affect device 

performance. Researchers should consider the role 

of these factors and how they can be captured in 

the data collection process. 

3.4 Drug/Device Combinations 

Device/drug combinations have become 

increasingly common over the past decade. 

Because the development processes for drugs and 

devices differ, combination products face different 

challenges. For example, drug-eluting stents (DES) 

are an example of a product where the device, a 

bare-metal stent, has been enhanced by the 

addition of an immunosuppressant or mitotic 

inhibitor8 and its elution polymer coating. In 2003, 

FDA approved the addition of drugs to stents for a 

subset of cardiac patients with uncomplicated 

coronary lesions.9 In these patients, there was a 

decrease in coronary restenosis requiring repeat 

revascularization 9 months after stent implantation, 

compared with bare-metal stents. However, as the 

adoption rate of DES increased, the population of 

patients in which they were implanted changed—

they tended be sicker patients. Moreover, stent 

thrombosis, a rare but serious adverse event, was 

higher in DES patients at 1 year compared with 

bare-metal stent patients.10-12 Stent thrombosis 

represented a localized device failure mode with 

serious and unique clinical manifestations. In cases 

like this, registries are a critical tool for 

understanding the long-term safety and 

effectiveness of the technology.13, 14 Special 

considerations in registry design include separate 

collection of concomitant drug dosing information 

and attention to the medications that the patient is 

taking during and post implantation in order to flag 

possible drug interactions. It is also important to 

prospectively collect concurrent medications that 

the patient is using over time, again in order to 

understand potential interactions.

3.5 Obtaining Sufficient Followup 

Information 

Obtaining sufficient followup information and as 

complete as possible case ascertainment are issues 

for all studies, and many of these challenges are 

addressed in Chapters 3 and 10. However, longterm followup is a particular concern with 

implantable devices, as well as other products such 

as ablation and radiation therapy devices. Clinical 

trials have relatively short followup for 

implantable devices that are expected to stay in the 

body indefinitely or until replaced with a similar 

device. These devices are typically studied for less 

than 5 years premarket, but are intended to work 

for decades. While followup time in the initial 

period of implantation is useful, an indefinite 

followup registry imbedded within a clinical 

practice has the ability to answer questions 

concerning device safety and effectiveness over 

the full product lifecycle. Only a few registries 

have sufficient followup for endpoints of device 

performance, continuous effectiveness, and safety. 

One of the best examples is Australia’s National 

Joint Replacement Registry,15 which has followup 

of more than 10 years.

A unique challenge for device registries is that 

once a device is implanted, a patient does not have 

to return to the doctor if he or she does not have 

any issues, in contrast to a therapeutic situation in 

which patients return for prescription refills. As a 

result, collecting followup data both directly from 

patients and through the health care provider is a 

useful tool for patient retention. Loss to followup 

differentially for patients who do not experience 

complications is a risk, and underscores the 

importance of achieving reasonably complete 

followup on all patients through delivery settings 

well designed for continuity of care.16 Long-term 

data on medical devices also can be obtained 

through linking registries with administrative 

billing data, electronic health records, or other 

clinically rich data sources, although data 
203

Chapter 23. Registries for Medical Devices

limitations such as lack of test results or reasons 

for procedures may remain. 

3.6 Provider Experience and Training

Provider experience and training can influence the 

selection of device, device performance, and 

patient outcomes, particularly for implantable 

devices. Surgeons generally have a preference in 

terms of selection of devices, and although 

sometimes this preference is based on clinical 

appropriateness, it also can be based on marketing 

or familiarity with a particular brand. Group 

purchasing organizations, cost, provider contracts, 

reimbursement, and other market forces may also 

influence selection and, in some of these cases, 

surgeons may not have a choice of device. 

Additionally, provider experience and surgical/

procedural skill can greatly influence the 

effectiveness and safety of devices.17

Device-specific training is an important element of 

a medical device registry that is not an issue in a 

drug registry. Device premarket studies are 

typically smaller than drug studies. Post-approval, 

regulatory agencies are concerned about training 

program quality. Often, an observational study is 

required to formally evaluate the appropriateness 

of physician training. Regardless of whether 

regulators have mandated this type of study, they 

recommend integrating training information into 

data collection. The importance of this training 

varies based on device type. For example, 

physicians with experience using balloonexpandable or self-expanding stents may not need 

additional device-specific training to use a new 

carotid stent. In contrast, prior experience may not 

be translatable to use of a new percutaneous valve; 

in this case, specific training on the delivery 

technique required for that specific valve is 

important. The amount of training required to 

ensure safe application of a technology is often 

unclear.9

Experience-related factors also should be 

considered in analyses and training evaluation: 

practitioner annual volume; practitioner lifetime 

volume; facility volume; and facility 

characteristics such as academic teaching status. It 

is important to distinguish between these factors 

because each has a threshold effect. Some factors, 

particularly lifetime volume, have not been well 

documented or analyzed. For others, such as 

hospital volume and academic teaching status, 

relationships with complications, revision surgery, 

length of hospital stay, and mortality are well 

documented.18-20 It is ideal to have training and 

volume information in the registry, but this may 

not always be realistic. If this is deemed critical, 

information needs to be collected on provider 

experience and training at registry initiation and 

supplemented if any training programs occur 

during the registry development. 

Registry design teams should consider how 

provider training and learning curves can be 

handled during analysis. Particularly for devices 

with few qualified surgeons, clustering may be an 

issue in analysis. Sample size may need to 

increase, and statistical methods that account for 

clustering, such as generalized estimating 

equations, should be used. Adjustment by surgical 

volume, either on the hospital or provider level, 

might also be appropriate.21 For example, some 

studies have shown that categories integrating both 

of these components as one adjustment variable 

(e.g., high-volume hospital and surgeon; highvolume hospital with low-volume surgeon; lowvolume hospital with high-volume surgeon; both 

low volume) are useful.22 

3.7 Summary of Design and Data 

Collection Considerations

Although device registries are similar to other 

registries operationally, the challenges outlined 

above are critical to consider during the registry 

design phase. Careful review of the unique 

features of medical device registries can result in 

high-quality, useful studies of device performance. 

Medical devices must be identified accurately and 

their attributes classified according to standard 

nomenclature. Minimum recommended identity 

variables include manufacturer, product name, lot 

number (where applicable), catalog number, serial 

number (where applicable and with appropriate 

consideration for protected health information), 

description of device, and device attributes. These 

implant characteristics can then be linked to 
204

Section V. Special Applications in Patient Registries

patient, surgeon, hospital, and procedural data 

along with outcomes of interest. For these reasons, 

diligence in information security is warranted 

when identity variables are being collected, and 

these data should not be included in limited data 

sets. (See Chapter 7 for a full discussion of the 

implications of collecting individually identifiable 

data within a patient registry.) Device 

identification and attribute classification require 

constant maintenance as new devices are 

introduced into the market. In the future, UDIs 

will significantly facilitate this process. A standard 

minimum data set targeted toward device registries 

would be useful to support strong registry designs 

and to facilitate linkages with other data sources. 

The definition and validity of device performance 

issues is another important element. In developing 

a device registry, the modes of failures and 

definitions must be clearly defined. Pragmatic 

systems for outcome verification need to be in 

place. It is also crucial to track all device 

components in case a single component fails. 

Collecting information to track drug/device 

interactions is also critical. Finally, sufficient 

followup of patients must be established in order 

for registries to provide longitudinal outcomes. 

4. Regulatory Uses and 

Considerations

FDA has for a long time been actively engaged in 

the use and development of registries for both 

pre- and postmarket assessments of device safety 

and effectiveness. For premarket considerations, 

device registries have: (1) provided data to support 

the development, design, and use of comparators 

(both concurrent and historical) in clinical trials 

(e.g., the National Heart, Lung, and Blood 

Institute’s (NHLBI) Interagency Registry for 

Mechanically Assisted Circulatory Support 

[INTERMACSTM ], heart valves); (2) provided 

access to products (outside of IDE [investigational 

device exemption] trials) (e.g., PFO [patent 

foramen ovale] occluders); (3) enhanced safety 

assessments via broader analysis of adverse events 

(e.g., adhesion barriers); and (4) expedited 

approval of device modifications or labeling (e.g., 

intraocular lenses). Postmarket applications of 

device registries have included frequent use in 

FDA-mandated studies (e.g., drug-eluting 

coronary stents), use in enhanced passive 

surveillance efforts (e.g., INTERMACS; the 

Department of Veterans Affairs’ Clinical 

Assessment, Reporting, and Tracking System for 

Cardiac Catherization [CART-CL]), use in 

exploratory efforts to expand FDA Sentinel 

capabilities, both in active surveillance and data 

source linkage (e.g., DELTA), and use in 

discretionary applied research as noted below. 

Over time, various stakeholder communities have 

increased their efforts to foster the development of 

clinical registries as a valuable postmarket tool for 

capturing utilization of devices, identifying early 

signals, and studying postmarket performance of 

medical technology. Some of these efforts 

transformed what were primarily procedure 

registries into procedure/device registries for 

certain targeted devices of interest. Examples of 

longstanding collaborative efforts include those 

between FDA and professional society databases 

such as the American College of Cardiology 

NCDR.23 This collaboration resulted in one of the 

largest observational studies on hemostasis devices 

using NCDR registry data.24 In addition, FDA 

collaborated with Duke University and the Society 

of Thoracic Surgeons to study the outcomes of 

transmyocardial revascularization procedures using 

the Adult Cardiac Surgery Database.25 More 

recently, the national multistakeholder community, 

including professional societies, FDA, and the 

Centers for Medicare and Medicaid Services, 

worked together to establish the first national 

transcatheter valve therapy (TVT) registry to 

capture transcathether aortic valve replacement 

therapies.26 This effort aims to foster both pre- and 

postmarket uses for future TVT devices and 

indications. In the orthopedic arena, FDA led the 

development of the International Consortium of 

Orthopedic Registries to advance the 

methodological infrastructure for studying 

performance and clinical outcomes of orthopedic 

implants.27

FDA continues to foster the development of 

registries in key product areas. It also recognizes 

the need for a national perspective on device 

registry development that considers how best to: 
205

Chapter 23. Registries for Medical Devices

(1) leverage existing experience and expertise; 

(2) establish common data elements across 

registries; (3) share evolving methodological tools; 

(4) enhance interoperability between standard 

electronic health records and registries; (5) create 

sustainable business models; and (6) adopt robust 

and transparent governance practices.

5. Potential Uses of Emerging 

Technology

Registries may soon be able to take advantage of 

emerging technology for data transmission. New 

technologies can enable medical devices to 

transmit data directly to electronic medical records 

and other patient management systems. Ultimately, 

this type of data may be sent directly to a patient 

registry, reducing the burden of data entry and 

increasing the timeliness of registry data.

These new technologies are currently at various 

stages of development. Automatic measurement 

and adjustment of programming to provide optimal 

settings is a potential area of innovation that would 

provide efficiencies in the use of pacemakers and 

other implantable devices. Feasibility for this has 

been demonstrated by the AUTOMATICITY 

registry, which aims to evaluate physicians’ 

acceptance of automatic algorithms for ventricular 

capture, automatic sensing, and automatic 

optimization of sensor settings.28, 29 The 

AUTOMATICITY team concluded that project 

team followup and avoidance of reprogramming 

due to the automated programming can increase 

effective use of hospital time and resources. This 

would be a useful technology for registries because 

all automated changes can be collected at one 

followup time point, rather than collecting each 

change ad hoc as it occurs.

Diagnostics for implantable devices are another 

area of technical improvement. Implantable 

devices, such as implantable cardioverter 

defibrillators, pacemakers, and cardiac 

resynchronization therapy devices, can track heart 

rate, heart rate variability, respiration rate, atrial 

tachyarrhythmia and ventricular tachyarrhythmia 

recurrence and duration, intrathoracic impedance, 

symptom markers, and patient activity.30 This 

diagnostic information can be provided directly to 

EMRs and fed into registries, in many cases 

continuously. Although the clinical application of 

this capability is still being examined, the benefits 

in efficient, timely data capture are clear. 

The fascinating pace of emerging medical 

technologies and information science applications 

are expected to further shape health care research. 

Further development and integration of devicebased registries into the national postmarket 

infrastructure creates opportunities for novel 

methodology developments, harmonization, 

sharing, and combining of data. 

6. Summary

Medical device registries can be designed for a 

variety of purposes. They can provide useful 

information on long-term effectiveness and safety 

of devices, as well as the impact of factors such as 

surgical technique, surgeon, hospital, and patient 

characteristics. Like all observational studies, 

medical device registries have some limitations. 

Failure to control for often complex confounding 

variables and the inability to take into account 

device version changes, surgical technique, and 

other unique factors can lead to erroneous 

conclusions. However, design and analysis of 

medical device registries can often provide critical 

information for decisionmaking by regulators, 

clinicians, patients, and policymakers.
206

Section V. Special Applications in Patient Registries

Case Examples for Chapter 23

Case Example 58. Designing a registry to 

study the effectiveness of a device training 

program for providers

Description The Carotid Artery Stenting 

with Emboli Protection 

Surveillance Post-Marketing 

Study (CASES-PMS) was 

designed to assess the outcomes 

of carotid artery stent procedures 

for the treatment of obstructive 

artery disease during real-world 

use. The primary purpose of the 

registry was to evaluate 

outcomes in the periapproval 

setting, including the use of a 

detailed training program for 

physicians not experienced in 

carotid artery stenting.

Sponsor Cordis Corporation

Year Started 2004

Year Ended 2006

No. of Sites 74

No. of Patients 1,493

Challenge

In 2004, the sponsor received approval for a 

carotid stent procedure from the U.S. Food and 

Drug Administration (FDA), largely because of 

the results of the Stenting and Angioplasty With 

Protection in Patients at HIgh Risk for 

Endarterectomy (SAPPHIRE) clinical trial. The 

SAPPHIRE trial studied the results of stent 

procedures performed by experts in the field. 

While the trial provided strong data to support the 

approval of the carotid stent, FDA and the 

Centers for Medicare & Medicaid Services 

(CMS) both questioned whether the outcomes of 

the trial were generalizable to procedures 

performed by physicians without prior experience 

in carotid artery stenting.

To respond to the FDA and CMS requests, the 

sponsor needed to design a study to confirm the 

safety and effectiveness of carotid artery stenting 

in a variety of settings. The study needed to 

gather data from academic and nonacademic 

settings, from physicians with various levels of 

carotid stenting experience, from settings with 

varying levels of carotid stenting volume, and 

from a geographically diverse mix of sites. The 

study would also need to examine the 

effectiveness of a training program that the 

sponsor had designed to teach physicians about 

the stenting procedure.

Proposed Solution

The sponsor designed a comprehensive training 

program for physicians and other health care 

professionals. The training program, which began 

in 2004, included didactic review, case 

observations and simulation training, and handson experience. Depending on their prior 

experience with carotid artery stenting, 

physicians received a 2-day in-person training 

plus online training; online training only; or no 

additional training. To study the effectiveness of 

the training program and to provide data on the 

clinical safety and effectiveness of carotid 

stenting in a variety of settings, the sponsor 

designed and launched the registry in 2004.

The registry was a multicenter, prospective, 

observational study designed to assess stenting 

outcomes in relation to the outcomes of the 

SAPPHIRE trial (the historic comparison group). 

The study enrolled 1,493 patients from 74 sites, 

using inclusion and exclusion criteria that 

matched those of the SAPPHIRE trial. The 

patients in the study were high-surgical-risk 

patients with de novo atherosclerotic or 

postendarterectomy restenotic obstructive lesions 

in native carotid arteries. Study participants 

completed clinical followups at 30 days and again 

at 1 year after the procedure. The 30-day 

assessments included a neurological examination 

by an independent neurologist and an evaluation 

of adverse events. The study defined the 30-day 

major adverse event rate as the 30-day composite 

of all deaths, myocardial infarctions, and strokes.
207

Chapter 23. Registries for Medical Devices

Case Example 58. Designing a registry to 

study the effectiveness of a device training 

program for providers (continued)

Results

The 30-day major adverse event rate of 5.0 

percent met the criteria for noninferiority to the 

outcomes of stented patients from the pivotal 

SAPPHIRE trial. Outcomes were similar across 

levels of physician experience, carotid stent 

volume, geographic location, and presence/

absence of the training program. The initial 

findings show that a comprehensive, formal 

training program in carotid stenting enables 

physicians from multiple specialties with varying 

levels of experience in carotid stenting to achieve 

outcomes similar to those achieved by the experts 

in the clinical trial.

Key Point

An observational registry can provide the 

necessary data for a postmarket evaluation of 

devices that are dependent on newly acquired 

skills. The registry can provide data to assess both 

the clinical safety of the device and the 

effectiveness and success of a training program.

For More Information

Katzen B, Criado F, Ramee S, et al. on behalf of 

the CASES-PMS Investigators. Carotid artery 

stenting with emboli protection surveillance 

study: 30-day results of the CASES-PMS study. 

Catheter Cardiovasc Interv. 2007;70:316–23.

Yadav JS, Wholey MH, Kuntz RE, et al. Protected 

carotid-artery stenting versus endarterectomy in 

high-risk patients. N Engl J Med. 2004;351:

1493–501.
208

Section V. Special Applications in Patient Registries

Case Example 59. Identifying and responding 

to adverse events found in a registry database

Description The Kaiser Permanente National 

Total Joint Replacement 

Registry (TJRR) was developed 

by orthopedic surgeons to 

improve patient safety and 

quality and to support research 

activities. The TJRR tracks all 

Kaiser Foundation Health Plan 

members undergoing elective 

primary and revision total knee 

and hip replacement. The 

purposes of the registry are to 

(1) monitor revision, failure, 

and rates of key complications; 

(2) identify patients at risk for 

complications and failures; 

(3) identify the most effective 

techniques and implant devices; 

(4) track implant usage; and 

(5) monitor and support implant 

recalls and advisories in 

cooperation with the U.S. Food 

and Drug Administration. The 

TJRR uses an electronic medical 

record (EMR) system to collect 

uniform data at the point of care. 

Data are abstracted from the 

EMR to the registry, and 

followup data are collected 

through several methods.

Sponsor Kaiser Foundation Health Plan

Year Started 2001

Year Ended Ongoing

No. of Sites 350 surgeons at 50 medical 

centers

No. of Patients 140,000 total joint replacements

Challenge

The registry collects standardized total joint 

preoperative, operative, and postoperative data to 

supplement administrative data collected through 

the EMR. The registry database includes 

information on patient demographics, implant 

characteristics, surgical techniques, and 

outcomes. As a result, the registry provides 

opportunities for total joint replacement 

surveillance and monitoring, but the depth and 

breadth of the data make manual data reviews for 

adverse events (AEs) too resource intensive and 

time consuming.

Proposed Solution

Electronic screening algorithms were developed 

to detect AEs in the registry database in a timely, 

efficient manner. The algorithms use ICD-9 codes 

and CPT codes to identify complications of joint 

replacement surgery, such as revisions, 

reoperations, infection, and pulmonary embolism. 

All complications picked up by the screening 

algorithms are validated with a chart review. The 

screening algorithms are run and the results 

monitored on a regular basis to identify trends.

The registry can also run specific queries to 

respond to physician concerns. For example, if 

physicians at participating medical centers notice 

a problem with an implant or hear about a 

problem from colleagues, they can request an ad 

hoc query of the registry database. The query can 

identify all patients receiving a particular implant 

and assess outcomes. When the outcome of 

interest is not part of the registry database, the 

registry staff may perform additional followup 

through chart review. The staff may also check 

the U.S. Food and Drug Administration’s Medical 

Product Surveillance Network (MedSun) to 

validate their findings against other data sources.
209

Chapter 23. Registries for Medical Devices

Case Example 59. Identifying and responding 

to adverse events found in a registry database 

(continued)

Proposed Solution (continued)

Once an implant has been recalled or when there 

is an advisory or concern, the registry can 

immediately generate a list of all patients who 

received that implant and notify their physicians. 

The registry can also identify complications and 

assess revision rates among its patients who 

received that implant. Registry staff continue to 

monitor outcomes of patients who undergo 

revision surgery, until the patient dies or is lost to 

followup. 

Results

Since its launch in 2001, the registry has assisted 

participating physicians with their responses to 

several implant recalls and advisories. Data from 

the registry were used to identify surgical 

techniques that resulted in higher revision rates. 

The registry staff shared this information with 

physicians, resulting in reduced use of these 

techniques.

Key Point

Electronic screening algorithms offer an efficient 

method of identifying potential AEs in large data 

sets in a timely manner. For such algorithms to be 

effective, the registry database must collect 

detailed information on the implants’ lots and 

catalog numbers, and must be updated frequently 

as new and modified products become available. 

In addition, when using medical codes, it is 

important to validate the results of the screening 

algorithm to ensure that coding errors have not 

affected the findings.
210

Section V. Special Applications in Patient Registries

Case Example 60. Receiving data from 

medical imaging devices

Description The Dose Index Registry 

collects data on radiation doses 

administered during computed 

tomography (CT) examinations. 

Facilities can compare their 

average radiation dose for a 

particular examination (e.g., CT 

head examination) to that of 

similar facilities and to the 

national average. Such 

comparisons help facilities 

identify examinations for which 

their dose indices may be higher 

than others and adjust their 

protocols accordingly. Because 

it does not collect patient 

outcomes, the registry is 

considered a registration registry 

rather than a patient registry. 

The example is presented to 

illustrate the ability to use 

medical devices to report 

information directly to registries 

or other databases.

Sponsor American College of Radiology

Year Started 2011

Year Ended Ongoing

No. of Sites Over 200

No. of Patients Over 1,000,000 CT 

examinations

Challenge

Safety concerns over the effects of ionizing 

radiation exposure from diagnostic imaging have 

been described in numerous publications and 

summarized in a Joint Commission Sentinel 

Event Alert posted August 24, 2011 (Issue 47). To 

address some of these concerns, the American 

College of Cardiology established the Dose Index 

Registry to collect and compare data on radiation 

doses administered during CT examinations. 

For the data collection system to be a viable tool 

for participating facilities, it was imperative that 

the data be collected without interrupting the 

workflow at the facility (i.e., without adding 

additional workload for the CT technologist). The 

biggest challenge facing the registry was to 

automate and standardize the collection of dose 

information provided by several CT 

manufacturers in a variety of formats. 

Proposed Solution

The sponsor worked with the organization 

Integrating the Healthcare Enterprise to develop 

the Radiation Exposure Monitoring (REM) 

profile. This profile describes the way dose 

information should be transmitted across different 

health care settings and specifies that information 

should be transmitted to a registry in the form of 

a Radiation Dose Structured Report (RDSR). 

However, only the most recent versions of CT 

scanner models and software support RDSR. In 

order to accept dose information provided by 

older scanners, the registry developed software 

that could convert the data into the RDSR format. 

The software that collects dose information also 

removes patient identifiers before sending data to 

the registry. 

The second hurdle in standardization was the 

development of a common nomenclature for CT 

examinations. Different names are used for the 

same examination both within and among 

imaging facilities, and the registry needed a 

standard terminology for meaningful reporting. 

While it would have been possible to develop a 

new standard, the registry was aware of other 

lexicons under development, such as the Radlex 

Playbook. Conversations between the registry and 

the Radlex Playbook developers allowed the two 

groups to understand and meet each other’s 

needs, and the registry was able to adopt the 

Radlex Playbook as the standard terminology for 

exam names. For facilities that submit data using 

non-standard terminology, these terms are 

mapped to Radlex Playbook terminology using a 

mapping tool developed by the registry.
211

Chapter 23. Registries for Medical Devices

Case Example 60. Receiving data from 

medical imaging devices (continued)

Results

To date, the data collection system has collected 

information related to CT radiation dose from 

over 200 facilities nationwide and has collected 

dose information from over 1,000,000 CT 

examinations. In addition to allowing comparison 

of dose indices between facilities, data collected 

from the registry will also be used to establish 

national benchmarks for CT dose indices.

Key Point

While not a patient registry, this example 

demonstrates that registries that collect 

information from medical devices may be able to 

reduce data entry burden by incorporating data 

transmitted directly from the device. When 

considering this option, registries may benefit 

from communicating with industry to find 

solutions that are not manufacturer specific and 

that can be implemented within a reasonable 

timeframe. Registries may also benefit from 

working with existing standards to determine if 

they can be modified to fit the registry’s use case.

For More Information

http://nrdr.acr.org

Case Example 61. Combining registry data 

with EHR data to measure real-world 

outcomes of implantable devices

Description Members of the Cardiovascular 

Research Network (CVRN) are 

conducting a longitudinal study 

of the characteristics, clinical 

outcomes, resource utilization, 

and costs among “real-world” 

patients receiving implantable 

cardioverter defibrillators(ICDs) 

for primary prevention of 

sudden cardiac death.

Sponsors National Heart, Lung, and Blood 

Institute (NHLBI), Agency for 

Healthcare Research and Quality 

(AHRQ), American College of 

Cardiology Foundation, Heart 

Rhythm Society

Year Started 2009

Year Ended Ongoing (planned completion of 

outcome ascertainment in 2013)

No. of Sites Seven health care systems, with 

15 participating hospital partners

No. of Patients 3,600

Challenge

ICDs have revolutionized the approach to 

treatment for hundreds of thousands of patients in 

the United States with left ventricular systolic 

dysfunction who are at risk for sudden cardiac 

death. Despite broadened indications for ICD 

therapy, use varies across patient subgroups. Most 

existing data on complication rates, mortality, 

morbidity, and cost of primary prevention ICD 

therapy come from clinical trial samples, which 

enroll subjects who may not be representative of 

patients cared for in routine practice. Baseline 

data on patients and devices are available in the 

National Cardiovascular Data Registry’s (NCDR) 

ICD Registry, but longitudinal outcomes are not 

routinely included in the registry system. 

Evaluation of longitudinal, real-world data is 

needed, but following large groups of patients in 

community care settings can present logistical 

challenges. 

Proposed Solution

CVRN is a national research collaborative funded 

by NHLBI that leverages expertise, populations, 

and data sources from a consortium of 14 health 

plans in the United States. Seven sites in the 

CVRN are sponsored by NHLBI, AHRQ, the 

American College of Cardiology Foundation, and 

the Heart Rhythm Society to conduct a 

longitudinal study of patients receiving ICDs for 

primary prevention of sudden cardiac death. The 

aims of the study are to (1) evaluate the extent to 

which patients receiving ICDs for primary 

prevention meet guideline-based eligibility 
212

Section V. Special Applications in Patient Registries

Case Example 61. Combining registry data 

with EHR data to measure real-world 

outcomes of implantable devices (continued)

Proposed Solution (continued)

case criteria; (2) assess longitudinal outcomes, 

including complications, hospitalization, 

mortality, and delivery of device therapies 

among primary prevention ICD patients; and 

(3) identify the characteristics associated with 

these outcomes in real-world community 

practice. To achieve these goals, the study 

developed a new database, which links a 

national device registry with information from 

health system medical records archived in 

electronic and other forms.

Baseline information is obtained from the 

NCDR ICD Registry, which captures national 

data on primary-prevention ICD implants for 

Medicare beneficiaries, although most 

participating hospitals, including the study 

facilities, submit data to the registry for all ICD 

recipients regardless of insurance status. Clinical 

and administrative followup data for three years 

post-implant are collected through the electronic 

health record (EHR) systems of the health plans 

participating in the CVRN. Finally, a new 

repository of arrhythmic episodes treated by 

ICD is being generated through review and 

abstraction of archived device followup records 

at the study sites. 

ICD patients have periodic followup visits which 

include downloading data from their devices; 

this data can include the number and type of 

arrhythmic episodes detected by the device since 

the last check, what therapy (e.g., shock) was 

administered by the device, and the outcome of 

that therapy. The health systems download the 

device data from patients during an office visit 

or via remote transmission over the telephone. 

The device followup data are then incorporated 

into the health system medical record archives. 

Since the study includes devices from a variety 

of manufacturers and includes device followup 

records in a variety of formats, study staff have 

created a standardized format for collecting 

therapy data so they can be adjudicated and 

analyzed. All three data sources are then 

combined into a single analytic data set for 

addressing the study’s specific aims, using a 

unique subject identifier to link all data elements 

together for the same subject. 

Results

The study includes 3,600 subjects with primaryprevention ICDs implanted from 2006 through 

2010. Data collection began in 2009. A report on 

the study methods and the baseline 

characteristics of the study population has been 

published, demonstrating that important 

demographic and clinical characteristics of 

patients receiving ICDs in real-world clinical 

practice are significantly different from the 

population enrolled in the landmark clinical 

trials conducted on ICDs in the early 2000s. 

Key Point

Existing registries and EHR data can be valuable 

data sources for measuring the long-term 

outcomes of devices in real-world settings. This 

is especially true for implantable devices, where 

data may be downloaded automatically from the 

device to an EHR system.

For More Information

Go AS, Magid DJ, Wells B, et al. The 

Cardiovascular Research Network: a new 

paradigm for cardiovascular quality and 

outcomes research. Circ Cardiovasc Qual 

Outcomes. 2008 Nov;1(2):138-47.

Masoudi FA, Go AS, Magid DJ, et al. The 

Longitudinal Study of Implantable Cardioverter 

Defibrillators: methods and clinical 

characteristics of patients receiving implantable 

cardioverter defibrillators for primary prevention 

in contemporary practice. Circ Cardiovasc Qual 

Outcomes. 2012 Nov;5(6):e78-85.
213

Chapter 23. Registries for Medical Devices

References for Chapter 23

1. U.S. Food and Drug Administration. Total Product 

Life Cycle. http://www.fda.gov/AboutFDA/

CentersOffices/

OfficeofMedicalProductsandTobacco/CDRH/

CDRHTransparency/ucm199906.htm. Accessed 

August 6, 2012.

2. Boam AB. Innovative Systems for Delivery of 

Drugs and Biologics. PowerPoint Presentation. 

http://www.fda.gov/ohrms/dockets/

dockets/03n0203/03n-0203-ts00010-Boam.ppt. 

Accessed August 6, 2012.

3. Institute of Medicine. New Medical Devices: 

Invention, Development, and Use. Washington, 

DC: National Academies Press; 1988.

4. Sedrakyan A, Marinac-Dabic D, Normand SL, et 

al. A framework for evidence evaluation and 

methodological issues in implantable device 

studies. Med Care. 2010 Jun;48(6 Suppl):S121-8. 

PMID: 20421824.

5. Paxton E. Kaiser Permanente Implant Registries. 

PowerPoint Presentation. http://www.fda.gov/

downloads/MedicalDevices/NewsEvents/

WorkshopsConferences/UCM272080.pdf. 

Accessed August 6, 2012.

6. U.S. Food and Drug Administration. Medical 

Device Report (MDR) Forms and Instructions. 

http://www.fda.gov/MedicalDevices/Safety/

ReportaProblem/FormsandInstructions/default.

htm. Accessed February 20, 2013.

7. Vidi VD, Matheny ME, Donnelly S, et al. An 

evaluation of a distributed medical device safety 

surveillance system: the DELTA network study. 

Contemp Clin Trials. 2011 May;32(3):309-17. 

PMID: 21356331. PMCID: 3070041.

8. Daemen J, Wenaweser P, Tsuchida K, et al. Early 

and late coronary stent thrombosis of sirolimuseluting and paclitaxel-eluting stents in routine 

clinical practice: data from a large twoinstitutional cohort study. Lancet. 2007 Feb 

24;369(9562):667-78. PMID: 17321312.

9. Serruys PW, Kutryk MJ, Ong AT. Coronary-artery 

stents. N Engl J Med. 2006 Feb 2;354(5):483-95. 

PMID: 16452560.

10. Camenzind E, Steg PG, Wijns W. Stent thrombosis 

late after implantation of first-generation drugeluting stents: a cause for concern. Circulation. 

2007 Mar 20;115(11):1440-55; discussion 55. 

PMID: 17344324.

11. Bavry AA, Kumbhani DJ, Helton TJ, et al. What is 

the risk of stent thrombosis associated with the 

use of paclitaxel-eluting stents for percutaneous 

coronary intervention?: a meta-analysis. J Am 

Coll Cardiol. 2005 Mar 15;45(6):941-6. 

PMID: 15766833.

12. Babapulle MN, Joseph L, Belisle P, et al. 

A hierarchical Bayesian meta-analysis of 

randomised clinical trials of drug-eluting stents. 

Lancet. 2004 Aug 14-20;364(9434):583-91. 

PMID: 15313358.

13. Urban P, Gershlick AH, Guagliumi G, et al. Safety 

of Coronary Sirolimus-Eluting Stents in Daily 

Clinical Practice: One-Year Follow-Up of the 

e-Cypher Registry. Circulation. 2006;113:

1434-41.

14. O’Malley AJ, Normand SL, Kuntz RE. 

Application of models for multivariate mixed 

outcomes to medical device trials: coronary artery 

stenting. Stat Med. 2003 Jan 30;22(2):313-36. 

PMID: 12520564.

15. National Joint Replacement Registry. 

https://aoanjrr.dmac.adelaide.edu.au/en. 

Accessed December 20, 2013.

16. Kuntz RE, Keaney KM, Senerchia C, et al. 

A predictive method for estimating the late 

angiographic results of coronary intervention 

despite incomplete ascertainment. Circulation. 

1993 Mar;87(3):815-30. PMID: 8443902.

17. Barker FG, 2nd, Amin-Hanjani S, Butler WE, et 

al. In-hospital mortality and morbidity after 

surgical treatment of unruptured intracranial 

aneurysms in the United States, 1996-2000: the 

effect of hospital and surgeon volume. 

Neurosurgery. 2003 May;52(5):995-1007; 

discussion 1007-9. PMID: 12699540.

18. Hannan EL, Racz M, Ryan TJ, et al. Coronary 

angioplasty volume-outcome relationships for 

hospitals and cardiologists. JAMA. 1997 Mar 

19;277(11):892-8. PMID: 9062327.
214

Section V. Special Applications in Patient Registries

19. Birkmeyer JD, Stukel TA, Siewers AE, et al. 

Surgeon volume and operative mortality in the 

United States. N Engl J Med. 2003 Nov 

27;349(22):2117-27. PMID: 14645640.

20. Halm EA, Lee C, Chassin MR. Is volume related 

to outcome in health care? A systematic review 

and methodologic critique of the literature. Ann 

Intern Med. 2002 Sep 17;137(6):511-20. 

PMID: 12230353.

21. Kuntz RE, Normand SL. Measuring percutaneous 

coronary intervention quality by simple case 

volume. Circulation. 2005 Aug 23;112(8):

1088-91. PMID: 16116067.

22. Birkmeyer JD, Siewers AE, Finlayson EV, et al. 

Hospital volume and surgical mortality in the 

United States. N Engl J Med. 2002 Apr 

11;346(15):1128-37. PMID: 11948273.

23. National Cardiovascular Data Registry. 

http://www.ncdr.com. Accessed August 20, 2012.

24. Tavris DR, Dey S, Albrecht-Gallauresi B, et al. 

Risk of local adverse events following cardiac 

catheterization by hemostasis device use - phase 

II. J Invasive Cardiol. 2005 Dec;17(12):644-50. 

PMID: 16327045.

25. Tavris DR, Brennan JM, Sedrakyan A, et al. 

Long-term outcomes after transmyocardial 

revascularization. Ann Thorac Surg. 2012 

Nov;94(5):1500-8. PMID: 22835557.

26. STS/ACC TVT Registry. https://www.ncdr.com/

TVT/Home/Default.aspx. Accessed August 20, 

2012.

27. Sedrakyan A, Paxton EW, Phillips C, et al. The 

International Consortium of Orthopaedic 

Registries: overview and summary. J Bone Joint 

Surg Am. 2011 Dec 21;93 Suppl 3:1-12. 

PMID: 22262417.

28. Alings M, Vorstenbosch JM, Reeve H. 

Automaticity: design of a registry to assess 

long-term acceptance and clinical impact of 

Automatic Algorithms in Insignia pacemakers. 

Europace. 2009 Mar;11(3):370-3. 

PMID: 19240111.

29. Alings M, Vireca E, Bastian D, et al. Clinical use 

of automatic pacemaker algorithms: results of the 

AUTOMATICITY registry. Europace. 2011 

Jul;13(7):976-83. PMID: 21422022.

30. Andriulli J. Device monitoring of intrathoracic 

impedance: clinical observations from a patient 

registry. Am J Cardiol. 2007 May 

21;99(10A):23G-8G. PMID: 17512419.
215

Chapter 24. Public-Private Partnerships

1. Introduction 

As both government and private groups have 

shown increased interest in patient registries, 

public-private partnerships (PPPs) have become 

more common as a means to develop and support 

patient registries and data linkage projects. These 

types of partnerships may become more common, 

as recent legislative actions have suggested PPPs 

as a potential approach to registry development.1

More information is needed on what types of 

public-private partnerships are possible, what 

issues should be considered when using such a 

partnership to develop or support a registry, and 

what characteristics and practices are likely to 

enhance the success of such efforts. This chapter 

defines PPPs in the context of patient registries, 

provides examples of existing PPPs, discusses 

considerations for setting up and operating PPPs, 

and reviews key factors for successful 

partnerships. While the discussion in this chapter 

focuses primarily on PPPs within the United 

States, some considerations for international 

partnerships are also reviewed. Case Examples 62, 

63, and 64 offer descriptions of PPPs for registries.

2. Definition of a PublicPrivate Partnership

“Public-private partnership” is a broad term that 

refers to any partnership in which at least one 

entity is a public agency (e.g., a government 

entity) and at least one other entity is a private 

organization. The scope can range from 

partnerships at the local level, including local and 

regional health agencies, to national and 

international health agencies and other private 

institutions or organizations (e.g., professional 

associations, patient advocacy groups). In a 

research context, a partnership implies some joint 

collaboration to achieve a common scientific goal. 

Partners may contribute intellectual capital, 

funding, data, or other services.

3. PPP Models

PPPs may take many forms. Some possible models 

include partnerships among Federal agencies to 

examine safety and effectiveness (e.g., 

INTERMACSTM); partnerships among health 

agencies from several countries on an international 

level to describe the clinical course of a disease 

and understand whether there are any effective 

treatments (e.g., Avian Flu Registry); partnerships 

with State agencies for quality improvement (e.g., 

Get With The Guidelines®); and partnerships for 

evidence development for coverage decisions (e.g., 

Centers for Medicare and Medicaid Services). 

These models are described below, as case studies.

3.1 INTERMACS

The Interagency Registry for Mechanically 

Assisted Circulatory Support (INTERMACS) is 

the U.S. national registry for patients who have 

received durable, U.S. Food and Drug 

Administration (FDA)-approved mechanical 

circulatory support device (MCSD) therapy to 

treat advanced heart failure. This registry was 

devised as a joint effort of the National Heart, 

Lung, and Blood Institute (NHLBI), the Centers 

for Medicare and Medicaid Services (CMS), the 

FDA, clinicians, scientists, and industry 

representatives. The goals of the registry are to—

• Facilitate the refinement of patient selection to 

maximize outcomes with current and new 

device options; 

• Identify predictors of good outcomes as well as 

risk factors for adverse events after device 

implantation; 

• Develop consensus “best practice” guidelines 

to improve clinical management by reducing 

short and long term complications of MCSD 

therapy; and 

• Use registry information to guide clinical 

application and evolution of next generation 

devices.
216

Section V. Special Applications in Patient Registries

A major challenge to INTERMACS was to create 

a registry with sufficient data quality, regulatory 

rigor, and sophistication to be able to achieve these 

goals. INTERMACS used the quality of a highlevel clinical trial as its standard, realizing that it 

could never totally meet these standards but could 

emulate them as closely as possible in a structured, 

protocol-driven manner. See Table 24–1 for a 

listing of the regulatory, data quality, and scientific 

components of a clinical trial, and an indication of 

which of these components are contained in 

INTERMACS. 

Table 24–1. Regulatory, data quality, and scientific components of a typical FDA clinical trial and 

INTERMACSa

Typical FDA Clinical Trial INTERMACS

DSMB 3 3

Informed consent 3 3

IRB approval 3 3

Data use agreement 3 3

Human subjects training 3 3

Information security 3 3

Active Web site 3 3

Protocol 3 3

CLIA certification 3 3

Adjudication 3

Local principal investigator 

certification

3 3

Data freezes 3 3

Audits 3 3

Complete enrollment 3 3

Complete data 3 3

AE definitions 3 3

Inclusion/exclusion 3 3

Nurse monitors 3 3

Site training 3 3

Site reports 3

Standardized data sets ? 3

Medical device reports to FDA 3 3

Mandatory data entry 3 3

Planned analyses 3 3

DAAP: research requests ? 3

Annual meetings 3 3

Committees 3 3

AE = adverse events; CLIA = Clinical Laboratory Improvement Amendments; DAAP =Data Access. Analysis, and 

Publications Committee; DSMB = Data Safety Monitoring Board; FDA = U.S. Food and Drug Administration; IRB 

= Institutional Review Board.

aCopyright 2014 INTERMACS. Table reprinted with permission.
217

Chapter 24. Public-Private Partnerships

Another major challenge to INTERMACS is to 

maintain focus on its mission while many 

tangential efforts and registry “byproducts” have 

appeared. For example, INTERMACS has offered 

a new regulatory pathway for industry, as FDA 

approval is sought for new devices. It also has 

provided the control arm for one FDA pre–marketapproval trial, and is in the process of providing 

control data for several ongoing and new trials. 

While these efforts were not part of the initial 

goals and contract deliverables of INTERMACS, 

they do, in general, fit its mission of moving the 

field forward.

In 2005, the original contract between NHLBI and 

the Division of Cardiothoracic Surgery at the 

University of Alabama at Birmingham specified a 

target enrollment of 40 to 60 hospitals. As of July 

2011, 120 hospitals have enrolled and have entered 

data on more than 5,000 patients.

The complexity of managing a patient with a 

mechanical circulatory assist device requires a 

similarly complex registry. Implantation of a left 

ventricular device, a right ventricular device, and/

or a total heart replacement device must be 

captured along with subsequent device explants, 

multiple adverse events, functional capacity, and 

quality of life. The INTERMACS clinical research 

forms are numerous and detailed, with more than 

1,500 data elements.

A unique feature of INTERMACS is that it is 

assessing a rapidly changing clinical and 

technological field. INTERMACS must be poised to 

quickly assess newly approved devices and to 

quantify the evolution in patient selection. Figure 

24–1 shows survival based on two types of devices. 

These devices correspond to eras, with the 

intracorporeal continuous flow pump being the most 

recently approved MCSD. The improvement in 

survival is dramatic, and INTERMACS has been 

the best way to quantify this improvement. 
218

Section V. Special Applications in Patient Registries

Figure 24–1. Overall survival of adult INTERMACS subjects receiving primary left ventricular 

assist devices, by pump typeb

Initially, INTERMACS was the result of an 

NHLBI initiative in collaboration with FDA and 

CMS. Other stakeholders quickly joined in the 

planning stage, and they have continued to be 

INTERMACS partners. These multiple partners 

each have their own agenda and their own reasons 

for participating in INTERMACS. While their 

goals do not always align, there is considerable 

overlap, and INTERMACS has been able to fulfill 

most needs for each partner. At the intersection of 

these agendas are the common goals of assessing 

current devices and contributing to the 

development of new devices by analyzing registry 

data. The ultimate goal for all of the partners is to 

improve patient outcomes.

Figure 24–2 is a schematic representative of the 

partners involved in INTERMACS. The 

relationships are necessarily complex and must be 

managed by clear expectations, deliverables, 

standard operating procedures, and lines of 

authority.

CFP = continuous flow pump; LVAD = left ventricular assist device; PFP = pulsatile flow pump

bReprinted from the Journal of Heart and Lung Transplantation. Vol. 30, Kirklin J, Naftel D, Kormos R, et al. Annual 

Report: The evolution of destination therapy in the United States, Pages 115-123, Copyright 2011, with permission 

from Elsevier.
219

Chapter 24. Public-Private Partnerships

Figure 24–2. Structure of INTERMACS partnershipsc

CCD = Continuity of Care Document; CMS = Centers for Medicare and Medicaid Services; 

Co-PIs = Co-Principal Investigators; DCC = data coordinating center; EHR = Electronic Health Record; FDA = U.S. 

Food and Drug Administration; NHLBI = National Heart, Lung, and Blood Institute; NIH = National Institutes of 

Health; UAB = University of Alabama at Birmingham; UNOS = United Network for Organ Sharing.

cCopyright 2014 INTERMACS. Figure reprinted with permission.

3.1.1 Stakeholders

NHLBI. As the sponsoring agency, NHLBI is both 

the primary partner and primary regulator of the 

registry. In addition to its oversight role, NHLBI 

has been involved in many of the day-to-day 

activities of INTERMACS, including the 

important role of ensuring scientific and regulatory 

integrity and patient protection.

FDA. Through their regulatory role in approving 

and monitoring new devices, the FDA functions as 

the “gatekeeper” for devices. INTERMACS 

benefited from early interactions with FDA in 

developing the specifications of data elements and 

definitions of adverse events. As INTERMACS 

evolved, it worked with two separate components 

of FDA. The premarket personnel at the Center for 

Device and Radiological Health (CDRH) helped 

create a registry that would build on the previous 

premarket approval studies of MCSD. 

INTERMACS also worked with the postmarket 

approval personnel of CDRH to explore ways to 

facilitate the analyses of approved devices. The 

partnership with FDA has evolved as 
220

Section V. Special Applications in Patient Registries

INTERMACS has become a major postmarket 

study vehicle for approved MCSDs, as evidenced 

by the collaboration with Thoratec and FDA to 

perform the post-market studies for HeartMate II, 

the first FDA-approved adult non-pulsatile pump.

CMS. When INTERMACS began, CMS was 

reimbursing hospitals for FDA-approved MCSDs 

that were implanted as destination therapy (DT) at 

approved centers. One of the requirements of the 

reimbursement was that data on implanted patients 

be entered into a national database. By the third 

year of INTERMACS, CMS changed the 

requirement to explicitly specify INTERMACS as 

the data repository and stated that a certified DT 

center must be in good standing with 

INTERMACS. This partnership with CMS has 

been critical to the development of a 

comprehensive database that captures the vast 

majority of approved durable devices implanted as 

DT or as bridge-to-transplant therapy. 

Joint Commission. The Joint Commission is 

responsible for certifying hospitals as DT centers. 

INTERMACS collaborates with CMS, The Joint 

Commission, and hospitals to assist in the 

quantitative summaries necessary for certification.

Industry. Essentially every company that 

manufactures approved MCSDs or is in the 

process of gaining approval for an MCSD has been 

involved with INTERMACS. Industry was “at the 

table” during the meetings to develop 

INTERMACS. Many companies saw great 

potential for using INTERMACS in both 

premarket clinical trials and postmarket studies. 

The FDA has encouraged companies to work with 

INTERMACS. Some of these activities fall outside 

of the strict deliverables of INTERMACS but do 

fall within its goals. 

Hospital collaborators (physicians, surgeons, 

coordinators, administrators, and quality 

assurance officers). The scientific and clinical 

energy of INTERMACS comes from physicians 

who care for heart failure patients and surgeons 

who implant the devices. The hospitals, via their 

coordinators, provide the data that populates the 

registry. INTERMACS serves as an important 

resource for the hospitals in activities related to 

mechanical circulatory support. For example, 

hospitals can submit requests for scientific studies, 

obtain their own electronic data from 

INTERMACS, and participate in an INTERMACS 

forum (the Coordinators Council) for coordinator 

feedback and discussion of relevant mechanical 

circulatory support topics. INTERMACS provides 

quarterly reports to participating hospitals that 

summarize and analyze their patients and provide 

benchmarking against registrywide data. Patientlevel reports that provide a chronological history 

of the patient’s MCSD-related events are also 

available. These clinical summaries are an 

important tool in the data quality process. 

Other entities. In addition to the formal partners of 

INTERMACS, a number of other entities have 

requested collaboration. These include regulatory 

bodies of foreign governments, scientific societies, 

foreign hospitals, insurance companies, investment 

firms, and the media. Each request for 

collaboration is handled on an individual basis and 

considered within the framework of the goals and 

regulatory structure of INTERMACS.

3.2 Avian Flu Registry

Highly pathogenic infectious diseases continue to 

emerge, with substantial public health and 

financial tolls. Three features of newly emerged 

communicable diseases are immediately salient to 

registry development and use: 

• Communicable diseases do not respect 

international borders.

• Communicable diseases, by their very nature, 

usually constitute a significant public health 

threat.

• Emerging communicable diseases usually enjoy 

a high media profile and are the subject of 

significant interest to the public.

Consider the recent H1N1 influenza pandemic and 

SARS (severe acute respiratory syndrome) as 

examples. In addition, while many newly emerged 

infections first manifest themselves in exotic or 

tropical locations, this is not an invariable rule, as 

shown by the emergence of legionellosis in 

Philadelphia. 

The facility with which communicable diseases are 

able to cross international borders means that they 

typically receive global attention, especially in our 
221

Chapter 24. Public-Private Partnerships

current era of mass international travel and 

globalization of trade. The fact that newly emerged 

infections usually represent a threat to public 

health means governments and their agencies 

usually become involved in their investigation and 

management, typically at an early stage. Public 

concern, often fueled by the news media, may add 

to pressures on public health authorities to react 

and to be seen as reacting to newly emerged 

threats. As a consequence, entities wishing to 

investigate newly emerged infections will 

generally need to engage with public health 

authorities, typically at a national government 

level. 

A prime example of such a collaboration is the 

Avian Flu Registry, set up to investigate infection 

with influenza A/H5N1, a disease with almost 90 

percent mortality if untreated.2, 3 The registry, 

which began in 2007, is a multicountry, 

observational study of the diagnosis, treatment, 

and outcomes of human cases of the A/H5N1 

virus. Data are collected from health care 

professionals, and information abstracted from 

detailed, published case studies are also included. 

The registry has built a multinational, multicenter 

collaboration that houses the world’s largest 

collection of human avian influenza cases and has 

made important contributions to the understanding 

of the treatment effectiveness for this highly lethal 

disease.4, 5 Its success has been built upon 

recognition of the unique nature of emerging 

infections, recognition of the differing needs of 

developing countries and collaborators, and 

adoption of a flexible and pragmatic approach. Its 

success is also attributable at least in part to the 

establishment of successful collaborations with 

national public health agencies in a number of 

countries. 

However, the establishment of such collaborations 

is not always a straightforward matter, especially 

when initiated by the private sector.6 Newly 

emerged infections usually become politicized 

quite soon after their initial appearance. The 

classic example of this phenomenon is HIV, but 

SARS and pandemic influenza were also 

politicized rapidly after emergence. This 

politicization is seen in both economically 

developed democracies and developing countries. 

Further and deeper politicization may ensue when 

the newly emerged infection is viewed by afflicted 

countries as stigmatizing them in some way or is 

seen as a matter of national security; the response 

of some governments to avian flu exemplifies these 

types of responses. Similar reactions were seen in 

Indonesia with H5N1 and in China with the early 

stages of SARS. Developing countries may also be 

sensitive to the fact that their health care systems 

do not offer the same level of care as is available in 

developed countries. These countries may also lack 

developed disease surveillance systems and may 

feel uncomfortable at the exposure of this 

deficiency. 

Considering these sensitivities, the establishment 

of registries to study newly emerged infections 

may require a different approach to that typically 

adopted in other disease areas. An understanding 

of local sensitivities and a willingness to attend to 

local needs and to answer local questions will be 

helpful. An avoidance of a “one size fits all” 

approach should also prove helpful, with flexibility 

to react to different countries in different ways 

being important. A useful guiding principle in the 

establishment of such multinational collaborations 

is to place the needs of the collaborator first, rather 

than the needs of the entity establishing the 

registry. While national public health authorities 

may well understand the altruistic nature of much 

global public health research, their constituencies 

remain local, and they are answerable to their local 

political masters and public. Working in this type 

of environment adds an additional layer of 

complexity, but one which has to be successfully 

navigated if success is desired.

The Avian Flu Registry provides a good example 

of these political issues and how they might be 

surmounted. A complaint frequently heard when 

approaching ministries of health for collaboration 

was that such previous efforts had yielded little or 

no benefit to the participating country, with little 

or no feedback once collaboration had been agreed 

and data entry completed. The Avian Flu Registry, 

from inception, took pains to ensure prompt 

feedback to collaborators of data analyses and 

registry findings and to respond to requests from 

collaborators for further analyses in a positive and 

timely manner. 
222

Section V. Special Applications in Patient Registries

The funding for the Avian Flu Registry came from 

a pharmaceutical company that had a marketed 

product for treatment of seasonal influenza. Since 

hardly any information was available about avian 

influenza, the registry sponsor wanted to learn 

more about the illness with an eye toward 

understanding if their product would be effective 

for this more lethal flu strain. While some may see 

primary funding from industry as a disadvantage, 

the apolitical nature of this funding may actually 

have been advantageous. The relationship between 

the funder and the scientists charged with building 

the registry was clearly established at the earliest 

stage of planning and documented in a clearly 

worded binding contract. It was in the interests of 

the industry sponsor to step back from operational 

issues, allowing the investigators to build an 

international collaboration with the sole purpose 

of understanding the disease, with the expectation 

this would be done as efficiently as possible and 

with findings to be shared with all participants.

In addition, the registry was created in its earliest 

stages to conform to principles of good practice 

for registry science, including formal ethical 

review, a steering committee, and various other 

governance structures that proved useful 

throughout the program. A complexity of the 

registry was its broad global reach, which included 

collaborators from 13 different countries. 

Regulations varied by country and by collaborator, 

but were in all instances compatible with the 

founding documents of the registry, as enshrined 

in the agreement between the industry funder and 

the investigators, and as presented to an 

independent ethics review board. A formal 

memorandum of understanding outlined all the 

key principles for data sharing, protection of 

privacy, ethical review, et cetera. Original 

documents guaranteed protection of the identity of 

individual reporting countries, a restriction that 

was later lifted by mutual agreement once it 

become apparent that country-specific factors like 

viral clade and barriers to access to care tempered 

treatment effectiveness. The Data Access and 

Publications Committee also proved to be useful 

by providing a formal mechanism for recording, 

reviewing, and prioritizing research questions that 

were posed to the registry. 

3.3 Get With The Guidelines®

Get With The Guidelines® is a hospital-based 

quality improvement program operated by the 

American Heart Association. The program aims to 

improve in-hospital care for patients by providing 

tools to support adherence to clinical practice 

guidelines. Hospitals pay a fee to participate in the 

program, which involves collecting and submitting 

data on patients. The program uses the data to 

generate benchmarking reports and to provide realtime feedback on adherence to the clinical practice 

guidelines. The program has been successful at 

demonstrating sustained quality improvement at 

participating hospitals.7

State-level departments of health also have an 

interest in improving quality at hospitals within 

their State. However, the development of a 

comprehensive quality improvement program is 

often not feasible, given resource and staff 

constraints. In several cases, State departments of 

health partnered with the American Heart 

Association to sponsor hospitals in the Get With 

The Guidelines program. The State agencies paid 

the program fee for participating hospitals and, in 

return, received reports on hospital performance 

on a quarterly basis. Hospitals agreed to share 

their performance data, which the program would 

normally keep confidential, in return for receiving 

free access to the Get With The Guidelines 

program.

3.4 CMS Coverage With Evidence 

Development

In 2006, CMS issued a guidance titled “National 

Coverage Determinations with Data Collection as 

a Condition of Coverage: Coverage with Evidence 

Development”8 that presented a new option for 

CMS when determining whether a drug or device 

would be covered under Medicare or Medicaid. In 

addition to the existing possible decisions of “no 

change in current coverage,” “non-coverage,” and 

“coverage without special conditions,” CMS could 

now grant “coverage with special conditions,” in 

which:
223

Chapter 24. Public-Private Partnerships

“The medical evidence is adequate to conclude 

that the item or service is reasonable and 

necessary [...] only under one or more of the 

following circumstances:

a. The item or service is covered only for 

patients with specific clinical or 

demographic characteristics.

b. The item or service is covered only when 

provided by physicians and/or facilities that 

meet specific criteria.

c. The item or service is covered only when 

specific data are submitted in addition to 

claims data to demonstrate that the item or 

service was provided as specified in the 

[national coverage determination].”8

Registries are particularly suited to this type of 

prospective data collection. This new Coverage 

with Evidence Development (CED) requirement 

spurred the creation of multi-stakeholder registries 

to facilitate data collection for drugs and devices 

receiving CMS coverage conditional on evidence 

development. In 2012, CMS published a draft 

guidance for the public, industry, and CMS staff 

on CED in the context of coverage decision, 

further clarifying best practices in an attempt to 

improve the application of CED.9 Aside from 

CMS, which provides the incentive for the data 

collection, major partners in CED studies often 

include professional associations (which contribute 

scientific guidance) and industry (which 

contributes funding). Registries that have been 

created or adapted to meet CED requirements 

include the National Oncologic PET Registry for 

the use of positron emission tomography to treat 

certain types of cancers10 and the ICD Registry for 

the use of implantable cardioverter defibrillators.11

4. Considerations for Setting 

Up a PPP

4.1 Governance

A public-private registry is, by definition, a 

collection of stakeholders who have different 

purposes and agendas that hopefully overlap at the 

intersection of clinical science and improved 

patient care The registry needs a central authority 

to keep it focused. Often, this authority is the 

principal investigator (PI), who oversees the 

registry and is responsible for developing 

consensus among stakeholders. The PI is also 

responsible for ensuring that the registry and the 

analyses of the registry data remain scientifically 

relevant and unbiased. The PI’s scientific and 

operational oversight can be augmented by an 

Advisory Committee, which can include co-PIs 

and representatives from various partners in the 

registry (e.g., funding sources, reporting entities, 

or subcontractors that handle operational aspects 

of the registry). 

4.2 Involving Patients

As with many other types of outcomes research, 

there is a growing trend to involve patients and 

patient advocacy groups in the planning and 

operation of patient registries. Working with 

patients brings certain unique considerations, and 

varying levels of patient involvement may be 

appropriate and/or feasible for different PPPs. For 

example, some diseases such as influenza A/H5N1 

lack a cohesive patient population or advocacy 

group, because of the disease’s rarity and high 

mortality rate. However, patients can offer valuable 

contributions to PPPs when it is feasible for them 

to do so, especially around such areas as user 

burden (e.g., when the registry is collecting 

patient-reported outcomes), registry feasibility, 

and training and support needs. 

4.3 Operational Decisions

Many registries are complex in nature with 

operational components including regulatory, 

financial, informed consent, data entry software, 

progress reports, periodic meetings, and scientific 

analyses. These registries are essentially small 

businesses that require intense day-to-day 

operations that should be conducted as a wellstructured effort. The structure of the registry 

efforts should be clear, with well-defined lines of 

authority and responsibility. The structure should 

also have the flexibility to adapt to changing 

science and the changing national landscape of 

regulatory requirements, such as the periodic 

updates to the Health Insurance Portability and 

Accountability Act (HIPAA) Privacy Rule. 

A representative Operations Committee that meets 

regularly to review the ongoing progress of the 
224

Section V. Special Applications in Patient Registries

registry and to address issues as they arise may be 

desirable. This group can make decisions by 

consensus rather than a formal vote. 

Documentation (i.e., meeting minutes) should be 

created and distributed to memorialize decisions 

and actions taken. 

4.3.1 Plans for Transparency and 

Communication

Transparency and ongoing communication are 

vital to the success of any complex registry, 

especially a PPP. An important vehicle for 

transparency can be a registry’s public Web site, 

which can contain regulatory documents including 

the protocol and user’s guide (see http://www.uab.

edu/medicine/intermacs/). The public face of the 

Web site for the Avian Flu Registry contains the 

registry prospectus and information about data 

security, along with an updated list of published 

scientific articles and presentations (see https://

www.avianfluregistry.org/), including many of the 

actual posters and slide sets for public viewing. 

Other options for engagement, transparency, and 

communication include periodic public 

stakeholder meetings, newsletters, and email 

listservs. 

4.3.2 Dispute Resolution

Disagreements, or even disputes, are inevitable 

when a group of diverse stakeholders collaborate 

on a single registry. As with any complex 

endeavor, the key to symbiotic working 

relationships lies in the initial formulation of the 

goals and expectations of the registry and of each 

collaborator. The responsibility of mediation and 

dispute resolution can be assigned to a leader 

within the registry, such as a Study Chair or PI, or 

can be handled by committee, as in the Avian Flu 

Registry.

4.3.3 Data Security

The data contained in any registry must be 

managed according to strict rules for data security, 

which can include secure password-protected 

access to data entry, secure transmission of data, 

background checks on personnel, personnel 

training on data security, virus scans of all 

computers, and offsite backup of data. Anyone 

creating a new registry is strongly advised to 

collaborate with information security experts, who 

can lead the registry through the data security 

requirements and can create protocols for security 

breaches. Additional information on data 

management, data quality, and data security can be 

found in Chapter 11. 

4.4 Data Ownership, Data Access, and 

Publications

As discussed in Chapter 7, ownership of registry 

data is a complex issue. In many registries, the 

registry sponsor owns the registry data. However, 

PPPs may have multiple sponsors and multiple 

stakeholders. Because of these complexities, it is 

important for the registry to specify clearly in 

registry partnership agreements and contracts who 

will have ownership rights to the registry data.

4.4.1 Data Access

A related question is who has access to the data. 

While some registries may rely on a scientific 

advisory board or other governing body to handle 

data access requests, PPPs should consider a 

formal Data Access and Publications Committee 

(DAPC). A formal DAPC can develop policies and 

manage requests for data access in a transparent, 

consistent way that is agreeable to all stakeholders. 

All data access should conform to HIPAA 

regulations, informed consent documents, and data 

use agreements between contributing sites and the 

registry. Many entities may request access to 

registry data, including some listed below:

• Data provider or participating site. Typically, 

the data use agreement between the site and the 

registry specifies that the site can request to 

receive all of its own data at any time, but may 

not request identified data from another site. 

• Registry sponsor. The registry sponsor owns 

the data and therefore has complete access to 

all data. When the registry ends, the entire 

database is often transferred to the sponsor. 

• Regulatory agency (e.g., FDA, CMS). A 

government regulatory agency may request 

registry data to fulfill safety reporting 

requirements or other obligations. In particular, 

if a sponsor has a marketed product that is used 

by any patients in the registry, that sponsor is 

subject to mandatory safety reporting 
225

Chapter 24. Public-Private Partnerships

requirements (see Chapter 12).

• Industry. Pharmaceutical companies or device 

manufacturers may request data of patients who 

receive their products, or may request registry 

data to use as controls for a clinical trial. 

• Investigators (within or outside the registry). 

Investigators may request registry data for a 

particular research project; each request should 

be reviewed by the DAPC so that only the 

necessary data elements for the research project 

are shared. 

• Public. Although rare, data requests from the 

media or the public are possible. Any 

information released to the public by the 

registry (via newsletters, public Web site, or 

other methods) should be reviewed prior to 

release, to ensure that data confidentiality is not 

compromised.

• Standardized data sets. Some registries produce 

de-identified, standardized data sets that are 

available to researchers on a periodic basis. 

These data sets contain no protected health 

information, no product or treatment brand 

names, and no site identifiers, and they are 

often constructed to provide the information 

believed to be most helpful to researchers. The 

actual content of these standardized data sets 

and the policy for distribution should be 

governed by the DAPC, with approval by the 

registry sponsor.

4.4.2 Process for Publications

As noted above, PPPs may find it particularly 

useful to form a DAPC to prioritize research 

projects and handle data access requests. The 

committee should meet regularly to formally 

review, prioritize, and evaluate the requests based 

on the potential impact on clinical practice and the 

amount of data available to answer the research 

question. The DAPC can also work directly with 

an advisory or operations committee to identify 

and facilitate internal research projects that 

directly address the stated research goals of the 

registry.

4.4.3 Process for Analyses

Depending on available resources, a registry can 

either conduct analyses to support publications 

in-house, contract with an outside agency to 

conduct analyses, or leave this task to the data 

requestors themselves. If an outside agency or data 

requestor will be conducting analysis on registry 

data, a secure mechanism should be in place for 

sending the data to them. The DAPC should retain 

oversight of these activities, especially those that 

are intended to be used for manuscripts submitted 

to peer-reviewed publications.

4.4.4 Formal Documentation of Roles and 

Responsibilities

Whether a registry resembles a traditional PPP 

(i.e., a group of stakeholders who come together to 

create and fund a registry) or a more unusual 

structure (i.e., a series of contracts and 

subcontracts that have precise deliverables), each 

entity is a collaborator in the sense that each 

partner provides something to the registry and 

receives something from the registry. For example, 

each hospital participating in INTERMACS 

provides the local effort for participation and data 

entry. The hospital also pays $10,000 per year for 

participation. In return, the hospital receives many 

deliverables and benefits such as quarterly quality 

assurance reports, clinical summaries on each 

patient, electronic copies of their data, 

participation in research projects, and 

representation on the INTERMACS committees.

Because each entity may have numerous functions 

within a registry, it is important that roles and 

responsibilities be clearly defined and documented 

at the beginning of the registry. An operations 

committee can be charged with producing the roles 

and responsibilities document and updating it 

periodically as needed. 

4.5 Funding

Registries can obtain their funding from a variety 

of sources. For example, INTERMACS was 

initially funded by a contract from NHLBI. During 

its second 5-year contract (December 2010–

November 2015), NHLBI asked the University of 

Alabama at Birmingham to develop a cost-sharing 

plan that would allow NHLBI to significantly 
226

Section V. Special Applications in Patient Registries

decrease their contribution while obtaining 

funding from private sources. The primary goal of 

this new arrangement was to obtain the necessary 

ongoing funding in order to achieve sustainability. 

This transition in funding is not unique to 

INTERMACS. Changes in funding are particularly 

common in PPPs, where funding often comes from 

multiple sources. When funding sources change, it 

is often necessary to revisit the roles and 

responsibilities and data access policies to ensure 

that all stakeholders are represented appropriately.

4.6 Ethics

4.6.1 Conflicts of Interest

Because of the variety of stakeholders involved, a 

plan for identifying and managing actual and 

perceived conflicts of interest (COI) is essential. In 

this context, COIs can be financial or intellectual. 

The plan should clearly spell out the timeline and 

process for obtaining completed COI and financial 

disclosure forms from participating members and 

for reviewing and managing any potential 

conflicts, particularly given any unique working 

relationships with the Federal government, 

academic institutions, or industry. The plan should 

also define what constitutes a problematic COI, 

and this definition should be reviewed on a 

periodic basis and revised if needed. It is 

suggested that the PI, co-PIs, study chair, 

operations and steering committee members, 

subcommittee members, and individuals named on 

the contract (including subcontractors and their 

staff) be required to complete annual COI forms. 

Once collected, the forms can be reviewed by 

registry staff and any conflicts forwarded to the 

operations committee for review. Any individuals 

that have a financial disclosure identified through 

the COI review process should declare it prior to 

participation in any scientific meetings, 

government meetings, presentations at sites, 

registry annual meetings, steering committee 

meetings, et cetera.

4.6.2 Informed Consent

The informed consent documents are key elements 

in determining the unique relationship between a 

patient’s medical information and the ultimate use 

of this information in achieving the goals of the 

registry. The document must contain an explicit 

description of who will see what data and how 

confidentiality will be maintained. For registries 

with many partners as is common with PPPs, it is 

desirable to have a common informed consent 

form. This document can be combined with a 

HIPAA authorization to use or disclose protected 

health information for registry purposes. (See 

Chapter 7 for additional information on HIPAA 

authorization requirements.) If a common form is 

not used, the reasons for the different forms should 

be documented and transparent to all key parties. 

The data coordinating center for INTERMACS 

created an informed consent template in 

collaboration with NHLBI that contains the 

necessary elements as determined by NHLBI and 

the institutional review board at the data 

coordinating center. Chapter 8 discusses informed 

consent in detail. 

5. Evolution of PPPs

Registries that are PPPs may undergo many 

changes over their lifetime. The registry goals and 

roles of stakeholders may change, and new 

stakeholders may become involved. A registry not 

initially set up as a PPP may later evolve into one. 

The general topic of registry transitions is covered 

in Chapter 14, but there are several changes and 

transitions that are unique to PPPs.

For example, INTERMACS began as a 

collaboration between NHLBI, FDA, and CMS. 

The other partners currently involved in 

INTERMACS (and shown in Figure 24–2) joined 

later, and each brought their own agenda and goals 

for the registry. As these new partners joined, 

INTERMACS had to evaluate the many different 

goals they brought to the table, identify areas of 

overlap, and determine how INTERMACS could 

meet the needs of each partner while remaining 

focused on the ultimate goal of the registry: to 

improve patient outcomes.

Sometimes a registry is not initially organized as a 

PPP but later evolves into one. This often happens 

when potential stakeholders do not see the value of 

being involved in a registry in the beginning 

stages, particularly when the registry has not yet 

published any results or provided proof of concept. 
227

Chapter 24. Public-Private Partnerships

In these situations, it is incumbent on the registry 

originators to operate the registry and produce 

results that will entice stakeholders to participate. 

For example, the Avian Flu Registry (funded by 

industry and operated by a private contract 

research organization) found much more success 

in partnering with international ministries of 

health after the Registry published its results in 

peer-reviewed journals and presented abstracts at 

well-known scientific conferences. Similarly, Get 

With The Guidelines was able to partner with 

State-level health departments only after 

consistently demonstrating its success in 

improving patient quality of care. 

6. Considerations for 

Managing a PPP

6.1 Stakeholder Engagement

Once a PPP has been established, it becomes 

critical to focus on proper management of the 

project. Major stakeholders may be involved, 

including clinicians, payers, patients/consumers, 

Federal agencies, and industry/manufacturers. 

Inclusion of varying perspectives ensures balance, 

yet decisionmakers from different sectors may 

have conflicting priorities. Engaging each of these 

groups with the common goal of improving health 

care quality and patient outcomes through sharing 

of data and other resources is vital to the 

achievement of the partnership. Such 

collaborations have occurred successfully in 

several industries where no single entity had the 

resources or expertise to drive an entire field.12, 13

Eliciting trust among decisionmakers combined 

with advice and/or participation from reputable 

associations are valuable incentives for 

maintaining the interest and engagement of 

collaborators.14-16 Successful collaborations satisfy 

the needs of multiple stakeholders, providing 

immediate value and long-term returns, while 

driving innovation and efficient productivity and 

leading to the development of best practices. 

It is also vitally important to set appropriate 

expectations for the participation of each group 

within the partnership. The utility of preproject 

meetings involving discussion of priorities and 

policies that will govern the collaborative efforts 

cannot be overemphasized.17, 18 Roles and 

responsibilities must be clearly defined and 

mutually agreed upon so that all stakeholders 

benefit equally.19 Evaluation of the available 

literature may reveal which practices have worked 

for other partnerships. Establishing guidelines that 

dictate partnership activities, including conflict of 

interest procedures, will allow accountability.16, 19

Identifying a PI with strong leadership skills, a 

project manager to drive timelines, and other 

properly trained team members will ensure 

successful execution of project goals.14

Establishing agreement between participating 

groups on the time commitments required of them 

from the beginning will help set appropriate 

expectations. Resources that increase ease of 

communication and minimize time commitments, 

such as shared Web sites or databases,14 can speed 

development and improve participation. Although 

the importance of timelines is paramount, the 

ability to be flexible is also important in the 

changing landscape of health care policies and for 

PPPs that add partners and collaborators and adapt 

over time.20

6.2 Communication

Communication tools for generating and 

maintaining interest among stakeholders and 

participants are beneficial when used effectively. 

Initiation of interactive workshops or exchange 

forums between public and private sectors, 

dissemination of publications and news releases, 

and updates at professional meetings are all 

effective ways of communicating the necessary 

information to drive the partnership forward.21

Periodic updates and exchanges of data have been 

shown to have positive effects on collaborations.15 

Overly frequent distribution of printed 

communications, required teleconferences, or 

excessive meetings will generate unwanted 

frustration or lack of continued support/

participation. However, the value of a reasonable 

number of written updates, fairly regular calls 

(monthly, for example), and at least two in-person 

meetings (at onset and before distribution of 

results) are essential for building strong team 

morale, maintaining commitments, and achieving 

successful outcomes. Clearly these processes must 
228

Section V. Special Applications in Patient Registries

be adapted to accommodate national and regional 

cultural sensitivities. 

6.3 Visibility

Visibility of results and the breadth of 

dissemination of information to be obtained 

through the partnership should be discussed in the 

early planning phase of the project. Preparing 

results for wide dissemination requires 

considerable time and effort, which may not fall 

within the scope of the project team. However, if 

such a distribution is desired and the funding and 

resources are available, the results can benefit a 

more widespread audience. Visibility of potential 

and perceived conflicts of interest should also be 

discussed at the onset of the partnership. An 

internal and/or external monitoring committee can 

reinforce ethical standards and trust among 

stakeholders.16 The priorities with respect to 

transparency and diffusion of information will 

depend on the nature of the partnership, the initial 

agenda, and the resources available.

6.4 Change Management

Anticipating and planning for change is good 

practice for all patient registries. Because of their 

nature and the variety of their stakeholders, PPPs 

in particular may be more subject to changes in 

registry goals, stakeholders, budget, processes, and 

other areas. For this reason, it is important for 

PPPs to have a plan for how change will be 

managed. Tools that can assist in change 

management planning include a manual of 

procedures, a governing body, infrastructure for 

ongoing personnel training, and a plan for 

communicating change. 

Protocols, governance documents, and other 

related documents may change from time to time 

as a registry matures and adapts. Documents 

should be reviewed periodically and updated as 

needed. Resubmission for ethical review may be 

required, depending on the extent of the changes. 

The use of versioning (e.g., naming a protocol 

“Registry Protocol v1.0”) can reduce 

miscommunications and ensure that all 

stakeholders refer to the same document. It is also 

important to document major decisions that will 

affect the scope or budget, or otherwise impact the 

registry, and to share information about these 

decisions with key stakeholders. For more 

information on managing change in registries, see 

Chapter 11, Section 2.6, and Chapter 14.

7. Special Considerations for 

International PPPs

International PPPs face some unique challenges, in 

addition to the usual challenges of language and 

cultural barriers. While some investigators may 

complain about the burden of compliance with 

regulation in developed countries, the opposite 

problem may exist in some less-developed 

jurisdictions. The absence of a clear regulatory 

framework within which to operate may create 

problems in both the investigator’s home country 

and in the host collaborating country. One example 

may be lack of clarity in determining the 

responsible office for establishment of 

collaborations; another example may be changes in 

the local political landscape that alter this locus of 

responsibility. An issue that should be clarified in 

advance is the right to publish findings and to 

confirm authorship. Early attention to these details 

will avoid later issues. 

8. Key Factors for Success and 

Potential Challenges 

8.1 Key Factors for Success

A PPP represents a valuable business model for 

the development of multi-stakeholder registries. 

The shared-risk and shared-benefit nature of PPPs 

presents an ideal opportunity for attracting 

involvement from risk-averse elements in any 

sector, but these benefits coincide with challenges 

that may derail the success of a project as a whole.

A PPP starts with an identified public health issue 

in need of a solution. There is no shortage of 

strong, scientifically valid and important topics 

relating to the delivery of medical care and use of 

medical products; the challenge is in prioritizing 

these issues and focusing on pragmatic solutions 

for high-impact projects. For example, a registry 

tracking care patterns for a well-understood rare 

disease would likely generate less support than one 
229

Chapter 24. Public-Private Partnerships

that would collect acute and chronic data on a 

novel treatment for a highly prevalent condition. 

To ensure success of a given PPP, it is vital to 

communicate with a broad array of stakeholders 

early in the process to ensure that the problem is 

appropriately conceptualized and that the goals 

mesh with priorities of stakeholders.

While PPPs represent a variety of interests and 

viewpoints, the value of a strong leader cannot be 

overstressed. Because of the nature of professional 

life, few people have the necessary time to devote 

to the difficult task of managing not only the 

scientific aspects of developing a registry, but also 

the equally challenging task of developing and 

managing an interdisciplinary team with diverse 

interests and leading it toward a common goal. The 

presence of a trusted and dedicated individual who 

is willing to commit substantial time to the 

development of a PPP is critical to the success of 

the project. This individual needs to be a 

recognized expert voice and have skills as a 

moderator, mediator, business developer, and 

salesman. Individuals who are open to pragmatic 

approaches that accommodate stakeholders 

without sacrificing the scientific integrity of the 

project will have a high likelihood of success. 

Similarly, an active and dedicated core team that 

represents an array of stakeholders is also 

necessary to support the goals of the PPP.

Many PPPs, like any project, are started with small 

conversations that grow into grand ideas. The 

formative stages of a PPP involve many steps of 

developing and refining the issues and potential 

solutions long before the first data entry form is 

ever filled out, and often consist of preparing 

documents, attending calls, holding workshops, 

and other collaborative activities. While talk is 

indeed cheap, there comes a point where the 

project cannot move further without some 

substantial funding. It is good practice to begin 

development of a funding strategy early, often 

alongside the development of the scientific 

strategy. Funding options should not depend solely 

on any one source or sector. A broader base of 

support is more likely to be a sustainable funding 

model, and has the added benefit of potentially 

reducing the appearance of conflicts of interest.

If one views the PPP as a business model, the 

necessity to provide accurate and timely reports to 

shareholders becomes more readily apparent. In 

the planning process and throughout the 

development of the project, it is important to set 

goals and produce meaningful deliverables within 

a reasonable time frame. Projects that appear to 

drag on, or that have a dearth of outputs for an 

extended period, are likely to lose support and 

jeopardize funding. Likewise, reporting of the 

progress of a project is critical to sustain interest 

and support. For PPPs that involve professional or 

academic societies, the annual scientific sessions 

of these organizations often provide an ideal 

opportunity to update the community. 

Clear communication in open forums that 

encourage and allow for buy-in and feedback is 

another critical component of success in a PPP 

environment. A registry is a unique application of 

the PPP model in that successful implementation 

of the final project is heavily dependent on 

individual hospitals and practitioners. Having 

stakeholders represented at the leadership levels of 

organizations is necessary for good governance; 

however, communicating with the physicians, 

hospitals, nurses, and associated staff to address 

their concerns will promote enrollment. Further, 

the case must be made to this group that the 

registry will add value to their organizations, and 

not just represent a further drain on their already 

sparse time.

Some registry characteristics that increase the 

probability of success include—

• The registry should have goals that address a 

clear and current clinical need in a well-defined 

population. These goals become the rallying 

point for the diverse partners.

• The expectations of each partner should be 

explicitly numerated, pragmatic, transparent, 

and measurable.

• The registry should return value to all partners 

who are financial contributors. As much as 

possible, the value should equal or exceed the 

financial contribution for each partner.
230

Section V. Special Applications in Patient Registries

• The registry should have strong, respected 

leaders who have national or international 

reputations. Mutual respect among all partners 

is also necessary for a strong working 

relationship.

• High quality data is essential to the success of 

the registry. Data must be collected 

consistently, using agreed-upon definitions. 

Protocol-driven efforts to assess compliance 

with the registry protocols and well-defined 

efforts to repair any deficient areas are critical.

• While the registry should be built for 

consistency, it still must have an element of 

flexibility to allow it to react to changes in the 

clinical landscape.

• The registry should have policies and 

procedures in place to support transparency and 

ongoing communication to partners and 

participants.

8.2 Common Challenges

The first challenge is involving stakeholders in 

designing the registry and implementing the 

registry procedures (e.g., governance, operational 

management, analysis). The second challenge is 

creating a data collection plan and registry 

procedures that are realistic and will capture the 

data necessary to meet the goals, but flexible 

enough to accommodate change when necessary. 

The third challenge is adhering to the registry 

procedures and data collection plan. If a registry is 

successful, many spinoff projects and additional 

uses of the registry may appear. Maintaining focus 

on the original goals of the registry while 

responding to increasing registry demands is 

clearly a challenge. Creating a business plan that 

will allow for sustainability of the registry is one 

of the biggest challenges. Assessing quality of life 

and other patient-reported outcomes, including 

clinical assessments (e.g., neurocognitive 

assessment) is a challenge because direct 

interaction with the patient is required. The biggest 

challenge is to provide daily high-level effort that 

simultaneously focuses on regulatory and data 

quality issues while continuing the scientific 

mission of the registry.

9. Summary

PPPs are increasingly being used as a model for 

operating patient registries in the United States and 

internationally. Government regulators and payers 

are increasingly requiring evidence development to 

inform decisions about approval, coverage, and 

expanded indications, and patient registries 

governed by PPPs are in a unique position to fulfill 

those requirements. In the future, PPPs that 

include international partners will continue to be 

important. While there are special considerations 

for planning and operating PPPs, they offer a 

unique way for varied stakeholders to contribute 

their particular strengths to achieve a common 

scientific goal.
231

Chapter 24. Public-Private Partnerships

Case Example 62. Developing a public-private 

partnership for comparative effectiveness 

research

Description The Registry In Glaucoma 

Outcomes Research (RiGOR) is 

a prospective observational study 

comparing the effectiveness of 

treatment strategies for openangle glaucoma.

Sponsor Agency for Healthcare Research 

and Quality (AHRQ)

Year Started 2011

Year Ended Ongoing

No. of Sites 47 community and academic 

ophthalmologic practices

No. of Patients 2,625

Challenge

In 2009, the Institute of Medicine disseminated a 

landmark report, “Initial National Priorities for 

Comparative Effectiveness Research,” which 

listed research priorities for the newly enacted 

American Recovery and Reinvestment Act 

(ARRA). Among the 100 priority research topics 

identified was evaluating the different treatment 

strategies for primary open-angle glaucoma. 

Since the disease disproportionately affects 

African-Americans, understanding the 

effectiveness of treatment strategies in minority 

populations was also of special interest. With 

ARRA funding, AHRQ sought to develop 

high-quality scientific evidence to inform 

decisionmaking by clinicians and patients. An 

approach was needed to obtain continued and 

expanded input from the various stakeholders 

while addressing existing evidence gaps.

Proposed Solution

A diverse group of stakeholders was assembled to 

implement the registry, provide scientific 

guidance, develop dissemination plans, and 

further key research based on study findings. The 

principal investigator and co-principal 

investigators represent AHRQ, the American 

Academy of Ophthalmology (AAO), the 

University of California at Los Angeles (UCLA) 

Jules Stein Eye Institute, and the Outcome 

DEcIDE (Developing Evidence to Inform 

Decisions about Effectiveness) Center. AHRQ 

provides oversight and financial support to the 

project, with scientific leadership from the 

principal and co-principal investigators; the 

Outcome DEcIDE Center manages the 

operational aspects of the study; AAO and UCLA 

engage sites and investigators and provide 

guidance on clinical issues. The stakeholder 

committee is comprised of individual clinical 

advisors and representatives from the Glaucoma 

Research Foundation, American Glaucoma 

Society, National Medical Association, and 

State-level health care organizations. 

Developing the study protocol, initiating startup 

activities and decisions, and analyzing and 

reporting the findings require continued 

communications among all stakeholders. 

A communication plan was developed to outline 

project team roles and organizational structures 

for each stakeholder. Regular stakeholder 

committee meetings have created a forum to 

discuss design issues, share study status, solicit 

input on unexpected challenges, and discuss 

future research. Site- and patient-recruitment 

efforts were designed to maximize geographic 

diversity and enrollment of minority populations. 

Quarterly study newsletters and investigator 

meetings coinciding with the AAO annual 

meeting were also implemented to maintain site 

interest.

Results

Launched in 2011, RiGOR is a prospective, 

observational, cohort study designed to compare 

the effectiveness of treatment strategies for 

primary open-angle glaucoma. Different 

treatment strategies studied in the registry include 

laser surgery, other procedures (such as incisional 

surgery or other glaucoma procedures), and 

medications. All treatment decisions are at the 

discretion of the treating physician according to 

their usual practice. Data collection includes 

 Case Examples for Chapter 24
232

Section V. Special Applications in Patient Registries

Case Example 62. Developing a public-private 

partnership for comparative effectiveness 

research (continued)

Results (continued)

patient demographics, medication, visual 

measures, glaucoma severity, surgical 

characteristics, adverse events, and patientreported outcomes, and occurs at baseline, 

3 months, 6 months, and 12 months. 

The registry has been successful in meeting its 

objective of enrolling a high percentage of 

minority patients. An interim report describing 

baseline findings is currently in process, and full 

analyses are expected to be published in 2013. 

The current AHRQ funding will allow RiGOR to 

operate through 2013. A future challenge for the 

registry will be identifying and transitioning to a 

new funding source once the initial funding ends. 

Key Point

The public-private partnership model can be an 

effective approach to engaging multiple 

stakeholders in an effort to address a comparative 

effectiveness research question. When working 

with multiple stakeholders, it is critical to clearly 

identify roles and communicate regularly with all 

stakeholders to address any design, operational, 

or analytical issues, solicit input from all 

contributors, share study findings, and maintain 

stakeholder engagement.

For More Information

http://www.effectivehealthcare.ahrq.gov/searchfor-guides-reviews-and-reports/?pageaction=

displayproduct&productid=841.

Case Example 63. Leveraging a public-private 

partnership for a postmarketing commitment

Description The Longitudinal Study of Urea 

Cycle Disorders is operated by 

the Urea Cycles Disorder 

Consortium (UCDC). Its 

primary purpose is to collect 

data on the natural history, 

disease progression, treatment, 

and outcomes of individuals 

with urea cycle disorders (UCD). 

The Orphan Europe Carbaglu 

Surveillance Protocol is a 

mandated postapproval registry 

that is collaborating with the 

study to monitor the long-term 

safety and effectiveness of 

Carbaglu, a treatment for UCD.

Sponsor Orphan Europe, UCDC, National 

Institutes of Health (NIH)

Year Started 2011

Year Ended Ongoing

No. of Sites 14

No. of Patients 1

Challenge

Orphan Europe is the manufacturer of Carbaglu, 

a drug used to treat hyperammonemia (high blood 

ammonia levels) due to N-Acetylglutamate 

synthetase (NAGS) deficiency, a type of UCD. In 

2010, the U.S. Food and Drug Administration 

(FDA) approved Carbaglu for use in the United 

States and mandated a 15-year postmarketing 

surveillance program to monitor its long-term 

safety and effectiveness. The sponsor recognized 

that data collection for this rare disease would be 

difficult because of the small number of NAGS 

patients in the United States and the extended 

timeframe for data collection; thus, the sponsor 

sought to meet FDA’s commitment while 

avoiding redundancy in research efforts and 

overburdening the small patient population.

Proposed Solution

NIH established UCDC in 2003 as part of the 

Rare Diseases Clinical Research Network. The 

UCDC is governed by a steering committee and 

maintains relationships with 16 clinical sites, a 

data monitoring and coordinating center, and 

patient advocacy groups. One of the functions of 

the UCDC is to operate the Longitudinal Study of 

Urea Cycle Disorders, initiated in 2006 and now
233

Chapter 24. Public-Private Partnerships

Case Example 63. Leveraging a public-private 

partnership for a postmarketing commitment 

(continued)

Proposed Solution (continued)

containing baseline and longitudinal data on 515 

patients. 

Orphan Europe recognized the value of the 

UCDC study’s existing database and 

infrastructure and pursued collaboration with the 

Consortium. After initial discussions, it became 

clear that the data elements already being 

collected in the study would need to be 

supplemented by only a few additional elements 

in order to fulfill the sponsor’s postmarketing 

registry commitment to FDA. A protocol was 

written for the registry to specify which study 

data the sponsor would have access to in their 

registry and which new data elements would be 

added to the study for registry purposes. 

One unique challenge encountered early on 

related to the execution of a legal agreement on 

which to base the collaboration. Because the 

UCDC does not have a legal entity, it was not 

possible to contract directly with them. Thus, the 

sponsor contracted with the Children’s Research 

Institute at Children’s National Medical Center, 

which served as a coordinating center for the 

remaining sites and had existing contractual 

agreements in place with them. The Institute then 

executed data use agreements with the sites for 

participation in the postmarketing registry. 

Results

The registry has been operating since 2011 and 

includes 14 active sites. To date, one patient has 

been enrolled in the registry. By collaborating 

with UCDC, Orphan Europe was able to 

operationalize the registry in a shorter timeframe 

and at a lower cost than building a new, 

independent registry. Collaboration with UCDC 

also resulted in only a few more data elements 

required from sites and patients participating in 

the UCDC study; this represents a smaller data 

collection burden on the already limited patient 

population. Future challenges for the registry 

include sustainability, as NIH funding for the 

UCDC study is expected to end in 2014. UCDC 

and Orphan Europe are in the process of 

collaborating with the European Registry and 

Network for Intoxication Type Metabolic 

Diseases (E-IMD), a registry collecting 

information on this patient population in Europe.

Key Point

Even after FDA approval, rare disease treatments 

can face logistical and financial challenges in 

fulfilling postmarketing obligations. 

Collaborating with existing studies that are 

collecting longitudinal data on the patient 

population can reduce the burden on participants 

and streamline the operationalization of a 

registry.

For More Information

Seminara J, Tuchman M, Krivitzky L, et al. 

Establishing a consortium for the study of rare 

diseases: The Urea Cycle Disorders Consortium. 

Mol Genet Metab. 2010;100 S97-105.

http://rarediseasesnetwork.epi.usf.edu/ucdc/index.

htm 

http://clinicaltrials.gov/ct2/show/NCT00237315 

http://www.e-imd.org 
234

Section V. Special Applications in Patient Registries

Case Example 64. Public-private partnerships 

for rare diseases

Description The American Thrombosis and 

Hemostasis Network (ATHN) is 

a nonprofit organization 

operating the ATHNdataset, a 

registry that serves as a secure 

resource of longitudinal, 

individual-level demographic, 

clinical, and genetic information 

from U.S. patients with rare 

bleeding and clotting disorders. 

The data are used for research, 

outcomes analysis, public health 

surveillance, and advocacy. 

Sponsor Initial infrastructure development: 

Novo Nordisk, Inc.

Project-specific extensions and 

applications: public sector 

partners such as Centers for 

Disease Control and Prevention 

(CDC), National Heart, Lung, 

and Blood Institute, Health 

Research and Services 

Administration (HRSA) 

Maternal and Child Health 

Bureau; as well as private-sector 

sponsors from industry, 

treatment centers, and the 

patient community (e.g., Baxter 

Bioscience, Hemophilia of 

Georgia, Indiana Hemophilia 

and Thrombosis Center)

Year Started 2010

Year Ended Ongoing

No. of Sites 129

No. of Patients 12,154 as of September 6, 2012

Challenge

In the late 1970s, the Maternal and Child Health 

Bureau of the Health Resources and Services 

Administration (HRSA), a branch of the U.S. 

Department of Health and Human Services, 

established a network of hemophilia treatment 

centers across the United States. As HIV became 

a concern for these patients in the 1980s, CDC 

also lent its support to this network. Today, more 

than 130 centers in the network deliver 

comprehensive care to patients with rare bleeding 

and clotting disorders, offering the 

multidisciplinary services of care teams that 

include hematologists, nurses, physical therapists, 

and social workers. Care coordination often 

includes orthopedists, dentists, genetic 

counselors, obstetrician-gynecologists, infectious 

disease practitioners, and emergency 

departments. 

While the network provided nationwide 

geographical coverage, centers were organized 

into independent regions that functioned as 

informal collaborations. Research often required 

pooling data from different regions in order to 

obtain a sample size sufficient for meaningful 

analysis, but these efforts were often hampered 

by the fact that treatment centers in different 

regions used different electronic data capture 

systems. 

A series of meetings was held at which 

government agencies and representatives from the 

network centers agreed that there was a clear 

need for standardized data to support the many 

data requests being received by the centers. There 

was also a need to efficiently use the limited 

resources available to these centers. 

Proposed Solution

In 2006, ATHN was established as a nonprofit 

organization, with initial motivation for startup 

supplied by CDC and the federally funded 

network centers, which continue to be supported 

by HRSA. After actively seeking a partner for 

ongoing, long-term funding, in 2007 ATHN 

secured support from Novo Nordisk, Inc. One of 

its first actions was establishing a new standardsbased electronic infrastructure, followed by the 

ATHNdata.quality.counts program, which 

provided funding to individual centers to increase 

data management capacity. Legacy data on more 

than 700,000 visits related to 101,610 patients 

was standardized and migrated to the new 

platform. Following the donation from Baxter 

Bioscience of the first electronic system for 
235

Chapter 24. Public-Private Partnerships

Case Example 64. Public-private partnerships 

for rare diseases (continued)

Proposed Solution (continued)

logging bleeds and infusions, patient self-reported 

data was integrated and made accessible to 

treatment centers in the same infrastructure.

In 2010, the ATHNdataset registry was created, 

with the goal of providing a consolidated, 

nationwide, longitudinal data resource that can be 

used for research, public health surveillance, and 

reporting. The registry is designed to study blood 

product safety, natural history of disease, 

effectiveness of care, prevention strategies, 

treatment patterns, and patient outcomes. 

Individual treatment centers execute Data Use 

Agreements and Business Associate Agreements, 

designating ATHN as the steward of data 

collected at the site, and allowing the data to be 

used for research purposes. Data are collected by 

the individual centers using a common electronic 

data capture system that also includes providerfocused Web-based tools and integrated 

electronic patient self-reporting systems. Data are 

stored as a limited data set (LDS) as defined by 

the HIPAA Privacy Rule, and the registry obtains 

authorization from patients for use of their data. 

Indiana Hemophilia and Thrombosis Center, 

Hemophilia of Georgia and other affiliated sites 

contributed to the design and testing of the 

registry, governance processes, and fundraising.

Results

Currently, 129 centers, representing about 12,154 

patients, are contributing data to the registry. 

ATHN provides a library of report templates and 

study management tools to participating centers, 

introducing efficiencies that allow the centers to 

devote more resources to providing care. ATHN 

itself has also received grants to conduct public 

health surveillance that leverages the registry data 

set, and recently received research funding from 

CDC to conduct a cross-sectional analysis on 

cardiovascular disease in older men with 

hemophilia. 

Key Point

PPPs can be especially valuable models for 

registries in rare diseases, where resources are 

scarce and research efforts may not be well 

coordinated. PPPs can provide the resources 

needed to create a common electronic 

infrastructure, data standards, and centralized 

data management and research functionalities, 

allowing disparate data sources to be combined 

for meaningful analysis. 

For More Information

Aschman DJ, Abshire T, Shapiro A, et al. 

Establishing community-based partnerships to 

create a standards-based information 

infrastructure. Am J Preventive Medicine. 2011; 

41(6) Suppl 4:S332-7.

Shapiro A, Peyvandi F, Soucie JM, et al. 

Knowledge and therapeutic gaps: A major public 

health problem highlighted in the rare bleeding 

disorders population. Am J Preventive Medicine. 

2001; 41(6) Suppl 4:S324-31.

Konkle B, Abshire T, Aschman D, et al. The 

ATHNdata set: A community resource for 

outcomes analysis, public health surveillance and 

research. Am J Hematol. 2012;87(Suppl.1):

S159-60.

Aschman D on behalf of ATHN Affiliates, Konkle 

B, Abshire T. The ATHNdatset: A U.S. based data 

set for outcomes analysis, public health 

surveillance and research. Haemophilia. 2012; 

18(Suppl.1):27.

Baker J, Riske B, Drake J, et al. US Hemophilia 

Treatment Center population trends 1990-2010: 

patient diagnosis, demographics, health services 

utilization. Haemophilia. 2013 Jan;19(1):21-6.
236

Section V. Special Applications in Patient Registries

References for Chapter 24

1. Food and Drug Administration Amendments Act 

of 2007., P. L. No. 110-85 (2007), Title VI. Sect. 

603. Critical Path Public-Private Partnerships.

2. Dreyer NA, Starzyk K, Wilcock K, et al. A global 

registry for understanding clinical presentation, 

treatment outcomes, and survival from human 

avian influenza. Bangkok International 

Conference on Avian Influenza; 2008 Jan 23; 

Bangkok: National Center for Genetic 

Engineering and Biotechology; 2008. p. 155.

3. Adisasmito W, Chan PKS, Lee N, et al. Global 

patient registry for influenza A/H5N1: 

strengthening results using multiple imputation. 

XIII International Symposium on Respiratory 

Viral Infections. Rome, Italy, March 13-16, 2011.

4. Adisasmito W, Chan PK, Lee N, et al. 

Effectiveness of antiviral treatment in human 

influenza A(H5N1) infections: analysis of a 

Global Patient Registry. J Infect Dis. 2010 Oct 

15;202(8):1154-60. PMID: 20831384.

5. Adisasmito W, Chan PK, Lee N, et al. 

Strengthening observational evidence for antiviral 

effectiveness in influenza A (H5N1). J Infect Dis. 

2011 Sep 1;204(5):810-1. PMID: 21844308.

6. Dreyer NA, Toovey S, Oner AF, et al. Investigating 

outbreaks of novel infectious disease: an 

international case study. Journal of Clinical 

Studies. 2013;i 5(2):52-53.

7. Schwamm LH, Fonarow GC, Reeves MJ, et al. 

Get With the Guidelines–Stroke is associated with 

sustained improvement in care for patients 

hospitalized with acute stroke or transient 

ischemic attack. Circulation. 2009 Jan 

6;119(1):107-15. PMID: 19075103.

8. Centers for Medicare & Medicaid Services. 

Guidance for the Public, Industry, and CMS Staff: 

National Coverage Determinations with Data 

Collection as a Condition of Coverage: Coverage 

with Evidence Development. July 12, 2006. 

Accessed December 26, 2013.

9. Centers for Medicare & Medicaid Services. Draft 

Guidance for the Public, Industry, and CMS Staff: 

Coverage with Evidence Development in the 

context of coverage decisions. http://www.cms.

gov/medicare-coverage-database/details/medicarecoverage-document-details.aspx?MCDId=23. 

Accessed February 20, 2013.

10. Lindsay MJ, Siegel BA, Tunis SR, et al. The 

National Oncologic PET Registry: expanded 

Medicare coverage for PET under coverage with 

evidence development. AJR Am J Roentgenol. 

2007 Apr;188(4):1109-13. PMID: 17377055.

11. Hammill S, Phurrough S, Brindis R. The National 

ICD Registry: now and into the future. Heart 

Rhythm. 2006 Apr;3(4):470-3. PMID: 16567298.

12. Reich MR, ed. Public-Private Partnerships for 

Public Health. Cambridge, MA: Harvard Center 

for Population and Development Studies; 2002.

13. Nikolic IA, Maikisch H. Health, Nutrition and 

Population (HNP) Discussion Paper. PublicPrivate Partnerships and Collaboration in the 

Health Sector. An Overview with Case Studies 

from Recent European Experience.Washington, 

D.C.: The International Bank for Reconstruction 

and Development/The World Bank; 2006.

14. Wagner JA, Prince M, Wright EC, et al. The 

Biomarkers Consortium: practice and pitfalls of 

open-source precompetitive collaboration. Clin 

Pharmacol Ther. 2010 May;87(5):539-42. 

PMID: 20407460.

15. Goodman M, Almon L, Bayakly R, et al. Cancer 

outcomes research in a rural area: a multiinstitution partnership model. J Community 

Health. 2009 Feb;34(1):23-32. PMID: 18850070.

16. Omobowale EB, Kuziw M, Naylor MT, et al. 

Addressing conflicts of interest in Public Private 

Partnerships. BMC Int Health Hum Rights. 

2010;10:19. PMID: 20615242. PMCID: 2914055.

17. Nishtar S. Public-private “partnerships” in 

health—a global call to action. Health Res Policy 

Syst. 2004 Jul 28;2(1):5. PMID: 15282025. 

PMCID: 514532.

18. Bloom GS, Frew D. Regulation of research 

through research governance: within and beyond 

NSW Health. N S W Public Health Bull. 2008 

Nov-Dec;19(11-12):199-202. PMID: 19126392.

19. Ciccone DK. Arguing for a centralized 

coordination solution to the public-private 

partnership explosion in global health. Glob 

Health Promot. 2010 Jun;17(2):48-51. 

PMID: 20587631.
237

Chapter 24. Public-Private Partnerships

20. McKee M, Edwards N, Atun R. Public-private 

partnerships for hospitals. Bull World Health 

Organ. 2006 Nov;84(11):890-6. 

PMID: 17143463. PMCID: 2627548.

21. HIV-related Public-Private Partnerships and 

Health Systems Strengthening. Joint United 

Nations Programme on HIV/AIDS (UNAIDS); 

Geneva, Switzerland; 2009.

Section VI 

Evaluating 

Registries

241

Chapter 25. Assessing Quality 

1. Introduction

As described throughout this guide, registries are 

created for many purposes, including scientific, 

clinical, and policy. Registries may also serve 

more than one purpose and potentially may add or 

change purposes over time. This leads to variations 

in design, operations, or quality assurance that are 

sometimes viewed as methodological 

inadequacies. It is not generally appreciated that 

the attributes important for some purposes may be 

less important for others. As a result, it is 

important to distinguish these purposes with 

respect to recommending particular practices.

For example, in describing a very rare disease or 

small subgroup of patients for whom there is little 

other information, some relevant data from a 

registry are better than no data. Further, even 

registries that fall short of including many of the 

essential elements of good registry practice 

described in this chapter may still provide valuable 

insights for some purposes. As a general rule, 

quality should be evaluated by elements that 

directly impact the ability of the registry to achieve 

its main objectives. In other words, a registry must 

be fit for its purpose.

Nonetheless, while all registries can provide useful 

information, there are levels of rigor that enhance 

validity and make the information from some 

registries more useful for guiding decisions than 

others. For example, there are certain practices that 

enhance the validity and reliability of registries 

intended to evaluate safety and comparative 

effectiveness in terms of design and confirmation 

of key outcomes. 

Prior to the publication of the first edition of this 

User’s Guide,1 no criteria had been developed to 

guide evaluation of registries. Research into the 

quality aspects of registries, whatever their 

purpose, remains relatively sparse, especially when 

compared with the rich information available to 

guide quality in clinical trials. The aim of this 

chapter is to provide a simple and user-friendly 

listing of attributes and practices that allow 

registries to be described and evaluated for their 

essential elements and enhancements in the 

context of the purpose for which they are 

conducted. Information is presented to help 

distinguish between—

• Essential registry practices that are desirable 

for every study.

• Practices that could enhance scientific rigor and 

that are particularly important for certain 

purposes, but may not be achievable because of 

practical constraints.

The items listed as “essential” elements of good 

practice are applicable to all patient registries. 

While it may not be practical or feasible to achieve 

all of the essential elements of good practice, it is 

useful to consider these characteristics in planning 

and evaluating registries. It is also important to 

remind readers that some of the fundamental 

differences between clinical trials and registries 

affect how quality is evaluated. For example, a 

clinical trial will have a rigorously maintained 

schedule of visits and assessment. A clinical trial 

patient who does not adhere to the schedule may 

be viewed as noncompliant with the protocol and 

potentially could be discontinued from the trial. In 

a registry, treatments and assessments may be 

recommended, but the registry participant who 

does not adhere to the schedule typically is 

allowed to remain in the registry, and this is 

considered good practice. Moreover, some argue 

that the kind of data produced by registries may be 

more valid for inferences needed in clinical 

decisionmaking because few exclusion criteria are 

used and inferences are drawn from measurements 

customarily used by clinicians.2

The information described in this User’s Guide, 

and particularly in this chapter, is also designed to 

be used in reporting registry study results, much as 

CONSORT (Consolidated Standards of Reporting 

Trials) guidelines have been used to improve 

reporting of clinical trials,3 and STROBE 

(Strengthening the Reporting of Observational 
242

Section VI. Evaluating Registries

Studies in Epidemiology) guidelines are being 

used for observational studies.4, 5

2. Defining Quality

This chapter has adapted a definition of quality 

that was developed for randomized controlled 

trials;6 the term is used to refer to the confidence 

that the design, conduct, and analysis of the trial or 

registry can be shown to protect against bias 

(systematic error) and errors in inference—that is, 

erroneous conclusions drawn from a study.7 As 

used here, quality refers both to the data and to the 

conclusions drawn from analyses of these data. For 

more information about the types of biases that 

can affect observational studies, as well as 

strategies for addressing and even avoiding these 

biases to the extent feasible, see Chapters 3 and 

13. For more information about bias, validity, and 

inference, readers are encouraged to consult 

epidemiologic textbooks.8-11

3. Measuring Quality

There are two major difficulties with assessing 

quality in registries:

• It can often be difficult to differentiate between 

the quality of the design, the study conduct, 

and the resultant information available.

• There is a lack of empirical evidence for 

evaluating the parameters purported to indicate 

quality and their impact on the utility of the 

evidence produced from registries.

Evaluations of the quality of any registry must be 

done with respect to the essential elements of the 

registry and those aspects that are important in the 

context of the registry’s main purpose and the 

purpose for which the data are being used. Both 

the internal and external validity of the data must 

be taken into account along with considerations of 

cost and feasibility.

The most commonly used method to assess quality 

of studies is a quality scale; there are numerous 

quality scales of varying length and complexity in 

existence, with strong opinions both for and 

against their use.,6, 8,12 Different scales emphasize 

distinctive dimensions of quality and therefore can 

produce disparate results when applied to a given 

study. In most situations, a summary score is 

derived by adding individual item scores, with or 

without weighting. This method, however, ignores 

whether the various items may lead to a bias 

toward the null (suggesting the erroneous 

interpretation that there is no effect) or tend to 

exaggerate the appearance of an effect when none 

really exists, and the final score produced does not 

reflect individual components.13 Furthermore, 

validation of the scales is difficult; studies have 

found wide variation in the scores for a particular 

study both by different reviewers and the same 

reviewers at different times.14

The approach suggested here is to undertake a 

quality component analysis, which involves an 

investigation of the components that may affect the 

results obtained. In the quality component 

analysis, a differentiation is made between two 

domains: research quality, which pertains to the 

scientific process (in this instance, the design and 

operational aspects of the registry), and evidence 

quality, which relates to the data/findings 

emanating from the research process.15-17

According to Lohr,18 “[t]he level of confidence 

one might have in evidence turns on the underlying 

robustness of the research and the analysis done to 

synthesize that research.” The individual items 

highlighted as essential elements of good practice 

and evidence quality can be used to guide the 

evaluation of registries, though there are no criteria 

as yet as to what proportion of elements must be 

satisfied in order to be considered “good enough” 

for various purposes. 

To select the quality components for analysis, 

several key elements identified in previous 

research studies, among many consulted, were 

Guidelines for Good Pharmacoepidemiology 

Practice,19 the ICH (International Conference on 

Harmonisation of Technical Requirements for 

Registration of Pharmaceuticals for Human Use) 

Guideline on Good Clinical Practice,20 the Council 

for International Organizations of Medical 

Sciences (CIOMS) International Guidelines for 

Ethical Review of Epidemiological Studies,21

standards developed for the conduct of registry 
243

Chapter 25. Assessing Quality

studies for patient-centered-outcomes research,22

various reports on rating scientific evidence from 

observational studies12, 23 and surveillance 

systems,24 Goldberg’s review of registry evaluation 

methods,25 the MOOSE (Meta-analysis of 

Observational Studies in Epidemiology) 

proposal,26 the EULAR (European League Against 

Rheumatism) task force on biologic registers,27

and Guidance for Reporting Observational Studies 

maintained by the Equator Network.28 Special 

purpose quality guidance documents, including the 

GRACE principles for observational studies of 

comparative effectiveness (see http://www.

graceprinciples.org),29 were also reviewed.

The results of the quality component analysis must 

be considered in terms of the registry purpose and 

in the context of the disease area (See Table 25–1.) 

For example, a disease-specific registry that has 

been designed to look at natural history should not 

be deemed low quality simply because it is not 

large enough to detect rare treatment effects. 

Table 25–1. Overview of registry purposes

• Determine clinical effectiveness, cost effectiveness, or comparative effectiveness of a test or treatment, 

including evaluating the acceptability of drugs, devices, or procedures for reimbursement.

• Measure or monitor safety and harm of specific products and treatments, including conducting comparative 

evaluation of safety and effectiveness.

• Measure or improve quality of care, including conducting programs to measure and/or improve the practice of 

medicine and/or public health.

• Assess natural history, including estimating the magnitude of a problem; determining the underlying incidence 

or prevalence rate; examining trends of disease over time; conducting surveillance; assessing service delivery 

and identifying groups at high risk; documenting the types of patients served by a health provider; and 

describing and estimating survival.

4. Quality Domains

The quality domains shown here reflect the 

domains described earlier in this User’s Guide and 

have also been adapted from work undertaken for 

clinical trials. For research, the quality domains 

are research design and processes and procedures, 

which address planning, design, data elements and 

data sources, and ethics, privacy, and governance. 

Table 25–2 shows the essential elements of good 

registry practice for research, and Table 25–3 

shows optional indicators of quality that may 

enhance registry validity and reliability, subject, of 

course, to feasibility and applicability. 

For evidence, the quality domains are external 

validity, internal validity, and analysis and 

reporting. Table 25–4 shows the essential elements 

of good registry practice for evidence, and Table 

25–5 shows optional further indicators of quality, 

including those important for selected purposes. It 

is important to weigh efforts to promote the 

accuracy and completeness of evidence in balance 

with the public health urgency of a problem, the 

types of interventions that are available, and the 

risks to public health from coming to a wrong 

conclusion. These lists of components are most 

likely incomplete, but the level of detail provided 

should be useful for high-level quality distinctions.

Most importantly, the essential elements of good 

practice, as well as the optional further indicators 

of quality depend, to a great extent, on the 

resources and budget available to support registrybased research.
244

Section VI. Evaluating Registries

Table 25–2. Research quality—essential elements of good practice for establishing and operating 

registries

Research Design

• Develop objectives and/or research questions (main and supporting, as needed).

• Identify the target population, eligibility, and inclusion and exclusion criteria. For registries where practice 

characteristics may influence outcome, seek to include diverse clinical practices. Where possible, a broad range 

of patients (few exclusion criteria) is desirable to facilitate subgroup analysis.

• Identify important personal identifiers, exposures, risk factors, and mitigating (or protective) factors, and seek 

those that are reasonably feasible to collect. Use the literature to inform the choice of data elements.

• Choose outcomes that are clinically meaningful and relevant to patients and to the medical community for 

decisionmaking. Define patient outcomes clearly, especially for complex conditions or outcomes that may not 

have uniformly established criteria (e.g., define “injection site reaction” in operational terms). Consider whether 

these outcomes will be collected from medical care providers, patients, or other observers. 

• Use validated scales and tests when such tools exist for the purpose needed.

• Understand the followup time required to detect events of interest and whether or not the objective is feasible to 

achieve. Ensure that the followup time planned is adequate to address the main objective.

• Plan the main analyses, including specification of exposure and effect measures.

• Consider the size required to detect an effect should one exist, or to achieve a desired level of precision. 

Consider whether or not the sample size requirement can be achieved within the available time and budget 

constraints.

• Consider the most efficient and reliable means to consistently collect data of sufficient quality to meet the 

registry’s purpose and whether existing data can be used to supplement or minimize active data collection.

• Plan to report safety events according to regulatory requirements.

• Plan the data analysis to address the key objectives or research questions, including what comparative 

information, if any, will be used to support study hypotheses or objectives.

Processes and Procedures

• At the outset, reach agreement on key aspects of the registry and document them, including the goals, design, 

target population, methods for data collection, data elements and sources, high-level data management and data 

quality review, and how human subjects will be protected. It may be helpful for stakeholders to have input to 

ensure clinical relevance and feasibility.

• Establish a process for documenting any modifications to the plan, since the main objective may change over 

time as knowledge accumulates, and the plan for data collection and followup may need to be adapted.

• Carefully consider the issues of protection of human subjects—including privacy, informed consent, data 

security, and study ethics—and address them in accordance with local, national, and international regulations. 

Obtain review and approval by any required oversight committees (e.g., ethics committee, privacy committee, 

or institutional review board, as applicable). If linkage of registry data to other sources is planned, consider the 

additional issues of protection of human subjects.

• Define the role of any external sponsor, including data access and use.

• Provide clear, operational definitions of outcomes and other data elements. Establish a data and coding 

dictionary to provide explicit definitions and to describe coding used. Whenever possible, use standardized 

data dictionaries, such as the International Classification of Diseases, and use coding that is consistent with 

nationally or internationally approved coding systems to promote comparability of information among studies.

• If linkage of registry data to other sources is planned, consider any additional requirements that may influence 

successful linkage, such as selection of data elements and definitions used.

• Plan subject and physician recruitment targets and methods to achieve those targets, and plan the means for 

monitoring enrollment and retention.
245

Chapter 25. Assessing Quality

Processes and Procedures (continued)

• Plan to expend reasonable efforts to ensure that appropriate patients are enrolled systematically and followed in 

as unbiased a manner as possible.

• Identify appropriate personnel and facilities, including those for secure data storage. Identify the individual(s) 

responsible for the integrity of the data, computerized and hard copy, and make sure these individuals have the 

training and experience to perform the assigned tasks.

• Develop standard instructions for use in training data collectors. For safety studies, create a process for 

identifying and reporting serious events that is consistent with regulatory requirements. Plan training for study 

personnel about how to identify serious events, including:

– Asking about complaints or adverse events in a manner that is clear and specific (e.g., solicited vs. 

unsolicited).

– Knowing if and how information should be reported to manufacturers and health authorities.

• Create a quality assurance plan that addresses data editing and verification. Plan an approach for handling 

missing data (e.g., go back and collect those data or make a plan as to how “missing” data will be coded in the 

data files.)

• Anticipate how study results will be communicated on completion. 

Table 25–2. Research quality—essential elements of good practice for establishing and operating 

registries (continued)

Table 25–3. Research quality—further indicators of quality for establishing and operating registries 

(optional)

Research Design

• Formalize the study plan as a research protocol.

• For comparative effectiveness and safety— 

– Use concurrent comparators, since they may offer an advantage over historical or external comparison 

groups, especially in situations where treatments are evolving rapidly. The comparator cohort should be as 

similar as possible to the exposed cohort, aside from the exposure under study, and should reflect current 

clinical practice.

– Use formal statistical calculations to specify the number of patients or patient-years of observation needed 

to measure an effect with a certain level of precision or to meet a specified statistical power to detect an 

effect should one exist, although the desired size may not be achievable within the practical study 

constraints. Temper considerations about precision and power with budgetary and feasibility constraints, 

while also giving heed to the importance of conducting research in areas where little information exists.

• Develop formal analysis plans.

• Collect information to permit linkage (and therefore validation) with external databases such as the National 

Death Index, electronic health records, or claims data sets, as appropriate. 

• Post the registry on a public registry of registries (e.g., at the Registry of Patient Registries) or trials listing 

(e.g., at ClinicalTrials.gov).

Processes and Procedures

• Undertake a feasibility study or pilot test (e.g., when studying hard-to-reach populations, when sensitive data 

are sought, and when critical registry methods are new or have not otherwise been tested).

• When capturing composite scores, collect and record core components, if possible.

• Collect information on start and stop dates of treatments of interest and dose (if relevant) or other means to 

discriminate between high and low exposure.

• Use similar methods of followup for exposure and comparison cohorts, and for all subjects in each cohort, to 

the extent feasible. 
246

Section VI. Evaluating Registries

Table 25–3. Research quality—further indicators of quality for establishing and operating registries 

(optional) (continued)

Processes and Procedures (continued)

• To enhance transparency, consider using an advisory board, particularly a board that includes members 

who are external to the clinician, center, or company that sponsors the research. An advisory board or 

steering committee can promote clinical and public health relevance and may assist with governance and 

communication. If using an advisory board, consider rotating membership and/or term limits.

• Specify publication policies in advance of collecting data and reevaluate at regular intervals (e.g., annually).

• Develop a plan for stopping or transitioning the registry, including any archiving or transferring of data and 

notifying participants, as appropriate.

• Consider when and how to allow third parties access to data, if feasible, and the process for any such data 

access.

• For safety and comparative effectiveness— 

– Data collection methods should not limit site participation to the extent that the representativeness of sites is 

compromised. Although a single method of data collection is most efficient, multiple methods of data 

collection may be desirable for some purposes.

– Loss to followup should be monitored and characterized to ensure that followup is sufficiently complete for 

the main objective and to see if there is differential loss to followup by characteristics that may affect the 

likelihood of achieving the main objectives. Reasonable resources should be devoted to minimizing loss to 

followup.

– Quality assurance (QA) may include review or monitoring of a sample of data and/or data review by an 

adjudication committee for complex conditions or endpoints for which established procedures and/or 

coding are not used. For most purposes, a risk-based strategy should be used for QA, focused on detecting 

and quantifying the most likely causes of error and the types of error most likely to affect the registry 

purpose, with QA activities adapted based on observed performance (e.g., increase QA for sites that appear 

to be having difficulty in study conduct or data entry).

• For safety, comparative effectiveness, and quality improvement— 

– Maintain appropriate documentation, such as an audit trail, to ensure proper handling of information and to 

support transparency. 

– Establish processes and standards for creating analytic data files and maintaining such files to support 

publications and presentations, since registry analyses may be performed on live data (data that may change 

as the registry continues to collect and verify information through various quality control procedures) or on 

data that have been locked and have undergone formal review and editing. 

– Use open standard approaches to interoperability when health information systems are used for active data 

collection to permit more efficient collection of data from multiple systems.
247

Chapter 25. Assessing Quality

Table 25–4. Essential indicators of good evidence quality for registries

External Validity

• External validity was demonstrated by showing that registry participants were similar to the target population 

and, to the extent feasible, efforts were devoted to minimizing selection bias (e.g., rules for sequential 

enrollment were developed and codified in a manner that worked for all sites).

• Completeness of information on eligible patients was evaluated and described. 

Internal Validity

• For safety studies, a clear and specific approach was used (e.g., solicited vs. unsolicited) to ask about 

complaints or adverse events.

• Necessary information was collected for relevant key exposures, risk factors, and mitigating or protective 

factors.

• Exposure data used to support the main research questions were as specific as possible. For example, a specific 

product, including manufacturer, was identified to the extent feasible.

• Data checks were employed using range and consistency checks.

• For comparative effectiveness and safety—

– Followup period was reasonably sufficient to capture the main outcomes of interest.

– Comparators reflect current practice.

– A sample of data was validated with patient records, (e.g., 10–20% of patients’ records were compared with 

registry data). 

– Followup was reasonably complete for the registry purpose.

Analysis and Reporting

• The report describes the methods, including target population and selection of study subjects, compliance with 

applicable regulatory rules and regulations, data collection methods, any transformation of variables and/or 

construction of composite endpoints, how missing data were handled, statistical methods for data analysis, and 

any circumstances that may have affected the quality or integrity of the data. The information was reported with 

enough detail to allow replication of the methods in another study.

• Results were reported for all the main objectives, including estimates of effect for each.

• Accepted analytic techniques were used; these may have been augmented by new or novel approaches.

• Followup time was described to enable assessment of the impact of the observation period on the conclusions 

drawn.

• The role and impact of missing data and potential confounding factors were considered.

• The report includes a clear statement of any conclusions drawn from the analysis of the registry’s main 

objectives and any implications of study results. Alternative explanations for the observed results were 

considered; a variety of factors, including the strength of the association, biases, and temporal relations, were 

considered before drawing any causal inferences. The practice of making inferences about causation largely on 

the outcome of tests of statistical significance is discouraged.

• The consistency of results was compared and contrasted with other relevant research.
248

Section VI. Evaluating Registries

External Validity

• Eligibility (inclusion and exclusion criteria) was confirmed on enrollment.

• Selection bias was evaluated by describing the representativeness of the actual population in terms of how it 

was selected, how well the characteristics of the actual population match those of the target population, and to 

whom the results apply.

Internal Validity

• Results that can be confirmed by an unbiased observer—such as death, test results, and scores from validated 

measures for patient-reported results or clinical rating scales—were used to enhance accuracy and reliability.

• For safety and quality reporting (to third parties)— 

– Potential sources of error relating to accuracy and falsification were rigorously evaluated and quantified to 

the extent feasible (e.g., through database and/or site reviews).

– Reproducibility of coding was evaluated.

Analysis and Reporting

• Validated analytic tools were used for the main analysis. For example, commercially available analytic packages 

were used. The data elements used in any models were described. 

• Effect estimates among meaningful subgroups were described.

• Appropriate statistical techniques were used to address confounding.

• Sensitivity analyses were used to examine and quantify the effect on the association between the a priori 

exposure of interest and the outcome(s) by, for example, varying the definitions of exposure, potential 

confounders, and outcomes.

• For safety studies, the risks and/or benefits of products, devices, or processes under study were quantitatively 

evaluated beyond simply evaluating statistical significance (e.g., rates, proportions, and/or relative risks, as well 

as confidence intervals, were reported).

• For studies of comparative effectiveness and safety, contemporaneous data were collected for one or more 

comparison groups that reflect current clinical practice, when other reasonably accurate and relevant 

comparative data were not available.

Table 25–5. Further indicators of registry evidence quality (optional)
249

Chapter 25. Assessing Quality

References for Chapter 25

1. Gliklich RE, Dreyer NA, eds. Registries for 

Evaluating Patients Outcomes: A User’s Guide. 

Rockville, MD: Agency for Healthcare Research 

and Quality; April 2007.

2. Dreyer NA, Garner S. Registries for robust 

evidence. JAMA. 2009 Aug 19;302(7):790-1. 

PMID: 19690313.

3. Moher D, Schulz KF, Altman D, et al. The 

CONSORT statement: revised recommendations 

for improving the quality of reports of parallelgroup randomized trials. JAMA. 2001 Apr 

18;285(15):1987-91. PMID: 11308435.

4. von Elm E, Altman DG, Egger M, et al. The 

Strengthening the Reporting of Observational 

Studies in Epidemiology (STROBE) statement: 

guidelines for reporting observational studies. Ann 

Intern Med. 2007 Oct 16;147(8):573-7. 

PMID: 17938396.

5. Vandenbroucke JP, von Elm E, Altman DG, et al. 

Strengthening the Reporting of Observational 

Studies in Epidemiology (STROBE): explanation 

and elaboration. PLoS Med. 2007 Oct 

16;4(10):e297. PMID: 17941715. 

PMCID: 2020496.

6. Moher D, Jadad AR, Nichol G, et al. Assessing the 

quality of randomized controlled trials: an 

annotated bibliography of scales and checklists. 

Control Clin Trials. 1995 Feb;16(1):62-73. 

PMID: 7743790.

7. Rothman K, ed. Causal inference. Chestnut Hill, 

MA: Epidemiology Resources, Inc.; 1988.

8. Rothman K, Greenland S. Modern Epidemiology. 

2nd ed. Philadelphia: Lippincott Williams & 

Wilkins; 1998. pp. 135-61.

9. Weiss NS. Clinical epidemiology: The study of the 

outcome of illness. New York: Oxford University 

Press; 2006.

10. Fletcher RH, Fletcher SW, Fletcher EH. Clinical 

epidemiology: the essentials. Baltimore, MD: 

Williams & Wilkins; 1996.

11. Lash TL, Fox MP, Fink AK. Applying quantitative 

bias analysis to epidemiologic data. New York, 

NY: Springer; 2009.

12. West S, King V, Carey TS, et al. Systems to Rate 

the Strength of Scientific Evidence. Evidence 

Report/Technology Assessment No. 47 (Prepared 

by the Research Triangle Institute-University of 

North Carolina Evidence-based Practice Center 

under Contract No. 290-97-0011). AHRQ 

Publication No. 02-E016. Rockville, MD: Agency 

for Healthcare Research and Quality; 2002. 

http://www.effectivehealthcare.ahrq.gov/reports/

final.cfm.

13. Greenland S. Invited commentary: a critical look 

at some popular meta-analytic methods. Am J 

Epidemiol. 1994 Aug 1;140(3):290-6. 

PMID: 8030632.

14. Berkman ND, Lohr KN, Morgan LC, et al. 

Reliability Testing of the AHRQ EPC Approach to 

Grading the Strength of Evidence in Comparative 

Effectiveness Reviews. Methods Research Report. 

(Prepared by RTI International–University of 

North Carolina Evidence-based Practice Center 

under Contract No. 290-2007-10056-I.) AHRQ 

Publication No. 12-EHC067-EF. Rockville, MD: 

Agency for Healthcare Research and Quality. May 

2012. http://www.effectivehealthcare.ahrq.gov/

reports/final.cfm.

15. National Center for the Dissemination of 

Disability Research. http://www.ncddr.org. 

Accessed August 20, 2012.

16. Mosteller F, Boruch R, eds. Evidence matters: 

randomized trials in education research. 

Washington, DC: The Brookings Institute; 2002.

17. Shavelson RJ, Towne L, eds. Scientific research in 

education. Washington, DC: National Research 

Council, National Academy Press; 2002.

18. Lohr KN. Rating the strength of scientific 

evidence: relevance for quality improvement 

programs. Int J Qual Health Care. 2004 

Feb;16(1):9-18. PMID: 15020556.

19. Ispe. Guidelines for good pharmacoepidemiology 

practices (GPP). Pharmacoepidemiol Drug Saf. 

2008 Feb;17(2):200-8. PMID: 17868186.

20. European Medicines Agency. ICH Topic E 6 (R1) 

Guideline for Good Clinical Practice. http://www.

ema.europa.eu/docs/en_GB/document_library/

Scientific_guideline/2009/09/WC500002874.pdf. 

Accessed December 26, 2013.
250

Section VI. Evaluating Registries

21. CIOMS. 1991 International Guidelines for Ethical 

Review of Epidemiological Studies. http://www.

cioms.ch/publications/guidelines/1991_texts_of_

guidelines.htm. Accessed August 20, 2012.

22. Gliklich R, Dreyer NA, Leavy MB, et al. 

Standards in the conduct of registry studies for 

patient-centered outcomes research. Prepared for 

the Patient-Centered Outcomes Research Institute, 

Washington, DC, 2012. http://www.pcori.org/

assets/Standards-in-the-Conduct-of-RegistryStudies-for-Patient-Centered-Outcomes-Research.

pdf. Accessed August 21, 2012.

23. Shea BJ, Hamel C, Wells GA, et al. AMSTAR is a 

reliable and valid measurement tool to assess the 

methodological quality of systematic reviews. 

J Clin Epidemiol. 2009 Oct;62(10):1013-20. 

PMID: 19230606.

24. Centers for Disease C. Guidelines for evaluating 

surveillance systems. MMWR Morb Mortal Wkly 

Rep. 1988 May 6;37 Suppl 5:1-18. 

PMID: 3131659.

25. Goldberg J, Gelfand HM, Levy PS. Registry 

evaluation methods: a review and case study. 

Epidemiol Rev. 1980;2:210-20. PMID: 7000537.

26. Stroup DF, Berlin JA, Morton SC, et al. Metaanalysis of observational studies in epidemiology: 

a proposal for reporting. Meta-analysis Of 

Observational Studies in Epidemiology (MOOSE) 

group. JAMA. 2000 Apr 19;283(15):2008-12. 

PMID: 10789670.

27. Dixon WG, Carmona L, Finckh A, et al. EULAR 

points to consider when establishing, analysing 

and reporting safety data of biologics registers in 

rheumatology. Ann Rheum Dis. 2010 

Sep;69(9):1596-602. PMID: 20525843.

28. Equator Network. http://www.equator-network.

org/library/. Accessed December 26, 2013.

29. Dreyer NA, Schneeweiss S, McNeil BJ, et al. 

GRACE principles: recognizing high-quality 

observational studies of comparative effectiveness. 

Am J Manag Care. 2010 Jun;16(6):467-71. 

PMID: 20560690.
Contributors 

and 

Reviewers

253

* Chapter Coordinator

Contributors

The findings and conclusions in this document are those of the contributors and reviewers. The 

contributors and reviewers participated as individuals and not necessarily as representatives of their 

organizations. No statement in this document should be construed as an official position of any 

organization represented here.

Chapter 1. Patient Registries

*Richard E. Gliklich, M.D.

David Matchar, M.D.

Chapter 2. Planning a Registry

Richard E. Gliklich, M.D.

Thomas Gross, M.D., M.P.H.

*Joel M. Kremer, M.D.

C. Geoffrey McDonough

Chapter 3. Registry Design

*Nancy A. Dreyer, M.P.H., Ph.D.

Hesha J. Duggirala, Ph.D., M.P.H.

Gregory Samsa, Ph.D.

Paul Stang, Ph.D.

Priscilla Velentgas, Ph.D.

Chapter 4. Data Elements for Registries

Marc L. Berger, M.D.

Irene Katzan, M.D., M.S.

*Michelle Bertagna Leavy, M.P.H.

Christina DeFilippo Mack, Ph.D., M.S.P.H.

John Spertus, M.D., M.P.H., FACC, FAHA

Chapter 5. Use of Patient-Reported Outcomes in 

Registries

*Amy P. Abernethy, M.D.

Ethan Basch, M.D., M.Sc.

Kimary Kulig, Ph.D., M.P.H.

Benjamin Miriovsky, M.D.

Chapter 6. Data Sources for Registries

Paul Eggers, Ph.D.

David J. Malenka, M.D.

Eric Gemmen, M.A.

*Lisa Griffin Vincent, M.A., Ph.D.

Chapter 7. Principles of Registry Ethics, Data 

Ownership, and Privacy

Susan Adams, J.D.

Andy DeMayo, J.D.

Holly L. Howe, Ph.D.

David W. Kaufman, Sc.D.

Stuart Kim, J.D., M.S.

Debra R. Lappin, J.D.

*Jane Hyatt Thorpe, J.D.

Marianne Ulcickas Yood, D.Sc., M.P.H.

Chapter 8. Informed Consent for Registries

*Jessica Berg, J.D., M.P.H.

Laura D. Buccini, Dr.P.H., M.P.H.

Catherine Koepper

Chapter 9. Protecting Data: Confidentiality and 

Legal Concerns of Providers, Manufacturers, and 

Health Plans

*Jane Hyatt Thorpe, J.D.

Christopher Weiser, J.D., M.P.H.

Chapter 10. Recruiting and Retaining Participants 

in the Registry

*Fred Anderson, Ph.D.

Ralph G. Brindis, M.D., M.P.H., MACC, FSCAI

Jeanette M. Broering, R.N., Ph.D., M.P.H.

Richard E. Gliklich, M.D.

Robert E. Harbaugh, M.D., FAANS, FACS, FAHA

Chapter 11. Data Collection and Quality Assurance

*Richard E. Gliklich, M.D.

Shukri F. Khuri, M.D.

Neal Mantick, M.S.

Barry Mirrer, M.S., CMQ/OE

Neha Sheth, Pharm.D.

Joseph Terdiman, M.D., Ph.D.

Chapter 12. Adverse Event Detection, Processing, 

and Reporting

*Nancy A. Dreyer, M.P.H., Ph.D.

Richard E. Gliklich, M.D.

Neha Sheth, Pharm.D.

Anne E. Trontell, M.D., M.P.H.

Chapter 13. Analysis, Interpretation, and Reporting 

of Registry Data To Evaluate Outcomes

*Jaclyn L. F. Bosco, Ph.D., M.P.H.

Wei Dong, M.D., Ph.D.

Nancy A. Dreyer, M.P.H., Ph.D.

Harlan M. Krumholz, M.D., S.M.

Eric Peterson, M.D., M.P.H.

Gillian D. Sanders, Ph.D.

Dale R. Tavris, M.D., M.P.H.

Priscilla Velentgas, Ph.D.
254

Chapter 14. Modifying and Stopping Registries

J. Alexander Cole, D.Sc., M.P.H.

Alexander Elbert, Ph.D.

Joanna Haas, M.D., M.Sc.

Kathleen Hewitt, M.S.N., R.N., A.A.C.C. 

Michelle Bertagna Leavy, M.P.H.

*Bruce C. Marshall, M.D.

*Kenneth J. Rothman, Dr.P.H.

Michelle Tarver, M.D., Ph.D.

Chapter 15. Interfacing Registries With Electronic 

Health Records

Landen Bain

Richard E. Gliklich, M.D.

*Daniel Levy, M.S.

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

Susan Adams, J.D.

Maya A. Bernstein, J.D.

Andy DeMayo, J.D.

*Stephen E. Fienberg, Ph.D., FRSC 

Melissa M. Goldstein, J.D.

Alan F. Karr, Ph.D.

Bradley Malin, Ph.D.

*Sara Rosenbaum, J.D.

Chapter 17. Managing Patient Identity Across Data 

Sources

Laura Khurana, M.P.H.

*Daniel Levy, M.S.

Kumar Sivaraman

Chapter 18. Analysis of Linked Registry Data Sets

*Timothy L. Lash, D.Sc., M.P.H.

Henrik Toft Sørensen, M.D., Ph.D., Dr.Med.Sci.

Brian D. Bradbury, M.A., D.Sc.

Dale R. Tavris, M.D., M.P.H.

Chapter 19. Use of Registries in Product Safety 

Assessment

Stella Blackburn, M.B., B.S., M.Sc., FRCP (Edin), 

FISPE

Nancy A. Dreyer, M.P.H., Ph.D.

*Kathryn Starzyk, Sc.M.

Chapter 20. Rare Disease Registries

David C. Dale, M.D.

Stephen C. Groft, Pharm.D.

Melanie J. Harrison, M.D., M.S.C.E.

Rachel L. Richesson, Ph.D., M.P.H.

Yaffa Rubinstein, Ph.D.

*Kathryn Starzyk, Sc.M.

Charlie Strange, M.D.

Chapter 21. Pregnancy Registries

Christina Chambers, Ph.D., M.P.H.

Deborah Covington, Dr.P.H.

Janet D. Cragan, M.D., M.P.H.

Sara A. Ephross, Ph.D.

*Sonia Hernández-Díaz, M.D., M.P.H., Dr.P.H.

Melissa S. Tassinari, Ph.D.

Chapter 22. Quality Improvement Registries

Anthony L. Asher, M.D., FACS

*Richard E. Gliklich, M.D.

Adrian F. Hernandez, M.D., M.H.S.

Michelle Bertagna Leavy, M.P.H.

Keith Mandel, M.D.

Lee H. Schwamm, M.D., FAHA

David M. Shahian, M.D.

Chapter 23. Registries for Medical Devices

Thomas Gross, M.D., M.P.H.

Richard E. Kuntz, M.D., M.Sc.

Christina DeFilippo Mack, Ph.D., M.S.P.H.

*Danica Marinac-Dabic, M.D., Ph.D.

Sharon-Lise T. Normand, Ph.D.

Elizabeth Paxton, M.A.

Art Sedrakyan, M.D., Ph.D.

Chapter 24. Public-Private Partnerships

*Nancy A. Dreyer, M.P.H., Ph.D.

Benjamin C. Eloff, Ph.D.

James K. Kirklin, M.D.

David C. Naftel, Ph.D.

Stephen Toovey, M.D., Ph.D.

Chapter 25. Assessing Quality

*Nancy A. Dreyer, M.P.H., Ph.D.

Sarah Garner, Ph.D., B.Pharm., M.R.Pharm.S.

Registries for Evaulating Patient Outcomes

* Chapter Coordinator
255

 Reviewers

Chapter 1. Patient Registries

Kathleen Blake, M.D., M.P.H.

Emma James, Ph.D.

Bruce C. Marshall, M.D.

Lawrence H. Muhlbaier, Ph.D.

Anne E. Trontell, M.D., M.P.H.

Kelly Trout, B.S.N., R.N.

Chapter 2. Planning a Registry

Hesha J. Duggirala, Ph.D., M.P.H.

Paul Eggers, Ph.D.

Kathleen Hewitt, M.S.N., R.N., A.A.C.C., CPHQ

Solomon Iyasu, M.D., M.P.H.

Bruce C. Marshall, M.D.

Deven McGraw, J.D., M.P.H., L.L.M.

Lawrence H. Muhlbaier, Ph.D.

Paul Stang, Ph.D.

Anne E. Trontell, M.D., M.P.H.

Chapter 3. Registry Design

Jaclyn L. F. Bosco, Ph.D., M.P.H.

Melanie J. Harrison, M.D., M.S.C.E.

Kathleen Hewitt, M.S.N., R.N., A.A.C.C., CPHQ

Solomon Iyasu, M.D., M.P.H.

Theodore Lystig, Ph.D.

Bruce C. Marshall, M.D.

Lawrence H. Muhlbaier, Ph.D.

Dimitrios Pappas, M.D., M.P.H.

Chapter 4. Data Elements for Registries

Deven McGraw, J.D., M.P.H., L.L.M.

Lawrence H. Muhlbaier, Ph.D.

Chapter 5. Use of Patient-Reported Outcomes in 

Registries

Jeanette M. Broering, R.N., Ph.D., M.P.H.

Lori Frank, Ph.D.

Eric Gemmen, M.A.

Deven McGraw, J.D., M.P.H., L.L.M.

Albert Wu, M.D., M.P.H.

Chapter 6. Data Sources for Registries

Kathleen Blake, M.D., M.P.H.

David C. Dale, M.D.

Deven McGraw, J.D., M.P.H., L.L.M.

Lawrence H. Muhlbaier, Ph.D.

Dimitrios Pappas, M.D., M.P.H.

Paul Stang, Ph.D.

Chapter 7. Principles of Registry Ethics, Data 

Ownership, and Privacy

Kathleen Blake, M.D., M.P.H.

Paul Eggers, Ph.D.

Marissa Gordon-Nguyen, J.D.

Julia G. Gorey, J.D.

Christina M. Heide, J.D.

Solomon Iyasu, M.D., M.P.H.

Wendy Lipworth, M.D., Ph.D.

Deven McGraw, J.D., M.P.H., L.L.M.

Lawrence H. Muhlbaier, Ph.D.

Chapter 8. Informed Consent for Registries

Paul S. Appelbaum, M.D.

Jeanette M. Broering, R.N., Ph.D., M.P.H.

J. Alexander Cole, D.Sc., M.P.H.

Theresa Cullen, M.D., M.S.

Andy DeMayo, J.D.

Douglas S. Diekema, M.D., M.P.H.

Paul Eggers, Ph.D.

Barbara Evans, J.D., Ph.D.

Richard E. Gliklich, M.D.

Julia G. Gorey, J.D.

Emma James, Ph.D.

Deven McGraw, J.D., M.P.H., L.L.M.

Yaffa Rubinstein, Ph.D.

Michelle Tarver, M.D., Ph.D.

Jennifer Weisman, Ph.D.

David S. Wendler, M.A., Ph.D.

The findings and conclusions in this document are those of the contributors and reviewers. The 

contributors and reviewers participated as individuals and not necessarily as representatives of their 

organizations. No statement in this document should be construed as an official position of any 

organization represented here.
256

Chapter 9. Protecting Data: Confidentiality and 

Legal Concerns of Providers, Manufacturers, and 

Health Plans

Andy DeMayo, J.D.

Nancy A. Dreyer, M.P.H., Ph.D.

Mark J. Fox

Danica Marinac-Dabic, M.D., Ph.D.

Deven McGraw, J.D., M.P.H., L.L.M.

James G. Snell, J.D.

Chapter 10. Recruiting and Retaining Participants 

in the Registry

Kathleen Blake, M.D., M.P.H.

Eric Gemmen, M.A.

Kelly Trout, B.S.N., R.N.

Chapter 11. Data Collection and Quality Assurance

Theodore Lystig, Ph.D.

Bruce C. Marshall, M.D.

Deven McGraw, J.D., M.P.H., L.L.M.

Paul D. Varosy, M.D.

Chapter 12. Adverse Event Detection, Processing, 

and Reporting

Jaclyn L. F. Bosco, Ph.D., M.P.H.

Julia G. Gorey, J.D.

Vandana Menon, M.D.

Chapter 13. Analysis, Interpretation, and Reporting 

of Registry Data To Evaluate Outcomes

Marcella Blackledge, B.A.

Rosemarie Hakim, Ph.D.

Theodore Lystig, Ph.D.

Aaron B. Mendelsohn, Ph.D., M.P.H.

Chapter 14. Modifying and Stopping Registries

Marcella Blackledge, B.A.

Jack L. Cronenwett, M.D.

David C. Dale, M.D.

Andy DeMayo, J.D.

Paul Eggers, Ph.D.

Emma James, Ph.D.

Deven McGraw, J.D., M.P.H., L.L.M.

Yaffa Rubinstein, Ph.D.

Kelly Trout, B.S.N., R.N.

Chapter 15. Interfacing Registries With Electronic 

Health Records

Kathleen Blake, M.D., M.P.H.

Deven McGraw, J.D., M.P.H., L.L.M.

David C. Naftel, Ph.D.

Chapter 16. Linking Registry Data With Other Data 

Sources To Support New Studies

Deven McGraw, J.D., M.P.H., L.L.M.

Patricia Russo, Ph.D.

Chapter 17. Managing Patient Identity Across Data 

Sources

Leonard D’Avolio, Ph.D.

T. Bruce Ferguson, M.D., FAHA

Michael Kahn, M.D., Ph.D.

Theodore Lystig, Ph.D.

Deven McGraw, J.D., M.P.H., L.L.M.

J. Marc Overhage, M.D., Ph.D.

Kelly Trout, B.S.N., R.N.

Chapter 18. Analysis of Linked Registry Data Sets

Joseph A.C. Delaney, Ph.D.

Mark J. Fox

Brian C. Sauer, Ph.D., M.S.

Art Sedrakyan, M.D., Ph.D.

Paul Stang, Ph.D.

Priscilla Velentgas, Ph.D.

Chapter 19. Use of Registries in Product Safety 

Assessment

Kathleen Blake, M.D., M.P.H.

Christina DeFilippo Mack, Ph.D., M.S.P.H.

David C. Naftel, Ph.D.

Paul D. Varosy, M.D.

Chapter 20. Rare Disease Registries

Diane J. Aschman, M.S.

Marcella Blackledge, B.A.

Kathleen Blake, M.D., M.P.H.

J. Alexander Cole, D.Sc., M.P.H.

Barrie J. Hurwitz, M.B., MRCP

Emma James, Ph.D.

Robert A. Jenders, M.D., M.S., FACP, FACMI

Catherine Koepper

Deven McGraw, J.D., M.P.H., L.L.M.

Chapter 21. Pregnancy Registries

Yvonne Geissbuehler, Ph.D.

Solomon Iyasu, M.D., M.P.H.

Emma James, Ph.D.

Christina DeFilippo Mack, Ph.D., M.S.P.H.

Judy A. Staffa, Ph.D., R.Ph.

Hugh Tilson, M.D., M.P.H., Dr.P.H.

Registries for Evaulating Patient Outcomes
257

Chapter 22. Quality Improvement Registries

Kathleen Blake, M.D., M.P.H.

Joseph P. Drozda, Jr., M.D., FACC

Claudia Grossmann, Ph.D.

Clifford Ko, M.D.

J. Michael McGinnis, M.D., M.P.P.

David C. Naftel, Ph.D.

Richard L. Prager, M.D.

Karen Richards

Hugh Tilson, M.D., M.P.H., Dr.P.H.

Chapter 23. Registries for Medical Devices

Jay Crowley

Hesha J. Duggirala, Ph.D., M.P.H.

Julian M. Goldman, M.D.

Barbara Muth

David C. Naftel, Ph.D.

Michael Steinbuch, Ph.D.

Paul D. Varosy, M.D.

Chapter 24. Public-Private Partnerships

Kathleen Blake, M.D., M.P.H.

Mark J. Fox

Rosemarie Hakim, Ph.D.

Danica Marinac-Dabic, M.D., Ph.D.

Deven McGraw, J.D., M.P.H., L.L.M.

Marissa A. Miller, D.V.M., M.P.H.

Elizabeth J. Swain

Hugh Tilson, M.D., M.P.H., Dr.P.H.

Chapter 25. Assessing Quality

Kathleen Blake, M.D., M.P.H.

Leonard D’Avolio, Ph.D.

Benjamin C. Eloff, Ph.D.

Melanie J. Harrison, M.D., M.S.C.E.

Theodore Lystig, Ph.D.

Danica Marinac-Dabic, M.D., Ph.D.

Barry Mirrer, M.S., CMQ/OE

David C. Naftel, Ph.D.

We would also like to acknowledge the reviewers who 

contributed to previous editions of the user’s guide. 

These reviewers are listed below in alphabetical order:

Susan Adams, J.D.

Steven J. Atlas, M.D., M.P.H.

Stephen W. Bernstein, J.D.

Jeffrey Brown, Ph.D.

Daniel Caños

Michael A. Carome, M.D.

Patrice Desvigne-Nickens, M.D.

Rachel S. Dokholyan, M.P.H.

Pamela S. Douglas, M.D., FACC

Nancy A. Dreyer, M.P.H., Ph.D.

Brenda K. Edwards, Ph.D.

Paul Eggers, Ph.D.

Jonathan S. Einbinder, M.D., M.P.H.

Shamiram Feinglass, M.D., M.P.H.

Leslye K. Fitterman, Ph.D.

Sally Flanzer, Ph.D., CIP

Gregg C. Fonarow, M.D.

Sarah Garner, Ph.D., B.Pharm., M.R.Pharm.S.

Richard E. Gliklich, M.D.

Marie R. Griffin, M.D., M.P.H.

Thomas Gross, M.D., M.P.H.

Joanna Haas, M.D., M.Sc.

Rosemarie Hakim, Ph.D.

John D. Halamka, M.D., M.S.

Stephen C. Hammill, M.D.

Christina M. Heide, J.D.

Janet Hiller, M.P.H., Ph.D.

Michael Ibara, Pharm.D.

Peter I. Juhn, M.D., M.P.H.

Catherine Koepper

Rebecca D. Kush, Ph.D.

Joseph A. Ladapo, M.D., Ph.D.

Julia I. Lane

Peter Littlejohns, M.D.

Bruce C. Marshall, M.D.

Susan McAndrew, J.D.

Newell McElwee, Pharm.D., M.S.P.H.

J. Michael McGinnis, M.D., M.P.P.

Charles R. McKay, M.D., FACC

Kristi R. Mitchell, M.P.H.

Lawrence H. Muhlbaier, Ph.D.

Steven E. Nissen, M.D.

Gareth Parry, Ph.D.

Steven Pearson, M.D.

Eric Peterson, M.D., M.P.H.

Steve E. Phurrough, M.D., M.P.A.

Richard Platt, M.D., M.Sc.

Victoria M. Prescott, J.D.

Michael T. Rapp, M.D., J.D., FACEP

Robert Reynolds, Sc.D.

Gerald F. Riley, M.S.P.H.

Sara Rosenbaum, J.D.

Kenneth J. Rothman, Dr.P.H.

Susan T. Sacks, Ph.D.

Grannum Sant, M.D., FRCS, FACS

William B. Saunders, Ph.D., M.P.H.

Juergen Schmider, M.D., Ph.D.

Reviewers
258

Walter L. Straus, M.D., M.P.H.

Hugh Tilson, M.D., M.P.H., Dr.P.H.

Anne E. Trontell, M.D., M.P.H.

Jack V. Tu, M.D., Ph.D.

Cunlin Wang, M.D., Ph.D.

Suzanne L. West, M.P.H, Ph.D.

Marcy Wilder, J.D.

Marcus D. Wilson, Pharm.D.

John Yee, M.D., M.P.H.

Registries for Evaulating Patient Outcomes
259

Case Example Contributors

1. Creating a Registry To Fulfill Multiple Purposes 

and Using a Publications Committee To Review Data 

Requests

Edna Stoehr

Michelle Bertagna Leavy, M.P.H.

2. Designing a Registry for a Health Technology 

Assessment

Michelle Bertagna Leavy, M.P.H.

Hannah Patrick

3. Developing Prospective Nested Studies in Existing 

Registries

Joel M. Kremer, M.D.

Pattra Mattox, S.M.

Dimitrios Pappas, M.D., M.P.H.

4. Designing a Registry To Address Unique Patient 

Enrollment Challenges

Karen Domino, M.D., M.P.H.

Laura Khurana, M.P.H.

Karen L. Posner, Ph.D.

5. Selecting Data Elements for a Registry

Michelle Bertagna Leavy, M.P.H.

R. Scott McKenzie, M.D.

6. Understanding the Needs and Goals of Registry 

Participants

Michelle Bertagna Leavy, M.P.H.

John Spertus, M.D., M.P.H., FACC, FAHA

7. Using Standardized Data Elements in a Registry

Laura Khurana, M.P.H.

Sheri Sanders, M.B.A., B.S.N., R.N.

Wendy Schroeder, R.N., B.S.N., CCRC, CRCP, CHRC

8. Developing and Validating a Patient-Administered 

Questionnaire

Michelle Bertagna Leavy, M.P.H.

Grannum Sant, M.D., FRCS, FACS

9. Using Validated Measures To Collect PatientReported Outcomes

Kathleen M. Fox, Ph.D., M.H.S.

Susan Grandy, Ph.D., M.B.A.

Michelle Bertagna Leavy, M.P.H.

10. Challenges in the Collection of PROs in a 

Longitudinal Registry

Ethan Basch, M.D., M.Sc.

Antonia V. Bennett, Ph.D.

Pattra Mattox, S.M.

11. Collecting PRO Data in a Sensitive Patient 

Population

Waguih William IsHak, M.D., FAPA

Laura Khurana, M.P.H.

12. Using Claims Data Along With Patient-Reported 

Data To Identify Patients

Vinicius Antao, M.D., M.Sc., Ph.D.

D. Kevin Horton, Dr.P.H., M.S.P.H.

Pattra Mattox, S.M.

Oleg Muravov, M.D., Ph.D.

13. Obtaining a Waiver of Informed Consent

Mark J. Fox

Michelle Bertagna Leavy, M.P.H.

14. Issues With Obtaining Informed Consent

Moira K. Kapral, M.D., M.Sc., FRCPC

Janice A. Richards, R.N., B.A.

Jack V. Tu, M.D., Ph.D.

15. Operationalizing Informed Consent for Children

Catherine Birken, M.D., M.Sc., FRCPC

Laura Khurana, M.P.H.

Jonathon Maguire, M.D., M.Sc., FRCPC

Magda Nunes de Melo, B.Sc.Pharm., M.Sc.

Patricia Parkin, M.D., FRCPC

The findings and conclusions in this document are those of the contributors and reviewers. The 

contributors and reviewers participated as individuals and not necessarily as representatives of their 

organizations. No statement in this document should be construed as an official position of any 

organization represented here.
260

16. Using a Patient-Centered Study Design To 

Collect Informed Consent, Maximize Recruitment 

and Retention, and Provide Meaningful Clinical 

Data

Christina Pavetto Bond, M.S., FACHE

Patricia Franklin, M.D., M.B.A., M.P.H.

Laura Khurana, M.P.H.

17. Handling Discovery Requests for Registry Data

Mark J. Fox

Laura Khurana, M.P.H.

18. Meeting the Confidentiality and Quality 

Improvement Needs of Providers Through a Patient 

Safety Organization

Karen Domino, M.D., M.P.H.

Laura Khurana, M.P.H.

Karen L. Posner, Ph.D.

Donald C. Tyler, M.D., M.B.A.

19. Protections Available to Registry Data From 

Institutional Review Boards and Academic 

Institutions

Karen Domino, M.D., M.P.H.

Michelle Bertagna Leavy, M.P.H.

Karen L. Posner, Ph.D.

20. Building Value as a Means To Recruit Hospitals

Sara Camp, R.N., M.S.N., A.C.N.P., CCRN-CSC

Michelle Bertagna Leavy, M.P.H.

21. Using Registry Tools To Recruit Sites

Michelle Bertagna Leavy, M.P.H.

22. Using a Scientific Advisory Board To Support 

Investigator Research Projects

Michelle Bertagna Leavy, M.P.H.

Michael D. Taylor, Pharm.D., Ph.D.

23. Identifying and Addressing Recruitment and 

Retention Barriers in an Ongoing Registry

Katie Crotty, M.P.H., CCRP

Keith M. Hume, M.A.

Pattra Mattox, S.M.

Christopher Simmons, B.S., B.A.

24. Developing a Performance-Linked Access System

Michael Karukin, P.A., Ph.D.

Michelle Bertagna Leavy, M.P.H.

25. Using Audits To Monitor Data Quality

Jack L. Cronenwett, M.D.

Michelle Bertagna Leavy, M.P.H.

26. Using Registry Data To Evaluate Outcomes by 

Practice

Michelle Bertagna Leavy, M.P.H.

Jeffrey Wagener, M.D.

27. Using Registry Data To Study Patterns of Use 

and Outcomes

Marnie L. Boron, Pharm.D.

Michelle Bertagna Leavy, M.P.H.

28. Determining When To Stop an Open-Ended 

Registry

Jessica Albano, Ph.D., M.P.H.

Ann Mallard, M.P.H.

Pattra Mattox, S.M.

29. Challenges in Transitions and Changes in Data 

Collection

Alexander Elbert, Ph.D.

Emily A. Knapp

Bruce C. Marshall, M.D.

Pattra Mattox, S.M.

Kristofer M. Petren

30. Transitioning From Startup to Ongoing Registry 

Funding With Public and Private Partners

Ani John, B.S.N., M.P.H.

Laura Khurana, M.P.H.

Marc C. Levesque, M.D., Ph.D.

31. Modifying a Registry Due to Changes in 

Standards of Care

Dawn Eng, R.N., B.S.N.

Alexandra Gillmore, M.S.

32. Using System Integration Software To Capture 

Registry Data From Electronic Health Records

Mark J. Fox

Michelle Bertagna Leavy, M.P.H.

Kristi R. Mitchell, M.P.H.

33. Creating a Registry Interface To Incorporate 

Data From Multiple Electronic Health Records

Laura Khurana, M.P.H.

Jen Moore

34. Technical and Security Issues in Creating a 

Health Information Exchange

Jenifer Hughes, Med.I.T.

Michelle Bertagna Leavy, M.P.H.

35. Developing a New Model for Gathering and 

Reporting Adverse Drug Events

Daniel Levy, M.S.

36. Linking Registries at the International Level

Laura Khurana, M.P.H.

Luigi Naldi, M.D.

Registries for Evaulating Patient Outcomes
261

37. Linking a Procedure-Based Registry With 

Claims Data To Study Long-Term Outcomes

Mark J. Fox

Michelle Bertagna Leavy, M.P.H.

38. Linking Registry Data To Examine Long-Term 

Survival

Richard Feltbower, Ph.D.

Pattra Mattox, S.M.

Patricia McKinney, Ph.D.

Roger Parslow, Ph.D.

Marlous Van Laar, M.Sc.

39. Linking Longitudinal Registry Data to Medicaid 

Analytical Extract Files

Efe Eworuke, Ph.D.

Pattra Mattox, S.M.

Almut Winterstein, Ph.D.

40. Integrating Data From Multiple Sources With 

Patient ID Matching

Noam H. Arzt, Ph.D.

Michelle Bertagna Leavy, M.P.H.

41. Using Patient Identity Management Methods To 

Combine Health System Data

Laura Khurana, M.P.H.

Roshanthi Weerasinghe, M.P.H.

42. Combining De-Identified Data From Multiple 

Registries To Study Long-Term Outcomes in a Rare 

Disease

Monika Brand

Laura Khurana, M.P.H.

Erwan Muros Le Rouzic, M.P.H.

Daniel Rosenberg, Ph.D., M.P.H.

Barbara Schwierin, Ph.D.

43. Understanding Baseline Characteristics of 

Combined Data Sets Prior to Analysis

Maria Inacio, M.S.

Laura Khurana, M.P.H.

Gregory B. Maletis, M.D.

Elizabeth Paxton, M.A.

44. Using a Registry To Assess Long-Term Product 

Safety

Pattra Mattox, S.M.

Deborah Symmons, M.D.

45. Using a Registry To Monitor Long-Term Product 

Safety

Michelle Bertagna Leavy, M.P.H.

Rachel Sobel, Dr.P.H.

46. Using Registries To Understand Rare Diseases

Betsy Bogard, M.S.

Catherine Koepper

Nicole Robichaud

John Yee, M.D., M.P.H.

47. Studying Rare Diseases in an Existing Registry 

Population

Charles Barr, M.D., M.P.H.

Laura Khurana, M.P.H.

Barbara Lippe, M.D.

Anne Morris, Ph.D.

48. Site Motivation and Retention in Rare Disease 

Registries

Monika Brand

Alexandra Gillmore, M.S.

Erwan Muros Le Rouzic, M.P.H.

Daniel Rosenberg, Ph.D., M.P.H.

Barbara Schwierin, Ph.D.

49. Expanding an Ongoing Pregnancy Registry

Jessica Albano, Ph.D., M.P.H.

Deborah Covington, Dr.P.H.

Laura Khurana, M.P.H.

50. Using a Pregnancy Registry To Detect Major 

Teratogenicity

Marianne Cunnington 

Lianna Ishihara

Pattra Mattox, S.M.

51. Implementing a Non-Mandated Pregnancy 

Registry

Dawn Eng, R.N., B.S.N.

Laura Khurana, M.P.H.

52. Using Proactive Awareness Activities To Recruit 

Patients for a Pregnancy Exposure Registry

Michelle Bertagna Leavy, M.P.H.

Susan Sinclair Roberts, Ph.D.

53. Using Recognition Measures To Develop a Data 

Set

Kenneth A. LaBresh, M.D.

Michelle Bertagna Leavy, M.P.H.

Louise Morgan, M.S.N., CPHQ

Lee H. Schwamm, M.D., FAHA

54. Managing Care and Quality Improvement for 

Chronic Diseases

Carl Donisi

Keith Mandel, M.D.

Pattra Mattox, S.M.

Randy Woodward

Case Example Contributors 
262

55. Use of Reporting Tools To Promote Quality 

Improvement

Rebecca S. Brown

Pam Kadlubek, M.P.H.

Pattra Mattox, S.M.

56. Using Registries To Drive Quality Improvement 

in Chronic Conditions

Pattra Mattox, S.M.

Peter Schmidt, Ph.D.

57. Clarifying the Federal Regulatory Requirements 

for Quality Improvement Registries

Anthony L. Asher, M.D., FACS

Laura Khurana, M.P.H.

Koryn Y. Rubin

58. Designing a Registry To Study the Effectiveness 

of a Device Training Program for Providers

Sidney A. Cohen, M.D., Ph.D.

Michelle Bertagna Leavy, M.P.H.

59. Identifying and Responding to Adverse Events 

Found in a Registry Database

Michelle Bertagna Leavy, M.P.H.

Rebecca Love, M.P.H., R.N.

Elizabeth Paxton, M.A.

60. Receiving Data From Medical Imaging Devices

Mythreyi B. Chatfield, Ph.D.

Laura Khurana, M.P.H.

61. Combining Registry Data With EHR Data To 

Measure Real-World Outcomes of Implantable 

Devices

Robert Greenlee, Ph.D., M.P.H.

Nat Jackson, M.S.

Laura Khurana, M.P.H.

Frederick Masoudi, M.D., M.S.P.H.

62. Developing a Public-Private Partnership for 

Comparative Effectiveness Research

Elise Berliner, Ph.D.

Flora Lum, M.D.

Pattra Mattox, S.M.

63. Leveraging a Public-Private Partnership for a 

Post-Marketing Commitment

Laura Khurana, M.P.H.

Prof Dr Stefan Kölker

Samantha Parker

Marshall Summar, M.D.

64. Public-Private Partnerships for Rare Diseases

Diane J. Aschman, M.S.

Laura Khurana, M.P.H.

Barbara Konkle, M.D. 

Registries for Evaulating Patient Outcomes
263

Contributor and Reviewer Affiliations

Amy P. Abernethy, M.D.

Director, Center for Learning Health Care

Director, Duke Cancer Care Research Program

Associate Professor of Medicine, Duke University 

 School of Medicine

Associate Professor of Nursing, Duke University 

 School of Nursing

Durham, NC

Susan Adams, J.D.

Director, Committee for the Protection of Human 

Subjects

Dartmouth College

Hanover, NH

Jessica Albano, Ph.D., M.P.H.

Senior Director, Epidemiology, Post Approval & 

 Strategic Services

INC Research

Raleigh, NC

Fred Anderson, Ph.D.

Research Professor of Surgery

Director, Center for Outcomes Research

University of Massachusetts Medical School

Worcester, MA

Vinicius Antao, M.D., M.Sc., Ph.D.

Registries Team Lead

Agency for Toxic Substances & Disease Registry/

 Centers for Disease Control and Prevention

Atlanta, GA

Paul S. Appelbaum, M.D.

Elizabeth K. Dollard Professor of Psychiatry, 

 Medicine & Law

Director, Division of Law, Ethics, and Psychiatry

Department of Psychiatry, Columbia University

New York, NY

Noam H. Arzt, Ph.D.

President

HLN Consulting, LLC

Palm Desert, CA

Diane J. Aschman, M.S.

President and CEO

American Thrombosis and Hemostasis Network

Riverwoods, IL

Anthony L. Asher, M.D., FACS

Co-Medical Director, Neuroscience Institute, 

 Carolinas Healthcare System

Director, National Neurosurgery Quality and 

 Outcomes Database

Carolina Neurosurgery & Spine Associates

Charlotte, NC

Steven J. Atlas, M.D., M.P.H.

Assistant Professor of Medicine

Harvard Medical School

Associate Physician, General Medicine Division

Massachusetts General Hospital

Boston, MA

Landen Bain

Liaison to Healthcare

Clinical Data Interchange Standards Consortium

Winston Salem, NC

Charles Barr, M.D., M.P.H.

Group Medical Director and Head of Patient 

 Registries

US Medical Affairs

Genentech, Inc.

South San Francisco, CA

Ethan Basch, M.D., M.Sc.

Associate Professor of Medicine and Public Health

University of North Carolina

Chapel Hill, NC

Antonia V. Bennett, Ph.D.

Assistant Professor of Public Health

University of North Carolina

Chapel Hill, NC

Jessica Berg, J.D., M.P.H.

Professor of Law, Bioethics & Public Health

Case Western Reserve University School of Law 

 and School of Medicine

Cleveland, OH

The findings and conclusions in this document are those of the contributors and reviewers. The 

contributors and reviewers participated as individuals and not necessarily as representatives of their 

organizations. No statement in this document should be construed as an official position of any 

organization represented here.
264

Marc L. Berger, M.D.

Vice President, Real World Data and Analytics

Pfizer

New York, NY

Elise Berliner, Ph.D.

Director, Technology Assessment Program

Center for Outcomes and Evidence

Agency for Healthcare Research and Quality

Rockville, MD

Maya A. Bernstein, J.D.

Privacy Advocate

U.S. Department of Health and Human Services

Washington, DC

Stephen W. Bernstein, J.D.

Head, Health Industry Practice Group

McDermott Will & Emery, LLP

Boston, MA

Catherine Birken, M.D., M.Sc., FRCPC

Staff Pediatrician and Associate Scientist

University of Toronto

Department of Pediatrics

The Hospital for Sick Children

Toronto, Canada

Stella Blackburn, M.B., B.S., M.Sc., FRCP (Edin), 

FISPE

EMA Risk Management Development and 

 Scientific Lead

European Medicines Agency

London, United Kingdom

Marcella Blackledge, B.A.

CF Registry Coordinator 

Research Data Specialist II

Seattle Children’s Research Institute

Seattle Children’s Hospital

Seattle, WA

Kathleen Blake, M.D., M.P.H.

Vice President, Physician Consortium for 

 Performance Improvement

American Medical Association

Chicago, IL

Betsy Bogard, M.S.

Associate Director, Program Management

Genzyme, a Sanofi company

Cambridge, MA

Christina Pavetto Bond, M.S., FACHE

National Operations Director

FORCE TJR

Worcester, MA

Marnie L. Boron, Pharm.D.

Director, Scientific Communications

MedImmune

Gaithersburg, MD

Jaclyn L. F. Bosco, Ph.D., M.P.H.

Epidemiologist

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Brian D. Bradbury, M.A., D.Sc.

Director, Center for Observational Research

Amgen, Inc.

Adjunct Assistant Professor, Epidemiology

UCLA Fielding School of Public Health

Los Angeles, CA

Monika Brand

Associate Director, Observational Studies

Actelion Pharmaceuticals Ltd

Allschwil, Switzerland

Ralph G. Brindis, M.D., M.P.H., MACC, FSCAI

Clinical Professor of Medicine

Department of Medicine & the Philip R. Lee 

 Institute for Health Policy Studies 

University of California, San Francisco

Senior Medical Officer, External Affairs - ACC 

 National Cardiovascular Registry

San Francisco, CA

Jeanette M. Broering, R.N., Ph.D., M.P.H.

Director, Data Operations

Urology Clinical and Translational Science

Department of Urology

University of California San Francisco

San Francisco, CA

Jeffrey Brown, Ph.D.

Research Director, Therapeutics Research and 

 Infectious Disease Epidemiology Program

Lecturer, Department of Population Medicine

Harvard Pilgrim Health Care Institute/Harvard 

 Medical School

Boston, MA

Rebecca S. Brown

Quality Measures Coordinator, Quality and 

 Guidelines

American Society of Clinical Oncology

Alexandria, VA

Laura D. Buccini, Dr.P.H., M.P.H.

Associate Staff, Digestive Disease Institute and 

 Transplant Center

The Cleveland Clinic

Cleveland, OH

Registries for Evaulating Patient Outcomes
265

Sara Camp, R.N., M.S.N., A.C.N.P., CCRN-CSC

Director, Quality Marketing and Research

American Heart Association

Dallas, TX

Daniel Caños

Epidemiologist, Division of Epidemiology

Office of Surveillance and Biometrics

Center for Devices and Radiological Health

Food and Drug Administration

Silver Spring, MD

Christina Chambers, Ph.D., M.P.H.

Professor of Pediatrics

Director, Organization of Teratology Information 

 Specialists Collaborative Research Center 

University of California San Diego School of 

 Medicine 

La Jolla, CA

Mythreyi B. Chatfield, Ph.D.

Director, Data Registries

American College of Radiology

Reston, VA

Sidney A. Cohen, M.D., Ph.D.

Formerly Vice President, Medical Affairs

Cordis Corporation and Conor Medsystems

Menlo Park, CA

J. Alexander Cole, D.Sc., M.P.H.

Director, Epidemiology

Alexion Pharmaceuticals

Cambridge, MA

Deborah Covington, Dr.P.H.

Global Head, Observational Studies and Pregnancy 

 Registries

PPD

Wilmington, NC

Janet D. Cragan, M.D., M.P.H.

National Center on Birth Defects and 

 Developmental Disabilities

Centers for Disease Control and Prevention

Atlanta, GA

Jack L. Cronenwett, M.D.

Professor of Surgery

Dartmouth-Hitchcock Medical Center

Medical Director

Society for Vascular Surgery Patient Safety 

 Organization

Lebanon, NH

Katie Crotty, M.P.H., CCRP

Manager, Research and Scientific Affairs

American Society of Plastic Surgeons

Arlington Heights, IL

Jay Crowley

Senior Advisor for Patient Safety

Center for Devices and Radiological Health

Food and Drug Administration

Silver Spring, MD

Theresa Cullen, M.D., M.S.

Chief Medical Informatics Officer 

Director, Health Informatics

Veterans Health Administration

Washington, DC

Marianne Cunnington 

Formerly Manager, Worldwide Epidemiology

GlaxoSmithKline

Essex, United Kingdom

David C. Dale, M.D.

Professor of Medicine

University of Washington School of Medicine

Attending Physician, University of Washington 

 Medical Center 

Seattle, WA

Leonard D’Avolio, Ph.D.

Director of Biomedical Informatics

Massachusetts Veterans Epidemiology Research 

 and Information Center

Faculty, Harvard Medical School

Boston, MA

Joseph A.C. Delaney, Ph.D.

Research Assistant Professor, Epidemiology

University of Washington School of Public Health

Seattle, WA

Andy DeMayo, J.D.

Senior Corporate Counsel

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Patrice Desvigne-Nickens, M.D.

Medical Officer

Heart Failure and Arrhythmia Branch

Division of Cardiovascular Science

National Heart, Lung, and Blood Institute

Bethesda, MD

Contributor and Reviewer Affiliations 
266

Douglas S. Diekema, M.D., M.P.H.

Director of Education, Treuman Katz Center for 

 Pediatric Bioethics

Seattle Children’s Hospital

Professor of Pediatrics

Adjunct Assistant Professor, Medical History and 

 Ethics

University of Washington School of Medicine

Seattle, WA

Rachel S. Dokholyan, M.P.H.

Project Leader

Duke Clinical Research Institute

Durham, NC

Karen Domino, M.D., M.P.H.

Professor, Department of Anesthesiology and Pain 

 Medicine

University of Washington

Seattle, WA

Wei Dong, M.D., Ph.D.

Deputy Site Head of Global Product Development 

 in Asia Pacific (AP)

Head of AP Development Acceleration 

Roche

Shanghai, China 

Carl Donisi

Director of Clinical Operations, Physician-

 Hospital Organization

Cincinnati Children’s Hospital Medical Center

Cincinnati, OH

Pamela S. Douglas, M.D., FACC

Ursula Geller Professor of Research in 

 Cardiovascular Diseases

Director of Imaging Program at Duke Clinical 

 Research Institute

Duke University

Durham, NC

Nancy A. Dreyer, M.P.H., Ph.D.

Global Chief of Scientific Affairs

Senior Vice President

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Joseph P. Drozda, Jr., M.D., FACC

Director, Outcomes Research

Mercy Health

St. Louis, MO

Hesha J. Duggirala, Ph.D., M.P.H.

Epidemiologist

Food and Drug Administration

Rockville, MD

Brenda K. Edwards, Ph.D.

Associate Director, Surveillance Research 

 Program

Division of Cancer Control and Population 

 Sciences

National Cancer Institute

Bethesda, MD 

Paul Eggers, Ph.D.

Program Director for Kidney and Urology 

 Epidemiology

National Institute of Diabetes and Digestive and 

 Kidney Diseases

Bethesda, MD

Jonathan S. Einbinder, M.D., M.P.H.

Brigham and Women’s Hospital

Boston, MA

Alexander Elbert, Ph.D.

Director of CF Registry Program

Cystic Fibrosis Foundation

Bethesda, MD

Benjamin C. Eloff, Ph.D.

Senior Scientific Program Manager

Division of Epidemiology

Center for Devices and Radiological Health

Food and Drug Administration

Silver Spring, MD

Dawn Eng, R.N., B.S.N.

Senior Clinical Trial Head, Oncology Global 

 Medical Affairs

Novartis Pharmaceuticals Corporation

East Hanover, NJ

Sara A. Ephross, PhD

Director, Worldwide Epidemiology

GlaxoSmithKline

Research Triangle Park, NC

Barbara Evans, J.D., Ph.D.

Professor of Law

Co-director, Health Law & Policy Institute

Director, Center on Biotechnology

University of Houston Law Center

Houston, TX

Efe Eworuke, Ph.D.

Department of Pharmaceutical Outcomes and 

 Policy

University of Florida

Gainesville, FL

Registries for Evaulating Patient Outcomes
267

Shamiram Feinglass, M.D., M.P.H.

Senior Medical Officer

Coverage and Analysis Group

Office of Clinical Standards and Quality

Centers for Medicare & Medicaid Services

U.S. Public Health Service

Baltimore, MD

T. Bruce Ferguson, M.D., FAHA

Department of Cardiovascular Sciences

Eastern Carolina University

Greenville, NC

Richard Feltbower, Ph.D.

Senior Lecturer, Division of Epidemiology

School of Medicine, University of Leeds

Leeds, United Kingdom

Stephen E. Fienberg, Ph.D., FRSC 

Maurice Falk University Professor of Statistics and 

 Social Science

Carnegie Mellon University

Pittsburgh, PA 

Leslye K. Fitterman, Ph.D.

Centers for Medicare & Medicaid Services

Baltimore, MD

Sally Flanzer, Ph.D., CIP

Regional Program Manager, Child Welfare

Administration for Children and Families 

Region IX

San Francisco, CA

Formerly Human Protections Administrator

Office of Extramural Research, Education and 

 Priority Populations

Agency for Healthcare Research and Quality

Rockville, MD

Gregg C. Fonarow, M.D.

Professor of Medicine/Cardiology

Director, Ahmanson-UCLA Cardiomyopathy 

 Center

Associate Chief, UCLA Division of Cardiology

The Eliot Corday Chair in Cardiovascular 

 Medicine and Science

David Geffen School of Medicine at UCLA

Los Angeles, CA

Kathleen M. Fox, Ph.D., M.H.S.

President

Strategic Healthcare Solutions, LLC

Monkton, MD

Mark J. Fox

Senior Contract/Grants Administrator

Compliance Integrity Officer

American College of Cardiology Foundation

Washington, DC

Lori Frank, Ph.D.

Director of Engagement Research

Patient-Centered Outcomes Research Institute

Washington, DC

Patricia Franklin, M.D., M.B.A., M.P.H.

Principal Investigator, FORCE TJR

Professor and Director, Clinical and Outcomes 

 Research

Department of Orthopedics and Physical 

 Rehabilitation

University of Massachusetts Medical School

Worcester, MA

Sarah Garner, Ph.D., B.Pharm., M.R.Pharm.S.

Associate Director, Research and Development

National Institute for Health and Clinical 

 Excellence

London, United Kingdom

Yvonne Geissbuehler, Ph.D.

Senior Clinical Epidemiologist

Novartis Pharma AG

Basel, Switzerland

Eric Gemmen, M.A.

Senior Director, Biostatistics and Outcomes 

 Research

Real-World & Late Phase Research

Quintiles

Rockville, MD

Richard E. Gliklich, M.D.

Leffenfeld Professor Otology/Laryngology, 

 Harvard Medical School

Principal Investigator, Outcome DEcIDE Center

President, Real-World & Late Phase Research

Quintiles

Cambridge, MA

Marissa Gordon-Nguyen, J.D.

Health Information Privacy Specialist

Office for Civil Rights

U.S. Department of Health and Human Services

Washington, D.C.

Contributor and Reviewer Affiliations 
268

Julian M. Goldman, M.D.

Medical Director, Biomedical Engineering

Partners HealthCare System 

Attending Anesthesiologist, Massachusetts 

 General Hospital (MGH) 

Director, Program on Medical Device 

 Interoperability, MGH and CIMIT 

Boston, MA

Melissa M. Goldstein, J.D.

Associate Professor, Department of Health Policy

School of Public Health and Health Services

George Washington University

Washington, DC

Julia G. Gorey, J.D.

Public Health Analyst

Office for Human Research Protections

Department of Health and Human Services

Rockville, MD

Susan Grandy, Ph.D., M.B.A.

Global Payer Evidence Director

Global Medicines Development

AstraZeneca

Wilmington, DE

Robert Greenlee, Ph.D., M.P.H.

Research Scientist, Epidemiology Research Center

Marshfield Clinic Research Foundation

Marshfield, WI

Marie R. Griffin, M.D., M.P.H.

Professor of Preventive Medicine

Professor of Medicine

Vanderbilt University School of Medicine

Nashville, TN 

Stephen C. Groft, Pharm.D.

Director, Office of Rare Diseases Research

National Institutes of Health

Bethesda, MD

Thomas Gross, M.D., M.P.H.

Director, Office of Surveillance and Biometrics

Center for Devices and Radiological Health

Food and Drug Administration

Rockville, MD

Claudia Grossmann, Ph.D.

Senior Program Officer

Institute of Medicine

Washington, DC

Joanna Haas, M.D., M.Sc.

Vice President, Global Patient Safety and Risk 

 Management 

Genzyme Corporation

Cambridge, MA

Rosemarie Hakim, Ph.D.

Senior Research Advisor

Centers for Medicare & Medicaid Services

Baltimore, MD

John D. Halamka, M.D., M.S.

Beth Israel Deaconess Medical Center

Harvard Medical School

New England Health Electronic Data Interchange 

 Network

MA-SHARE

U.S. Healthcare Information Technology Standards 

 Panel

Boston, MA

Stephen C. Hammill, M.D.

Professor of Medicine

Mayo Clinic College of Medicine

Director, Electrocardiography Laboratory

Rochester, MN 

Robert E. Harbaugh, M.D., FAANS, FACS, FAHA

Director, Penn State Institute of the Neurosciences

University Distinguished Professor and Chair, 

 Department of Neurosurgery

Professor, Department of Engineering Science and 

 Mechanics

Penn State University, M.S. Hershey Medical 

 Center

Hershey, PA

Melanie J. Harrison, M.D., M.S.C.E.

Group Medical Director

Global Pharmacovigilance and Epidemiology

Bristol-Myers Squibb

Pennington, NJ

Christina M. Heide, J.D.

Senior Health Information Privacy Policy 

 Specialist

Office for Civil Rights

U.S. Department of Health and Human Services

Washington, DC

Adrian F. Hernandez, M.D., M.H.S.

Associate Professor of Medicine

Duke University School of Medicine

Durham, NC

Registries for Evaulating Patient Outcomes
269

Sonia Hernández-Díaz, M.D., M.P.H., Dr.P.H.

Associate Professor of Epidemiology

Director, Pharmacoepidemiology Program

Harvard School of Public Health

Boston, MA

Kathleen Hewitt, M.S.N., R.N., A.A.C.C., CPHQ

Associate Vice President

American College of Cardiology

Washington, DC

Janet Hiller, M.P.H., Ph.D.

Professor of Public Health

Director, Adelaide Health Technology Assessment

University of Adelaide

Adelaide, Australia

D. Kevin Horton, Dr.P.H., M.S.P.H.

Chief 

Environmental Health Surveillance Branch

Agency for Toxic Substances & Disease Registry

Centers for Disease Control and Prevention

Atlanta, GA

Holly L. Howe, Ph.D.

Executive Director

North American Association of Central Cancer 

 Registries, Inc.

Springfield, IL

Jenifer Hughes, Med.I.T.

Senior Vice President, 

Administrative Network, Inc.

Oakland Southfield Physicians, PC

Southfield, MI

Keith M. Hume, M.A.

Vice President, Research & Scientific Affairs

American Society of Plastic Surgeons

Chief Operating Officer

The Plastic Surgery Foundation

Arlington Heights, IL

Barrie J. Hurwitz, M.B., MRCP

Associate Professor Emeritus of Neurology and 

 Community and Family Medicine

Duke University Medical Center

Durham, NC

Michael Ibara, Pharm.D.

Head of Pharmacovigilance Information 

 Management

Pfizer, Inc.

New York, NY 

Waguih William IsHak, M.D., FAPA

Director, Psychiatry Residency Training and 

 Medical Student Education

Consultant Psychiatrist, Consultation-Liaison 

 Service

Cedars-Sinai Medical Center

Clinical Professor of Psychiatry

University of California Los Angeles

Los Angeles, CA

Maria Inacio, M.S.

Epidemiologist

Surgical Outcomes and Analysis

Kaiser Permanente

San Diego, CA

Lianna Ishihara

Manager, Worldwide Epidemiology

GlaxoSmithKline

Stockley Park, United Kingdom

Solomon Iyasu, M.D., M.P.H.

Director 

Division of Epidemiology

Office of Surveillance and Epidemiology 

Center for Drug Evaluation and Research

Food and Drug Administration

Silver Spring, MD

Nat Jackson, M.S.

Project Manager

Kaiser Permanente Colorado Institute for Health 

 Research

Denver, CO

Emma James, Ph.D.

Associate Director, Medical Affairs and Global 

 Registries

Synageva BioPharma Corp.

Lexington, MA

Robert A. Jenders, M.D., M.S., FACP, FACMI

Co-Director, Center for Biomedical Informatics

Professor of Medicine, Charles Drew University

Professor, David Geffen School of Medicine

University of California, Los Angeles

Los Angeles, CA

Ani John, B.S.N., M.P.H.

Head of Immunology Registries

Genentech, Inc. a member of the Roche Group

South San Francisco, CA

Contributor and Reviewer Affiliations 
270

Peter I. Juhn, M.D., M.P.H.

President

Therapeutic Resource Centers

Medco Health Solutions, Inc.

Franklin Lakes, NJ

Pam Kadlubek, M.P.H.

Assistant Director, Quality Oncology Practice 

 Initiative Operations

American Society of Clinical Oncology

Alexandria, VA

Michael Kahn, M.D., Ph.D.

Associate Professor of Epidemiology, Department 

 of Pediatrics

University of Colorado Denver

Co-Director and Biomedical Informatics Core 

 Director

Colorado Clinical and Translational Sciences 

 Institute

Director of Clinical Informatics, Department of 

 Quality & Patient Safety

Children’s Hospital Colorado

Aurora, CO

Moira K. Kapral, M.D., M.Sc., FRCPC

Associate Professor of Medicine, University of 

 Toronto

Senior Scientist, Institute for Clinical Evaluative 

 Sciences

Ontario, Canada

Alan F. Karr, Ph.D.

Director

National Institute of Statistical Sciences

Research Triangle Park, NC

Michael Karukin, P.A., Ph.D.

Director, Registries

Teva Pharmaceuticals USA

Miami, FL

Irene Katzan, M.D., M.S.

Director, Neurological Institute Center for 

 Outcomes Research & Evaluation

Cleveland Clinic Foundation

Center for Health Care Research & Policy, 

 MetroHealth Medical Center

Cleveland, OH

David W. Kaufman, Sc.D.

Associate Director, Slone Epidemiology Center

Professor of Epidemiology, Boston University 

 School of Public Health

Boston, MA

Laura Khurana, M.P.H.

Formerly Research Associate

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Shukri F. Khuri, M.D. (deceased)

Professor of Surgery, Harvard Medical School

Co-Founder, National Surgical Quality 

 Improvement Program

VA Boston Healthcare System

West Roxbury, MA

Stuart Kim, J.D., M.S.

Associate General Counsel, Regulatory Affairs

Covidien

Hazelwood, MO

James K. Kirklin, M.D.

Professor and Director, Division of Cardiothoracic 

 Surgery

University of Alabama at Birmingham

Birmingham, AL

Emily A. Knapp

Patient Registry Senior Coordinator

Cystic Fibrosis Foundation

Bethesda, MD

Clifford Ko, M.D.

Professor, Department of Surgery

Director, Center for Surgical Outcomes and 

 Quality

University of California, Los Angeles

Los Angeles, CA

Catherine Koepper

Director, Global Registry Program

Genzyme, a Sanofi company

Cambridge, MA

Prof Dr. Stefan Kölker

Head, Division of Inherited Metabolic Diseases

Centre for Paediatric and Adolescent Medicine

University of Heidelberg

Heidelberg, Germany

Barbara Konkle, M.D.

Director, Clinical and Translational Research

Medical Director, Hemostasis Reference 

 Laboratory

Puget Sound Blood Center

Professor of Medicine/Hematology

University of Washington

Seattle, WA

Registries for Evaulating Patient Outcomes
271

Joel M. Kremer, M.D.

Pfaff Family Professor of Medicine

Albany Medical College

Director of Research

The Center for Rheumatology

President

The Consortium for Rheumatology Researchers of 

 North America

Albany, NY

Harlan M. Krumholz, M.D., S.M.

Harold H. Hines, Jr. Professor of Medicine and 

 Epidemiology and Public Health

Director, Robert Wood Johnson Clinical Scholars 

 Program

Yale University School of Medicine

Director, Center for Outcomes Research and 

 Evaluation

Yale-New Haven Hospital

New Haven, CT

Kimary Kulig, Ph.D., M.P.H.

Group Director, Oncology Collaborative Institute

Bristol-Myers Squibb

Plainsboro, NJ

Richard E. Kuntz, M.D., M.Sc.

Senior Vice President and Chief Scientific, 

 Clinical and Regulatory Officer

Medtronic, Inc.

Minneapolis, MN

Rebecca D. Kush, Ph.D.

President and CEO

Clinical Data Interchange Standards Consortium 

Austin, TX

Kenneth A. LaBresh, M.D.

Senior Fellow

RTI International

Waltham, MA

Joseph A. Ladapo, M.D., Ph.D.

Beth Israel Deaconess Medical Center

Boston, MA

Julia I. Lane

National Science Foundation

Washington, DC 

Debra R. Lappin, J.D.

Senior Vice President

B&D Consulting LLC – Health and Life Sciences 

 Practice

Washington, D.C.

Timothy L. Lash, D.Sc., M.P.H.

Professor, Epidemiology

Rollins School of Public Health

Emory University

Atlanta, GA

Michelle Bertagna Leavy, M.P.H.

Manager, Health Policy

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Erwan Muros Le Rouzic, M.P.H.

Senior Epidemiologist

Actelion Pharmaceuticals Ltd

Allschwil, Switzerland

Marc C. Levesque, M.D., Ph.D.

Associate Professor, Division of Rheumatology 

 and Clinical Immunology

University of Pittsburgh

Pittsburgh, PA

Daniel Levy, M.S.

Chief Technology Officer

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Barbara Lippe, M.D.

Principal Medical Director

Genentech, Inc.

South San Francisco, CA

Wendy Lipworth, M.D., Ph.D.

Postdoctoral Research Fellow, Centre for Clinical 

 Governance Research in Health

Australian Institute of Health Innovation

Faculty of Medicine

University of New South Wales

Sydney, Australia

Peter Littlejohns, M.D.

Clinical and Public Health Director

National Institute for Health and Clinical 

 Excellence

London, UK

Rebecca Love, M.P.H., R.N.

Project Manager, Surgical Outcomes and Analysis

Southern California Permanente Medical Group

San Diego, CA

Contributor and Reviewer Affiliations 
272

Flora Lum, M.D.

Executive Director

H. Dunbar Hoskins Jr, MD Center for Quality Eye 

 Care

Director, Quality of Care & Knowledge Base 

 Development

American Academy of Ophthalmology

San Francisco, CA

Theodore Lystig, Ph.D.

Lead Corporate Biostatistician

Medtronic

Minneapolis, MN

Christina DeFilippo Mack, Ph.D., M.S.P.H.

Research Consultant

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Jonathon Maguire, M.D., M.Sc., FRCPC

Paediatrician and Scientist, Li Ka Shing 

 Knowledge Institute

Department of Pediatrics

St. Michael’s Hospital

Toronto, Canada

David J. Malenka, M.D.

Professor of Medicine, The Geisel School of 

 Medicine at Dartmouth

The Dartmouth Institute for Health Policy and 

 Clinical Practice

Hanover, NH

Section of Cardiology

Dartmouth-Hitchcock Medical Center

Lebanon, NH

Gregory B. Maletis, M.D.

Chief of Orthopedics

Baldwin Park Medical Center

Kaiser Permanente

Baldwin Park, CA

Bradley Malin, Ph.D.

Associate Professor

Vanderbilt University

Nashville, TN

Ann Mallard, M.P.H.

Formerly Associate Director, Registries and Health 

 Outcomes

Kendle International Inc.

Wilmington, NC

Keith Mandel, M.D.

Vice President of Medical Affairs, Physician 

 Hospital Organization

James M. Anderson Center for Health Systems 

 Excellence

Cincinnati Children’s Hospital Medical Center

Cincinnati, OH

Neal Mantick, M.S.

Senior Director, Observational Research

PAREXEL International

Boston, MA

Danica Marinac-Dabic, M.D., Ph.D.

Director, Division of Epidemiology

Center for Devices and Radiological Health

Food and Drug Administration

Silver Spring, MD

Bruce C. Marshall, M.D.

Vice President of Clinical Affairs

Cystic Fibrosis Foundation

Bethesda, MD

Frederick Masoudi, M.D., M.S.P.H.

Associate Professor of Medicine 

Division of Cardiology

University of Colorado Anschutz Medical Campus

Aurora, CO

David Matchar, M.D.

Professor of Medicine

Duke University Medical Center

Durham, NC

Professor and Director

Health Services and Systems Research

Duke-NUS Graduate Medical School

Singapore

Pattra Mattox, S.M.

Associate Epidemiologist

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Vandana Menon, M.D.

Formerly Director of Clinical Epidemiology

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Susan McAndrew, J.D.

Deputy Director for Health Information Privacy

Office for Civil Rights

U.S. Department of Health and Human Services

Washington, D.C.

Registries for Evaulating Patient Outcomes
273

C. Geoffrey McDonough

Vice President, Global Medical Affairs

Genzyme Therapeutics

Cambridge, MA

Newell McElwee, Pharm.D., M.S.P.H.

Senior Director/Team Leader

Global Outcomes Research

Pfizer, Inc.

New York, NY

J. Michael McGinnis, M.D., M.P.P.

Senior Scholar

Institute of Medicine

National Academy of Sciences

Washington, DC

Deven McGraw, J.D., M.P.H., L.L.M.

Director, Health Privacy Project

The Center for Democracy and Technology

Washington, DC

Charles R. McKay, M.D., FACC

Professor of Medicine, Chief, Division of 

 Cardiology

Los Angeles Biomedical Research Institute at 

 Harbor-UCLA Medical Center

Torrance, CA

R. Scott McKenzie, M.D.

Senior Director, Translational Science Team

Health Economics & Outcomes Research

Janssen Scientific Affairs

Dallas, TX

Patricia McKinney, Ph.D.

Emeritus Professor of Paediatric Epidemiology 

 Division of Epidemiology

School of Medicine 

University of Leeds

Leeds, United Kingdom

Magda Nunes de Melo, B.Sc.Pharm., M.Sc.

Manager, Observational Epidemiology and 

 Qualitative Research Unit

Applied Health Research Centre

St. Michael’s Hospital

Toronto, Canada

Aaron B. Mendelsohn, Ph.D., M.P.H.

Director of Epidemiology

Real-World & Late Phase Research

Quintiles

Rockville, MD

Marissa A. Miller, D.V.M., M.P.H.

Branch Chief, Advanced Technologies and Surgery

National Heart, Lung, and Blood Institute

National Institutes of Health

Bethesda, MD

Benjamin Miriovsky, M.D.

Fellow, Hematology-Oncology

Duke University School of Medicine

Durham, NC

Barry Mirrer, M.S., CMQ/OE

Senior Director, Quality Assurance

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Kristi R. Mitchell, M.P.H.

Formerly Senior Director, Research, Development, 

 and Quality Products

American College of Cardiology

Washington, DC

Jen Moore

Senior Director, Operations

MMC Physician-Hospital Organization

Portland, ME

Louise Morgan, M.S.N., CPHQ

National Director, Get With The Guidelines

American Heart Association

Scotch Plains, NJ

Anne Morris, Ph.D.

Formerly Associate Registry Epidemiologist

Genentech, Inc.

South San Francisco, CA

Lawrence H. Muhlbaier, Ph.D.

Associate Professor, Biostatistics and 

 Bioinformatics

Duke University School of Medicine

Durham, NC

Oleg Muravov, M.D., Ph.D.

Medical Epidemiologist

Agency for Toxic Substances & Disease Registry/

 Centers for Disease Control and Prevention

Atlanta, GA

Barbara Muth

Formerly Director of Research and Guidelines

American Society for Radiation Oncology

Washington, DC

Contributor and Reviewer Affiliations 
274

David C. Naftel, Ph.D.

Professor of Surgery, Division of Cardiothoracic 

 Surgery

Professor of Biostatistics, School of Public Health

University of Alabama at Birmingham

Birmingham, AL

Steven E. Nissen, M.D.

Chairman, Department of Cardiovascular 

 Medicine

Cleveland Clinic

Cleveland, OH

Luigi Naldi, M.D.

Director, Centro Studi GISED

Dermatologist, Department of Dermatology

Azienda Ospedaliera papa Giovanni XXIII 

Bergamo, Italy

Sharon-Lise T. Normand, Ph.D.

Professor, Department of Health Care Policy

Harvard Medical School

Professor, Department of Biostatistics

Harvard School of Public Health

Boston, MA

J. Marc Overhage, M.D., Ph.D.

Chief Medical Informatics Officer

Siemens Healthcare

Malvern, PA

Dimitrios Pappas, M.D., M.P.H.

Assistant Professor of Medicine

Columbia University College of Physicians and 

 Surgeons

Scientific Director

The Consortium for Rheumatology Researchers of 

 North America

New York, NY

Samantha Parker

Director of External Affairs and Rare Disease 

 Partnerships

Orphan Europe Recordati Group

Paris, France

Patricia Parkin, M.D., FRCPC

Research Director

Division of Pediatric Medicine and the Pediatric 

 Outcomes Research Team

The Hospital for Sick Children

Toronto, Canada

Alexandra Parnell, M.S.

Research Associate

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Gareth Parry, Ph.D.

Director, Quality Measurement & Analysis

Department of Medicine

Children’s Hospital

Boston, MA

Roger Parslow, Ph.D.

Senior Lecturer, Division of Epidemiology

School of Medicine 

University of Leeds

Leeds, United Kingdom

Hannah Patrick

Consultant Clinical Advisor to the IP Programme

National Institute for Health and Clinical 

 Excellence

London, United Kingdom

Elizabeth Paxton, M.A.

Director, Kaiser Permanente National Implant 

 Registries

Southern California Permanente Medical Group

San Diego, CA

Steven Pearson, M.D.

Associate Professor of Ambulatory Care and 

 Prevention

Harvard Medical School

Boston, MA

Eric Peterson, M.D., M.P.H.

Fred Cobb, M.D., Distinguished Professor of 

 Cardiology

Duke University Medical Center

Director, Duke Clinical Research Institute

Durham, NC

Kristofer M. Petren

Patient Registry Specialist

Cystic Fibrosis Foundation

Bethesda, MD

Steve E. Phurrough, M.D., M.P.A.

Medical Officer

Agency for Healthcare Research and Quality

Rockville, MD

Registries for Evaulating Patient Outcomes
275

Richard Platt, M.D., M.Sc.

Professor and Chair, Department of Population 

 Medicine

Harvard Medical School and Harvard Pilgrim 

 Health Care Institute

Boston, MA

Karen L. Posner, Ph.D.

Research Professor, Department of Anesthesiology 

 and Pain Medicine

University of Washington

Seattle, WA

Richard L. Prager, M.D.

Professor, Cardiac Surgery

Head, Section of Adult Cardiac Surgery

Director, Cardiovascular Center

University of Michigan Health System

Ann Arbor, MI

Victoria M. Prescott, J.D.

CEO

McBroom Consulting, LLC

Tampa, FL

Michael T. Rapp, M.D., J.D., FACEP

Centers for Medicare & Medicaid Services

Baltimore, MD

Robert Reynolds, Sc.D.

Senior Director, Global Head, Epidemiology 

 Safety and Risk Management

Pfizer, Inc.

New York, NY

Janice A. Richards, R.N., B.A.

Formerly Project Manager

Registry of the Canadian Stroke Network

Toronto, Ontario, Canada

Karen Richards

Administrative Director, Research and Optimal 

 Patient Care

American College of Surgeons

Chicago, IL

Rachel L. Richesson, Ph.D., M.P.H.

Associate Professor

Duke University 

Durham, NC

Gerald F. Riley, M.S.P.H.

Social Science Research Analyst

Office of Research, Development, and Information

Centers for Medicare & Medicaid Services

Baltimore, MD

Susan Sinclair Roberts, Ph.D.

Associate Professor, Clinical Research Program

University of North Carolina Wilmington

Senior Advisor

INC Research

Wilmington, NC

Nicole Robichaud

Formerly Principal Project Manager, Gaucher 

 Registry

Genzyme, a Sanofi company

Cambridge, MA

Sara Rosenbaum, J.D.

Harold and Jane Hirsh Professor, Health Law and 

 Policy

The George Washington University

School of Public Health and Health Services

Washington, DC

Daniel Rosenberg, Ph.D., M.P.H.

Senior Director, Head Epidemiology & 

 Observational Studies

Actelion Pharmaceuticals Ltd

Allschwil, Switzerland

Kenneth J. Rothman, Dr.P.H.

Vice President, Epidemiology Research

RTI Health Solutions

Research Triangle Park, NC

Koryn Y. Rubin

Senior Manager, Quality Improvement

American Association of Neurological Surgeons/

 Congress of Neurological Surgeons

Washington, DC

Yaffa Rubinstein, Ph.D.

Director of Patient Resources for Clinical and 

 Translational Research

Office of Rare Diseases Research

National Center for Advancing Translational 

 Sciences

National Institutes of Health

Bethesda, MD

Patricia Russo, Ph.D.

Vice President, Integrated Analytic Services

Real-World & Late Phase Research

Quintiles

Rockville, MD

Susan T. Sacks, Ph.D.

Director, Epidemiology

Roche Biometrics

Nutley, NJ

Contributor and Reviewer Affiliations 
276

Gregory Samsa, Ph.D.

Associate Professor, Department of Biostatistics 

 and Bioinformatics

Duke University School of Medicine

Durham, NC

Gillian D. Sanders, Ph.D.

Director, Duke Evidence-based Practice Center 

Associate Professor of Medicine 

Duke Clinical Research Institute 

Durham, NC 

Sheri Sanders, M.B.A., B.S.N., R.N.

Vice President, Registry and Biorepository 

 Services

Caris Life Sciences

Phoenix, AZ

Grannum Sant, M.D., FRCS, FACS

Vice President and Interim Head

Oncology Medical Unit

Head of Urology Portfolio

Sanofi-aventis U.S.

Bridgewater, NJ

Adjunct Professor of Urology, Tufts University School 

of Medicine

Boston, MA

Brian C. Sauer, Ph.D., M.S.

Assistant Professor, Internal Medicine

Adjunct Assistant Professor, Pharmacotherapy and 

 Family and Preventive Medicine

University of Utah School of Medicine

Salt Lake City, UT

William B. Saunders, Ph.D., M.P.H.

Manager, Research & Analytics

GE Medical

Charlotte, NC

Juergen Schmider M.D., Ph.D.

Corporate Safety Officer and Vice President, 

 Global Pharmacovigilance & Epidemiology 

Cephalon, Inc.

Frazer, PA

Peter Schmidt, Ph.D.

CIO and Vice President of Programs

National Parkinson Foundation

Miami, FL

Wendy Schroeder, R.N., B.S.N., CCRC, CRCP, 

CHRC

Formerly Senior Director, Registry and 

 Biorepository Network Management

Caris Life Sciences

Phoenix, AZ

Lee H. Schwamm, M.D., FAHA

Executive Vice Chairman, Department of 

 Neurology

C. Miller Fisher Endowed Chair

Director, MGH Stroke Services

Massachusetts General Hospital

Professor of Neurology

Harvard Medical School

Boston, MA

Barbara Schwierin, Ph.D.

Associate Director, Observational Studies

Actelion Pharmaceuticals, Ltd.

Allschwil, Switzerland

Art Sedrakyan, M.D., Ph.D.

Associate Professor

Director, Patient-Centered Comparative 

 Effectiveness Program and MDEpiNet Science 

 and Infrastructure Center

Weill Cornell Medical College and 

New York-Presbyterian Hospital

New York, NY

David M. Shahian, M.D.

Vice President, Center for Quality and Safety

Massachusetts General Hospital

Professor of Surgery, Harvard Medical School

Boston, MA

Neha Sheth, Pharm.D.

Executive Director, Risk Management Strategy & 

 Pharmacoepidemiology

Product Safety Pharmacovigilance

Astellas Pharma Global Development

Northbrook, IL

Christopher Simmons, B.S., B.A.

Clinical Research and Database Program Associate

American Society of Plastic Surgeons

Arlington Heights, IL

Kumar Sivaraman

Director of Product Strategy, Health Sciences 

 Global Business Unit

Oracle

Los Angeles, CA

James G. Snell, J.D.

Co-chair, Intellectual Property Group and Privacy 

 and Security Group 

Bingham McCutchen, LLP 

Palo Alto, CA

Rachel Sobel, Dr.P.H.

Senior Director, Epidemiology

Pfizer

New York, NY

Registries for Evaulating Patient Outcomes
277

Henrik Toft Sørensen, M.D., Ph.D., Dr.Med.Sci.

Professor, Chair

Department of Clinical Epidemiology

Aarhus University Hospital

Aarhus, Denmark

John Spertus, M.D., M.P.H., FACC, FAHA

Deputy Editor, Circulation: Cardiovascular Quality 

 and Outcomes Research

Professor, University of Missouri – Kansas City

Clinical Director of Outcomes Research

Mid America Heart Institute of Saint Luke’s 

 Hospital

Kansas City, MO

Judy A. Staffa, Ph.D., R.Ph.

Director, Division of Epidemiology II

Office of Surveillance & Epidemiology

Center for Drug Evaluation & Research

Food and Drug Administration

Silver Spring, MD

Paul Stang, Ph.D.

Senior Director: Epidemiology

Janssen Research & Development, LLC / Johnson 

 & Johnson

Titusville, NJ

Kathryn Starzyk, Sc.M.

Director, Epidemiology

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Charlie Strange, M.D.

Professor of Pulmonary, Critical Care, Allergy, and 

 Sleep Medicine

Medical University of South Carolina

Charleston, SC

Walter L. Straus, M.D., M.P.H.

Global Director for Scientific Affairs - Vaccines

Merck Research Laboratories

Adjunct Scholar, Center for Clinical Epidemiology 

 and Biostatistics

University of Pennsylvania School of Medicine

Upper Gwynedd, PA

Edna Stoehr

Global Study Manager

Genentech, Inc.

South San Francisco, CA

Marshall Summar, M.D.

Division Chief, Genetics and Metabolism

Children’s National Medical Center

Professor, Pediatrics

George Washington University School of Medicine 

 and Health Sciences

Washington, DC

Elizabeth J. Swain

Director of Pharmacovigilance Advocacy and 

 Policy

Global Clinical Safety and Pharmacovigilance

GlaxoSmithKline

Stevenage, United Kingdom

Deborah Symmons, M.D.

Professor of Rheumatology and Musculoskeletal 

 Epidemiology

University of Manchester

Manchester, United Kingdom

Michelle Tarver, M.D., Ph.D.

Medical Officer/Epidemiologist

Center for Devices and Radiological Health

Food and Drug Administration

Silver Spring, MD

Melissa S. Tassinari, Ph.D.

Senior Clinical Advisor, Pediatric and Maternal 

 Health Staff

Office of New Drugs 

Center for Drug Evaluation and Research

Food and Drug Administration

Silver Spring, MD

Dale R. Tavris, M.D., M.P.H.

Medical Officer/Epidemiologist

Center for Devices and Radiological Health 

Food and Drug Administration

Silver Spring, MD

Michael D. Taylor, Pharm.D., Ph.D.

Global Head of Epidemiology for Oncology

Genentech, A Member of the Roche Group

South San Francisco, CA

Joseph Terdiman, M.D., Ph.D.

Formerly Director of Information Technology

Division of Research

Kaiser Permanente

Oakland, CA

Contributor and Reviewer Affiliations 
278

Jane Hyatt Thorpe, J.D.

Associate Professor, School of Public Health and 

 Health Services

Director, Healthcare Corporate Compliance 

 Program

The George Washington University

Washington, DC

Hugh Tilson, M.D., M.P.H., Dr.P.H.

Adjunct Professor, Public Health Leadership

University of North Carolina Gillings School of 

 Global Public Health

Chapel Hill, NC

Stephen Toovey, M.D., Ph.D.

Chief Medical Officer

d3 Ltd

Hong Kong, China

Anne E. Trontell, M.D., M.P.H.

Program Director, Centers for Education and 

 Research on Therapeutics 

Center for Outcomes and Evidence 

Agency for Healthcare Research and Quality 

Rockville, MD 

Kelly Trout, B.S.N., R.N.

International WAGR Syndrome Association

San Antonio, TX

Donald C. Tyler, M.D., M.B.A.

Executive Director, Wake Up Safe

Society for Pediatric Anesthesia

Richmond, VA

Jack V. Tu, M.D., Ph.D.

Professor of Medicine, University of Toronto

Staff Physician, Sunnybrook Schulich Heart 

 Centre

Senior Scientist, Institute for Clinical Evaluative 

 Sciences

Ontario, Canada

Marlous Van Laar, M.Sc.

Research Statistician, Division of Epidemiology

School of Medicine 

University of Leeds

Leeds, United Kingdom

Paul D. Varosy, M.D.

Director of Cardiac Electrophysiology

VA Eastern Colorado Health Care System

Associate Professor of Medicine

University of Colorado Denver

Denver, CO

Priscilla Velentgas, Ph.D.

Senior Director of Epidemiology

Real-World & Late Phase Research

Quintiles

Cambridge, MA

Lisa Griffin Vincent, M.A., Ph.D.

Chief Regulatory Officer and President, Outcomes 

 Research Solutions

PatientPoint

Chicago, IL

Jeffrey Wagener, M.D.

Professor of Pediatrics

University of Colorado Medical School

Aurora, CO

Cunlin Wang, M.D., Ph.D.

Medical Epidemiologist, Center for Device and 

 Radiological Health

Food and Drug Administration

Silver Spring, MD

Assistant Professor

Johns Hopkins Medical Institutions

Baltimore, MD

Roshanthi Weerasinghe, M.P.H.

Research Associate, Providence Regional Breast 

 Health Registry

Providence Cancer Center

Portland, OR

Christopher Weiser, J.D., M.P.H.

Health Insurance Specialist

Centers for Medicare and Medicaid Services

Washington, DC

Jennifer Weisman, Ph.D.

Staff Scientist

Strategic Analysis, Inc.

Arlington, VA

Suzanne L. West, M.P.H, Ph.D.

Health, Social, and Economics Research

RTI International

Research Triangle Park, NC

David S. Wendler, M.A., Ph.D.

Head, Unit on Vulnerable Populations Department 

 of Bioethics

National Institutes of Health Clinical Center

Bethesda, MD

Registries for Evaulating Patient Outcomes
279

Marcy Wilder, J.D.

Partner

Hogan & Hartson LLP

Washington, DC

Marcus D. Wilson, Pharm.D.

President

HealthCore

Wilmington, DE

Almut Winterstein, Ph.D.

Professor, Department of Pharmaceutical 

 Outcomes and Policy

University of Florida

Gainesville, FL

Randy Woodward

Formerly Senior Application Developer

Division of Emergency Medicine, Center for 

 Simulation and Research

Cincinnati Children’s Hospital Medical Center

Cincinnati, OH

Albert Wu, M.D., M.P.H.

Professor of Health Policy and Management

Director, Center for Health Services and Outcomes 

 Research

Johns Hopkins Bloomberg School of Public Health

Baltimore, MD

John Yee, M.D., M.P.H.

Formerly Vice President, US and Global Medical 

 Affairs

Genzyme, a Sanofi company

Cambridge, MA

Marianne Ulcickas Yood, D.Sc., M.P.H.

Principal Epidemiologist, EpiSource

Hamden, CT

Associate Research Scientist, Epidemiology and 

 Public Health, Yale University School of Medicine

New Haven, CT

Contributor and Reviewer Affiliations 

Appendixes

283

Appendix A. An Illustration of Sample Size Calculations

As a general principle, sample size calculations 

depend on the study design, the study question, 

and the scale of measurement of the variables 

being measured. Indeed, one of the benefits of 

performing a sample size calculation is the 

requirement that each of these elements be 

specified, thus increasing the likelihood that the 

proper variables will be measured on the proper 

patients in the proper manner.

For concreteness, assume that the outcome of 

interest is a dichotomous variable measured for 

each patient, such as the presence/absence of a 

complication associated with carotid 

endarterectomy (CE). Typically, this literature 

considers complications within 30 days of the 

procedure. Nothing essential changes for outcome 

variables measured on other scales, such as 

continuous or survival data. The dichotomous 

outcome (i.e., presence or absence of a 

complication) is then aggregated across patients 

into a complication rate (e.g., 9 complications for 

300 patients equals a 3-percent complication rate).

For CE, some registry-based designs and study 

questions that might be of interest include the 

following. For the purpose of this discussion, 

case-mix adjustment is the incorporation of 

various patient characteristics believed to influence 

complications of CE into a mathematical model 

used to predict the likelihood of these 

complications. The most natural such model is a 

logistic regression.

Design 1: For patients at high risk of stroke, 

perhaps using an operational definition of 

“symptomatic with 70–99 percent stenosis of the 

carotid artery,” the study question is whether the 

surgeons within a larger entity (e.g., a national 

chain of hospitals) are, in aggregate, experiencing 

complication rates similar to those who 

participated in the randomized trials demonstrating 

the efficacy of CE. The reason this is an open 

question is that the surgeons and institutions in 

these randomized trials have undergone a high 

degree of selection, so that there is a concern that 

their surgical outcomes may be better than could 

be expected in usual practice.

The patient inclusion criteria for the registry are 

selected to be as close as possible to those of the 

randomized trials; thus, while various 

characteristics might be collected on each patient, 

no formal case-mix adjustment is required.

Further, suppose that the 30-day complication rate 

of CE in the randomized trials was 3 percent. The 

study question can then be translated into a 

statistical hypothesis of a one-sample comparison 

of an observed complication rate versus a 

prespecified value. In other words, the null 

hypothesis is that surgeons within the larger entity 

are, in aggregate, experiencing complication rates 

that are the same (3%) as those of surgeons who 

participated in the randomized trials. The final 

input required to perform the sample size 

calculation is the complication rate under the 

alternative hypothesis. For example, if it is 

determined that the goal of the registry is to have 

high power to flag results as statistically significant 

if the true complication rate is 6 percent or higher, 

then the complication rate under the alternative 

hypothesis is 6 percent.

In general, the value of the complication rate under 

the alternative hypothesis is derived using a 

combination of quantitative and qualitative 

reasoning. The precise methods used are context 

dependent and thus not discussed in detail here. In 

the present example, a cost-effectiveness analysis 

might suggest that complication rates of 6 percent 

and above would call into question the efficacy of 

CE. Given these inputs, it can be shown that the 

effect size is 0.21, and the sample size required for 

80-percent power is approximately 370.

Design 2: Continuing to follow patients at high 

risk of stroke, now suppose that the goal of the 

registry is to compare complication rates across 

hospitals. For simplicity, we continue to assume 

that patients are sufficiently similar to the 

comparator patients that no explicit adjustment for 

case mix is required.
284

Registries for Evaulating Patient Outcomes

Design 2 is a simple form of benchmarking 

application. For example, the CE complication 

rates for each hospital might be reported to a 

regulatory agency and/or the general public, on the 

presumption that statistically significant 

differences between complication rates can be 

used to identify hospitals with differences in 

quality of care. The particular danger in this design 

is that the complication rate for any particular 

hospital might be estimated with relatively little 

precision, thus generating results that have more 

noise than signal. Another danger, discussed later, 

is that case-mix adjustment is required and not 

performed, or performed, but not adequately.

We assume that the benchmarking will focus on 

comparing specific hospitals—i.e., in the 

underlying statistical model, hospital will represent 

a fixed rather than random effect. The null 

hypothesis is that the complication rates for all the 

hospitals are identical, and the alternative 

hypothesis is that the complication rates follow 

some pattern other than being identical. In this 

design, specifying the alternative hypothesis of 

interest is a potentially formidable task. One way 

to formulate this hypothesis is to focus on outlier 

hospitals. For example, suppose that there are 10 

hospitals in the registry, the overall complication 

rate among 9 of these is expected to be 3 percent, 

and the complication rate at the tenth hospital is 10 

percent. This information, along with expected 

number of cases in each hospital, is sufficient to 

calculate an effect size and thus perform the 

sample size calculation.

When comparing complication rates among 

specific hospitals, some adjustment may be made 

for multiple comparisons—that is, in any group of 

hospitals, there will always be a hospital with the 

highest complication rate, and focusing on 

differences between the outcomes of this particular 

hospital versus outcomes of the others will 

overstate the level of statistical significance. The 

initial statistical test used to assess the 

homogeneity of complication rates across all the 

hospitals in the registry implicitly takes this 

multiple-comparison problem into account. 

Subsequent tests, in particular those tests that 

compare apparent outlier hospitals with others, 

should include an explicit adjustment for multiple 

comparisons, and the sample size calculations 

should reflect the fact that an adjusted comparison 

is being made.

In practice, the approach to this design might 

reasonably depend on whether registry data are 

being collected electronically or manually. If data 

are being collected electronically, the most 

sensible policy is to collect information on all CE 

procedures performed within each hospital and to 

use the sample size formula as an assessment of 

whether the registry as a whole is likely to produce 

results that are sufficiently accurate to support 

decisionmaking. This assessment can be framed in 

terms of statistical power, as discussed above, or in 

terms of precision.

Considering precision, a 95-percent confidence 

interval for a nonzero complication rate for any 

hospital is p ± 1.96 sqrt (pq/n), where p is the 

observed complication rate, q = 1- p, and n is the 

sample size. Supposing that p = 3 percent and 

n = 300 per hospital, within any particular 

hospital, the width of this confidence interval is 

expected to be approximately ±1.9 percent. If data 

are being collected manually, and thus the 

marginal cost of data collection per patient is high, 

a reasonable policy would be to collect data on a 

sufficient number of patients in each hospital so 

that the precision of the estimates of the 

complication rate within a given hospital would be 

considered adequate.

As with hypothesis testing, the analysis to derive 

the width of the confidence interval usually applies 

a combination of qualitative and quantitative 

insights. In particular, the question can be 

reframed as the following: For what values of the 

complication rate will my decision (whether taken 

from the perspective of clinical medicine, public 

health, etc.) be the same? For example, if the 

decision is the same regardless of where the 

complication rate falls within the range of 2 to 4 

percent, an interval of this width is sufficiently 

precise.

Unless sample sizes are large, using registries to 

compare individual hospitals is potentially quite 

problematic. Although determining the inputs to 
285

Appendixes

the power calculations is not always a 

straightforward task, performing this analysis is 

quite useful, even if the result is only to suggest 

extreme caution in the interpretation of differences 

between hospitals.

Design 3: Continuing to follow patients 

undergoing CE, now suppose that the goal of the 

registry is to compare two different versions of the 

surgical procedure. For simplicity, continue to 

assume that patients are sufficiently similar to the 

comparator patients that no explicit adjustment for 

case mix is required. The following discussion 

(after including an adjustment for case mix, if 

appropriate) also applies to comparisons of two 

different versions of a medical device and similar 

applications. The key distinctions between this 

design and Design 2 are that in Design 3 the 

primary comparison or comparisons can be stated 

ahead of time and the number of comparisons is 

relatively small, so that the issue of multiple 

comparisons can be ignored.

The analytic approach to this design is a logistic 

regression, with the input file having one record 

per patient. The outcome variable is the presence 

or absence of a complication, the categorically 

scaled control variable is the hospital, and the 

primary predictor is the categorically scaled 

coding of the type of surgical procedure (i.e., CE 

using version A vs. CE using version B). The null 

hypothesis is that, after accounting for any 

differences in hospitals, the two different versions 

of the procedure have identical complication rates. 

The alternative hypothesis is that the rates differ by 

a specified amount, this amount being the 

minimum clinically significant difference

interpreted to be of concern. Power calculations 

proceed in the same fashion as for logistic 

regression with multiple predictors.

The main pitfall in this design is that patients who 

receive version A of the surgical procedure might 

differ from those who receive version B of the 

procedure along some dimension that has an 

impact on outcomes. (This pitfall is discussed in 

more detail under Design 4.)

In this application, the null and alternative 

hypotheses are sometimes structured the same way 

as in an equivalence trial—that is, differences in 

complication rates are not expected, and the goal 

of the study is to demonstrate that complication 

rates for the two versions of the surgical procedure 

are similar within a certain level of precision. The 

structure of the analysis is not fundamentally 

different. Indeed, sample size calculations for 

equivalence trials are sometimes not performed 

within a hypothesis-testing framework but instead 

by identifying a sample size of sufficient 

magnitude to make the confidence interval for the 

difference in the complication rates between the 

two versions of the surgical procedure a certain 

width. For simplicity of presentation, let us assume 

from now on that any equivalence-trial-type 

calculations can be reframed into confidenceinterval format, and thus need not be discussed 

separately.

Design 4: Continuing to follow patients at high 

risk of stroke, and continuing to assume that the 

goal of the registry is to compare two different 

versions of the surgical procedure, now 

additionally assume that this comparison will 

include an adjustment for case mix. Within the 

logistic regression paradigm, variables used to 

adjust for case mix are accounted for as covariates 

(i.e., additional predictors). Alternatively, 

propensity-scoring methods could be used to 

adjust for those variables that predict the 

assignment of patients to particular versions of the 

procedure. For concreteness, let us focus on 

logistic regression. In order to perform a sample 

size calculation for a logistic regression, the 

analyst must specify the predictive ability of the 

covariates and the odds ratio associated with the 

predictor of interest. (For example, version B of 

the procedure might increase the odds of 

complications by a factor of 1.5.) Once these 

inputs are specified, the sample size calculation is 

straightforward.

Both the logistic-regression and propensity-scoring 

approaches suffer from the fundamental drawback 

that they can adjust only for covariates that are 

observed. In particular, if there are variables that 

predict outcome that are unmeasured (e.g., a 

physician’s assessment of a patient’s likelihood to 

comply with treatment, or an assessment of “stroke 

in evolution” not included in the administrative 

database used as the source of data for the 
286

Registries for Evaulating Patient Outcomes

registry), then the comparison between the two 

versions of the surgical procedure is potentially 

biased. Accordingly, before proposing to use a 

registry to compare complication rates (e.g., across 

different versions of a procedure or a device) or 

other outcomes, it is critical to determine that the 

following three conditions do not all hold: 

(1) a patient, provider, system, or other 

characteristic affects the complication rate; 

(2) this characteristic is unmeasured within the 

registry; and (3) there is a reasonable likelihood 

that this characteristic might be differentially 

distributed across the different versions of the 

procedure or the device. If all three conditions (in 

epidemiologic terms, the conditions for 

confounding) hold, use of the registry to compare 

outcomes is potentially dangerous.

Critical to Designs 1–4 is the assumption that the 

CE complication rate is stable over time. If this is 

the case, it is appropriate to use the registry to 

estimate a single complication rate associated with 

version A of the procedure, estimate another single 

complication rate associated with version B of the 

procedure, and compare the rates. On the other 

hand, if the technology of CE (e.g., physical 

materials, surgical technique) is improving, then 

the registry should continue to monitor the 

performance of CE over time. Such an ongoing 

monitoring function seems particularly relevant for 

medical devices and similar applications.

Even when the associated technology is assumed 

stable, some registries are intended to provide 

ongoing assessments of outcomes. For example, in 

a quality assurance context, CE complication rates 

might be assessed at individual hospitals on an 

annual basis (e.g., in order to check for problems 

that have recently arisen). On the other hand, a 

registry whose purpose is to assess whether 

complication rates observed in randomized trials 

could be achieved in usual practice could be 

designed with a sunset provision to cease operation 

once this question is answered. The latter type of 

registry might, for example, support a coverage 

decision by the Centers for Medicare & Medicaid 

Services.

Having an ongoing monitoring function induces 

additional analytical complications, among others 

a multiple-comparisons problem. Traditional 

statistical power calculations are performed under 

the assumption that the sample size is fixed and 

that, unless otherwise noted, multiple comparisons 

are not a major issue. Sequential testing methods 

associated with randomized trials (where, for 

example, the type I error of .05 is apportioned into 

an early test with alpha = .001 and a subsequent 

test with alpha = .499) are not appropriate for this 

particular design, since most of these methods 

assume that the maximum sample size is fixed. 

Some methods assume that what is fixed is not the 

number of patients but the number of events, but 

these methods are also inappropriate for registry 

applications.

Design 5: Suppose the goal is to estimate the 

complication rate associated with CE at multiple 

time points for the foreseeable future. Control 

chart methodology might reasonably be applied to 

this class of problems. This methodology, often 

used in the quality assurance and quality 

improvement context, was originally developed for 

industrial applications. In this example, the null 

hypothesis, under which the system in question is 

“in control,” is that the CE complication rate 

remains at the desired value of 3 percent 

throughout the entire followup period. Samples are 

taken at given points in time (e.g., monthly). As an 

example, if these monthly samples are of size 100, 

then the standard error is approximately 1.7 

percent. The analyst then creates a control chart by 

plotting these monthly complication rates over 

time and forming channels based on the standard 

error. In this example, the channel extending from 

the point estimate to 1 standard error above the 

point estimate is 3 percent to 4.7 percent.

Once the basic control chart—which goes by 

different names depending on the scale of 

measurement of the outcome variable—is formed, 

the plot is checked for various violations of the 

null hypothesis of constant complication rates. The 

set of possible violations to be flagged as 

statistically significant might include (1) any 

observation more than 3 standard errors from the 

mean; (2) two of three consecutive observations 

more than 2 standard errors from the mean; 

(3) eight observations in a row that increase or 

decrease; and (4) eight observations in a row on 

one side of the mean. These rules of thumb 
287

Appendixes

implicitly take into account the multiplecomparisons problem by requiring noteworthy 

departures from the null hypothesis in order to be 

flagged; they are also based on the observed 

properties of physical machines as they fall out of 

adjustment: suddenly breaking down and 

producing an extreme outlier, or gradually heating 

and thus producing sequentially higher readings. 

Complication rates of CE might or might not 

follow the properties of physical machines, but the 

decision rules from control chart methodology are 

at least a good place to start.

289

Appendix B. Copyright Law

Copyright law confers exclusive legal rights on the 

owner of the copyright.1 The exclusive rights of 

copyright may be sold, assigned (transferred), or 

licensed (limited transfer of rights for use on 

specific terms or conditions) to others; these rights 

may also be waived (quit claim). Licensing 

ordinarily consists of a private agreement governed 

by contract rather than copyright law.2

However, the exclusive rights conferred by 

copyright to prepare derivative works and 

distribute copies of a health information registry 

may be limited by regulatory requirements. Health 

Insurance Portability and Accountability Act 

(HIPAA) Privacy Rule restrictions may limit data 

use, reuse, and disclosures or may require 

additional patient authorizations for subsequent 

research use. The conditions of institutional review 

board (IRB) approval under the Common Rule 

may also limit reuse and further disclosure of 

registry data. The terms of patient authorization 

and consent, a data use agreement, or a business 

associate agreement may modify the scope and 

nature of rights protected by copyright law. These 

limitations can be avoided by the use of deidentified health information, as defined by the 

Privacy Rule, plus information that is not subject 

to the Common Rule, if they suffice for the 

scientific or other purposes of the registry. Without 

resort to copyright protections, State laws may 

directly restrict access to registry data, as well as 

the use and disclosure of data from registries 

developed by public health agencies.

Formal copyright registration3 with the U.S. 

Copyright Office is not necessary but may be 

desirable for registries anticipated to have 

commercial value. The owner of a copyright is 

generally the author4 or author’s employer; 

ownership of the copyright for a compilation is not 

ownership of the underlying facts or data.5

Copyright law presumes that an employer owns the 

copyright in materials created by an employee 

within the scope of his or her employment as a 

“work made for hire.”6 Institutional policies and 

procedures frequently prescribe whether the 

registry developer, his or her employer, or a 

funding agency owns the copyright. Employee 

manuals often contain an employer’s position on 

the intellectual property created by employees. 

Research institutions frequently reserve the right 

to the intellectual property produced by their 

employees. Intellectual property issues are 

explicitly negotiated in most sponsored research 

contracts. Authors of a joint work are co-owners of 

copyright in the work.4

Several factors determine whether the use of a 

registry protected by copyright for scholarship, 

research, or certain other purposes is within the 

statutory fair use limitation on copyright.7 In 

general, these factors will support subsequent uses 

of registry data for research, even though it may be 

protected by copyright. In any given set of 

circumstances, a specific analysis of the statutory 

factors is necessary to determine whether use is 

likely to be viewed within the fair use limitation on 

copyright.1

Copyright law may provide some legal protections 

for compilations such as health information 

registries. The extent of this protection depends on 

the specific characteristics of the registry. In 

general, the concept of ownership does not 

comfortably apply to health information, even 

when limited to copyright. Nevertheless, some 

registry developers may want to consider adding 

the legal protections of copyright to reinforce 

controls on access to and use of registry data. 

Registry developers may also encounter copyright 

protections on health information held by health 

care providers. Use of health information protected 

by copyright for research purposes may constitute 

fair use under copyright law.
290

References for Appendix B

1. Harris RK, S.S. R. Copyright protection for 

genetic databases. Jurimetrics Journal. 

2005;45:225-50.

2. 17 U.S.C. § 106.

3. United States Copyright Office. http://www.

copyright.gov. Accessed July 29, 2010.

4. 17 U.S.C. § 201(a).

5. 17 U.S.C. § 102(b). Feist Publications, Inc., v. 

Rural Telephone Service, Co., Inc., 499 U.S. 340 

(1991).

6. 17 U.S.C. § 201(b).

7. 17 U.S.C. § §107

Registries for Evaulating Patient Outcomes
291

Appendix C. Relevant Entities in Health Information 

Technology Standards

The Clinical Data Interchange Standards 

Consortium, or CDISC, is a multidisciplinary 

nonprofit organization that is focused specifically 

on medical research and that works toward 

developing and supporting global, platformindependent data standards that enable information 

system interoperability. It is a membership 

organization made up of more than 170 academic 

research centers, global biopharmaceutical 

companies, technology and service providers, and 

institutional review boards.1 CDISC has 

established standards to support the acquisition, 

exchange, submission, and archiving of clinical 

research data and metadata, such as case report 

tabulation data definitions, submission data, and 

operational data modeling; these standards are 

intentionally vendor neutral, platform independent, 

and freely available. CDISC has formed key 

partnerships with other standards bodies, vendors, 

and research groups to further the creation and use 

of these and other industry standards. CDISC’s 

Healthcare Link project is an initiative that 

specifically focuses on the mission of 

interoperability between health care and clinical 

research. 

Health Level Seven, or HL7, is an American 

National Standards Institute (ANSI)–accredited 

nonprofit organization that produces specifications 

and protocols for clinical and administrative health 

care data.2 HL7 is a global organization with 

corporate and individual membership consisting of 

providers, vendors, payers, consultants, and 

government groups. Like CDISC, HL7 does not 

develop software, but instead creates 

specifications. HL7’s original specification was a 

messaging standard that enables disparate health 

care applications to exchange key sets of clinical 

and administrative data.3 This standard defines the 

structure and content of the messages that are 

exchanged between systems in either batch mode, 

which facilitates transfer of a collection of 

individual messages labeled by a single header, or 

interactive mode, which transmits a single 

message. HL7 then extended this idea to a Clinical 

Document Architecture (CDA®), which is 

designed to support standards for storing and 

retrieving file-level information such as electronic 

health records (EHRs).3 The Reference 

Information Model then specifies the details, 

results, and contexts of clinical informatics by 

defining subject areas, classes, attributes, use 

cases, and trigger events (such as a followup 

clinical visit). HL7 also houses important 

specifications and tools relating to electronic 

documentation of standards, for example, the 

Continuity of Care Document.

The Healthcare Information and Management 

Systems Society, or HIMSS, is an industry 

membership organization that focuses on 

knowledge sharing, advocacy, and collaboration 

among its members. HIMSS is a longstanding 

advocate of using information management 

systems to improve health care, and represents a 

large portion of the industry (more than 20,000 

individuals and 350 corporations).4 HIMSS plays a 

critical role in this discussion through the HIMSS 

Electronic Health Record Association, and also 

through its role in partnering with two other key 

standards groups: the Health Information 

Technology Standards Panel (HITSP) and 

Integrating the Healthcare Enterprise (IHE).

The HIMSS Electronic Health Record Association 

(EHRA) is a trade association specifically made up 

of EHR companies. This association is a key 

player in the interoperability discussion. EHRA 

focuses on creating interoperable EHRs in hospital 

and ambulatory care settings by providing a forum 

and structure for EHR leaders to work toward 

standards development, interoperability, the EHR 

certification process, performance and quality 

measures, health information technology 

legislation, and other EHR issues.5 
292

Registries for Evaulating Patient Outcomes

IHE is an initiative sponsored by HIMSS, the 

Radiological Society of North America, and the 

American College of Cardiology.6 It is designed 

specifically to bridge the gap between existing 

standards and the implementation of integrated 

systems. IHE does this by creating Profiles, which 

specify precisely how standards are to be used in 

integration implementations. It is important to note 

that IHE does not develop standards; instead, it 

provides a link between existing standards and the 

problems within the industry that need to be 

solved. The initiative is focused on eliminating 

ambiguities, reducing configuration and 

interfacing costs, and ensuring a higher level of 

practical interoperability for users and developers 

of health care information technology as they 

implement standards-based communication 

between systems and then perform tests to 

determine that the implementation conforms to the 

specifications.7 In recent years, IHE has developed 

the Patient Identifier Cross Referencing (PIX) 

Integration Profile, which supports the crossreferencing of patient identifiers from multiple 

domains,8 and the Patient Demographics Query 

(PDQ) Integration Profile, which facilitates the 

querying of a patient database to retrieve 

demographics data.9 Standards from different 

organizations that achieve the same goal can be 

inserted into an IHE Profile, and IHE will then 

produce technical specifications that can be used 

by developers and vendors to build products 

compliant with those standards. Because of IHE’s 

practical approach, its value has been recognized 

by other standards organizations, particularly 

CDISC. For example, IHE has defined a simple 

four-step process that carries a specific problem 

from problem definition, through implementation 

and testing, to the real world:

1. Identify interoperability problem.

2. Specify Integration Profiles.

3. Test systems at Connectathon (an annual 

weeklong interoperability-testing event); demo 

at HIMSS Interoperability Showcase.

4. Implement in real world.

HITSP serves as a partnership between the public 

and private sectors with the purpose of identifying 

a widely accepted set of standards for 

interoperability of health care applications. HITSP 

is funded by the U.S. Department of Health and 

Human Services, administered by ANSI, and 

tightly partnered with HIMSS; Federal agencies 

are mandated to use interoperability standards that 

have been harmonized by HITSP.10

The Certification Commission for Health 

Information Technology (CCHIT®) is a private 

nonprofit organization with the “sole public 

mission of accelerating the adoption of robust, 

interoperable health information technology by 

creating a credible, efficient certification 

process.”11 It is divided into workgroups that 

address the standards for specific functional areas 

such as ambulatory care, behavioral health, 

personal health records, and cardiovascular care. 

Since being recognized as a certifying body by the 

Department of Health and Human Services in 

2006, it remains the only federally approved 

organization to certify health information 

technology products and systems.

The Regenstrief Institute, Inc., is an informatics 

and health care research organization and a joint 

enterprise of the Regenstrief Foundation, Inc., the 

Indiana University School of Medicine, and the 

Health and Hospital Corporation of Marion 

County. Regenstrief is active in developing health 

care informatics standards, including the widelyused Logical Observation Identifiers Names and 

Codes (LOINC®) terminology.12 

Table C–1 provides details about the 

establishment, membership, and mission of the 

organizations described above, along with a listing 

of standards/specifications pertaining to each 

organization.
293

Appendixes

Table C–1. Relevant entities in health information technology standards

Group

Year 

Established

Number of 

Members Mission

Relevant Standards/

Specifications

CDISC 2000 >290 (corporate) Developing and supporting data 

standards.

CDASH

HL7 1987 >4,000 Producing specifications and protocols 

for clinical and administrative health 

care data.

CDA, RIM, CCD

HIMSS 1961 >570 (corporate)

>44,000 

(individuals)

Knowledge sharing, advocacy, and 

collaboration

EHRA 2004 ~40 (corporate) Creating interoperability between 

existing EHRs.

EHRA 

Interoperability 

Roadmap

IHE 1997 >540 

(organizations)

Providing a link point between the 

standards that exist and the problems 

among the industry that need to be 

solved.

RFD, CRD

HITSP 2005 >550 

(corporate and 

organizations)

Partnering with public and private 

sectors to achieve standards to support 

interoperability among health care 

software applications.

TP50, C76

CCHIT 2004 94 products 

certified under 

2011/2012 

CCHIT criteria

Defines the requirements for an EHR 

to be certified in the United States.

CCHIT certification 

criteria (available at 

ww.cchit.org/certify)

Regenstrief 

Institute

1969 >50 

(investigators)

Improvement of health through 

research that enhances the quality and 

cost-effectiveness of health care.

LOINC®

C76 = HITSP Case Report Pre-Populate Component; CCD = HL7 Continuity of Care Document; 

CCHIT = Certification Commission for Health Information Technology; CDA = HL7 Clinical Document 

Architecture; CDASH = Clinical Data Acquisitions Standards Harmonization; CDISC = Clinical Data Interchange 

Standards Consortium; CRD = IHE Clinical Research Data Capture; EHR = electronic health record; 

EHRA = Electronic Health Record Association; HIMSS = Healthcare Information and Management Systems 

Society; HITSP = Healthcare Information Technology Standards Panel; 

HL7 = Health Level Seven; IHE = Integrating the Healthcare Enterprise; LOINC = Logical Observation Identifiers 

Names and Codes; RFD = IHE Retrieve Form for Data Capture; RIM = HL7 Reference Information Model; 

TP50 = HITSP Retrieve Form for Data Capture Transaction Package.
294

References for Appendix C

1. Clinical Data Interchange Standards Consortium. 

http://www.cdisc.org. Accessed May 1, 2013.

2. Health Level Seven. http://www.hl7.org. Accessed 

May 1, 2013.

3. HL7. “2010 Standards Descriptions.” http://www.

hl7.org/documentcenter/public/calendarofevents/

himss/2010/About%20HL7%20and%202010%20

Standards%20Descriptions%20Combined.pdf. 

Accessed June 15, 2012.

4. HIMSS. “Interoperability and Standards.” 

http://www.himss.org/ASP/topics_ihe.asp. 

Accessed June 15, 2012.

5. HIMSS. “HIMSS EHR Association.” 

http://www.himssehra.org/ASP/index.asp. 

Accessed June 15, 2012.

6. HIMSS. “Integrating the Healthcare Enterprise 

(IHE).” http://www.himss.org/ASP/topics_ihe.asp. 

Accessed June 15, 2012.

7. Integrating the Healthcare Enterprise. “About 

IHE.” http://www.ihe.net/About_IHE/. Accessed 

December 23, 2013.

8. Integrating the Healthcare Enterprise. Patient 

Identifier Cross-Referencing. http://wiki.ihe.net/

index.php?title=Patient_Identifier_CrossReferencing. Accessed June 15, 2012.

9. Integrating the Healthcare Enterprise. Patient 

Demographics Query. http://wiki.ihe.net/index.

php?title=Patient_Demographics_Query. Accessed 

June 15, 2012.

10. HITSP. “Transaction Package.” http://www.hitsp.

org/ConstructSet_Details.aspx?&PrefixAlpha=2&

PrefixNumeric=50. Accessed June 15, 2012.

11. CCHIT. “About CCHIT.” http://www.cchit.org/

about/. Accessed June 15, 2012.

12. The Regenstrief Institute, Inc. 

http://www.regenstrief.org/. Accessed September 

30, 2013.

Registries for Evaulating Patient Outcomes
295

Appendix D. Linking Clinical Registry Data With 

Insurance Claims Files

A research project is being designed to compare 

the effectiveness for treating diabetes of one class 

of medication, or one specific generic medication 

within the class, to another. The results should 

provide scientific evidence for patients, physicians, 

and policymakers to use to make decisions about 

the use of these drugs.

Registry developer A will collect limited data sets 

of information on patients discharged with a 

diagnosis of diabetes from hospitals in three 

States. These limited data sets do not include 

patient names or direct identifiers, and so are not 

considered individually identifiable health 

information under the Health Insurance Portability 

and Accountability Act (HIPAA) Privacy Rule. The 

registry developer has institutional review board 

approval to use the data for research purposes. The 

hospitals will provide the data sets to group B 

under a data use agreement that complies with the 

HIPAA Privacy Rule.

Group B intends to perform probabilistic matching 

of the registry data to a health insurance claims 

database to determine diabetes treatment 

outcomes. Registry developer A and research 

group B have entered into a formal collaboration 

for this research project.

The health insurance database will be derived from 

the claims data of multiple health plans operating 

in the same three States. The insurers’ original data 

sets include direct beneficiary identifiers and 

constitute protected health information under the 

HIPAA Privacy Rule. Because the registry 

contains only limited data sets, the claims data 

collected in the insurance database will have to be 

linked to the registry data using probabilistic 

matching techniques.

Consequently, the research project will use only a 

limited data set of health insurance claims data to 

create the link with the registry data. The health 

insurance companies will provide the limited data 

sets of claims information to group B under data 

use agreements that comply with the HIPAA 

Privacy Rule.

The common data elements for the insurance 

database and the registry that will be used for 

linkage are date of birth, gender, race, hospital ID, 

State of hospital, date of admission, date of 

discharge, date of death (if the patient died), 

ICD-9 (International Classification of Diseases, 

9th Revision) code for primary diagnosis for the 

index hospitalization, primary procedure code for 

the index hospitalization, and ZIP Code for the 

patient’s address.

In order to protect the identity of the hospitals, the 

researchers were asked to sign a confidentiality 

agreement that specifically defined the registry 

operator’s proprietary information. Such 

proprietary information included the names or 

other identifiers of hospitals or other health care 

facilities participating in the registry. The 

researchers were precluded from using the names 

or other distinguishing characteristics of the 

hospitals in any public document, including 

publications or marketing materials. The 

confidentiality agreement did allow the researchers 

to retain an identifier number for each hospital, as 

long as that number identified only generic 

characteristics and excluded any information about 

the hospital that would enable anyone to identify 

the specific hospital. For example, the researchers 

could not retain information that classified a 

particular hospital with a number that identified it 

as an academic teaching hospital based in a 

particular State with a certain number of beds, 

since in many instances the identity of the hospital 

could be derived from such information. Due to 

the potential contractual liability that may arise, 

the possibility of identifying participating 

hospitals is a critical issue.

U.S. Department of 

Health and Human Services

Agency for Healthcare Research and Quality

540 Gaither Road

Rockville, MD 20850

AHRQ Pub. No. 13(14)-EHC111

April 2014

ISBN: 978-1-58763-432-1