# Presentation: 390917

## Good Practice for Patient RegistriesAHRQ Annual MeetingBethesda, MD|  September 9, 2008Speaker: Richard Gliklich, MD, Principal Investigator, Outcome DEcIDE Center,President, OutcomeCambridge, MA 

- Good Practice for Patient RegistriesAHRQ Annual MeetingBethesda, MD|  September 9, 2008Speaker: Richard Gliklich, MD, Principal Investigator, Outcome DEcIDE Center,President, OutcomeCambridge, MA

**Notes:**
Thank you Dr. Bartman

## Premise

*	**Registries can provide unique outcomes information on populations under real-world conditions that are not studied in clinical trials.*
*Reality*
*Generalizability*
*Applicability*
*Availability*
*	**	*
*	**What makes a registry good?*

**Notes:**
The premise of today’s panel are the registries are valuable, that they can serve some unique purposes that are important to decision making and complementary to randomized clinical trials.  I like to think of it in terms of filling the gaps left by RCTs for making clinical and policy decisions.  Those gaps fall into four areas—reality—registries can provide informaiton on real-world practice not accessible to clinical trials; generalizability—registries can provide information on subpopulations, like the elderly or infirm, that may not have been studied in clinical trials, applicability—registries can provide information that cannot be captured in protocol driven studies such as the impact of physician choice; and availability—there is not the time or money in the world to fund enough randomized effectiveness studies needed for current decision making.  Observational studies like registries can fill some of those gaps.  But need does not justify use.  Whether to use registries for these purposes depends on whether we can determine if these registries can provide valid and reliable information.  So, the question is, what makes a registry good?

## Current guidelines for pharmacoepidemiologic research do not fully address registries

- Experimental Research
- CONSORT Statement *(JAMA 2001; 285:1987-1991)*
- Observational Research
- Guidelines for good pharmacoepidemiology practices (Pharmacoepidemiology & Drug Safety 2005:14:589-595)
- Quality of Reporting  of Observational Longitudinal Research (AJE 2005:161:280-288)
- Guidance for Industry: Good pharmacovigilance practices & pharmacoepi. assessment.  DHHS, March 2005
- Guidance for Industry: Establishing Pregnancy Exposure Registries, DHHS, August 2002.

**Notes:**
A scan of the available information pre-2007 reveals no guidelines that fully address the quality of registries in the way that the Consort statement addressed randomized cliical trials.

## Registries for Evaluating Patient Outcomes: A User’s Guide

- Commissioned by the Agency for Healthcare Research and Quality and the Centers for Medicare and Medicaid Services in the United States
- Purpose: To promote development of high-quality evidence that can be relied on for public and  private purposes
- Product: Handbook addresses the creation, operation, analysis, interpretation, reporting and evaluation of registries designed to evaluate patient outcomes

**Notes:**
This promise of registries coupled with the lack of good practice indicators led to the development of the handbook, registries for evaluatiing patient outcomes: a user’s guide funded by AHRQ in 2006

## Registries for Evaluating Patient Outcomes: A User’s Guide

- 	PIs:  RE Gliklich, NA Dreyer, Outcome DEcIDE
- 	Collaborative effort
- Outcome Sciences DEcIDE center
- Duke University EPC
- CMS Coverage and Analysis Group
- 39 contributors from industry, academia, health plans, physician societies, U.S. government, NICE
- 35 invited peer reviewers and public comment, including OCR, OHRP, IOM among others
- 	Example driven: ~23 case studies illustrating specific challenges and solutions
- 	Available on-line and in print at http://effectivehealthcare.ahrq.gov

- Gliklich RE, Dreyer NA, eds.
- Registries for Evaluating Patient Outcomes: A User's Guide.
- Prepared by Outcome DEcIDE Center
- AHRQ Publication No. 07-EHC001-1. Rockville, MD:
- Agency for Healthcare Research and Quality. April 2007

**Notes:**
The project, led by the Outcome Decide center, was a collaborative effort that included 39 contributors representing industry, academia, health plans, physician societies, and government agencies. It was reviewed by a nearly equal number of reviewers including representatives of several agencies.  The handbook was designed to be pragmatic and 23 illustrative case studies were included.  It was released in may 2007 and is available on the AHRQ website.

## What is a Patient Registry?

- 	A patient registry:
- Is an organized system that uses observational study methods to collect uniform data (clinical and other)
- Evaluates specified outcomes for a population defined by a particular disease, condition, or exposure, and that
- Serves a predetermined scientific, clinical, or policy purpose

- Derived from: Gliklich RE, Dreyer NA: Registries for Evaluating ient Registries: A User’s Guide:  AHRQ publication No. 07-EHC001. Rockville, MD. April 2007

- Gliklich RE, Dreyer NA, eds. Registries for Evaluating Patient Outcomes: A User's Guide. Prepared by Outcome DEcIDE Center  AHRQ Publication No. 07-EHC001-1. Rockville, MD: Agency for Healthcare Research and Quality. April 2007

**Notes:**
So, what is a patient registry?  A registry is an organized system that uses observational study methods to collect uniform data to evaluate specified outcomes for a population defined by a particular dsiease, condition or exposure that serves a predetermiend scientific, clinical or policy purpose.

## Key Characteristics of Registries

- Data are collected in a naturalistic manner
- Registry is designed to fulfill specific purposes, and these purposes are defined in advance of collecting and analyzing the data
- Registry captures data elements with specific and consistent data definitions
- Data are collected in a uniform manner for every patient.
- Data collected derive from and are reflective of the clinical status of the patient (by history, examination, laboratory test, or patient reported)
- At least one element of registry data collection is active, meaning that some data are collected specifically for the purpose of the registry

**Notes:**
As you will hear from the panel, the key characteristics of a registry are that data are collected in a naturalistic manner, to fulfll specific predetermiend purposes.  

Registries that meet this definition exhibit the following characteristics:

They collect data in a naturalistic manner

They are designed to fulfill a specific purpose, which is defined before the data has been collected and analyzed

They collect data elements with specific and consistent data definitions

They collect data in a uniform manner for each patient, meaning with the use of CRFs

They collect data about the clinical status of the patient

They collect at least some data specifically for the registry

## Typical Goals of Patient Registries

- Track natural history of a disease process
- Measure or monitor safety and harm
- Evaluate clinical, comparative or cost effectiveness
- Measure and/or improve quality of care

**Notes:**
Registries are defined by the purpose or purposes that they are intended to serve. Typical goals of patient registries include describing the natural histroy of a disease, measuring safety and harm of products, evaluating effectiveness and measuring or improving qualiyt of care.

## Taxonomy of Registries: Product

- May include all or a subset of patients exposed to a drug, device or biologic
- Device registries
- Implantable cardioverter defibrillators (ICD)
- Stents
- Orthopedic devices
- Pharmaceutical product registries
- Cox 2 inhibitors
- Thalidomide
- Pregnancy registries
- Exposed population = mother *and* fetus

**Notes:**
In the handbook, we set out to establish a taxonomy to categorize patient registries. Three categories that emerged were product, service and disease or condition.  A product registry includse all or a subset of patients that are exposed to a drug or device. These are some examples.

## Taxonomy of Registries: Service

-  Targeting a health care service, procedure or clinical encounter
- Procedure registries
- Primary coronary intervention
- Normal pressure hydrocephalus registry
- Society Thoracic Surgeons (STS) database
- Clinical service (and quality measurement) registries
- Hospitalization registries
- P4P

**Notes:**
A service registry focuses on a health care service, procedure or set of encounters.  A registry of patients undergoing coronary catheterization is an example.

## Taxonomy of Registries: Disease/Event

- Patients enrolled share a common disease or event experience, regardless of treatment or other exposures
- Acute diseases or events
- National Registry Myocardial Infarction (NRMI)
- Organ transplant registries
- Chronic diseases
- ESRD registry
- Heart failure registry
- Cancer registries (SEER)
- Rare diseases
- Lysosomal storage disorders
- Cystic fibrosis
- Hemophilia

**Notes:**
Disease or event registries are those in which patients enrolled share a common disease or an acute event within such a dsease, such as acute events like myocardial infraction, chronic diseases like heart failure or the rare, lifelong diseases like Pompe.

## Registry types and perspective

- Disease registry

- Procedure registry

- Device

- Drug registry

- Clinical event
- registry

- Pregnancy
- registry

**Notes:**
As we will hear with the Intermacs case study, choosing a type in registry design is really the perspective on how you can view the inforamtion.  A devide registry can only evaluate patients who receive the device, while a procedure registry might be able to look at more than one devices or alternative devices.  A disease registry may be able to look at pharmacueticals and devices as they are used in that disease, but not in other diseases, and so forth.

## Sections of Handbook

- Creating a Registry
- Operating Registries
- Evaluating Registries

**Notes:**
I’d like to spend a few minutes on the handbook as it lays out the components of a registry that will be useful in I nterpreting  the information my colleagues will be presenting.  The handbook is divided into 3 sections, which cover the creation of a registry, the operation, and the evaluation

## Sections of Handbook

- Creating a Registry
- Planning
- Design
- Data Elements
- Other Data Sources
- Ethical and Legal Issues
- Operating Registries
- Evaluating Registries

**Notes:**
The first section focuses on how to create a registry and includes planning, design, data elements, data sources and ethical and legal issues..  

## 
State the purpose
Identify stakeholders
Establish governance
Define scope of registry
Define target population
Assess feasibility
Secure funding

- State the purpose
- Identify stakeholders
- Establish governance
- Define scope of registry
- Define target population
- Assess feasibility
- Secure funding

- Planning

**Notes:**
Planning factors range from defining the purpose to target popoulations, scope, governance, feasibility and funding.  

There are several key steps in planning a patient registry, including articulating the purpose of the registry; determining whether the registry is an appropriate means for addressing the research question; identifying stakeholders; defining the scope and target population; assessing feasibility; and securing funding. 

## Design

- Design registry with respect to its major purposes
- Different levels of rigor are necessary for different types of registries
- The specific clinical questions of interest will guide definitions of study subjects, exposures, and outcome measures
- Choose a study design
- Select data sources, populations, comparison groups
- Determine whether sampling is needed
- Identify possible sources of bias (systematic error) and address them to the extent that is practical and achievable

**Notes:**
Design addresses a number of issues with respect to the major purposes, incluidng identifying upfront potential sources of bias.

A patient registry should be designed with respect to its major purpose, with the understanding that different levels of rigor may be required for registries that are intended primarily for descriptive purposes, in contrast to those designed to address focused analytical questions to support decisionmaking.  The key points to consider in designing a registry include choosing a study design; selecting data sources, populations, and comparison groups; determining whether sampling is needed and if so, how; identifying possible sources of bias (systematic error); and addressing them to the extent that is practical and achievable. 

## Data Elements

- Select based on importance and relationship to the primary outcome
- Consider data collection burden and incremental costs for collection
- Whenever possible, use established standards and common data definitions or validated instruments
- Weigh pros/cons of using patient identifiers
- Use pilot testing to assess feasibility and burden as well as reliability, validity, and potential for missing data

**Notes:**
Data element selection is key to the feasibility burden of a registry, its scientific value and the potential future ability to interface with other registries.  

The selection of data elements requires balancing such factors as their importance for the integrity of the registry and for the analysis of primary outcomes, their reliability, their contribution to the overall burden for respondents, and the incremental costs associated with their collection.  Selection begins with identifying relevant domains.  Specific data elements are then selected with consideration for established clinical data standards, common data definitions, and the use of patient identifiers.  It is important to determine which elements are absolutely necessary and which are desirable but not essential.  In choosing measurement scales for assessing patient-reported outcomes, it is preferable to use scales that have been appropriately validated.  Once data elements have been selected, a data map should be created and the data collection tools should be pilot tested.  Testing allows assessment of respondent burden, accuracy and completeness of questions, and potential areas for missing data.  Inter-rater agreement for data collection instruments can also be assessed, especially in registries that rely on chart abstraction.  

## Data Sources

- Registries can include data from many sources
- Primary data are collected for direct purposes of the registry
- Secondary data were originally collected for other purposes
- Medical records
- Institutional or organizational databases
- Administrative and claims data
- Death and birth records
- Census databases
- Existing registry databases
- When selecting data sources, consider cost, timeliness, structure, availability and quality

**Notes:**
Data sources include primary data—or what is collected specifically for the purposes of the registry and secondary data which were collected for other purposes.  

 A single registry may integrate data from various sources.  The form, structure, availability, and timeliness of required data are important considerations.  Data sources may be classified as primary or secondary.  Primary data are collected for direct purposes of the registry.  Secondary data are comprised of information that has been collected for purposes other than the registry, and they may not be uniformly structured or validated with the same rigor as primary data.  Sufficient identifiers are necessary to guarantee an accurate match between secondary sources and registry patients.  Furthermore, a solid understanding of the original purpose of the secondary data and how they were collected is advised, because how those data were collected and verified or validated will help shape their use in a registry.  Common secondary sources of data linked to registries include medical records systems, institutional or organizational databases, administrative health insurance claims data, death and birth records, census databases, and related existing registry databases

## Ethics, data ownership, and privacy

- Review ethical and data privacy requirements early in planning phase to ensure compliance
- Common Rule
- HIPAA
- Local requirements
- “The research purpose of a registry, the status of its developer, and the extent to which registry data are individually identifiable largely determine applicable regulatory requirements.”
- Also important:  registry transparency, oversight/governance and data ownership

**Notes:**
With privacy and protection of human subjects a primary concern in undertaking registries, a number of issues were explored in the handbook.  

Ethics, data ownership, and privacy.  Critical ethical and legal considerations should guide the development and use of patient registries.  The Common Rule is the uniform set of regulations on the ethical conduct of human subjects research from the Federal agencies that fund such research.  Institutions that conduct research agree to comply with the Common Rule for federally funded research and may opt to apply that rule to all human subjects activities conducted within their facilities or by their employees and agents, regardless of the source of funding.  The Health Insurance Portability and Accountability Act of 1996 (HIPAA) and its implementing regulations (collectively, the Privacy Rule) are the legal protections for the privacy and security of individually identifiable health information created and maintained by so-called “covered entities.” The research purpose of a registry, the status of its developer, and the extent to which registry data are individually identifiable largely determine applicable regulatory requirements.  Other important ethical and legal concerns include transparency of activities, oversight, and data ownership. 

## Slide 20

| 
Registry Developer or Purpose of Registry | Extent An Individual May Be Identified from Health Information |  |  | Waiver of Authorization, Documentation of Consent, or Consent Process |
| --- | --- | --- | --- | --- |
|  | 1.  De-identified health information** | 2.  Health information excludes direct identifiers | 3.  Health information includes direct identifiers |  |
| 1A.  Federal or State Public Health Agency: Registry for public health practice within agency’s legal authority not involving research | No requirements. | The Privacy Rule permits use or disclosure to a public health authority for public health activities.
The Common Rule is not applicable. | The Privacy Rule permits use or disclosure to a public health authority for public health activities.
The Common Rule is not applicable. | Waivers are not applicable. |
| 1B.  Federal or State Public Health Agency: Registry as agency research project  | No requirements, unless identifiable health information is disclosed to the registry developer to create the de-identified data, in which case the Privacy Rule requires a business associate agreement with the data source.  
If the Common Rule applies*, it permits an IRB grant of exemption from review, unless a re-identification code is used. | Privacy Rule permits use or disclosure of limited data set provided the data source and registry developer enter into a data use agreement.  If information containing direct identifiers is disclosed to the registry developer to create the limited data set, the Privacy Rule requires a business associate agreement with the data source.  If the Common Rule applies*, it permits an IRB grant of exemption, unless a re-identification code used. | Privacy Rule permits use or disclosure with a patient authorization or a waiver of authorization.
If the Common Rule applies*, IRB review and documented consent are required, unless an IRB grants a waiver of documentation or waiver for the consent process. | Privacy Board or IRB approval of a waiver of authorization depends on satisfaction of specific regulatory criteria. 
If the Common Rule applies*, IRB approval of a waiver of consent documentation or process depends on satisfaction of specific regulatory criteria. |
| 2. Registry produces evidence in support of labeling for a FDA regulated product. | No requirements. | Privacy Rule permits use or disclosure to a person responsible for an FDA-regulated product. | Privacy Rule permits use or disclosure to a person responsible for an FDA-regulated product 
FDA regulations, and Common Rule if applicable*, require IRB review, a documented consent process, and protection of confidentiality of research data. | Waivers are not applicable. |

**Notes:**

one of the key contributions of the document is to put a range of regulations--privacy and common rule--in perspective as they specifically relate to registries.  for example, in the chapter 6, the document provides a series of tables describing how HIPAA applies based on who the developer is (column one), how identifiable the data is (next three columns) and what the purpose of the registry is-(research or healthcare operations or both).  i used the diagram primarily to show the headings and then to provide one example.  i then also commented that the common rule applies to research performed by sites under the fwa (facility wide assurances) and those requirements are slightly different and need to be understood for those sites.  i made a comment that one of the interesting case examples in the handbook covers the NOPET (Nat’l Oncologic PET registry) registry by cms, where the sites are using the registry for healthcare operations (i.e. to get paid by cms) and the only group actually doing research is the radiology society running the registry--and that OHRP agreed with this and waived the need for the sites to go through IRB (only the registry organizer did as the central site).  Also important, the handbook in its entirety an this chapter specifically were reviewed and edited by both the Office of Civil Rights (which enforces hipaa) and Office of Human Research Protections (which interprets the Common Rule)

## Sections

- Creating a Registry
- Operating Registries
- Recruitment and management of providers and patients
- Data collection and quality assurance
- Adverse event collection and management
- Data analysis and interpretationinterpretation
- Evaluating Registries

**Notes:**
Next section – operating a registry provides insights into recruitment and management of providers and patients; collecting data and assuring its quality; adverse event detection and reporting and data analysis and interpretation.

## Patient and Provider Recruitment & Management

- Recruitment occurs at many levels
- Facilities (hospital, practice, pharmacy)
- Providers
- Patients
- Motivation for participation at each level differs
- Relevance, importance, scientific credibility, risks, burdens, incentives
- Goals for recruitment, retention and follow-up should be explicit and deviations continuously evaluated for risk of introducing bias

**Notes:**
Selecting and recruiting both providers and patients, recognizing and utilizing incentives and overcoming obstacles are important in the success for these programs. Understanding apriori goals for recruitment, retention and followup are critical for ultilmately evaluating the quality of the operations.   

Patient and provider recruitment and management.  Recruitment and retention of providers (as registry sites) and patients are essential to the success of a registry.  Recruitment typically occurs at several levels, including facilities (hospital, practice, pharmacy), providers, and patients.  The motivating factors for participation at each level and the factors necessary to achieve retention differ according to the registry.  

Factors that motivate participation include the perceived relevance, importance, or scientific credibility of the registry as well as the risks and burdens of participation and any incentives for participation.  Because provider and patient recruitment and retention can affect how well a registry accurately represents the target population, well-planned strategies for enrollment and retention are critical.  

Goals for recruitment, retention, and follow-up should be explicitly laid out in the registry planning phase, and deviations during the conduct of the registry should be continuously evaluated for their risk of introducing bias.  Include example.

## Data Collection & Quality Assurance

- Data collection
- Includes collecting, cleaning, storing, monitoring and reporting registry data
- Broad range of data collection procedures and systems available
- Critical factors in data quality
- Data element structure and definition, training of personnel, how data problems are handled
- Quality assurance
- Define requirements at registry creation
- Risk-based approach
- Most important or likely sources of error or potential lapses in procedures that may impact quality in the context of intended purpose

**Notes:**
The handbook covers data collectoin and quality assurance as well adverse event detection.  One of the key contributions I believe that the handbook has made is in advocating a risk based approach to quality assurance. In other words, given that every registry and every study for that matter, has a fixed limit of resources, the wisest allocation of those resources for quality assurance should be to the most important and likely sources of error that can impact quality in the context of the intended purpose.  Determining that upfront is both efficient and  provides a measurable bar for assessing quality at the end.  

Data collection and quality assurance.  The integrated system for collecting, cleaning, storing, monitoring, reviewing, and reporting on registry data determines the utility of those data for meeting the registry’s goals.  There are a broad range of data collection procedures and systems that are available.  Some are more suitable than others for particular purposes.  Critical factors in the ultimate quality of the data include how data elements are structured and defined, how personnel are trained, and how data problems are handled (e.g., missing, out of range or logically inconsistent values).  Registries may also be required to conform to guidelines or standards of specific end-users of the data (e.g., 21 CFR Part 11).  Quality assurance (QA) aims to affirm that the data were, in fact, collected in accordance with established procedures and that they meet the requisite standards of quality to accomplish the registry’s intended purposes and the intended use of the data.  Requirements for quality assurance should be defined during the registry’s inception and creation.  Because certain requirements may have significant cost implications, a risk-based approach to developing a QA plan is recommended and should be based on identifying the most important or likely sources of error or potential lapses in procedures that may impact the quality of the registry in the context of its intended purpose. 

Cost implication 

## Adverse Event Detection, Processing, & Reporting

- Important for any registry, especially with direct patient and/or physician contact
- Collection of spontaneously reported or solicited safety information
- Protocol-defined procedures strongly recommended
- Investigators and site staff appropriately trained
- Processing, coding and managing of AE data
- Reporting and regulatory requirements

**Notes:**
The role of registries in adverse event detection and reporting is  increasingly important both from a regluatory perspective but also as an approach to active surveillance and improved signal detection.  The recent call for proposals from the fda for a registry of registries for orthopedic devices is an example of this renewed interest in registries for signal detection. 

Adverse event detection, processing, and reporting.  Adverse events (AE) are defined by the U.S. Food and Drug Administration (FDA) as any untoward medical occurrence in a patient administered a pharmaceutical product, whether or not related or considered to have a causal relationship with the treatment, and they are categorized according to the seriousness and, for drugs, the expectedness of the event.  Although AE reporting for all marketed products is dependent on the principle of “becoming aware,” collection of adverse event (AE) data falls into two categories:  those events that are intentionally solicited (meaning data that are part of the uniform collection of information in the registry) and those that are unsolicited (meaning that the AE is volunteered or noted in an unsolicited manner).  Determining whether the registry should use a case report form to collect AEs should be based on the scientific importance of the information for evaluating the specified outcomes of interest. 

Regardless of whether or not AEs constitute outcomes for the registry, it is important to develop a plan for detection, processing, and reporting AEs for any registry that has direct patient contact.  If the registry receives sponsorship, in whole or in part, from a regulated industry (for drugs or devices), the sponsor has mandated reporting requirements and the process for detecting and reporting AEs should be established and investigators trained about how to identify AEs and to whom they should be reported.  Sponsors are encouraged to hold discussions with health authorities about the most appropriate process for serious AE reporting for registries designed specifically to meet requirements for surveillance of drug or device safety.

## Best Practices for Adverse Event Reporting to FDA in Registries of Post-Marketed Products

- Does the registry receive sponsorship or financial support from any regulated industry?

- Follow good public health practices for reporting new or serious AEs (recommended practice; not mandated)

- Notify company and/or FDA about new or serious AEs*

- Establish rules, roles, responsibilities for involved parties for oversight and reporting in conformance with registry design and applicable regulations.

- Does the registry have data collection with individual patient interaction ?

- Are SAEs recognized by a knowledgeable person in temporal  association with a drug* under study?

- Is there a reasonable possibility that the drug caused the SAE?

- Notify responsible entity (e.g., company) ASAP, ideally within 24 hours

- Company reports SAEs considered unexpected and possibly related for own drugs to FDA within 15 calendar days of original report; reports for device-related deaths, serious injuries, or malfunctions are due within 10-30 calendar days.

- Aggregate study findings of adverse events

- Report AEs in FDA periodic reports or PSUR  if applicable

- Company Contact

- FDA

**No**

**Yes**

**Yes**

**No**

**No**

**Yes**

**No**

**No**

- **For devices**, no attribution of expectedness is required; “device-relatedness” is based on whether the device caused or contributed to death or serious injury, or, in the case of malfunction, if the chance of death or serious injury is not remote if the malfunction were to recur

- Registry trains site(s) on identification and reporting of AEs including events of special interest and serious AEs (SAE).

**Yes**

**Yes**

**Best Practices for Adverse Event Reporting to FDA in Registries of Post-Marketed Products**
- Gliklich R, Dreyer N, eds.  Registries for Evaluating Patient Outcomes: A User’s Guide.  AHRQ Pub. 07-EHC001-01 2007.

**Notes:**
This is one useful figure in the handbook describing the adverse event reporting requirements by product type, level of interaction and sponsorship.  Tom Gross contributed to this diagram.

## Analysis and Interpretation

- Analysis
- Importance of a statistical analysis plan
- Analytic plans and statistical techniques for primary and secondary objectives
- Report on characteristics of the patient population, exposures of interest and endpoints
- Interpretation
- Who was studied?
- Is the actual population representative of the target population?
- How were the data collected, edited and verified?
- Completeness of data collection and data quality
- How were missing data handled and reported
- How were the analyses performed?

**Notes:**
Analysis and interpretation of registries is increasingly important as registries move from pure descriptive analyses to utlizing more sophisticated comparative effectiveness methodologies

Analysis and interpretation.  Analysis and interpretation of registry data begins with answering a series of core questions.  Who was studied?  How were the data collected, edited, and verified, and how were missing data handled?  How were the analyses performed?  Four populations are of interest in describing who was studied:  the target population, the accessible population, the intended population, and the population actually studied (the “actual population”).  The representativeness of the actual population to the target population is referred to as generalizability.  Analysis of registry outcomes first requires an analysis of the completeness of data collection and data quality.  Considerations include an evaluation of completeness for most if not all important covariates and an understanding of how missing data were handled and reported.  Analysis of a registry should provide information on the characteristics of the patient population, the exposures of interest, and the end points.  Descriptive registry studies focus on describing frequency and patterns of various elements in a patient population, whereas analytical studies concentrate on associations between patients or treatment characteristics and health outcomes of interest.  A statistical analysis plan (SAP) describes the analytical plans and statistical techniques that will be used to evaluate the primary and secondary objectives specified in the study plan.  Interpretation of registry data should be provided so that the conclusions can be understood in the appropriate context and so that any lessons from the registry may be applied to the target population and used to improve patient care and outcomes. 

The analysis and interpretation chapter discusses how to conduct analyses of registry data and how to interpret the results of those analyses.  This chapter really highlights why the design issues discussed in chapter 3 are so important.  In particular, it emphasizes the importance of having a statistical analysis plan in pl ace before the registry begins.  Since most of our registries have as external reporting and publications as a goal, this chapter can help us to understand what goes into that reporting and what we need to do to make sure that the analysis phase of a registry goes smoothly. 

## Case Examples

- 	23 case examples are included to illustrate practical challenges and how  they were addressed.  E.g.,
**NRMI** – “*Creating a Registry to Fulfill Multiple Purposes and Using a Publications Committee to Review Data Requests”*
**GWTG-Stroke** – “*Using Performance Measures to Develop a Data Set*”
**BPH** – “*Developing and Validating a Patient-Administered Questionnaire*”
**GWTG** – “*Using Recognition Programs to Recruit Sites”*
**OPTIMIZE-HF** – “*Using Registry Tools to Recruit Sites”*
**ESCF** – “*Using Registry Data to Evaluate Outcomes by Practice”*

**Notes:**
The handbook provides 23 practical examples drawn from well known registries such as the national registry of myocardial infarction and get with the guidelines, and highlights issues ranging from incentives for participation to systems to publications policies

## How do you know what makes a registry “good enough”?

**Notes:**
Once the registry is done, how can decision makers know that its good enough?

## Evaluating Registries

- Although all registries can provide useful information, there are levels of rigor that enhance validity and make the information from some registries more useful for guiding decisions
- We adapted quality evaluations used for trials, that address the confidence that the design, conduct and analysis of the registry protect against systematic errors and errors in inference

**Notes:**
Evaluating registries.  Although most registries can provide useful information, there are levels of rigor that enhance validity and make the information from some registries more useful for guiding decisions than others.  The term “quality” can be applied to registries to describe the confidence that the design, conduct, and analysis of the registry can be shown to protect against bias and errors in inference, that is, erroneous conclusions drawn from a study.  Although there are limitations in any assessment of quality, this handbook uses a quality component analysis to evaluate high-level factors that may affect results, and differentiates between research quality (which pertains to the scientific process) and evidence quality (which pertains to the data/findings emanating from the research process).  Quality components are classified as either “necessary aspects,” which can be viewed as a basic checklist that should be considered for all patient registries, or as “enhancements” that improve the information value.  The results of such an evaluation should be considered in the context of the disease area(s), the type of registry, and the purpose of the registry, and should also take into account feasibility and affordability 

## Evaluating Registries

- Quality component analysis
- Research quality (scientific process)
- Planning; design; data elements & data sources; ethics, privacy and governance
- Evidence quality (data/findings)
- Patients; data elements & data sources; QA; analysis; reporting
- Components classified as
- Basic Practice (draft names)
- Future Directions/Potential Enhancement (draft names)

**Notes:**
Evaluating registries.  Although most registries can provide useful information, there are levels of rigor that enhance validity and make the information from some registries more useful for guiding decisions than others.  The term “quality” can be applied to registries to describe the confidence that the design, conduct, and analysis of the registry can be shown to protect against bias and errors in inference, that is, erroneous conclusions drawn from a study.  Although there are limitations in any assessment of quality, this handbook uses a quality component analysis to evaluate high-level factors that may affect results, and differentiates between research quality (which pertains to the scientific process) and evidence quality (which pertains to the data/findings emanating from the research process).  Quality components are classified as either “necessary aspects,” which can be viewed as a basic checklist that should be considered for all patient registries, or as “enhancements” that improve the information value.  The results of such an evaluation should be considered in the context of the disease area(s), the type of registry, and the purpose of the registry, and should also take into account feasibility and affordability 

## Research Quality: Basic 

- PLANNING: A written study plan documents: goals, design, study population, recruitment, data collection, human subject protection, data element, sources, & review/QA.  Feasibility is considered at the outset.
- Plans address how data will be evaluated, incl. what comparative information, if any, will be used to support study hypotheses or objectives.
- DESIGN: The size required to detect an effect, should it exist, or achieve a desired level of precision is acknowledged, whether or not met.
- Follow-up time needed to detect events of interest is acknowledged, whether or not feasible to achieve.  To the extent feasible, follow-up time is adequate to address the main objective.
- DATA ELEMENTS: Outcomes are clinically meaningful and relevant, i.e., useful to the medical community for decision-making.
- ETHICS, PRIVACY, GOVERNANCE: Registry has received review by required oversight committees
- COMMUNICATION PLAN for results is addressed.

## Research Quality:  Potential Enhancements

- PLANNING: Formal study protocol with review from key stakeholders prior to finalization.  Pilot studies are useful when studying hard to reach populations or when sensitive data are sought.
- DESIGN: Use of concurrent comparators in situations where treatments are evolving rapidly.
- Formal statistical calculations to support sample size, whether or not that size is achievable within practical constraints.
- DATA ELEMENTS & SOURCES: Multiple methods of data collection may be required for some purposes.
- Use validated scales and tests when such tools exist for purpose needed.
- Adapt levels of QA based on observed performance.
- Coding consistent with nationally approved coding systems; standardized data dictionaries, validated assessment tools....
- Publication policies are specified in advance of collecting data.

## Evidence Quality: Basic

- PATIENTS: Participants are similar to the target population; attention is paid to minimize selection bias to the extent feasible.
- For safety studies, registry personnel are trained to ask about AEs in a consistent, clear & specific manner, and know how to report.
- DATA ELEMENTS & SOURCES: Data are reasonably complete.
- QA: Reasonable efforts have been expended to assure that appropriate patients have been systematically enrolled and followed in as unbiased a manner as possible; reasonable efforts have been devoted to minimize losses to follow-up.  Data are checked using range and consistency checks.
- ANALYSIS: Accepted analytic techniques are used; they may be augmented by new or novel approaches.
- REPORTING: Results are reported for all main objectives; follow-up time is described so readers can assess its impact on conclusions drawn; report clearly states any conclusions drawn and implications of results, as appropriate.

## Evidence Quality: Potential Enhancements 

- PATIENTS: External validity is described; for comparative effectiveness and safety, contemporaneous comparison data are collected.
- DATA: Results that can be confirmed by an unbiased observer enhance accuracy and reliability (e.g., death, test results, scores from validated measurements for PRO or clinical rating scales).
- QA:  Potential sources of errors relating to accuracy and falsification are rigorously evaluated and quantified (e.g., through database and site reviews).  For studies of safety, effectiveness and comparative effectiveness, a sample of data are compared with patient records.
- ANALYSIS: Loss to follow-up is characterized at all stages of study conduct. For safety studies, risks and/or benefits of “exposure” are evaluated quantitatively, *beyond statistical significance*.  Sensitivity analyses are used to examine the effect of varying the inclusion/exclusion criteria & other assumptions.
- REPORTING: Inferences about causality are based on a variety of factors, including the strength of the association, biases, temporal relation, etc.

## Some Unanswered Questions

- When should a registry end?
- What are the privacy implications of linking multiple data sources into registries?
- How can different registries or other data sources working together, interfacing
- How should a registry be designed specifically for adverse event collection?

**Notes:**
The User’s Guide was a first step in an area of vital importance with little standardization. I am pleased to report that there is now new funding from AHRQ to turn our attention to some new questions and challenges as we are now starting on a new edition of the handbook.

## Thank You

- Contact Information:
- Richard Gliklich MD
- President
- Outcome
- 201 Broadway
- Cambridge, MA 02139
- _[richg@outcome.com](mailto:richg@outcome.com)_
- 617-621-1600 x530
- www.outcome.com