Guidance for Industry and FDA Staff 

  
Best Practices for Conducting 
and Reporting 
Pharmacoepidemiologic Safety 
Studies Using Electronic 
Healthcare Data 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
U.S. Department of Health and Human Services 

Food and Drug Administration 

Center for Drug Evaluation and Research (CDER) 

Center for Biologics Evaluation and Research (CBER) 

 
May 2013 

Drug Safety 

 
   
Guidance for Industry and FDA Staff 

  
Best Practices for Conducting 
and Reporting 
Pharmacoepidemiologic Safety 
Studies Using Electronic 
Healthcare Data 
 
 
Additional copies are available from: 

Office of Communications 
 
Division of  Drug Information, WO51, Room  2201 
 
Center for Drug Evaluation  and Research  

 Food and Drug  Administration 

10903 New Hampshire Ave.  

Silver Spring, MD 20993-0002 
  
Phone: 301-796-3400;  Fax: 301-847-8714 
 
druginfo@fda.hhs.gov 

http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/default.htm  

and/or 

 Office of Communication, Outreach, and 
  
Development (OCOD) 
  
Center for Biologics Evaluation and Research 

 Food and Drug  Administration 

1401 Rockville Pike, Rockville, MD  20852-1448  

 (Tel) 800-835-4709  or  301-827-1800  

Email: ocod@fda.hhs.gov 

http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guidances/default.htm  

 
 
 
 
U.S. Department of Health and Human Services 

Food and Drug Administration 

Center for Drug Evaluation and Research (CDER) 

Center for Biologics Evaluation and Research (CBER) 

 
May 2013 

Drug Safety
 
  Contains Nonbinding Recommendations 
TABLE OF CONTENTS 

 
 
 
I.   INTRODUCTION............................................................................................................. 1 
 
II.   BACKGROUND ............................................................................................................... 3 
 
A.   Electronic Healthcare Data in Pharmacoepidemiologic Safety Studies....................................4
  
B.   Prior Guidelines and Guidance Documents ................................................................................5
  
III.   BEST PRACTICES  GENERAL CONSIDERATIONS........................................... 6 
 
A.   Title and Detailed Study Summary..............................................................................................6
  
B.   Background ....................................................................................................................................7
  
C.   Study Approach Considerations...................................................................................................7
  
D.   Study Team Expertise and Credentials .......................................................................................8 
 
E.   Interpretation of Findings.............................................................................................................8
  
IV.   BEST PRACTICES  DATA SOURCES..................................................................... 9 
 
A.   Appropriateness of Data Source(s) in Addressing Safety Questions of Interest......................9
  
B.   Enrollment and Comprehensive Capture of Care....................................................................10
  
C.   Country of Origin and Health System:  Relevance to the United States ................................11 
 
D.   Selection of Study Population.....................................................................................................11
  
E.   Quality Assurance (QA) and Quality Control (QC).................................................................12
  
F.   Study Time Frame and Lag Time Issues...................................................................................13
  
V.   BEST PRACTICES  STUDY DESIGN .................................................................... 13 
 
A.   Study Design Considerations ......................................................................................................13
  
1.  Choice of Study Design..................................................................................................................13
  
2.  Examples of Study Designs (Not All-Inclusive) .............................................................................13 
 
3.  Comparator Selection ....................................................................................................................13
  
4.  Study Time Frame..........................................................................................................................14 
 
5.  Identification and Handling of Confounders and Effect Modifiers ............................................... 14 
 
6.  Sample Size and Statistical Power.................................................................................................16
  
B.   Study Design: Exposure Definition and Ascertainment..........................................................16 
 
1.  Exposure Definition .......................................................................................................................16 
 
2.  Exposure Ascertainment  Study Design......................................................................................17
  
3.  Exposure Ascertainment  Data Source ......................................................................................17
  
4.  Exposure Ascertainment  Gaps in Therapy and Censoring.......................................................18 
 
5.  Exposure Ascertainment  Dose ..................................................................................................19 
 
6.  Exposure  Other Factors............................................................................................................19 
 
C.   Study Design: Outcome Definition and Ascertainment...........................................................19 
 
1.  Medically and Scientifically Relevant Case Definition of Safety Outcomes of Interest................. 20 
 
2.  Validation of Outcomes..................................................................................................................20
  
3.  Outcome Definition  Procedures or Diagnoses.........................................................................21
  
i 
  Contains Nonbinding Recommendations 
VI.   BEST PRACTICES  ANALYSES............................................................................. 22
  
A.   Prespecified Analysis Plan ..........................................................................................................22
  
B.   Study Analyses .............................................................................................................................22
  
C.   Use of Specific Statistical Techniques (e.g., To Minimize  Confounding)................................23
  
D.   Sensitivity Analyses......................................................................................................................23
  
E.   Linking or Pooling Data  From Different Sources.....................................................................23 
 
F.   Assessment and Handling of Missing and Uninterpretable Data............................................24 
 
G.   Quality Assurance (QA) and Quality Control (QC).................................................................24
  
H.   Procedures To Ensure Accuracy of Data Management and Analysis Process.......................25
  
GLOSSARY................................................................................................................................. 26 
 
BIBLIOGRAPHY....................................................................................................................... 29 
 
ii 
  Contains Nonbinding Recommendations 
1 
Guidance for Industry and FDA Staff   
 
Best Practices for Conducting and Reporting 

Pharmacoepidemiologic Safety Studies Using 

Electronic Healthcare Data
   
 
 
This guidance represents the Food and Drug Administrations (FDAs) current thinking on this topic.  It 
does not create or confer any rights for or on any  person and does not operate to bind FDA or the public.  
You can use an alternative approach if the approach satisfies the requirements of the applicable statutes 
and regulations. If you want to discuss an alternative approach, contact the FDA staff responsible for 
implementing this guidance.  If you cannot identify the appropriate FDA staff, call the appropriate 
number listed on the title page of this guidance.  
 
 
I.  INTRODUCTION  
 
This guidance describes best practices pertaining to conducting and reporting on 
pharmacoepidemiologic safety studies2 that use electronic healthcare data, which include 
administrative claims data and electronic medical record (EMR) data.3  The guidance includes 
recommendations for documenting the design, analysis, and results of pharmacoepidemiologic 
safety studies to optimize FDAs review of protocols and final reports that are submitted to the 
Agency. For purposes of this guidance, the term  pharmacoepidemiologic safety study refers to 
an observational study  designed to assess the risk associated with a drug exposure and to test 
prespecified hypotheses. For ease of reference, this guidance uses the term drug to refer to drug 
and biological products regulated by CDER or CBER.  Medical devices are not within the scope 
of this guidance. 
 
This guidance is intended to provide the following: 
 
	   Consistent guidance for industry and FDA to use when designing, conducting, and 
analyzing pharmacoepidemiologic safety studies;  
                                                 
1 This guidance has been  prepared by the Office of Surveillance and Epidemiology in the Center for Drug  
Evaluation and Research (CDER) in cooperation  with the Center for Biologics Evaluation and Research (CBER) at 
the Food and Drug Administration.  
 
2 All terms presented in  bold italics at first use in this guidance are defined in the Glossary. 
 
3 This guidance uses the term EMR instead of electronic health record (EHR) because at this time, currently 
available electronic data sources used for pharmacoepidemiologic safety studies contain mainly EMR versus EHR  
data.  Thus, although the adoption of EHRs  is increasing, it is not yet possible to recommend best practices on their 
use in  pharmacoepidemiologic safety studies.   
 
  1
  Contains Nonbinding Recommendations 
	   A framework for industry to use when submitting pharmacoepidemiologic safety study 
protocols and final reports to FDA; and 
	   A framework for FDA reviewers to use when reviewing and interpreting 

pharmacoepidemiologic safety study protocols and final reports.  

 
The focus of this guidance is on best practices that specifically apply to pharmacoepidemiologic 
safety studies using electronic healthcare data. Although the guidance is not intended to address 
basic epidemiologic principles, many of the concepts discussed in the guidance may apply more 
broadly to pharmacoepidemiologic safety studies using other types of data, as well as descriptive 
studies.  FDA encourages industry to inform FDA of all pharmacoepidemiologic safety studies 
that it conducts; to submit plans and protocols for such studies before study initiation; and to 
submit comprehensive final reports with detailed methods and results to FDA in a timely 
manner.4 
 
The Prescription Drug User Fee Amendments of 2007 (PDUFA IV) authorized a significant 
expansion of the postmarket focus under the PDUFA program.5  Under PDUFA IV, FDA agreed 
to specific commitments to enhance and modernize the drug safety system.6  One FDA 
commitment was to identify pharmacoepidemiologic safety study best practices and to develop a 
guidance describing these practices; the current guidance is intended to fulfill this commitment.  
The guidance focuses on the conduct and reporting of pharmacoepidemiologic safety studies that  
use electronic healthcare data because: (1) existing guidelines and checklists are not specifically 
                                                 
4 The Food and  Drug Administration Amendments Act of  2007  (FDAAA) (Public Law  110-85) amended the 
Federal Food, Drug, and Cosmetic Act (FD&C Act) to  require submission  of information pertaining to  certain  
postapproval (hereinafter referred to as postmarketing) studies, including  the status  of any required  
pharmacoepidemiologic studies undertaken to investigate a safety issue.  See section 505(o)(3)(A), 505(o)(3)(B),  
and 505(o)(3)(E)(ii)  of the FD&C Act (21  U.S.C.  355(o)(3)(A),  (B), and (E)(ii)).   In  April 2011, FDA published a 
guidance for industry,  Postmarketing Studies and Clinical Trials  Implementation of  Section 505(o)(3) of the 
Federal Food, Drug, and Cosmetic Act, that addresses submission requirements, including  timetables and 
schedules, for  postmarketing requirements (PMRs) and postmarketing commitments (PMCs).  We update guidances  
periodically.  To make sure you have the most recent version of a guidance, check the CDER guidance page at 
http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/ Guidances/default.htm or the CBER 
guidance page  at  
http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guidances/default.htm.  
 
Section 506B  of the FD&C  Act (21 U.S.C. 356b) requires sponsors to report  on postmarketing studies  that the 
sponsors have  agreed to conduct.  These requirements are included in the current FDA regulations for annual 
reports:  21 CFR 314.81(b)(2)(vii)  for new drug applications, 21 CFR 314.81(b)(2)(vii) and 21 CFR  314.98(c) for  
abbreviated new drug applications.  Section 601.70(b) of  FDA regulations (21 CFR  601.70(b))   requires annual  
progress reports for required or agreed-upon postmarketing studies for biologics license applications, but it should  
be  noted that  601.70(a) limits the definition  of postmarketing studies to those concerning clinical safety, clinical 
efficacy, clinical pharmacology, and nonclinical toxicology. 
 
5 See FDAAA, Title I  Prescription  Drug User Fee Amendments of 2007. 
 
6
 See the letter from the Secretary of  Health and Human Services to the Chairman of the Committee on Health, 
Education, Labor, and Pensions of the Senate and the  Chairman of the Committee on Energy and Commerce of the  
House of Representatives, as set forth in the Congressional Record, at 
http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm119243.htm.  
 
  2
  Contains Nonbinding Recommendations 
directed to the use of electronic healthcare data in pharmacoepidemiologic safety studies; (2) an 
increasing number of these studies using electronic healthcare data have been conducted and 
submitted to FDA in recent years; and (3) studies  that involve the use of electronic healthcare 
data have complex and unique characteristics that warrant more specific guidance.  
 
FDA held a public workshop in May 2008 to obtain input from experts in the field and the public 
regarding the use of electronic healthcare data in pharmacoepidemiologic safety studies of drug 
safety issues and the draft guidance was published in February 2011.  FDA carefully reviewed 
and considered all oral and written comments from the workshop and the draft guidance docket 
to create the current guidance.   
 
This guidance does not address real-time active safety surveillance  assessments  (e.g., Mini-
Sentinel and Sentinel (Robb, et al. 2012)), as this field continues to evolve and it is not yet 
possible to recommend sound best practices.  This guidance is not intended to be prescriptive 
with regard to choice of study design or type of analysis and does not endorse any particular type 
of data resource or methodology.  It does not provide a framework for determining the 
appropriate weight of evidence to be given to studies from this data stream in the overall 
assessment of drug safety, as this appraisal represents a separate aspect of the regulatory 
decision-making process and is currently best accomplished in the context of the specific safety 
issue under investigation. 
 
This guidance also does not address the decision to pursue a pharmacoepidemiologic safety 
study using electronic healthcare data in any particular case over any other type of study, as this 
decision is unique to each specific safety issue of interest.7  Generally, however, these studies 
may be particularly useful when clinical trials would be infeasible (e.g., when the safety outcome  
is very rare) or when the study of outcomes or exposures in an interventional or prospective 
study would be unethical. There are also circumstances when a pharmacoepidemiologic safety 
study may not be appropriate or adequate to answer the safety question of interest; for example, 
if the safety outcome of interest is a subjective patient-reported outcome  that is not typically 
available in electronic healthcare data.   
 
FDAs guidance documents, including this one, do not establish legally enforceable 
responsibilities. Instead, guidance documents describe the Agency's current thinking on a topic 
and should be viewed only as recommendations, unless specific regulatory or statutory 
requirements are cited.  The use of the word should in Agency guidances means that something 
is suggested or recommended, but not required.  
 
II.  BACKGROUND 
 
                                                 
7 In some instances, when  FDA is concerned about a serious risk, applicants are required  to complete postmarketing 
studies, which are one of an array of potential PMRs.  For a full discussion  of PMRs, refer to the guidance for 
industry, Postmarketing Studies and Clinical Trials  Implementation of  Section 505(o)(3) of the Federal Food, 
Drug, and Cosmetic Act. FDAAA  provides the specific circumstances when FDA can  require the conduct of  
postmarketing studies (see section 505(o)(3)(D)(i)  of  the FD&C Act). 
 
 
  3
  Contains Nonbinding Recommendations 
FDA postmarket regulatory decision-making on a drug safety issue generally involves several 
interrelated steps. 
  
 	  FDA receives evidence of an association between a particular drug and an adverse event.  
This evidence generally comes from one or more of the following data streams:  randomized 
controlled trials (RCTs), spontaneous adverse event case reports, pharmacoepidemiologic 
safety studies, and meta-analyses. 
  
 	  FDA assesses the quality of both the design and conduct of the studies and reviews the final 
reports pertaining to the purported association of drug and outcome; additional studies might 
be initiated to further examine the association. 
   
 	  The accumulated evidence is then integrated and weighed by a multidisciplinary team to 
reach an overall conclusion, to the extent possible based on the available data, regarding the 
relationship between the drug and the outcome. This is followed by an overall examination 
and a reassessment of the drugs benefits and risks. 
   
 	  FDA then determines whether any regulatory action is required (e.g., labeling changes, 
requiring or modifying a Risk Evaluation and Mitigation Strategy, generation of additional 
data from one or more data streams, such as a postmarketing pharmacoepidemiologic safety 
study or trial, etc.) 
   
FDA regulatory decision-making on drug safety issues is an iterative process because regulatory 
decisions are informed by emerging evidence, including any additional studies that are initiated 
as mentioned above. 
 
For many potential associations between a drug and an adverse event, findings across studies can 
be inconsistent for a variety of reasons.  However, because drug-related adverse events have the 
potential to broadly affect the public health, there is often an urgency to take regulatory action to 
address drug safety issues based on the available evidence, even if the data are less than optimal.   
 
One aspect of regulatory decision-making is evaluating the evidence from  
pharmacoepidemiologic safety studies.  Electronic healthcare data are often useful in conducting 
these studies that are designed to formally test drug safety hypotheses.  As described in this 
guidance, the best practices for the conduct and reporting of pharmacoepidemiologic safety 
studies using electronic healthcare data are intended to facilitate a more consistent and objective 
interpretation of findings from these studies.   
 
A.  Electronic Healthcare Data in Pharmacoepidemiologic Safety Studies  
 
Recent technological advances now allow investigators to efficiently assemble electronic 
healthcare data and, relative to alternative approaches, rapidly conduct pharmacoepidemiologic 
studies of drug safety issues in real-world healthcare settings and with large numbers of patients.  
In addition, innovative statistical methods allow investigators to study complex drug safety 
questions previously considered too difficult to examine outside of a clinical trial setting.  
  4
  Contains Nonbinding Recommendations 
Indeed, these advances have opened up many new opportunities for conducting 
pharmacoepidemiologic studies.   
 
B.  Prior Guidelines and Guidance Documents 
 
Guidance documents previously published by the FDA, in addition to guidelines published by 
several other organizations, have informed the development of best practices for conducting 
pharmacoepidemiologic safety studies using electronic healthcare data.   FDAs 2005 guidance 
Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment (FDA 2005 
guidance), which is much broader in scope than the current guidance, largely focuses on the 
practice of pharmacovigilance and the various types of pharmacoepidemiologic studies and  
includes an abbreviated section on observational studies..   
 
The International Society for Pharmacoepidemiologys (ISPEs) Guidelines for Good  
Pharmacoepidemiology Practices (GPP) (ISPE guidelines 2008) recommends that all 
pharmacoepidemiologic studies address the following: 
 
 	  Providing a written protocol, with dated amendments and justifications; 
 
 	  Performing a critical review of the literature to facilitate the identification of knowledge 
gaps in the current evidence base for safety issue(s) of interest and how the current or 
proposed study contributes to this evidence base; 
 
 	  Ensuring human subject protection; FDA considers the investigator to be responsible for 
ensuring that the data are Health Insurance Portability and Accountability Act (HIPAA)  
compliant and that all research performed complies with standards of privacy of 
individually identifiable health information and protects the rights of human subjects in 
research; 
 
	   Providing confidence intervals about estimates of risk in addition to p-values; although p-
values address the issue of statistical significance, confidence intervals quantify the 
precision of the risk estimates;  
 
	   Including both absolute and relative risk estimates to assist in the interpretation of the 
public health impact of the findings; and 
 
	   Archiving of relevant study documents and data. 
 
The Consolidated Standards of Reporting Trials (CONSORT) statement (Moher, et al. 2001), 
exemplifies how basic reporting standards can improve the quality of reports on clinical  trials. 
The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) 
statement (von Elm, et al. 2008) provides guidelines for reporting observational studies.8  The 
                                                 
8 The term  reporting in these documents means the transparent disclosure of information  to the public describing  
critical methodological and scientific aspects of the study to enable the public to assess the strengths and  
weaknesses of the study  design, conduct, and analysis.  The term does not refer to regulatory reporting 
requirements (von Elm, et al. 344). 
  5
  Contains Nonbinding Recommendations 
STROBE recommendations were developed in response to the pervasiveness of missing 
information in published observational epidemiologic studies (von Elm, et al. 2008). The 
European Network of Centers for Pharmacoepidemiology and Pharmacovigilance (ENCePP) has 
developed a Checklist of Methodological Standards for ENCePP Study Protocols and a Guide on 
Methodological Standards in Pharmacoepidemiology to provide standards for 
pharmacoepidemiologic studies and to enhance awareness about scientific and methodological 
developments in pharmacoepidemiology.9  
 
The FDA 2005 guidance, the ISPE guidelines, the STROBE reporting framework, and the 
ENCePP methods checklist and guide provide general guidance applicable to all 
pharmacoepidemiologic safety studies.  Studies that involve the use of electronic healthcare data 
have complex and unique characteristics that warrant more specific guidance.  This best practices 
guidance specifically addresses the design, analysis, conduct, and documentation of 
pharmacoepidemiologic safety studies using electronic healthcare data  with protocols and 
reported results submitted to FDA.   
   
 
III.  BEST PRACTICES  GENERAL CONSIDERATIONS 
 
For all pharmacoepidemiologic safety studies  using electronic healthcare data submitted to the 
Agency,10 investigators should submit protocols to FDA before study initiation and final reports 
upon completion.  
 
 The scientifically valid protocol should include the names and roles of all individuals involved 
in its development.  Investigators should predefine aspects of the study related to the design, 
analysis, conduct, and reporting of the study and articulate them in the study protocol and final 
reports, along with a science-based rationale for the scientific study choices pertaining to these 
aspects of the study; more information regarding these aspects is provided in sections IV, V, and 
VI of this guidance. Any changes to the initial protocol after data collection has commenced 
should be justified and documented in the final protocol and study report, along with a discussion 
of the impact of those changes on the final interpretation of study results.  Published studies 
submitted to FDA for safety-related regulatory decision-making should be accompanied by 
supplemental documentation that addresses the aspects of study design, analysis, conduct, and 
reporting discussed in this guidance.  All of the issues described within this guidance should 
be addressed in the protocol, in the study report, or in accompanying documentation.  At 
this time, FDA is not proposing a specific format for any of these documents when 
reporting on pharmacoepidemiologic safety studies that use electronic healthcare data.  
   
A.  Title and Detailed Study Summary 
 
                                                                                                                                                             
 
9 http://www.encepp.eu/standards_and_guidances/index.html. Last updated February 2,  2012. 
 
10 21 CFR 314.81(b)(2)(viii). 
 
  6  Contains Nonbinding Recommendations 
Each protocol and final report should include a study title that indicates the type of 
pharmacoepidemiologic safety study design (e.g., cohort, case-control) employed in the study.  
The report should contain a detailed study summary that concisely describes the critical elements 
listed below. 
  
   Scientific goals, study objectives, and prespecified hypotheses; 
   Study design, including comparator groups; 
   Study population and time period of study, including: (1) study time frame  and 
(2) scheduled milestones (final protocol submission, study completion, and the 
final report); 
   Data sources used; 
   Drug exposures of interest; 
   Drug safety outcomes of interest; 
   Methods to control for sources of bias and confounding; 
   Statistical analysis plan; 
   Brief, balanced description of the results, interpretation of study findings, and key 
  study limitations; and 

   Public health impact. 

 
B.  Background  
 
A background section should be provided that includes a brief description of prior evidence or  
suspicions prompting the study initiation, a brief  discussion of the strengths and weaknesses of 
previous studies on the safety issue, and some  general information about the therapeutic class 
and use of the study drug(s). Investigators should then describe the concise study objectives and 
feasible hypotheses upon which the study design has been based.   
 
C.  Study Approach Considerations 
 
Once the prespecified hypotheses are identified, the study approach, including the selection of 
data sources, study design, and analysis plan, can be developed.  The investigators should 
describe the reasons for their choices of study design, selection of databases, and analysis plan as 
they pertain to the hypotheses. The investigators should also briefly describe any alternative 
study approaches and databases they considered before deciding upon the proposed approach, 
and explain why the alternatives were neither feasible nor optimal in answering the specific 
study questions. The discussion should reflect an in-depth understanding of the use of the 
drug(s) of interest, the safety outcome(s) of interest, the usual treatment of the safety outcome(s) 
of interest, and the capture of both the exposure and safety outcome in relevant patient 
populations using electronic healthcare data  sources.  Results of any preliminary or feasibility 
studies should be included. 
 
Use of multiple study designs and data sources can assist in addressing the hypotheses by 
increasing the generalizability and robustness of findings and allowing for the study of different 
sub-populations of interest (Vandenbrouke 2008).  The protocol should address whether the use 
of more than one data source is appropriate.  The study sample size can be increased by 
designing a study that uses the same design and analysis plan across multiple electronic 
  7
  Contains Nonbinding Recommendations 
healthcare data sources. Such protocols should address the critical factors related to assembly of 
analytic data and data pooling in the development of the analysis plan.  Multiple data sources can 
also be used separately, using the same and/or different design and analysis plans, to verify and 
replicate study findings. 
 
Specific aspects of best practices for the selection of the data sources, study design, and analysis 
plan to be described and included in any regulatory submission are discussed in detail in sections 
IV, V, and VI of this guidance. 
 
D.  Study Team Expertise and Credentials 
 
The protocol should include a description of the expertise and credentials of the study team, 
including their experience in using the specific data sources to be employed in the study.  
Because all existing electronic healthcare data sources used for pharmacoepidemiologic safety 
studies have unique features based on their original purpose and methods for collecting data and 
including patients, the inclusion of personnel on the study team with hands-on experience and 
knowledge of the data source will increase the probability of appropriate use of the data.  An 
experienced, balanced study team with the appropriate expertise is crucial to the successful 
execution of a safety study. 
 
E.  Interpretation of Findings 
 
When interpreting study findings, investigators should summarize the key results from (1) 
unadjusted analyses, if conducted, and (2) the prespecified primary and secondary  analyses. The 
summary should address the objectives and hypotheses of the study as specified in the protocol.  
Providing both absolute and relative effect measures whenever possible will provide the most 
information on the public health impact of the study results. 
 
In interpreting the findings, the precision of the estimated effect measure from the study should 
be discussed. Confidence intervals are crucial in interpreting the precision of the risk estimates 
from the study.  Confidence intervals can be used to provide a range of the effect measures 
consistent with the study findings.  Findings of no association between the drug and safety 
outcome of interest should be presented in the context of the initial statistical power calculations; 
investigators should attempt to determine the level of risk that can be ruled out, given the study 
findings. 
 
Conversely, statistical significance alone does not exclusively determine the clinical importance 
of the findings, because when dealing with large amounts of electronic healthcare data, statistical 
significance of very small effect measures can be common.  Consideration should be given to 
both the clinical significance and potential biases.  
 
Investigators should discuss potential biases and confounding, the suspected magnitude and 
direction of these effects, and their potential impact on the interpretation of the study findings. 
Results from sensitivity analyses might prove helpful in discussing these issues (refer to section 
VI.D). Investigators should also discuss the limitations of the database and the study design, and 
the impact of those limitations on generalizability.  Finally, investigators should place the study 
  8
  Contains Nonbinding Recommendations 
findings in the context of other evidence, including studies previously conducted on the same  
issue using other databases, populations, and study designs. 
 
IV.  BEST PRACTICES  DATA SOURCES 
 
FDA does not endorse one type of electronic healthcare data source over another.  Investigators 
should select data sources that are most appropriate to address the specific study hypotheses.  
When submitting protocols or final reports to FDA, investigators should include in an appendix 
the names of all data sources used for the study and other relevant descriptive information 
discussed in more detail below. 
 
A.   Appropriateness of Data Source(s) in Addressing Safety Questions of Interest  
 
Investigators should demonstrate a complete understanding of the electronic healthcare data 
source and its appropriateness to address specific hypotheses.  Because existing electronic 
healthcare data systems were generated for purposes other than drug safety investigations, it is 
important that investigators understand their potential limitations and make provisions to use the 
data systems appropriately.  For example: 
  
	   Administrative claims data are generated to support payment for care; the payors 
policies governing the approval and denial of such payments should be considered before 
using these data for study investigations. 
   
	   EMR data are generated in the course of routine clinical care provision; therefore, it is 
important for investigators to consider the particular healthcare systems characteristics 
and guidelines for patient care and their common clinical practices, as these factors will 
have an impact on the data collected. 
   
Investigators should also describe the historical accessibility of the data source(s) proposed to be 
used in the study. This description should include:  
 
   How long the data source has been available to the research community;  
   How often this data source has been used for pharmacoepidemiologic safety studies;  
   The capability of the selected data source to validate the outcome and other study 
elements (e.g., exposures, key covariates, inclusion/exclusion criteria) based on the safety 
question; and 
	   References for any relevant publications, including validation studies of safety outcomes 
of interest in the proposed study that are captured in the database (to be described further 
in section IV.E). 
 
This information will allow FDA reviewers to better understand how experienced the research 
community is in using the data source(s) that will be employed. 
 
In addition, investigators should demonstrate that each data source contains sufficient clinical 
granularity to capture the exposures and outcomes of interest in the appropriate setting of care.  
For example, outpatient data sources that do not include access to hospitalization data would not 
  9
  Contains Nonbinding Recommendations 
be appropriate for studying safety outcomes likely to result in hospitalization.  It is also 
important to address the data standardization and coding and explain why the coding is sufficient 
for ascertainment of outcomes of interest and other important variables.  For example, safety 
outcomes that cannot be identified using International Classification of Diseases (ICD) codes 
cannot be appropriately studied using data sources that rely solely on ICD codes in claims data. 
 
The protocol should describe the relevant populations to be used in a study, including whether 
access to specific patient populations (e.g., psychiatric, pediatric) may be important.  
Investigators should include a description of the continuity of coverage (see section IV.B) for 
patients included in the data source, and specify how all relevant exposures and outcomes will be 
captured during the study period. It is important for investigators to ensure that the data 
source(s) contain a sufficient number of patients or patient follow-up time to ascertain outcomes 
of interest based on the hypothesized exposure risk window. Investigators should provide 
information about the churn rate, as it may indicate whether the selected data source(s) are 
appropriate for ascertaining long latency safety outcomes.  
 
B.  Enrollment and Comprehensive Capture of Care 
 
Investigators using administrative claims data sources should address continuity of coverage  
(enrollment and disenrollment), particularly for claims data sources in the United States, because  
patients often enroll and disenroll in different health plans in relation to changes in employment 
or other life circumstances (Strom 2005).  The validity of the findings of a study using these data 
depends in part on ensuring that the migration of patients in and out of the electronic healthcare 
data sources can be documented.  Such documentation allows only periods of enrollment during 
which data are available on the patients of interest to be included in the study, and periods of 
disenrollment when data are not available on patients can be appropriately excluded.  Definitions 
of enrollment or continuous coverage should be developed and documented, particularly in 
studies using more than one data source.   
 
It is important for investigators to address comprehensiveness and continuity of coverage when 
employing EMR data sources, as the entire continuum of the patients care might not be 
available in a single EMR system.  For example, patients receiving treatment from multiple 
physicians, offices, or hospitals might have their course of care captured in several different 
EMR data sources. In addition, patients in the United States do not typically enroll in physician 
practices, but rather see physicians as needed or as their insurance coverage allows.  Therefore, 
investigators using an EMR data source should describe the steps taken to ensure complete 
capture of patient care over time to facilitate the likelihood that all exposures and safety 
outcomes of interest will be captured.   Furthermore, primary care-based EMR networks in the 
United States may not capture hospitalizations or visits to specialists.  If these are events of 
interest, investigators should specify how these events will be captured.   
 
If a protocol specifies hospital data as the sole data source, the investigator should discuss 
whether outpatient care data are relevant to the study because it is often not captured in this type 
of data source. Similarly, if an outpatient data source alone is proposed, detailed data on drug 
exposures in the hospital setting is usually not available.  For either of these scenarios, the 
investigator should fully describe the relevant data source characteristics and their potential 
  10
  Contains Nonbinding Recommendations 
impact on the interpretation of study results.  In general, over-the-counter (OTC) medications 
and dietary supplements are not captured systematically in electronic healthcare data because 
they are not prescribed by physicians and their costs are not always reimbursable under insurance 
plans. If these exposures are particularly relevant to the study question, then investigators should 
describe how they will address this information gap. 
 
C.  Country of Origin and Health System:  Relevance to the United States  
 
There are differences in the practice of medicine across the world, some of which are particular 
to the healthcare system.  Certain differences affect whether a non-U.S. data source can be used 
to address specific drug safety hypotheses in a way that is relevant to the U.S. population.  
Various factors in non-U.S. healthcare systems, such as medication tiering (e.g., first-line, 
second-line) and patient coverage selection, influence the degree to which patients on a given 
therapy in other countries might differ in disease severity from patients on the same therapy in 
the United States. 
 
Even within the United States, patients in different electronic healthcare databases may differ in 
disease severity. In addition, as we begin to learn more about the role of pharmacogenetics in 
drug-related harms, it will become increasingly important to consider the impact of potential 
pharmacogenetic profile variations on the feasibility and generalizability of safety studies relying 
on non-U.S. data sources. 
 
For studies that propose the use of a data source from a country other than the United States, the 
investigator should provide: 
  
 	  The rationale for selecting the particular data source(s) to address the specific 
hypotheses; 
  
 	  Background information about the healthcare system, including method(s) of 
diagnosis and preferred patterns of treatment for the disease(s) of interest, and the 
degree to which such information is collected in the proposed data sources; 
  
 	  A description of prescribing and utilization practices, including approved indications, 
formulations, and doses for non-U.S. settings; 
  
 	  Information on the market availability of the treatment(s) of interest; and 
  
 	  An explanation of how all these factors might affect the generalizability of the study 
results to the U.S. population. 
 
D.  Selection of Study Population 
 
Investigators should use explicit inclusion and exclusion criteria for the selection of the study 
population and provide an appropriate explanation in the protocol for the criteria selected.  For 
proposed studies, we recommend providing specific estimates of relevant population size in the 
proposed data source, including the size of the exposed population.  For studies involving elderly 
  11
  Contains Nonbinding Recommendations 
patients (age 65 and older) in the United States, it is important to describe the completeness of 
medical care and drug coverage, including direct access or linkage to Medicare data.11    
 
Obtaining comprehensive drug coverage and medical care data on patients with some types of 
serious and life-threatening conditions (e.g., HIV/AIDS or cancer) can present a unique 
challenge. These patients may receive treatment in state- or federal-based clinics, in 
experimental clinical trials, or under pharmaceutical company assistance programs, which may 
not be fully captured in most electronic healthcare data sources.  If these issues are relevant to 
the study question of interest, the investigators should report how they will be addressed in the 
protocol. 
 
FDA recommends the use of a flow diagram  or other accounting scheme to display the 
disposition of study subjects at various stages of inclusion or exclusion.  A diagram or schematic 
provides an easily visible record of study process (see Esposito 2009; Friedman 2008; 
Schneeweiss 2008; Weiner 2008). 
 
E.  Quality Assurance (QA) and Quality Control (QC) 
 
Investigators should fully understand the quality assurance (QA)  and quality control  (QC)  
procedures used by the data holders and how these procedures could have an effect on the 
integrity of the data and the overall validity of the study.  FDA recommends that investigators 
address the following topics: 
 
	   The general procedures used by the data holders to ensure completeness, consistency, 
and accuracy of data collection and management; 
  
	   The frequency and type of any data error corrections or changes in data adjudication 
policies implemented by the data holders during the relevant period of data collection; 
 
	   A description of any peer-reviewed publications examining data quality and/or 
validity, including the relationships of the investigators with the data source(s); 
 
	   Any updates and changes in coding practices (e.g., ICD codes) across the study 
period that are relevant to the outcomes of interest; 
  
	   Any changes in key data elements during the study time frame and their potential 
effect on the study; and 
 
	   A report on the extent of missing data over time (i.e., the percentage of data not 
available for a particular variable of interest) and a discussion on the procedures (e.g., 
exclusion, imputation) employed to handle this issue.  Investigators should also 
address the implications of the extent of missing data on study findings and the 
missing data methods used. 
                                                 
11 For more information on Medicare data, please access the Centers for Medicare and Medicaid Services (CMS) 
Web site at http://www.cms.hhs.gov/medicareGenInfo. 
 
  12
  Contains Nonbinding Recommendations 
 
F.  Study Time Frame and Lag Time Issues 
 
Investigators should define the study time frame (which spans from the beginning of the look 
back period, when the investigator looks back in the database before drug exposure to ascertain 
baseline patient covariate data to the end of the follow up period) in the protocol and discuss the 
suitability of the time frame in addressing the specific hypotheses associated with the study. The 
discussion should address temporal changes in the standard of care, the availability of other 
treatments, and any other relevant factors.  For example, a given drugs patterns of use may 
change over time, and any potential corresponding differences in the patients exposed to the drug 
over time could be relevant to the safety outcome(s) of interest.  Investigators should discuss the 
impact of changes in use patterns over time on the exposed group and develop the study 
approach accordingly. 
 
Lag times in data availability can be of particular concern for studies that use electronic 
healthcare data. Investigators should specifically designate the time period for ascertainment of  
the relevant outcomes and covariates in the protocol and ensure that complete data are available 
for the selected time frame. The use of clear diagrams and pictorial displays to describe study 
time frames is encouraged (Schneeweiss and Avorn 2005).   
 
V.  BEST PRACTICES  STUDY DESIGN 
 
  A.  Study Design Considerations 
 
  1.  Choice of Study Design 
 
FDA does not endorse a specific type of study design for pharmacoepidemiologic safety studies 
that use electronic healthcare data because the choice should be made uniquely in the context of 
the drug, the safety issue, and the specific hypotheses of interest.  Investigators should first 
establish the study questions of interest and then determine which data source(s) and design are 
most appropriate to address these questions. Investigators should discuss their rationale for 
selecting a particular study design in the study protocol and final report.  
 
  2.  Examples of Study Designs (Not All-Inclusive)  
 
Several study designs are commonly used in observational pharmacoepidemiologic safety 
studies, including, among others, cohort, case-control, nested case-control, case-cohort, and case-
crossover studies. The selection of the most appropriate study design depends on the study 
question of interest and what is known about the postulated relationship between drug exposure 
and the specific safety outcomes of interest.  Overall, different study designs will be appropriate 
depending on the study question(s). FDA discourages investigators from applying a one size fits 
all approach to study design, irrespective of the designs suitability in addressing the particular 
study questions and hypotheses. 
 
3.  Comparator Selection 
   
  13
  Contains Nonbinding Recommendations 
Selection of an appropriate comparator group or control group is a critical part of a 
pharmacoepidemiologic safety study.  FDA encourages the use of multiple comparator groups in 
those study designs where it is feasible and relevant, as this strategy can enhance the validity of 
safety studies (Waning 2001).   If multiple comparator or control groups are employed, the 
protocol should identify the primary comparator and include a discussion of the rationale for 
each group and its importance to the study questions of interest.   
 
In cohort studies, it is ideal to use a comparator group taking a drug used to treat the same  
disease, with the same level of disease severity, and from the same time period as the exposed 
cohort. For example, when selecting a comparator group, investigators should consider the 
impact that formulary status (e.g., medication tiering or prior authorization issues) could have on 
the level of disease severity of the exposed group and the comparator group.  If it is not possible 
to find an appropriate comparator group from the same time period as the exposed group, 
investigators might elect to use historical comparators (i.e., a comparator group selected from a 
different time period than the exposed cases).  If investigators choose to use historical 
comparators, they should explain the rationale for their choice and discuss how the use of 
historical comparators might have an impact on the interpretation of study results (e.g., 
consideration of the impact on changes in medical practice over time).   
 
Pharmacoepidemiologic safety studies conducted on preventative therapeutics, such as vaccines, 
present issues that call for special considerations in study design.  One such issue is the healthy 
vaccinee effect. Because vaccines are generally given to persons who are healthy, confounding 
could occur if vaccinated persons are compared to unvaccinated persons, as the latter may, for 
example, be more ill or lack access to primary medical care.  The sheer avoidance of or delay in, 
vaccination may be associated with an increased risk of the outcomes of interest (Fine 1992).  
Although there are various ways to address this issue, it may be appropriate to use self-control or 
case-crossover designs, where the same person serves as his or her own control (Maclure 1991).  
Regardless of which study approach is used, the protocol and final report should describe the 
relevant limitations of the comparator groups selection. 
 
4.  Study Time Frame 
 
Refer to section IV.F, Study Time Frame and Lag Time Issues. 
 
  5.  Identification and Handling of Confounders and Effect Modifiers 
  
Biases that are either unidentified or inadequately addressed can compromise the validity of  
pharmacoepidemiologic safety studies.  Therefore, it is important for investigators to describe the 
processes used to identify potential biases and to provide a scientific and clinical rationale for the 
methods selected to handle them.  Channeling  and confounding by indication are two related 
biases that can be particularly problematic in pharmacoepidemiologic safety studies (Strom  
2005). Although these biases can be managed by using certain methods, such as propensity or 
disease risk scores, restriction, or by using multiple or self-comparison groups, they could be so 
pervasive as to entirely preclude the use of an observational study to examine the safety issue 
(Petri and Urguhart 1991, McMahon 2003). 
 
  14
  Contains Nonbinding Recommendations 
Channeling refers to the situation where drugs are prescribed to patients differently based on the  
presence or absence of factors prognostic of patient outcomes.  In other words, the potential for a  
given outcome differs between the groups (those who receive the drug or medical product of 
interest and the comparison group) because of patient characteristics that were present prior to 
drug treatment.  When these characteristics are also related to prescribing patterns of the drug of 
interest, a biased estimate of risk may result.  As such, channeling may introduce a bias that can 
make a drug seem either more beneficial or more harmful than it actually is. (Petri 1991; Journal 
of American Medical Association Online Dictionary).  Confounding by indication is a type of 
channeling bias that occurs when the indication, which is associated with the drug exposure, is 
an independent risk factor for the outcome; this is of particular concern in 
pharmacoepidemiologic safety studies using electronic health data because these data do not 
generally identify the indication for drug use. 
 
There are various epidemiologic and statistical methods for identifying and handling 
confounding in pharmacoepidemiologic safety studies using electronic healthcare data, and FDA 
does not endorse or suggest any particular method.  FDA encourages the continued development, 
use, and evaluation of innovative methods to adjust for confounding.   
 
Propensity score modeling is one approach that is often used to address confounding in 
pharmacoepidemiologic studies.  A propensity score for an individual is the predicted probability 
for treatment with a particular drug (usually the drug under study) conditioned on the 
individuals measured covariate values within the relevant database(s).  The score can be used to 
achieve balance in the distribution of potential confounding factors between those exposed to the 
drug of interest and the comparator group, with respect to the measured covariates (Rosenbaum  
and Rubin 1983; DAgostino et al. 1998).  When propensity score modeling is used, 
investigators should present diagnostics of the propensity score model to allow for an assessment 
of its performance and fit.  A full discussion of propensity scores is beyond the scope of this 
guidance; however, many articles in the literature, some of which are cited in this guidances 
bibliography, provide a more in-depth discussion of this model and its appropriate application to  
pharmacoepidemiologic safety studies.  
 
Another approach to address confounding in pharmacoepidemiologic safety studies is to exclude 
patients who have risk factors for the safety outcome that are unrelated to drug use.  This strategy 
can be appropriate, but can also have the unintended consequence of reducing the size of the 
study population, precluding examination for effect modification, and limiting the 
generalizability of the results.  An alternative approach to excluding such patients is for the 
investigator to stratify the patients by the relevant risk factors.  The decision-making process for 
excluding or stratifying should be made in the context of the specific study hypotheses and fully 
explained and documented in the protocol.  FDA discourages the exclusion of patients because it 
prevents investigators from enhancing the generalizability of the study results, compromises 
statistical power, and precludes the examination for effect modification by these other risk 
factors. However, if a particular group of patients is to be excluded from a study, the investigator 
should justify this approach, provide a detailed explanation of the exclusions, and discuss the 
resulting limitations in study interpretation. 
 
  15
  Contains Nonbinding Recommendations 
All confounders, particularly time-varying confounders, and effect modifiers should be 
operationally defined and justified. 
 
Unmeasured confounders can affect the study validity.  Investigators should consider how they 
might explore the impact of these unmeasured confounders.  Any assumptions regarding these 
variables and any sensitivity analyses used to test these assumptions should be fully described.  
Sometimes supplemental information can be used to explore the potential impact of unmeasured 
confounders on the study findings. For example, in a pharmacoepidemiologic safety study of 
nonsteroidal anti-inflammatory medications, investigators supplemented electronic pharmacy 
data with survey data on OTC medications, which served as a proxy for OTC nonsteroidal anti-
inflammatory drug use (Graham, et al. 2005).  In another example, the functional status of study 
subjects was obtained through medical chart review, which enabled researchers to more 
accurately estimate the impact of the subjects frailty (a confounder that typically would not be 
available in electronic health data) on the risk of harms associated with a vaccine (Jackson, et al. 
2006; Jackson, et al. 2008). 
 
The potential for effect modification by main demographic variables (e.g., age, gender, and 
race), or pertinent co-morbidities, should be examined in the study.  If significant effect 
modification is found, the risk estimates should be presented appropriately.   
 
  6.  Sample Size and Statistical Power 
 
Sample size and statistical power should be estimated before initiating the study.  In addition, 
investigators should explain how the sample size was determined and provide relevant 
assumptions with pertinent justifications; formulas used to calculate the sample size; and a 
description accounting for the impact of anticipated exclusion criteria applied to the study 
population and database that was selected.  It is especially important to provide the rationale 
behind the determination of sample size for rare outcomes (e.g., specific vaccine issues related to 
an adverse outcome) and long latency outcomes.  The initial power calculations and the validity 
of underlying assumptions should be revisited at the end of the study in the context of the results, 
particularly in the case of negative findings. 
 
  B.  Study Design: Exposure Definition and Ascertainment 
 
  1.  Exposure Definition 
   
The investigator should define the exposure risk window for the outcome of interest and describe 
in detail the measurement of the window in the selected data source(s).  FDA recommends that 
the investigators obtain information about the postulated exposure risk window from other 
sources, such as spontaneous report data, to increase the likelihood that only relevant periods of 
exposure are examined.  For example, if an adverse outcome is known to only occur immediately 
after initial use of a drug and the exposure definition includes all of the patients time on a drug, 
a significant amount of nonrelevant exposure time could be included, which could result in 
biased risk estimates.  All assumptions made in defining the exposure risk window should be 
clearly articulated and justified, including when information about the timing of exposure and 
  16
  Contains Nonbinding Recommendations 
outcome is not known during the study design process.  Sensitivity analyses might prove helpful 
in testing these assumptions (refer to section VI.D). 
 
Investigators should justify the units selected for the exposure risk window (e.g., person-time, 
patients, and prescriptions). Relevant drug interactions that could affect the measurement of the 
exposure risk window for the drug of interest should also be operationally defined.  For example, 
if investigators are operationally defining concomitancy, they should state whether drugs are 
considered as being used concomitantly by the same patient if: 
  
   they are dispensed on the same day;  
   they have overlapping days supply; 
   they have ever received prescriptions for the two drugs during the study period; or  
   another relevant definition is appropriate. 
  
If a different definition of concomitancy is used, it should be described in detail. 
 
Dispensed prescriptions are commonly used to define the exposure risk window for studies 
conducted using administrative claims data.  If investigators elect to define the exposure in other 
ways (e.g., using other clinical characteristics or medical diagnoses, or prescription orders from  
EMR data), they should discuss the demonstrated validity of these definitions. 
 
  2.  Exposure Ascertainment  Study Design 
 
As previously mentioned, the study design should be tailored to the study question of interest and 
justified in the study protocol; FDA does not advocate the use of any specific study designs.  
However, one design increasingly being used is the new (incident) user design, which is based 
on the first exposure to or use of the drug of interest.  This approach merits discussion here, as it 
provides advantages over the prevalent user design, which is susceptible to survivor bias. The 
prevalent user design can also be susceptible to other biases that arise from controlling for 
covariates in the causal pathway. If employing a new user design (Ray 2003), the investigator 
should operationally define new use and provide a rationale for this definition in the context of 
the study question.  This information will allow reviewers to critically evaluate the accuracy of 
the exposure definition. 
 
Investigators should bear in mind that patients may have entered the electronic healthcare data 
system already using the drug of interest; therefore, look back periods should be defined to 
ensure that such patients are not incorrectly classified as new users.  A new user design might 
focus on patients newly switching from a first-line to second-line therapy or on patients adding a 
new medication to existing therapy.  If investigators choose to employ a prevalent user design, 
they should provide sound justification and fully describe the time during which follow up to 
prevalent use begins; the rationale for this selection; and the implications for covariate 
measurement. Investigators should include a discussion on the limitations of this approach and 
the potential impact on study findings. 
 
3.  Exposure Ascertainment  Data Source 
 
  17
  Contains Nonbinding Recommendations 
It is critical that the proposed data source sufficiently capture the drug exposures during the 
defined study period, and a relevant discussion should be included in the study protocol.  The 
operational exposure definitions should be based on the coding system of the selected data 
source(s) and reflect an understanding of the prescription, delivery and reimbursement 
characteristics of the drug in that data source.  For example, in the United States, the operational 
definition should include the appropriate pharmacy codes (National Drug Codes or NDC codes) 
and/or procedure (J) codes12 to capture the use of the drug(s) in various settings. This is 
particularly important in the case of non-oral drugs, which may be assigned different codes 
depending on how they are obtained. For example, patients using an injectable drug can 
purchase it from the pharmacy, in which case the NDC code would be recorded, or the provider 
can purchase it for the patient and bill for both the drug and its administration using the J code.   
 
When investigators use an insurance-based data source, it is important for them to address lack 
of capture of prescriptions not associated with insurance claims if these drugs are relevant 
exposures for the study. Uncaptured prescriptions might include low cost generics and drugs 
obtained through programs at a standardized discount price; samples provided by pharmaceutical 
companies and dispensed by healthcare providers; and drugs sold through the Internet.  In 
addition, OTC medications and dietary supplements are not generally captured in insurance-
based data sources. Study investigators should address the presence and extent of incomplete 
exposure ascertainment and its effect on study validity.  
 
4.  Exposure Ascertainment  Gaps in Therapy and Censoring 
 
Because patients often do not obtain refills exactly on time, gaps in therapy could exist in 
electronic healthcare data, and decisions should be made as to when these gaps are long enough 
to suggest true interruption of chronic therapy.  FDA recommends that investigators clearly 
describe and justify in the protocol how they will address potential gaps in therapy, especially for 
chronic therapies. Intermittent therapies (e.g., drugs used to treat pain on an as needed basis) and 
therapies for which samples are often provided to patients (e.g., oral contraceptives) present 
special challenges in accurately assessing the actual exposure time.  It is critical that 
investigators establish operational definitions of exposure when studying these types of 
therapies. 
 
It is also important that investigators define the window of time after discontinuation during 
which the events of interest might still be attributed to the drug.  It might be most appropriate for 
this period of time to be the same as the gap allowance used to define continuous enrollment. 
Alternatively, the period of time can be established based on the relationship between the known 
mechanisms of action of the drug and the outcome of interest. Overall, the censoring of follow-
up time and events should be clearly defined by the investigator.  For example, events can be 
counted using an approach in which only time on drug (or perhaps time on drug plus a window 
of time after drug exposure) is included in the analysis.  Alternatively, the approach might be to 
include all follow-up time in the analysis, including time off the drug.  For further discussion of 
the exposure risk window, refer to section V.B.1. 
                                                 
12 A drugs  J code  more properly, the Health Care Financing Administration (HCFA) common procedures coding  
system code  is  used for submitting Medicare claims for reimbursement of  outpatient care. 
 
  18
  Contains Nonbinding Recommendations 
 
5.  Exposure Ascertainment  Dose 
 
Although electronic healthcare data capture drugs that are either prescribed for or dispensed to 
patients, they do not capture patients actual drug exposure because this depends on patients 
adherence to the prescribed therapy.  In certain cases, such as drugs used chronically or drugs 
with fixed-dose regimens, it can be appropriate to infer dosage information from electronic 
healthcare data.  Investigators should provide the specific assumptions made when estimating the 
dose of the exposure (drug) of interest, especially for pediatric patients.  It is also important for 
investigators to report how different dosage forms will figure into the dosage calculation, if  
multiple forms are available.  Sensitivity analyses may prove helpful in testing these assumptions 
(refer to sections VI.D and V.B.1). 
 
6.  Exposure  Other Factors 
 
In pharmacoepidemiologic studies of drugs using inpatient data, it is important for investigators 
to consider the order in which inpatient drugs are administered, both in relation to each other and 
to the outcomes of interest; this information may not be readily available in electronic healthcare 
data. In addition, if the exposure of interest is available as both a fixed combination product and 
the individual component of a fixed combination product, it is important to account for this fact 
when ascertaining total exposure (e.g., drugs to treat hypertension are commonly prescribed as 
combination drugs).  Another important consideration in defining and ascertaining exposure is 
switching between drugs within the same therapeutic drug class.  Repeated switching can 
significantly complicate exposure definition, but for many drugs this will reflect real-world 
patient experience. In these cases, the investigator should describe the approach taken to account 
for switching and discuss the potential impact on the studys generalizability. 
 
In most EMR data, information on exposure is generally limited to products prescribed by 
healthcare providers.  In the absence of a linkage between prescribing and dispensing systems, it 
cannot be assumed that the patient actually filled the prescription and thus the investigator cannot 
assume the patient is exposed.  It is important for investigators to describe their approaches to 
ensure the validity of EMR prescribing information before using it to define patient drug 
exposures. 
 
  C.  Study Design: Outcome Definition and Ascertainment 
 
One of the most crucial steps in selecting a data source is determining whether it is appropriate 
for capturing the outcomes of interest.  Because electronic healthcare data typically capture 
outcomes that are treated (or at least brought to the attention of a healthcare professional), 
outcomes representing mild symptoms (or the other extreme of sudden death without medical 
care) will not be well captured.  Outcomes on the continuum between these two extremes can be 
captured to varying degrees by different types of data sources and should be assessed carefully 
before study initiation. 
 
  19
 Contains Nonbinding Recommendations 
1. 	 Medically and Scientifically Relevant Case Definition of Safety Outcomes 
of Interest 
 
When developing the case definition for the outcomes of interest, it is important to obtain both 
epidemiologic and clinical input.  Case definitions for outcomes should be developed 
independently of drug exposure status, and exposure to the drug should not be an inherent 
component of the outcome definition.  To avoid selection bias when evaluating the effect of drug 
exposures, FDA recommends that investigators refrain from defining cases (and their controls if 
part of the study design) based on the presence or absence of these exposures before or after the 
occurrence of the outcome of interest.  
 
  2. 	 Validation of Outcomes 
 
Because administrative claims data are not collected for investigative purposes, but rather for 
patient care or reimbursement purposes, investigators should ensure that the selected medical 
outcomes of interest are validated (Lanes 2006).  Specifically, outcome validation involves 
establishing a clinically appropriate outcome definition and determining the positive predictive 
value of that definition. Although validation can be performed using different techniques, the 
determination of the positive predictive value of a code-based (e.g., ICD) operational outcome  
definition often involves selecting all or a sample of cases with the codes of interest from the 
data source and conducting a review of their primary medical data (generally medical charts) to 
determine whether or not each patient actually experienced the coded event. Although validation 
is critical for all safety studies, it is especially important for certain vaccine outcomes for which 
coding practices cannot be known or assumed. 
  
If the outcome has previously been validated, the investigator should cite the specific literature 
references. The validated algorithm should be described in detail, including the population and 
database in which the validation was performed, its performance characteristics, and the time 
frame during which the validation was performed.  For studies lacking prior outcome validation, 
the investigator should provide appropriate justification of the outcome definition used.13  
 
For pharmacoepidemiologic safety studies, the sensitivity of the outcome definition is an 
important consideration because it is often not possible to completely ascertain the outcome of 
interest when using electronic healthcare data. For example, although an operational outcome  
definition could have high positive predictive value, it could have low sensitivity if the outcome  
of interest does not often come to medical attention and therefore is missed.  Thus, investigators 
should discuss (1) the positive predictive value of the outcome definition and potential 
implications on the studys internal validity and (2) the sensitivity of the outcome definition, 
namely, the extent to which the outcome can be ascertained in the selected data sources and 
                                                 
13 One example of justification is referencing standardized case definitions.  For example, for vaccine studies, 
investigators could reference collections of standardized case definitions such as the International Brighton  
Collaboration  (http://www.brightoncollaboration.org), which provides a growing repertoire of such definitions  for 
vaccine safety investigations. The  use  or adaptation of definitions from these types of standardized case definition 
collections can facilitate comparisons of analyses between  different studies. 
 
  20
  Contains Nonbinding Recommendations 
study population. The discussion should include potential implications for the external validity, 
or generalizability of the study findings, to the real world patient population. 
   
FDA recommends that outcome definitions be specified and explained a priori and incorporate 
the coding system of the data source(s) used.  In some situations, it can be appropriate to validate 
only a sample of cases.  Information gathered in the outcome validation process should be 
incorporated into the analysis plan and protocol and, if appropriate, be submitted as a final 
protocol amendment.  Investigators should consider the rarity of the outcome when considering 
the desired sensitivity and specificity of the coding algorithm.  It is important to consider the 
often arbitrary ranking of coded primary and secondary hospital discharge diagnoses, and the 
associated limitations of these categories when selecting which diagnoses to choose as outcomes 
(e.g., the order of discharge diagnoses may not correspond to their medical importance).  ICD 
codes in claims data are generally considered more reliable for inpatient outcomes than for 
outpatient outcomes, where upcoding and downcoding practices are commonly used to 
maximize reimbursement (Strom 2005).  Therefore, when using claims data, it is preferable to 
use and validate inpatient codes in defining outcomes whenever possible because these codes are 
generally more reliable and tend to reflect more serious diseases.  In addition, investigators 
should report their ability to capture outcome  severity in the databases employed.  This is 
especially pertinent for outcomes that patients can manifest differently and with varying degrees 
of severity. For example, an allergic reaction can range from a  skin rash to a life-threatening 
experience such as anaphylactic shock. 
 
The strategy described above for validation in administrative claims data might not be relevant 
when EMR data are used, because the EMR might represent the only primary medical data 
available for validating the patient encounter.  There is still a scientific need, however, to 
develop and employ strategies for ensuring that the coded electronic data accurately reflect 
patient experience.  To gain more information, investigators might, for example, review pertinent 
paper files and documents, review clinical detail from free text in the EMR, or follow up with the 
healthcare providers. As implementation of EMRs becomes more widespread, FDA encourages 
investigators to develop innovative strategies to confirm electronic exposure and outcome data. 
This will serve to strengthen the validity of studies relying upon these data. 
 
3.  Outcome Definition  Procedures or Diagnoses 
 
When defining outcomes, if the investigator is using procedure data instead of or in addition to 
diagnosis data as outcomes, he or she should explain the rationale behind this choice.  Validation 
of the codes used, or a justification for not performing validation, should also be included.  
 
    4.   Mortality as an Outcome 
 
Death is a particularly difficult outcome to ascertain reliably and completely using electronic 
healthcare data. Although deaths that occur while a patient is under medical care are often 
documented in these data systems, reliable ascertainment of deaths can only be accomplished 
through linkage with vital statistics or other systems such as the Social Security Administration 
  21
  Contains Nonbinding Recommendations 
(SSA) or National Death Index (NDI) (NDI Web site).14  These linkages can provide 
confirmation of death and date of death; however, determining the cause of death may call for 
further information obtained from death certificates (NDI data provide cause of death 
information from the state death certificate but SSA data do not) (MacMahon and Pugh 1970).  
The use of death certificate data is subject to all the known limitations of such data (Lenfant 
1067; Cambridge and Cina 2010). 
 
Because deaths that occur while the patient is not under medical care might not be captured in 
electronic healthcare data systems, patients who die might only be observed in electronic 
healthcare data as either not filing any further claims or not receiving any additional care past a 
particular date. For studies in which the outcomes may often be fatal, it is important that 
investigators not exclude patients who appear to be lost to follow up at any time following their 
exposure to the study drug. These patients should be included in searches of NDI or other 
systems to see whether their absence (disenrollment) from the system has been caused by death, 
specifically by death related to the study outcome of interest. 
 
VI.  BEST PRACTICES  ANALYSES 
 
A.  Prespecified Analysis Plan 
 
If investigators have conducted any feasibility analyses, the study protocol should discuss the 
results, including their impact on the prespecified analysis plan and study.   
 
The prespecified analysis plan should address the specific study objectives and specify the 
details of the primary analysis and any  secondary analyses. The plan should be included in the 
protocol; however, depending on the level of detail, it may be preferable to include the plan in an 
appendix to the protocol. At a minimum, the plan should include details on the statistical models 
and tests, sample size estimation, the significance level and power of the study, handling of 
missing values, subgroup analyses and the assessment of effect modification, and the method of 
confounding adjustment.  Operational definitions of exposures, covariates, and outcomes should 
also be delineated. 
 
Careful consideration of the adjustment for confounders is critical for studies that employ 
electronic healthcare databases.  Because these databases are often large, the studies can have 
high power to obtain statistically significant results.  However, results might be biased because 
of unmeasured confounding and small effects can be statistically significant.  The potential 
impact of unmeasured confounders and other relevant sources of bias should be discussed and, 
when possible, evaluated analytically.  Unadjusted results and results that have been adjusted for 
confounding both should be presented and discussed in the final analysis to facilitate both a 
qualitative and quantitative evaluation of the adjustments. 
 
B.  Study Analyses 
 
                                                 
14 National Death  Index  Web site: http://www.cdc.gov/nchs/ndi.htm. 
 
  22
  Contains Nonbinding Recommendations 
Unadjusted analyses/analyses with nonvalidated outcomes: If investigators plan to perform  
unadjusted analyses (analyses that do not adjust for confounders and examine effect 
modification) or analyses with nonvalidated outcomes, in addition to adjusted analyses and 
analyses with validated outcomes, they should prespecify the plan for these analyses and explain 
the rationale for conducting these analyses (e.g., to describe study covariates).  For further 
discussion of the identification and handling of confounders, examination of effect modification, 
and outcome validation, refer to sections V.A.5 and V.C.2. 
 
Primary analyses address the prespecified study hypotheses.  
 
Secondary analyses (e.g., prespecified analyses to address secondary objectives, subgroup 
analyses, post hoc analyses), which are defined as analyses conducted in addition to the primary 
study analyses, should also be described. Significant findings in subgroup analyses can be 
considered hypothesis-generating unless prespecified and adequately powered.  If subgroup 
analyses are employed, investigators should describe methods used to examine the subgroups 
and interactions (von Elm, et al. 2008).  Post hoc analyses  are analyses that were not anticipated 
or described in the analysis plan or study protocol (Begaud 2000).  It is recommended that post 
hoc analyses be clearly described as such to aid interpretation.  
 
C.  Use of Specific Statistical Techniques (e.g., To Minimize Confounding) 
 
Investigators should ensure that appropriate statistical techniques are used to address 
confounding and assess effect modification. These techniques should be well described and 
justified, including a clear delineation of relevant assumptions and limitations.  Diagnostics, both 
graphical and analytical, are often relevant and facilitate the evaluation of assumptions and 
performance of the techniques.  The statistical analysis plan should specify the planned statistical 
techniques and diagnostic methods before the data are reviewed.  The study report should present 
the results of the statistical analysis and accompanying diagnostics.  When there are key effect 
modifiers, the report should also present results for the subgroups defined by the effect 
modifiers. 
 
D.  Sensitivity Analyses 
 
FDA recommends the use of sensitivity analyses to determine the impact of various study 
decisions relating to design, exposure definition and outcome definition, and of deficiencies in 
the data source(s) selected. Such analyses can be very helpful in determining the potential 
impact of varying assumptions (e.g., changing cutpoints for categorical variables, limiting or  
expanding case or outcome definitions, changing definitions of current exposure) on study 
results, and can facilitate better interpretation of study results in light of significant uncertainty.  
It is important for investigators to clearly identify and describe sensitivity analyses that are 
performed and to provide their own interpretation of the impact of these analyses on the 
interpretation of the study findings. 
 
E.  Linking or Pooling Data From Different Sources 
 
  23
  Contains Nonbinding Recommendations 
If multiple data sources are to be used in the study, the reason for the multiple data sources and 
the details on how they will be used should be provided. As mentioned in the study approach 
section above (III.C), multiple data sources can be linked to provide more complete information 
on a set of patients (e.g., a healthcare database can be linked with national mortality records).  
For this use, the details of linking and the expected accuracy of the linking should be provided.  
 
Another use of multiple data sources is to provide a larger or more diverse sample of patients. 
For this use, details on whether and how the data sources are to be pooled should be provided.  
The data sources can be pooled on a patient level, or because of privacy or other limitations, the 
data sources can be pooled at a summary statistical level.  In some cases, it may be possible to 
use summary statistics or other de-identified information to pool information across the data 
sources to produce an analysis that is equivalent to pooling the patient-level data. In other cases, 
it may be important to employ study-level meta-analytical techniques to obtain a pooled 
estimate.  In either case, careful consideration should be given to the heterogeneity of the data 
sources and individual data source summaries (e.g., baseline characteristics, follow-up time) and 
results should be provided. 
 
F.  Assessment and Handling of Missing and Uninterpretable Data 
 
Investigators should develop a plan to assess and handle missing and uninterpretable data.  It is 
important to provide the percentage of missing data for key variables of interest.  Missing 
information is sometimes falsely interpreted.  For example, lack of positive information on the 
occurrence of an event (such as myocardial infarction) or a risk factor (such as smoking) might 
not mean that the event or risk factor was nonexistent.  Care should be given to default values of 
data and the implications of lack of information on data values.  
 
G.  Quality Assurance (QA) and Quality Control (QC) 
   
The QC and QA plan for the construction of the analytical data and the analysis of data should be 
clearly described. QC consists of the steps taken during the analysis to ensure that it meets 
prespecified standards and that it is reproducible.  QA consists of activities undertaken to 
evaluate quality control.  CDERs Manual of Policy and Procedures (MAPP) 6700.2 Standards 
for Data Management and Analytic Processes  in the Office of Surveillance and Epidemiology 
(OSE), (FDA MAPP 6700.2), provides an example of how QC and QA planning and 
implementation can be accomplished .15     
 
Investigators should describe the approaches taken to ensure data integrity (confidentiality and 
security of information from authorized access or  revision) and data validity (correctness of data 
that is collected and stored). 
 
In certain circumstances, FDA might request access to the original analytic data set to conduct 
re-analyses of the data to verify study results; thus, the lead study investigator should ensure that 
analytic data sets used in the study are archived in a way that provides access for the purpose of 
                                                 
15 MAPPs are available on the Internet at 
http://www.fda.gov/AboutFDA/CentersOffices/CDER/ManualofPoliciesProcedures/default.htm#ODS. 
 
  24
  Contains Nonbinding Recommendations 
such re-analyses while ensuring personal data protection.  FDA will address the need for 
accessing original analytic data on a case-by-case basis. 
 
H.  Procedures To Ensure Accuracy of Data Management and Analysis Process 
 
The 2008 ISPE guidelines highlight the importance of describing data management and 
statistical software programs and hardware to be used in the study and data preparation and 
analytical procedures as well as the methods for data retrieval and collection (ISPE guidelines 
2008). FDA encourages investigators to describe these processes used for managing and 
preparing their data. It is important that analysts performing and reviewing data management 
and analysis have appropriate training or prior experience in the use of the particular analytic 
software that is being used.  FDA considers documentation a very important component of the 
analytic process and recommends that all analytic programs be thoroughly annotated with 
comments that clearly describe the intent or purpose of each step.  
  25
  Contains Nonbinding Recommendations 
 
GLOSSARY 
 
 
 
The following terms are defined for the purposes of this guidance. 
 
Administrative claims data: Claims data arise from a person's use of the healthcare system  
[and reimbursement of healthcare providers for that care]. (Strom 2005). 
 
Channeling: A situation where drugs are prescribed to patients differently based on the  presence 
or absence of factors prognostic of patient outcomes.   
 
Confounding by indication: Confounding by indication is a type of channeling bias that occurs 
when the indication, which is associated with the drug exposure, is an independent risk factor for 
the outcome  
 
Churn rate:  Individuals may enroll or disenroll from a health plan over time, especially as, in 
the United States, most healthcare coverage is employer-based and thus coverage changes over 
time with changes in employer. The churn rate, also known as attrition rate, is the proportion of 
enrollees that disenroll from a health plan in a given time period.  
 
Comparator (comparison) group: Any group to which patients with either the exposure or 
outcome of interest are compared (Porta 2008).  
 
Continuous coverage/Continuity of coverage: The period of time over which a patient is 
enrolled in a healthcare system and during which any medical service or drug prescription would 
be captured in the healthcare systems electronic record system. 
 
Diagnostics: Methods used to assess the performance of a statistical model and/or evaluate the 
fit of the method or model to the data. 
 
Electronic healthcare data:  Analytic data that is an organized set of [healthcare] data or 
collection of files available by computer through electronic format which can be used for the 
conduct of pharmacoepidemiologic [safety] studies (Hartzema 2008).  It is derived from a raw 
electronic healthcare database. 
 
Electronic medical record (EMR): An electronic record of health-related information on an 
individual that can be created, gathered, managed, and consulted by authorized clinicians and 
staff within one healthcare organization (National Alliance for Health Information Technology 
(NAHIT) Report 2008).  
 
Exposure risk window: Interval of exposure (to the drug of interest) time considered to be 
relevant in the design or analysis of a pharmacoepidemiologic study.  In case-control studies, it is 
essential to define a priori the period during which the possible exposure to the drug of interest 
will be investigated in the previous history of  the cases and comparators.  An equivalent period 
should be defined for the comparators.  Similarly, in a cohort study, the time window defines the 
  26
  Contains Nonbinding Recommendations 
period after the beginning of exposure during which the occurrence of an event (safety outcome) 
of interest will be attributed to the exposure.  (Begaud 2000).  
 
Feasibility analyses: Analyses conducted before the development of the protocol and analysis 
plan to determine if a study is feasible (i.e., the enumeration of patients exposed to the therapies 
of interest, the evaluation of the magnitude of the safety outcome in the population of interest). 
 
Historical comparators: Comparator group for whom data were collected at a time preceding 
that at which the data were gathered on the group of interest.  (Porta 2008). 
 
Lag time: Lag time is the time of a patient's actual date of service to the date when the 
adjudicated claim first appears in the data source.  Different types of claims (pharmacy, 
physician, and hospital) have different average lag times.  This phenomenon causes varying 
degrees of under-ascertainment near the data cut-off date.  
 
Observational studies: In observational studies, the investigator does not control the therapy, 
but observes and evaluates the results of ongoing medical care. (Strom 2005). For purposes of 
this guidance, observational study designs do not involve random allocation and   include case-
control, nested case-control, case-cohort, and case-crossover designs. 
  
Pharmacoepidemiologic safety study: An observational study designed to assess the risk 
associated with a drug exposure and test prespecified hypotheses (2005 FDA guidance).  
 
Post hoc analysis: An analysis that was not anticipated or described in the analysis plan or 
study protocol (Begaud 2000). 
 
Primary analyses: Analyses that address the prespecified primary study hypotheses. 
 
Quality assurance (QA): Quality assurance consists of activities undertaken to evaluate quality 
control (FDA MAPP 6700.2). 
 
Quality control (QC): Quality control consists of steps taken during the generation of a drug or 
service to ensure that it meets prespecified standards and that the drug or service is reproducible 
(FDA MAPP 6700.2).  
 
Secondary analyses: Analyses conducted in addition to the primary analyses (e.g., prespecified 
analyses to address secondary objectives, subgroup analyses, post hoc analyses). 
 
Study time frame:  The time frame from the beginning of the look back period, when the 
investigator looks back in the database before drug exposure to ascertain baseline patient 
covariate data, to the end of the follow-up period.  
 
Survivor bias: A specific type of selection bias arising from differences in competing risks or 
loss to follow-up between exposure groups (Rothman 2008). 
 
  27
  Contains Nonbinding Recommendations 
Time-varying confounder: A  confounder variable whose values change over the study time 
frame (Platt 2009).  
 
Validation: Validation involves the establishment of a clinically appropriate definition for the 
variable (e.g., patient attribute, medical condition, outcome) of interest and the determination of 
the positive predictive value of that definition.  
 
 
 
 
 
 
 
 
 
 
 
 
 
  28
  Contains Nonbinding Recommendations 
 
BIBLIOGRAPHY  

 
Begaud, Bernard. Dictionary of Pharmacoepidemiology. Chichester, England: Wiley and Sons, 
2000. 
 
Cambridge and Cina. The accuracy of death certificate completion in a suburban 
community. Am J Forensic Med Pathol 2010;31:232-235. 
 
Charlson, et al. A new method of classifying prognostic comorbidity in longitudinal studies: 
development and validation. J Chronic Dis 1987;40(5):373-383. 
 
DAgostino, et al. Propensity score methods for bias reduction in the comparison of a treatment 
to a nonrandomized control group.  Stat Med 1998;2265-2281. 
 
Esposito, et al. Results of a retrospective claims database analysis of differences in 
antidepressant treatment persistence associated with escitalopram and other selective serotonin 
reuptake inhibitors in the United States. Clin Ther 2009;31:644-656. 
 
FDA guidance for industry: Good Pharmacovigilance Practices and Pharmacoepidemiologic 
Assessment (March 2005). 
 
FDA Manual of Policies and Procedures (MAPP) 6700.2: Standards for Data Management and 
Analytic Processes in the Office of Surveillance and Epidemiology (2008). 
 
Fine, et al. Confounding in studies of adverse reactions to vaccines. Am J Epidemiol 
1992;136:121-135. 
 
Friedman, et al. A retrospective study of the use of fluticasone propionate/salmeterol 
combination as initial asthma controller therapy in a commercially insured population. Clin Ther 
2008;30:1908-1917. 
 
Graham, et al. Risk of acute myocardial infarction and sudden cardiac death in patients treated 
with cyclo-oxygenase 2 selective and non-selective non-steroidal anti-inflammatory drugs: 
nested case-control study. Lancet 2005;365:475-481. 
 
Hartzema, Tilson, and Chan. Pharmacoepidemiology and Therapeutic Risk Management. 
Cincinnati: Harvey Whitney Books, 2008. 
 
Hernan, et al. Causal knowledge as a prerequisite for confounding evaluation: an application to 
birth defects epidemiology. Am J Epidemiol 2002;155:176-184. 
 
HTA 101: Glossary, National Information Center on Health Services Research and Health Care 
Technology (NICHSR), National Library of Medicine. 
<http://nlm.nih.gov/nichsr/hta101/ta101014.html>. 
 
  29
  Contains Nonbinding Recommendations 
ISPE guidelines. Guidelines for good pharmacoepidemiology practices (GPP). 
Pharmacoepidemiol Drug Saf 2008;17:200-208. 
 
Jackson, et al. Evidence of bias in estimates of influenza vaccine effectiveness in seniors.  Int J 
Epidemiol 2006;35:337-344. 
 
Jackson, et al. Influenza vaccination and risk of community-acquired pneumonia in 
immunocompetent elderly people: a population-based, nested case-control study. Lancet 
2008;372:398-405. 
 
Journal of the American Medical Association. Online Dictionary. 
<http://jamaevidence.com/search/result/57441>. 
 
Lanes, et al. Bias due to false-positive diagnoses in an automated health insurance claims  
database. Drug Saf 2006;29(14):1069-1075. 
 
Lenfant, et al. Fifty years of death certificates: The Framingham Heart Study. Ann Int Med 
1998;129:1067. 
 
Maclure, Malcolm. The case-crossover design: a method for studying transient effects on the risk 
of acute events. Am J Epidemiol 1991;133:144-153. 
 
MacMahon and Pugh. Epidemiology Principles and Methods. Boston: Little, Brown, and 
Company, 1970.   
 
McMahon, Alex. Approaches to combat with confounding by indication in observational studies 
of intended drug effects. Pharmacoepidemiol Drug Saf 2003;12:551558. 
 
Moher D, Schulz KF, Altman DG, for the CONSORT Group. The CONSORT statement: revised 
recommendations for improving the quality of reports of parallel group randomized trials. Ann 
Intern Med 2001;134:657-662. 
 
National Alliance for Health Information Technology (NAHIT) Report to the Office of the 
National Coordinator for Health Information Technology (ONCHIT) on Defining Key Health 
Information Technology; 5/15/08. 
 
Petri, H. and Urquhart, J. Channeling bias in the interpretation of drug effects. Stat  Med 
1991;10:577-581. 
 
Platt, Robert W, et al. Time-modified confounding. Am J Epidemiol 2009;170(6):687-694. 
 
Porta, Miquel. A Dictionary of Epidemiology. New York: Oxford University Press, 2008. 
 
National Death Index, National Center for Health Statistics, CDC.  
<http://www.cdc.gov/nchs/data_access/ndi/about_ndi.htm>. 
  30
  Contains Nonbinding Recommendations 
Ray, Wayne.  Evaluating medication effects outside clinical trials: new-user designs.  Am J 
Epidemiol 2003;158:915-920.    
 
Robb, et al. The U.S. Food and Drug Administrations Sentinel Initiative: expanding the 
horizons of medical product safety.  Pharmacoepidemiol Drug Saf 2012;21(S1):9-11. 
 
Rosenbaum and Rubin. The Central Role of the Propensity Score in Observational Studies for 
Causal Effects. Biometrika 1983;70:41-55. 
 
Rothman KJ, Greenland S, Lash TL. Modern Epidemiology. Philadelphia: Lippincott Williams  
& Wilkins, 2008:198.  
 
Strom, Brian. Pharmacoepidemiology. Chichester, England: John Wiley and Sons, 2005. 
 
Schneeweiss, et al. Aprotinin during coronary-artery bypass grafting and risk of death.  N Engl J 
Med 2008;358:771-783. 
 
Schneeweiss and Avorn. A review of uses of health care utilization databases for epidemiologic 
research on therapeutics. J Clin Epidemiol 2005;58:323-337. 
 
Suissa, Samy. Immortal time bias in observational studies of drug effects. Pharmacoepidemiol 
Drug Saf 2007;16:241-249. 
 
Vandenbroucke, Jan. Observational Research, Randomised Trials, and Two Views of Medical 
Science. PLoS Med 2008;5(3):e67. 
 
Von Elm, et al. Strengthening the reporting of observational studies in epidemiology (STROBE) 
statement: guidelines for reporting observational studies. J Clin Epidemiol 2008;61:344-349. 
 
Waning, et al. Pharmacoepidemiology Principles and Practice. USA: McGraw Hill, 2001. 
  
Weiner, et al. Replication of the Scandinavian Simvastatin Survival Study using a primary care 
medical record database prompted exploration of a new method to address unmeasured 
confounding. Pharmacoepidemiol Drug Saf 2008;17:661-670. 
 
  31