National Institutes of Health: Additional Data Would Enhance the Stewardship of Clinical Trials across the Agency
Published: 2016-03-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In fiscal year 2014, NIH spent nearly $3.2 billion on clinical trials as part of its research activities. NIH's OD oversees the operations of 27 ICs to ensure that NIH's research portfolio is balanced, not unnecessarily duplicative, and utilizes cross-cutting research. In 2010, IOM made recommendations for clinical trials supported by NCI, one of NIH's ICs. In 2012, NIH was directed to conduct a review of the applicability of IOM's recommendations across all NIH ICs that conduct clinical trials.
A Joint Explanatory Statement accompanying a 2015 appropriations act included a provision for GAO to review how NIH applied the IOM recommendations. This report examines (1) the steps that NIH took, if any, to apply the IOM recommendations across its ICs other than the NCI, and (2) the extent to which NIH's OD uses data to oversee clinical trial activity across the ICs. GAO reviewed NIH documentation on the applicability of the IOM recommendations, data the OD uses to oversee clinical trial activity, and its process for using such data. GAO compared these to federal standards for internal control. GAO also interviewed NIH and IC officials, IOM officials, and stakeholders, such as a group representing researchers.
section_title: What GAO Found
paragraphs: Although the National Institutes of Health (NIH) assessed the applicability of recommendations made by the Institute of Medicine (IOM) in 2010 to improve clinical trials—studies involving human subjects that test the effects of interventions on health-related outcomes—within one of its Institutes and Centers (IC), NIH did not apply the recommendations across its ICs. In response to a conference report provision that it review the applicability of the IOM recommendations across its ICs, NIH administered a survey to all 24 of the ICs that fund clinical trials and presented the findings at a leadership forum and in a report to Congress. These findings showed that over half of the ICs surveyed indicated that the majority of the recommendations were applicable. NIH decided not to apply the recommendations across its ICs because more analysis was needed before proposing any NIH-wide recommendations, given the variation across ICs. Officials explained that the IOM recommendations were designed for one program within the National Cancer Institute (NCI) and that most ICs do not support clinical trial networks that operate with the size and volume of the program, thus making the recommendations more pertinent to NCI. Leaders from NIH and the ICs indicated that more analysis was needed to account for the ICs' portfolios and management activities. As a result, NIH developed its own recommendations that aimed to enhance its stewardship of clinical trials, including several to improve data collection across the ICs. For example, its recommendation to improve monitoring systems, tools, and processes could assist NIH in identifying additional data that could be collected across the ICs.
NIH's Office of the Director (OD) reviews some data on clinical trial activity across NIH but has not finalized what additional data it needs or established a process for using these data to enhance its stewardship of clinical trials, as intended by NIH's own recommendations. The OD only reviews two types of data related to clinical trial activity on a regular basis: financial data and data on the inclusion of minorities and women. Beyond these data, OD officials review other data from the ICs on clinical trial activity if there is a specific inquiry. Officials from the OD acknowledged that they do not regularly review much data specifically related to clinical trial activity, but they are considering reviewing additional data collected from the ICs to inform the OD's stewardship of clinical trial activity across NIH. However, the OD has not finalized what data it needs from the ICs. In addition, the OD has not established a process that specifies how and when the OD will use the additional data it decides to review. As a result, it is unclear how often the OD will review the data, for what purpose, and what the product of its analysis will be. Federal Standards for Internal Control state that agencies need operational and financial data to determine whether they are meeting their goals for effective and efficient use of resources. Given that ICs oversee specific clinical trials, the OD may not need the same data or level of detail collected by ICs. However, until the OD determines which additional data it will review and the process it will use to review these data, NIH is limited in its ability to make data-driven decisions regarding the use of its roughly $3 billion annual investment in clinical trials.
section_title: What GAO Recommends
paragraphs: GAO recommends that the NIH OD (1) finalize data on clinical trial activity that the OD needs to collect from ICs, and (2) establish and implement a process for using those data. HHS concurred with the recommendations.

section_title: Letter
paragraphs: As part of its mission to improve the nation’s health, the National  Institutes of Health (NIH), an agency within the Department of Health and  Human Services (HHS), supports research on the effectiveness of  various health care interventions, such as medications, therapies, or other  clinical treatments. This research includes clinical trials, which are studies  involving human subjects that test the effects of interventions on health- related outcomes. In fiscal year 2014, NIH spent nearly $3.2 billion on  clinical trials, an amount representing about 11 percent of the agency’s  overall budget, through 24 of NIH’s Institutes and Centers (IC).  According to NIH, clinical trials are a critically important component of  NIH’s mission and among the most complex and challenging research  activities the agency funds. In addition, because they involve human  subjects, including some with life-threatening conditions, clinical trials  inherently carry more risk than other types of research.
While NIH’s ICs are directly responsible for overseeing the clinical trials  they fund, NIH’s Office of the Director (OD), the agency’s central policy  office, is responsible for overseeing the operations of the ICs and for  program coordination across the ICs, including ensuring that NIH’s  research portfolio across the ICs is balanced, not unnecessarily  duplicative, and utilizes cross-cutting research. Because of the multi- billion dollar investment associated with clinical trials and the risk to  patients who participate in them, NIH has stated that it has an obligation  to exercise good stewardship over these studies, such as by reviewing  data to help set agency-wide policies and priorities. According to NIH, the  OD provides leadership to the ICs and helps coordinate research priority- setting activities across the ICs.
One of NIH’s ICs, the National Cancer Institute (NCI), asked the Institute  of Medicine (IOM) to review NCI’s Cooperative Group Program, a  component of a clinical trial network that comprised more than 3,100  institutions, 14,000 investigators, and enrolled more than 25,000 patients  in clinical trials each year. According to IOM, NCI requested this review  to address several challenges faced by the Cooperative Group Program,  such as redundancies and inefficiencies in its process for designing,  reviewing, and initiating clinical trials. In 2010, IOM released its report  detailing its review of the Cooperative Group Program and provided 12  recommendations—referred to in this report as the IOM  recommendations—to improve the efficiency and effectiveness of clinical  trials supported by the program. For example, IOM recommended that  NCI develop and evaluate novel clinical trial designs. NCI has efforts  underway to implement each of the recommendations.
A provision in the Conference Report accompanying the Consolidated  Appropriations Act, 2012, directed NIH to conduct a review of the  applicability of the IOM recommendations to all of NIH’s ICs that conduct  clinical trials. The Conference Report indicated that the review should  examine ways to develop and strengthen NIH-wide policies, with a focus  on identifying opportunities to incorporate innovative science, increasing  the speed with which clinical trials are initiated and completed, improving  the means of setting priorities, and developing better incentives to  encourage participation in clinical trials. In March 2013, NIH provided a  report to Congress on the findings of its review.
The Joint Explanatory Statement to Accompany the Consolidated and  Further Continuing Appropriations Act, 2015, includes a provision for us  to review how NIH applied the IOM recommendations across its ICs.  This report examines  1.  the steps that NIH took, if any, to apply the IOM recommendations  across its ICs other than the NCI, and  2.  the extent to which NIH’s OD uses data to oversee clinical trial activity  across the ICs.
To identify the steps that NIH took, if any, to apply the IOM  recommendations across its ICs other than NCI, we reviewed  documentation from NIH’s Clinical Trials Working Group (CTWG), NIH’s  Task Force on Clinical Trials Stewardship Reform—referred to in this  report as the Task Force, and NIH’s OD, such as a presentation on the  applicability of the IOM recommendations. We interviewed officials who  were involved in determining the applicability of the IOM  recommendations across NIH, including officials from the OD. To  supplement our understanding of the steps NIH took to apply the IOM  recommendations, we interviewed officials from a nongeneralizable  selection of 4 of the 24 ICs that fund clinical trials. Our selection included  ICs with a range of low, medium, and high levels of funding for clinical  trials in fiscal year 2014 and included two ICs with clinical trial networks  and two ICs without clinical trial networks. We interviewed officials from  the selected ICs about the applicability of the IOM recommendations to  clinical trials funded by their ICs and how NIH assessed the applicability  of the IOM recommendations across the ICs. We also interviewed officials  from IOM about the IOM recommendations. Finally, we interviewed  stakeholders involved in clinical trial activity, such as groups representing  pharmaceutical companies and clinical researchers, to obtain their  perspectives on NIH’s clinical trial activity.
To determine the extent to which NIH’s OD uses data to oversee clinical  trial activity across the ICs, we reviewed documentation of the data that  the OD reviews related to clinical trial activity, such as documentation of  the funding provided for each clinical trial supported by the ICs. We  compared the types of data the OD reviews on clinical trial activity with  the federal internal control standards described in Standards for Internal  Control in the Federal Government, which specify that agencies should  have operational and financial data to determine whether they are  meeting their goals. We interviewed OD officials about the process they  use to review data on clinical trial activity, and we compared the OD’s  process with requirements for comparing actual performance to planned  or expected results outlined in Standards for Internal Control in the  Federal Government. To supplement our understanding of how the OD  reviews data from IC officials, we interviewed officials from the four  selected ICs described above about the data that they provide to the OD  and about how the OD uses that information. We also reviewed  documentation from the selected ICs on the types of information the ICs  collect on clinical trial activity, such as information collected through  progress reports submitted to NIH by recipients of NIH clinical trial  funding.
We conducted this performance audit from July 2015 through March 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: NIH comprises 27 ICs and an OD and is headed by a presidentially  appointed and Senate-confirmed director. NIH funds clinical trials through  24 of its 27 ICs. Each IC has its own mission and generally focuses on  specific diseases, systems, organs, or stages in life (e.g., childhood). In  addition, ICs each have their own director and staff, which help to support  and oversee the IC’s work. The OD includes various offices responsible  for issues, programs, and activities that span NIH components, such as  research initiatives involving multiple ICs. By law, the OD has several  responsibilities, including coordinating and overseeing the operation of  the ICs and ensuring, in consultation with the ICs, that NIH’s research  portfolio is balanced, not unnecessarily duplicative, and utilizes cross- cutting research.
subsections: section_title: Clinical Trials Supported by NIH
paragraphs: NIH’s ICs support clinical trials predominantly through extramural  research—awarding funds to researchers at universities or other research  entities through grants, contracts, and cooperative agreements. These  ICs use a standard peer review process to help determine which clinical  trials and other extramural research projects to fund. The size and  composition of the ICs’ clinical trial portfolios vary substantially,  depending on factors such as the IC’s budget and mission.
Officials at each IC oversee clinical trial activity within their IC through the  review of various reports, correspondence, site visits, and other available  information. For example, NIH requires that all grant recipients, including  those conducting clinical trials, submit progress reports annually to ICs  that describe the major goals of the trial and what activities have been  accomplished and that provide financial information. According to NIH  officials, IC staff generally use this information to identify potential  problems and areas where assistance might be necessary for individual  trials, and to help determine whether the IC will continue funding the trial.
While ICs are responsible for determining which clinical trials to fund and  for overseeing the progress of individual clinical trials, officials from the  OD are responsible for overseeing the operations of the ICs and, in  consultation with the ICs, ensuring that NIH’s research portfolio across  the ICs is balanced, not unnecessarily duplicative, and utilizes cross- cutting research. For example, the OD has indicated that it aims for a  balance of short- and long-term research activities across the agency,  among other things.
section_title: Overview of IOM Recommendations
paragraphs: The 2010 IOM report to NCI made 12 recommendations across 4 goals to  reduce redundancy and improve the effectiveness and efficiency of the  clinical trials supported by the Cooperative Group Program. (See Table  1.)
section_title: NIH Assessed the Applicability of the IOM Recommendations and Determined Not to Apply Them across ICs, but Developed Its Own Recommendations
paragraphs: To assess the applicability of the IOM recommendations from a 2010  review of NCI’s Cooperative Group Program across the agency, NIH  administered a survey to its ICs that fund clinical trials. While NIH decided  not to apply the IOM recommendations across its ICs, NIH developed its  own recommendations with an aim to enhance NIH’s stewardship of its  clinical trials, including several recommendations to improve data  collection across the ICs.
subsections: section_title: NIH Surveyed ICs to Assess the Applicability of the IOM Recommendations and Determined Not to Apply the Recommendations across ICs
paragraphs: In response to a conference report provision to review the applicability of  the IOM recommendations, NIH administered a survey to all 24 of the ICs  that fund clinical trials and then aggregated the results. NIH presented  these results at an annual leadership forum in September 2012 and  reported these results to Congress in March 2013. The survey results  showed that over half of the ICs surveyed indicated that 7 of the 12 IOM  recommendations were applicable to their IC. For example, 75 percent  of the ICs surveyed indicated that the recommendation to increase the  volume, diversity, and speed of accrual was applicable.
Similarly, stakeholders we spoke with also indicated that some of the  recommendations, such as the ones relating to patient accrual and the  development and evaluation of novel trial designs, may be relevant  across NIH’s ICs, in addition to NCI. For example, one stakeholder we  spoke with said that the recommendation to develop and evaluate novel  trial designs is important because it could ultimately lead to trials that are  not as costly or lengthy without compromising the information obtained  from the trial.
OD officials told us that NIH decided not to apply the IOM  recommendations across all of its ICs that fund clinical trials because  more analysis was needed before proposing any NIH-wide  recommendations, given the variation across ICs. Officials explained that  the IOM recommendations were designed for one program within NCI— the Cooperative Group Program—and most ICs do not support clinical  trial networks that operate with the size and volume of NCI’s Cooperative  Group Program, thus making the recommendations more pertinent to  NCI. Leaders from NIH and the ICs indicated that more analysis was  needed to account for the ICs’ portfolios and management activities.
section_title: NIH Developed Its Own Recommendations That Aim to Enhance its Stewardship of Clinical Trials, Including Several to Improve Data Collection across its ICs
paragraphs: Through its CTWG, NIH developed recommendations that aim to  enhance its stewardship of clinical trials, including several  recommendations to improve NIH’s ability to collect data on clinical trial  activity across its ICs. According to officials, the CTWG—a working group  which included officials from NIH’s OD and the ICs—began in February  2013 in response to discussions among NIH officials at the agency’s  annual leadership forum, where officials presented the results of NIH’s  survey on the applicability of the IOM recommendations. Although it  decided not to apply the IOM recommendations across ICs, NIH officials  recognized that they could enhance aspects of NIH’s stewardship of  clinical trials, and so they developed the CTWG to provide  recommendations on how NIH could do so. In August 2013, the CTWG  issued eight recommendations intended to enhance NIH’s stewardship of  clinical trials. (See Table 2.)
Several of the CTWG recommendations aim to improve the data NIH  collects from across the ICs. For example, in developing the fourth  recommendation to improve monitoring systems, tools, and processes,  the CTWG concluded that there is a need for standardized data collected  across NIH’s ICs that would help NIH identify the factors or characteristics  that contribute to a successful clinical trial. The CTWG indicated that one  type of additional data that could be collected across the ICs is accrual  data—that is, information on the number of patients enrolled in a research  study, such as a clinical trial. According to the CTWG, some ICs have  their own systems for monitoring clinical trials, which could be used to  collect accrual data, while other ICs do not have these systems and  instead rely on communication with awardees and grant reports to  monitor their clinical trials. The CTWG explained that these different  monitoring practices result in variation in the types of clinical trial data  currently collected at the IC level.
To implement the recommendations of the CTWG, NIH formed the Task  Force in April 2015. Officials on the Task Force told us that they are  taking steps to address the CTWG recommendations by developing  policies and procedures. For example, officials stated that the Task Force  is currently working on identifying a standard set of data on clinical trials  that each IC would collect to respond to the CTWG’s fourth  recommendation to improve monitoring systems, tools, and processes.  NIH said that this could improve monitoring systems by ensuring that all  ICs collect similar data on clinical trials, which could facilitate analysis of  such data across ICs. NIH could not provide an expected date for when  the Task Force would complete its work; however, officials expect that the  Task Force will continue its efforts at least through July 2016.
Although NIH officials said that the CTWG and the IOM recommendations  are not related, we identified similarities between two of the CTWG  recommendations and the IOM recommendations. Specifically, the  CTWG recommendation to streamline Institutional Review Board review  of multi-site studies aligns with the IOM recommendation to streamline  and harmonize government oversight. Additionally, the CTWG  recommendation to provide Good Clinical Practice training for  investigators and staff is similar to the IOM recommendation to support  clinical investigators through training and mentoring, paid protected  research time, sufficient resources, and recognition.
section_title: NIH’s OD Reviews Some Data on Clinical Trial Activity across ICs, but Has Not Finalized What Additional Data it Needs or Established a Process for Using These Data
paragraphs: NIH’s OD reviews some data on clinical trial activity across the agency,  but has not finalized what other data it needs to enhance its stewardship  of clinical trials, as intended by the CTWG recommendations, or  established a process for using these data. Specifically, the OD only  reviews two types of data related to clinical trial activity on a regular basis:  financial data and inclusion data. The OD collects financial data from its  ICs to determine how much funding is allocated across all of NIH’s clinical  research—which include clinical trials as well as other types of research.  The OD reviews this information as part of the agency’s efforts to develop  its annual budget. The OD also reviews inclusion data it collects from the  ICs on the enrollment of minorities and women in clinical trials— information which NIH provides to the Congress in a biennial report  required by law. However, as we have previously reported, the inclusion  data the OD reviews is limited because the data only provide information  on aggregate-level enrollment across NIH and for each IC, which may  inadvertently mask low enrollment in clinical trials related to specific  diseases or research areas.
Beyond financial and inclusion data, OD and IC officials said that the OD  does not routinely review additional data on clinical trial activity. Rather,  OD officials review data from the ICs on clinical trial activity in response to  specific inquiries. For example, OD officials reviewed information from the  ICs on their clinical trial activities in India after regulatory requirements in  India changed during the past two years and the officials received an  inquiry about it. According to OD officials, ICs are responsible for  overseeing the clinical trials they fund and for collecting data for this  purpose. Officials from the four ICs we spoke with provided various  examples of data on clinical trial activity reviewed at the IC level, such as  data on the number of clinical trials funded by the ICs and the amount  allocated for each clinical trial per year. While the frequency of these  reviews varies, IC officials review these data at least once a year.
While officials from the OD acknowledged that they do not regularly  review much data specifically related to clinical trial activity, they are also  considering changing this practice and reviewing additional data collected  from the ICs to inform the OD’s stewardship of clinical trial activity across  NIH. According to OD officials, the additional data that they could review  may include the standardized data that NIH’s Task Force is responsible  for identifying. As we previously mentioned, NIH’s Task Force is  developing a set of standard data that the ICs would collect on the clinical  trials they support. Some or all of these data could potentially be used by  officials from the OD to review NIH’s clinical trial portfolio. For example,  OD officials told us that one example of standard data could be accrual  data, which could be used by the OD to determine whether or not clinical  trials across NIH are enrolling patients according to proposed timelines.
While OD officials have speculated about what additional data from the  ICs they may review, the OD has not finalized which of these data the  office needs. According to OD officials, they are waiting for the Task  Force to complete its efforts to identify a standardized set of data to be  collected across the ICs. In addition, the OD has not established a  process that specifies how and when the OD will use the additional data it  decides to review. As a result, it is unclear how often the OD will review  the data, for what purpose, and what the product of its analysis will be.
Federal Standards for Internal Control state that agencies need  operational and financial data to determine whether they are meeting their  goals for effective and efficient use of resources. Given that ICs oversee  the clinical trials they award, the OD may not need the same data or level  of detail collected by ICs, or need to review the data with the same  frequency. However, until the OD determines which additional data it will  review and the process it will use to review these data, NIH is limited in its  ability to make data-driven decisions about the use of its roughly $3 billion  annual investment in clinical trials. Furthermore, reviewing operational  and financial data could help NIH reach its strategic plan goal of ensuring  a continued high return on the public investment in research. Finally,  reviewing standard data on clinical trials could help OD officials ensure  that NIH meets its responsibility to ensure that its research portfolio is  balanced, not unnecessarily duplicative, and utilizes cross-cutting  research.
section_title: Conclusions
paragraphs: As part of its mission, NIH is responsible for exercising good stewardship  of its multi-billion dollar public investment in clinical trials. The outcomes  of these trials are vital for improving public health and advancing science,  as they are used to identify the effects of medications and other health  care interventions on people, some with life-threatening illnesses and  conditions. Although NIH’s ICs are responsible for the clinical trials they  fund, federal law requires NIH’s OD, in consultation with the heads of the  ICs, to coordinate NIH’s research portfolio to ensure that it is balanced,  not unnecessarily duplicative, and utilizes cross-cutting research. NIH  decided not to apply the IOM recommendations across its ICs; however, it  does acknowledge the need for enhanced stewardship of its clinical trials  and developed its own recommendations for this purpose. Consistent with  our finding that NIH’s OD only uses some data to oversee clinical trial  activity across the ICs, some of NIH’s own recommendations recognize  the need for additional data. However, NIH’s OD has not determined  which data it needs or how it will use the data to strengthen its  stewardship of NIH’s clinical trials and research portfolio. Until the OD  identifies what data to review, and how the data will be used and when,  NIH is missing an opportunity to maximize the public investment in clinical  trials.
section_title: Recommendations for Executive Action
paragraphs: To enhance its stewardship of clinical trials across the ICs, we  recommend that the Secretary of HHS direct the NIH OD to take the  following two actions:  1.  finalize data on clinical trial activity that the OD needs to collect from  2.  establish and implement a process for using those data.
section_title: Agency Comments
paragraphs: We provided a draft of this report to HHS for comment. HHS provided  written comments, which are reprinted in appendix I. HHS concurred with  our recommendations and stated that the agency plans to complete its  efforts to identify additional data that will be collected from the ICs on a  regular basis. HHS indicated that these data will provide the OD with  readily available information on the number, type, and progress of clinical  trials funded across the ICs. Although HHS did not specify plans to  establish and implement a process for using the data, it said its goal is to  complete its current efforts during 2016 and to implement other strategies  to enhance the quality and transparency of HHS-funded clinical trials.
In its comments, NIH also stated that the report over-emphasized NIH's  decision to not apply the IOM recommendations across the ICs and  under-emphasized the role and scientific expertise of the ICs in managing  and overseeing clinical trials. We maintain that the report appropriately  describes the steps taken by NIH to assess the applicability of IOM's  recommendations and to enhance its stewardship of clinical trials.  Further, the report notes that the ICs and the OD both play a critical, but  different, role in ensuring the stewardship of clinical trials across the  agency. In addition, HHS provided technical comments, which we  incorporated as appropriate.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Health and Human Services, and other  interested parties. In addition, the report will be available at no charge on  GAO’s website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Office  of Congressional Relations and Office of Public Affairs can be found on  the last page of this report. Other major contributors to this report are  listed in appendix II.
section_title: Appendix I: Comments from the Department of Health and Human Services
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Tom Conahan, Assistant  Director; Krister Friday; Q. Akbar Husain; Morgan Jones; Rebecca Rust  Williamson; and Emily Wilson made key contributions to this report.