National Institutes of Health: Additional Data Would Enhance the Stewardship of Clinical Trials across the Agency
Released: 2016-03-10T00:00:00

Letter

Background

NIH comprises 27 ICs and an OD and is headed by a presidentially appointed and Senate-confirmed director. NIH funds clinical trials through 24 of its 27 ICs. Each IC has its own mission and generally focuses on specific diseases, systems, organs, or stages in life (e.g., childhood). In addition, ICs each have their own director and staff, which help to support and oversee the IC’s work. The OD includes various offices responsible for issues, programs, and activities that span NIH components, such as research initiatives involving multiple ICs. By law, the OD has several responsibilities, including coordinating and overseeing the operation of the ICs and ensuring, in consultation with the ICs, that NIH’s research portfolio is balanced, not unnecessarily duplicative, and utilizes cross- cutting research.

Clinical Trials Supported by NIH

NIH’s ICs support clinical trials predominantly through extramural research—awarding funds to researchers at universities or other research entities through grants, contracts, and cooperative agreements. These ICs use a standard peer review process to help determine which clinical trials and other extramural research projects to fund. The size and composition of the ICs’ clinical trial portfolios vary substantially, depending on factors such as the IC’s budget and mission.

Officials at each IC oversee clinical trial activity within their IC through the review of various reports, correspondence, site visits, and other available information. For example, NIH requires that all grant recipients, including those conducting clinical trials, submit progress reports annually to ICs that describe the major goals of the trial and what activities have been accomplished and that provide financial information. According to NIH officials, IC staff generally use this information to identify potential problems and areas where assistance might be necessary for individual trials, and to help determine whether the IC will continue funding the trial.

While ICs are responsible for determining which clinical trials to fund and for overseeing the progress of individual clinical trials, officials from the OD are responsible for overseeing the operations of the ICs and, in consultation with the ICs, ensuring that NIH’s research portfolio across the ICs is balanced, not unnecessarily duplicative, and utilizes cross- cutting research. For example, the OD has indicated that it aims for a balance of short- and long-term research activities across the agency, among other things.

Overview of IOM Recommendations

The 2010 IOM report to NCI made 12 recommendations across 4 goals to reduce redundancy and improve the effectiveness and efficiency of the clinical trials supported by the Cooperative Group Program. (See Table 1.)

NIH Assessed the Applicability of the IOM Recommendations and Determined Not to Apply Them across ICs, but Developed Its Own Recommendations

To assess the applicability of the IOM recommendations from a 2010 review of NCI’s Cooperative Group Program across the agency, NIH administered a survey to its ICs that fund clinical trials. While NIH decided not to apply the IOM recommendations across its ICs, NIH developed its own recommendations with an aim to enhance NIH’s stewardship of its clinical trials, including several recommendations to improve data collection across the ICs.

NIH Surveyed ICs to Assess the Applicability of the IOM Recommendations and Determined Not to Apply the Recommendations across ICs

In response to a conference report provision to review the applicability of the IOM recommendations, NIH administered a survey to all 24 of the ICs that fund clinical trials and then aggregated the results. NIH presented these results at an annual leadership forum in September 2012 and reported these results to Congress in March 2013. The survey results showed that over half of the ICs surveyed indicated that 7 of the 12 IOM recommendations were applicable to their IC. For example, 75 percent of the ICs surveyed indicated that the recommendation to increase the volume, diversity, and speed of accrual was applicable.

Similarly, stakeholders we spoke with also indicated that some of the recommendations, such as the ones relating to patient accrual and the development and evaluation of novel trial designs, may be relevant across NIH’s ICs, in addition to NCI. For example, one stakeholder we spoke with said that the recommendation to develop and evaluate novel trial designs is important because it could ultimately lead to trials that are not as costly or lengthy without compromising the information obtained from the trial.

OD officials told us that NIH decided not to apply the IOM recommendations across all of its ICs that fund clinical trials because more analysis was needed before proposing any NIH-wide recommendations, given the variation across ICs. Officials explained that the IOM recommendations were designed for one program within NCI— the Cooperative Group Program—and most ICs do not support clinical trial networks that operate with the size and volume of NCI’s Cooperative Group Program, thus making the recommendations more pertinent to NCI. Leaders from NIH and the ICs indicated that more analysis was needed to account for the ICs’ portfolios and management activities.

NIH Developed Its Own Recommendations That Aim to Enhance its Stewardship of Clinical Trials, Including Several to Improve Data Collection across its ICs

Through its CTWG, NIH developed recommendations that aim to enhance its stewardship of clinical trials, including several recommendations to improve NIH’s ability to collect data on clinical trial activity across its ICs. According to officials, the CTWG—a working group which included officials from NIH’s OD and the ICs—began in February 2013 in response to discussions among NIH officials at the agency’s annual leadership forum, where officials presented the results of NIH’s survey on the applicability of the IOM recommendations. Although it decided not to apply the IOM recommendations across ICs, NIH officials recognized that they could enhance aspects of NIH’s stewardship of clinical trials, and so they developed the CTWG to provide recommendations on how NIH could do so. In August 2013, the CTWG issued eight recommendations intended to enhance NIH’s stewardship of clinical trials. (See Table 2.)

Several of the CTWG recommendations aim to improve the data NIH collects from across the ICs. For example, in developing the fourth recommendation to improve monitoring systems, tools, and processes, the CTWG concluded that there is a need for standardized data collected across NIH’s ICs that would help NIH identify the factors or characteristics that contribute to a successful clinical trial. The CTWG indicated that one type of additional data that could be collected across the ICs is accrual data—that is, information on the number of patients enrolled in a research study, such as a clinical trial. According to the CTWG, some ICs have their own systems for monitoring clinical trials, which could be used to collect accrual data, while other ICs do not have these systems and instead rely on communication with awardees and grant reports to monitor their clinical trials. The CTWG explained that these different monitoring practices result in variation in the types of clinical trial data currently collected at the IC level.

To implement the recommendations of the CTWG, NIH formed the Task Force in April 2015. Officials on the Task Force told us that they are taking steps to address the CTWG recommendations by developing policies and procedures. For example, officials stated that the Task Force is currently working on identifying a standard set of data on clinical trials that each IC would collect to respond to the CTWG’s fourth recommendation to improve monitoring systems, tools, and processes. NIH said that this could improve monitoring systems by ensuring that all ICs collect similar data on clinical trials, which could facilitate analysis of such data across ICs. NIH could not provide an expected date for when the Task Force would complete its work; however, officials expect that the Task Force will continue its efforts at least through July 2016.

Although NIH officials said that the CTWG and the IOM recommendations are not related, we identified similarities between two of the CTWG recommendations and the IOM recommendations. Specifically, the CTWG recommendation to streamline Institutional Review Board review of multi-site studies aligns with the IOM recommendation to streamline and harmonize government oversight. Additionally, the CTWG recommendation to provide Good Clinical Practice training for investigators and staff is similar to the IOM recommendation to support clinical investigators through training and mentoring, paid protected research time, sufficient resources, and recognition.

NIH’s OD Reviews Some Data on Clinical Trial Activity across ICs, but Has Not Finalized What Additional Data it Needs or Established a Process for Using These Data

NIH’s OD reviews some data on clinical trial activity across the agency, but has not finalized what other data it needs to enhance its stewardship of clinical trials, as intended by the CTWG recommendations, or established a process for using these data. Specifically, the OD only reviews two types of data related to clinical trial activity on a regular basis: financial data and inclusion data. The OD collects financial data from its ICs to determine how much funding is allocated across all of NIH’s clinical research—which include clinical trials as well as other types of research. The OD reviews this information as part of the agency’s efforts to develop its annual budget. The OD also reviews inclusion data it collects from the ICs on the enrollment of minorities and women in clinical trials— information which NIH provides to the Congress in a biennial report required by law. However, as we have previously reported, the inclusion data the OD reviews is limited because the data only provide information on aggregate-level enrollment across NIH and for each IC, which may inadvertently mask low enrollment in clinical trials related to specific diseases or research areas.

Beyond financial and inclusion data, OD and IC officials said that the OD does not routinely review additional data on clinical trial activity. Rather, OD officials review data from the ICs on clinical trial activity in response to specific inquiries. For example, OD officials reviewed information from the ICs on their clinical trial activities in India after regulatory requirements in India changed during the past two years and the officials received an inquiry about it. According to OD officials, ICs are responsible for overseeing the clinical trials they fund and for collecting data for this purpose. Officials from the four ICs we spoke with provided various examples of data on clinical trial activity reviewed at the IC level, such as data on the number of clinical trials funded by the ICs and the amount allocated for each clinical trial per year. While the frequency of these reviews varies, IC officials review these data at least once a year.

While officials from the OD acknowledged that they do not regularly review much data specifically related to clinical trial activity, they are also considering changing this practice and reviewing additional data collected from the ICs to inform the OD’s stewardship of clinical trial activity across NIH. According to OD officials, the additional data that they could review may include the standardized data that NIH’s Task Force is responsible for identifying. As we previously mentioned, NIH’s Task Force is developing a set of standard data that the ICs would collect on the clinical trials they support. Some or all of these data could potentially be used by officials from the OD to review NIH’s clinical trial portfolio. For example, OD officials told us that one example of standard data could be accrual data, which could be used by the OD to determine whether or not clinical trials across NIH are enrolling patients according to proposed timelines.

While OD officials have speculated about what additional data from the ICs they may review, the OD has not finalized which of these data the office needs. According to OD officials, they are waiting for the Task Force to complete its efforts to identify a standardized set of data to be collected across the ICs. In addition, the OD has not established a process that specifies how and when the OD will use the additional data it decides to review. As a result, it is unclear how often the OD will review the data, for what purpose, and what the product of its analysis will be.

Federal Standards for Internal Control state that agencies need operational and financial data to determine whether they are meeting their goals for effective and efficient use of resources. Given that ICs oversee the clinical trials they award, the OD may not need the same data or level of detail collected by ICs, or need to review the data with the same frequency. However, until the OD determines which additional data it will review and the process it will use to review these data, NIH is limited in its ability to make data-driven decisions about the use of its roughly $3 billion annual investment in clinical trials. Furthermore, reviewing operational and financial data could help NIH reach its strategic plan goal of ensuring a continued high return on the public investment in research. Finally, reviewing standard data on clinical trials could help OD officials ensure that NIH meets its responsibility to ensure that its research portfolio is balanced, not unnecessarily duplicative, and utilizes cross-cutting research.

Conclusions

As part of its mission, NIH is responsible for exercising good stewardship of its multi-billion dollar public investment in clinical trials. The outcomes of these trials are vital for improving public health and advancing science, as they are used to identify the effects of medications and other health care interventions on people, some with life-threatening illnesses and conditions. Although NIH’s ICs are responsible for the clinical trials they fund, federal law requires NIH’s OD, in consultation with the heads of the ICs, to coordinate NIH’s research portfolio to ensure that it is balanced, not unnecessarily duplicative, and utilizes cross-cutting research. NIH decided not to apply the IOM recommendations across its ICs; however, it does acknowledge the need for enhanced stewardship of its clinical trials and developed its own recommendations for this purpose. Consistent with our finding that NIH’s OD only uses some data to oversee clinical trial activity across the ICs, some of NIH’s own recommendations recognize the need for additional data. However, NIH’s OD has not determined which data it needs or how it will use the data to strengthen its stewardship of NIH’s clinical trials and research portfolio. Until the OD identifies what data to review, and how the data will be used and when, NIH is missing an opportunity to maximize the public investment in clinical trials.

Recommendations for Executive Action

To enhance its stewardship of clinical trials across the ICs, we recommend that the Secretary of HHS direct the NIH OD to take the following two actions: 1. finalize data on clinical trial activity that the OD needs to collect from 2. establish and implement a process for using those data.

Agency Comments

We provided a draft of this report to HHS for comment. HHS provided written comments, which are reprinted in appendix I. HHS concurred with our recommendations and stated that the agency plans to complete its efforts to identify additional data that will be collected from the ICs on a regular basis. HHS indicated that these data will provide the OD with readily available information on the number, type, and progress of clinical trials funded across the ICs. Although HHS did not specify plans to establish and implement a process for using the data, it said its goal is to complete its current efforts during 2016 and to implement other strategies to enhance the quality and transparency of HHS-funded clinical trials.

In its comments, NIH also stated that the report over-emphasized NIH's decision to not apply the IOM recommendations across the ICs and under-emphasized the role and scientific expertise of the ICs in managing and overseeing clinical trials. We maintain that the report appropriately describes the steps taken by NIH to assess the applicability of IOM's recommendations and to enhance its stewardship of clinical trials. Further, the report notes that the ICs and the OD both play a critical, but different, role in ensuring the stewardship of clinical trials across the agency. In addition, HHS provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix II.

Appendix I: Comments from the Department of Health and Human Services

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Tom Conahan, Assistant Director; Krister Friday; Q. Akbar Husain; Morgan Jones; Rebecca Rust Williamson; and Emily Wilson made key contributions to this report.