Medical Product Oversight: FDA Needs More Strategic Planning to Guide Its Scientific Initiatives
Published: 2016-05-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: FDA has faced challenges regulating medical products, owing in part to rapid changes in science and technology. In 2010, FDA established a regulatory science initiative that identified eight priority areas for medical products where new research was needed to advance its mission. Legislation enacted in 2012 required FDA to establish a plan for measuring its progress on its regulatory science efforts.
GAO was asked to examine FDA's progress on its regulatory science efforts related to medical products. In this report, GAO (1) evaluates FDA's strategic planning efforts to address its regulatory science priorities, (2) describes FDA's funding targeted at regulatory science projects, and (3) describes the achievements of selected FDA regulatory science projects. GAO compared related FDA strategic planning documents to federal internal control standards and leading practices for strategic planning. GAO reviewed FDA data on obligations targeted at regulatory science projects for fiscal years 2010 through 2014 and reviewed the achievements FDA reported from a sample of 17 projects, chosen to ensure nine FDA centers and offices and priority areas are represented.
section_title: What GAO Found
paragraphs: The Food and Drug Administration (FDA) lacks measurable goals to assess its progress in advancing regulatory science—the science supporting its effort to assess the products it regulates. The agency issued strategic planning documents in 2011 and 2013 to guide its regulatory science efforts and identify priority areas for conducting work, but these documents do not specify the targets and time frames necessary for the agency to measure progress overall or within each of the eight priority areas related to medical products. According to leading practices for strategic planning, identifying and using consistent measurable goals in planning and progress documents is important to assessing effectiveness. While FDA cited examples of its achievements in regulatory science in a 2015 report, FDA cannot assess how those achievements constitute progress towards its goals. In addition, FDA lacks information about how funding targeted at regulatory science is distributed across the priority areas. Decisions to award these funds are made by individual FDA centers and offices, which generally did not collect information on the associated priority areas of funded projects. Rather, FDA retrospectively identified these areas for the purpose of GAO's review. The lack of consistent information limits FDA's ability to examine obligations across, or progress within, specific priority areas. Standards for internal control in the federal government state that complete and accurate data are needed to make operating decisions and allocate resources. Furthermore, multiple centers or offices fund projects toward a given priority area and leading practices for strategic planning encourage agencies to manage efforts that cut across the agency.
For the 17 regulatory science projects GAO reviewed, FDA identified achievements ranging from the dissemination of project findings to changes in both agency and external stakeholder practices. For example, FDA reported that all projects resulted in some type of change within FDA. About half of the projects resulted in the agency developing standards, methods, tools, or training that it could use internally, and about one-third of the projects affected guidance or regulations. FDA also reported that about half of the projects resulted in the development of new tools or standards for use by industry or other stakeholders, in areas such as setting new standards for defibrillators to account for radio interference.
section_title: What GAO Recommends
paragraphs: To improve strategic planning for regulatory science efforts, FDA should (1) develop and document measurable goals, including targets and time frames, and (2) systematically track funding across its regulatory science priority areas. The Department of Health and Human Services agreed with GAO's recommendations.

section_title: Letter
paragraphs: The Food and Drug Administration (FDA)—an agency within the  Department of Health and Human Services (HHS)—has faced challenges  carrying out its responsibilities to approve and oversee the safety and  efficacy of medical products sold in the United States. As a result, we  identified FDA’s oversight of medical products as an area of high risk, and  we noted that rapid changes in science and technology had contributed to  these challenges. However, these are not new concerns for the agency.  In 2007, FDA’s Science Board reported that the agency’s ability to fulfill  its mission was at risk because of a weak scientific base. Further, the  Institute of Medicine, following a series of reports and workshops  identifying gaps between the science of medical product technologies and  FDA’s capacity to regulate those technologies, held a workshop in 2011  on FDA’s critical role in the area of regulatory science—which is defined  by FDA as the science of developing new tools, standards, and  approaches to assess the safety, efficacy, quality, and performance of all  FDA-regulated products. At the workshop, it was noted that regulatory  science had been neglected at FDA and that the agency could take steps  internally and with external partners to better keep up with new  technologies and develop the expertise necessary to regulate those  technologies.
To accelerate the process from scientific breakthrough to the availability  of new medical products, FDA established the Advancing Regulatory  Science Initiative in 2010. Since then, the agency has issued multiple  documents to guide its strategic planning related to regulatory science. In  2011 and 2014, FDA named regulatory science as one of five  crosscutting strategic priorities for the agency. The agency also  developed a strategic plan in 2011 specifically for regulatory science that  identified priority areas in which FDA would apply funding and other  resources. In 2013, FDA issued another strategic plan for regulatory  science, as required by the Food and Drug Administration Safety and  Innovation Act of 2012 (FDASIA). This document identified regulatory  science priorities with respect to FDA’s decision making and included  categories by which progress on the priorities would be measured. Since  then, FDA issued two reports on its progress. Specifically, in May 2015, it  issued a report required by FDASIA on fiscal year 2013 and 2014  activities and in July 2015, it publicly issued a report to its Science  Board.
Given the concerns associated with FDA’s activities related to regulatory  science, you requested that we provide information on whether FDA has  been making progress in this area. This report  1.  evaluates FDA’s strategic planning efforts to address its regulatory  science priorities for medical products,  2.  describes FDA’s funding targeted at regulatory science projects  related to medical products, and  3.  describes the achievements of selected FDA regulatory science  projects related to medical products.
To evaluate FDA’s strategic planning efforts to address its regulatory  science priorities for medical products, we analyzed FDA documents,  including its 2011 and 2013 strategic planning documents, its reports on  strategic priorities, and a 2015 progress report. We analyzed whether the  methods FDA uses to collect data and report on progress are consistent  with relevant criteria identified in Standards for Internal Control in the  Federal Government. We also reviewed relevant criteria from GAO’s  body of work on effectively managing performance under the Government  Performance and Results Act of 1993 (GPRA) as enhanced by the GPRA  Modernization Act of 2010. We also interviewed FDA officials about how  they collected information on and assessed progress within the priority  areas they identified.
To describe FDA’s funding targeted at regulatory science projects related  to medical products, we analyzed available FDA data on funding for  grants or other awards (referred to as “projects”) targeting regulatory  science from fiscal years 2010 through 2014 and interviewed FDA  officials about these projects. FDA officials told us that the agency does  not keep a compilation of projects that it identifies as being targeted at  regulatory science. As a result, we relied on each center and office in our  review to individually identify such projects for our review. We obtained  data from FDA on funding obligations for these projects from FDA,  examined these data for inconsistencies and sought clarification as  necessary, and determined that the data were sufficiently reliable for our  purposes. FDA officials indicated that the agency also funds broader  efforts that benefit regulatory science along with other goals; however,  these efforts were not included in our funding totals because the agency  was not able to identify funding at the project level specifically for  regulatory science.
To describe the achievements of selected FDA regulatory science  projects related to medical products, we selected a sample of 17 projects  to review, which were chosen to ensure representation of all FDA centers  and offices with targeted regulatory science funding and all FDA  regulatory science priority areas. First, from among those 1,279 projects  that were obligated funds targeted at regulatory science from fiscal year  2010 through fiscal year 2014, we selected the project with the greatest  funding during this time period for each center and office, for a total of 8  projects. We excluded projects that received obligations in fiscal year  2014, as they were ongoing at the time of our selection and therefore may  not have had time to demonstrate an achievement. These 8 projects  represented less than 1 percent of the projects for which we had funding  data, yet accounted for 5 percent of the total funding obligations for  targeted regulatory science projects for fiscal years 2010 through 2013.  Second, we randomly selected a sample of 9 projects from among those  completed projects with achievements reported in FDA’s 2014 progress  report to the Science Board, ensuring representation by centers and  offices and for priority areas. For each of these 17 projects, we reviewed  information provided by FDA describing the results of regulatory science  achievements and interviewed FDA officials. We classified these  achievements into three categories: shorter-term achievements, such as  sharing information via dissemination of project findings; medium-term  achievements, such as internal changes at FDA; and longer-term  achievements such as changes by industry and other groups outside of  FDA.
We conducted this performance audit from February 2015 through May  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: According to FDA, the goal of its regulatory science initiative is to develop  and apply the best available scientific data, knowledge, methods, and  tools to reduce uncertainty and make regulatory decisions more efficient  and consistent. In doing so, the agency seeks to ensure public access to  products that are manufactured or processed in a high quality manner  and monitored to ensure safety and quality during real-world use. FDA’s  2011 strategic plan identified eight priority areas for regulatory science, of  which seven related to medical products where the agency determined  that new or enhanced engagement is essential to the continued success  of the agency’s public health and regulatory mission. The agency then  added an additional priority area related to global product safety in 2013.  (See table 1.)
FDA conducts work to advance regulatory science through intramural  research and extramural collaborations, such as collaboration with other  government agencies, academia, industry, patient organizations,  professional associations, and other stakeholders. Targeted funding for  regulatory science at FDA comes from a number of centers and offices,  including the Center for Biologics Evaluation and Research (CBER), the  Center for Drug Evaluation and Research (CDER), and the Center for  Devices and Radiologic Health (CDRH), all within FDA’s Office of Medical  Products and Tobacco. These centers are responsible for approving  medical products (biologics, drugs, and devices, respectively) and  monitoring their ongoing safety once approved. In addition, other offices  support the regulatory mission of FDA, such as the Office of International  Programs (OIP) in the Office of Global Regulatory Operations and Policy,  and those under the Office of the Chief Scientist—whose mission is to  provide strategic leadership, coordination and expertise—including the  National Center for Toxicological Research (NCTR), the Office of  Counterterrorism and Emerging Threats (OCET), the Office of Minority  Health (OMH), the Office of Regulatory Science and Innovation (ORSI),  and the Office of Women’s Health (OWH). (See fig. 1.) Additional broad  regulatory science efforts may occur within these centers and offices as  well as others at FDA.
section_title: FDA Lacks the Measurable Goals and Funding Information Necessary to Conduct Strategic Planning for Regulatory Science Priorities
paragraphs: FDA does not have measurable goals, such as targets and time frames,  to assess its progress in advancing regulatory science. Further, FDA  does not consistently track information on center and office funding  decisions for each FDA priority area, leaving it without important  information needed to conduct strategic planning on the agency’s  regulatory science priorities.
subsections: section_title: FDA Lacks Measurable Goals to Assess Progress Meeting Its Regulatory Science Priorities
paragraphs: FDA’s 2011 and 2013 strategic planning documents do not identify  measurable goals, such as targets and time frames, for assessing  progress in regulatory science. For example, the 2013 strategic  planning document states that FDA will measure progress by  enumerating and describing product-specific advisory committees and  staff training opportunities related to the priority areas. However, the  document neither sets any targets or time frames, nor establishes further  outcome-based measures for what FDA hopes to achieve for either a  given priority area or regulatory science in general. Likewise, FDA’s 2011  strategic planning document presents strategies for addressing the  priority areas, but the document does not include measurable goals within  these strategies. For example, it indicates that addressing the toxicology  priority area would involve developing human and animal models to  predict adverse responses, but it does not provide the number of models  FDA intends to develop or the adverse responses for which the models  are intended. FDA’s lack of specific measurable goals is reflected in the  progress report that it completed for FDASIA. In that report, FDA  includes examples of achievements but it does not specify overall results,  such as the number of tests and technologies developed for the efficacy  and safety of medical products, which the 2013 planning document  indicated would help the manufacturing and quality priority area. In  addition, FDA generally did not link the categories that it used to measure  the adoption of regulatory science to a specific priority area. Specifically,  in the progress report, only four of the eleven categories are linked to the  FDA priority areas that they address. For example, while FDA reported  one or more related FDA priority area for each guidance document  included in the progress report, it did not provide the same information for  the reported training examples or the projects included in the Drug  Development Tool Qualification programs.
According to our work on leading practices for strategic planning, an  agency’s strategic goals should explain what results are expected from  the agency and when to expect those results. It is also critical that the  strategic planning documents describing these goals include specific  actions and implementation schedules for how the agency is to achieve  these goals. Without measurable goals, clear targets, and  implementation time frames, FDA cannot provide a complete assessment  of progress made in the regulatory science areas it has designated as  priorities.
According to FDA officials, it is difficult to measure the progress made in  the priority areas because the priority areas are very broad and the  underlying science is continually changing. Furthermore, the adoption of  discoveries from regulatory science research can take years. Because  the priority areas are managed across multiple organizational units, they  are not being overseen by a specific center or office. However, we have  previously recognized that while measuring the performance of science- related projects can be difficult, science-related agencies like FDA should  still have clearly established goals.
One of the centers that targets funding at regulatory science has a plan to  track the progress of its regulatory science activities, but this plan does  not include measurable goals that could be used to assess progress.  Specifically, CDRH officials told us the center has drafted a logic model to  identify and track the short- and long-term outcomes of funds it spends on  regulatory science research. The center expects the model to be finalized  in 2017. However, the current draft does not include measurable goals to  assess progress. In addition, officials from most of the other centers and  offices that obligated funds targeted at regulatory science activities in  fiscal years 2010 through 2014 told us they are not developing such a  model. One office also stated that the CDRH model may not be  applicable to their work and that they may not have the resources needed  to customize and implement such a model. In another effort, FDA officials  indicated that the agency recently initiated internal discussions primarily  to share information, and if feasible and desired, harmonize best  practices within the agency. FDA officials also said that this group would  discuss evaluation strategies and processes and perhaps eventually  arrive at a common FDA approach to thinking about research outcomes  and impact. This group first met in October 2015 to discuss current  efforts, and FDA officials reported in April 2016 that it is still in the early  stages of sharing information.
section_title: FDA Lacks Information Necessary to Track Funding and Strategically Plan for Its Regulatory Science Priorities across the Agency
paragraphs: FDA does not have the information necessary to track funding and  conduct strategic planning agency-wide for its regulatory science priorities  because most of the centers and offices did not collect information on the  FDA priority areas that were addressed by the projects they funded. As a  result, information on funding for each priority area is generally not readily  available and therefore the majority of these centers and offices had to  retrospectively assign FDA priority areas to each project at the time of our  review.
Six centers and offices—CDRH, NCTR, OCET, OIP, OMH, and  OWH—fund regulatory science projects that address multiple FDA  priority areas, but generally did not collect information about how  funds are distributed across those priority areas. All six had to  retrospectively identify associated FDA priority areas for our review.  For example, while CDRH and NCTR collect some information about  research priorities for the projects they fund, the information they  collect does not fully align with the FDA priority areas. Specifically,  CDRH collects information on its own set of priorities and NCTR  collects information on FDA goals. While this information is partially  aligned with the FDA priority areas, there is not a one-to-one  correspondence, thus information on FDA priority areas is not readily  available. According to NCTR officials, they began capturing  information on the FDA priority areas in 2016. Further, OMH asks  researchers to identify the unmet regulatory science need that their  proposed research addresses, but the researchers are not asked to  identify specific FDA priority areas. In addition, OCET and OIP  officials told us that all of their funded projects are related to one FDA  priority area, and therefore they do not collect information about other  priority areas. However, there is no guarantee they will not fund  projects related to other priority areas in the future. Further, while OIP  told us they focus on just the FDA priority area global product safety,  the data they provided for our review also identified an additional  priority area, information sciences, as an FDA priority area being  addressed by some of their projects.
Three offices, CBER, CDER, and ORSI, collect some information on  the FDA priority areas addressed by their targeted projects. CBER  and CDER have not collected this information at the time of funding.  Rather, CBER asks researchers to provide it as part of required  annual reports and CDER officials indicated that CDER has made a  similar request of researchers since 2012. Officials from both CBER  and CDER said that in the near future their centers plan to track the  FDA priority areas at the time projects are proposed for funding. ORSI  asks researchers to identify FDA priority areas for its broad agency  announcements, which are competitive funding announcements for  extramural research programs and accounted for 21 percent of  ORSI’s funding for the projects that we reviewed. ORSI does not ask  this of researchers applying for its other regulatory science funding  programs, although the announcements for the intramural grants state  that the proposals should align with one or more of the eight original  priority areas. For these other funding programs, ORSI confirms that a  project proposal is related to regulatory science, but does not  document any specific FDA priority area for that project.
While each center or office may be funding projects that are generally  consistent with the priority areas that FDA established, without consistent  information from each center or office detailing those connections, the  agency is not able to examine obligations across specific priority areas.  Standards for internal control in the federal government state that  complete and accurate data are needed to make operating decisions and  to allocate resources. In addition, to ensure program goals are met, our  work encourages agencies to manage efforts that cut across the  agency. Without complete information on the allocation of funding  across priority areas by centers and offices, FDA cannot ensure that  funding is being distributed in line with its strategic plan.
section_title: FDA Annual Obligations for Regulatory Science Projects Generally Increased from 2010 through 2014 and Varied across Centers and Offices and Priority Areas
paragraphs: In response to our request, FDA identified projects targeted at regulatory  science funded from fiscal years 2010 through 2014 totaling more than  $507 million. Annual obligations for these projects generally increased  during that time. This funding varied across centers and offices, ranging  from approximately $450,000 to about $200 million. In addition, while  FDA does not systematically track regulatory science obligations by  priority area, the agency’s retrospective review showed wide funding  variation across priority areas, ranging from about $3 million to  approximately $203 million.
subsections: section_title: FDA Annual Obligations for Regulatory Science Projects Generally Increased over Time
paragraphs: FDA obligations targeted at regulatory science projects increased from  about $73 million in fiscal year 2010 to a peak of about $123 million in  fiscal year 2013, then a decline to about $110 million in fiscal year 2014.  These funds represent those obligations targeted at specific regulatory  science projects and do not include FDA obligations for other activities  benefitting regulatory science for which the agency was not able to  quantify spending at a project level. (See fig. 2.) Over the 5 years we  examined, FDA obligated nearly $507 million for 1,279 regulatory science  projects, an average of approximately $400,000 per project. Total  obligations for individual projects ranged from $430 to $9.1 million during  this time. While FDA obligated funds for some projects for a single year,  FDA typically obligated funds for projects in multiple years.
FDA obligated 80 percent of these regulatory science funds to intramural  projects—those led by FDA researchers. For example, the Office of the  Chief Scientist’s grant program includes five intramural grant programs;  for these programs, FDA scientists first submit concept papers that are  ranked and then submit full proposals that are peer-reviewed. The  remaining 20 percent of projects were either extramural or a combination  of intramural and extramural. For example:
Broad Agency Announcements are extramural, competitive funding  announcements supporting regulatory science research programs.
Centers of Excellence in Regulatory Science and Innovation are  extramural partnerships between FDA and universities to promote  cross-disciplinary regulatory science training and research.
The Critical Path Initiative started in 2004 to improve medical product  development, evaluation, and manufacturing and is used to support  intramural research and external collaborations. Most Critical Path  Initiative projects were intramural, except for ten projects that were  both intramural and extramural.
Of the total funding targeted at regulatory science projects, 48 percent  was obligated for projects awarded through a non-competitive award  process, 39 percent through a competitive award process, and 13 percent  through a combination of competitive and non-competitive processes.
Officials reported that FDA has traditionally funded regulatory science  projects with FDA general appropriations, but projects funded within  CDER have also been supplemented by funds collected under user fee  acts—specifically the Prescription Drug User Fee Act (PDUFA) and  Generic Drug User Fee Amendments of 2012 (GDUFA)—that authorize  the collection of funds from industry (including the pharmaceutical and  biotechnology industries). PDUFA funds accounted for 2 percent of  CDER’s total regulatory science obligations for fiscal years 2010 through  2014, ranging from 1 to 4 percent annually, with approximately $300,000  of PDUFA funds obligated each year for regulatory science. Starting in  fiscal year 2013, CDER committed to using a portion of its GDUFA funds  for regulatory science. GDUFA funds accounted for 65 percent of CDER’s  regulatory science obligations in fiscal years 2013 and 2014 combined,  about $17 million (70 percent) and $19 million (62 percent), respectively.  (See fig. 3.) In 2013, the addition of funds from GDUFA more than  doubled CDER’s annual obligations for regulatory science projects for  fiscal years 2010-2013. GDUFA and PDUFA are the only two user fee  programs that support targeted regulatory science obligations.
In addition, FDA indicated that it funded other efforts benefitting  regulatory science, but was unable to quantify spending at the project  level. For example, CDER officials told us that funds targeted at  regulatory science identified for our review represent only a portion of  CDER’s investment in regulatory science. They added that lab programs,  like those in the Office of Pharmaceutical Quality and the Division of  Applied Regulatory Science within the Office of Translational Sciences,  fund the bulk of their projects through the normal CDER budgeting  process and therefore are not included among the targeted funds. One  such CDER project focused on developing an analytical method that  sponsors could use to support that proposed generic and brand name  forms of estrogen are chemically equivalent. The work resulted in new  recommendations for some estrogen analyses, which were incorporated  into a guidance document for the development of generic estrogen  products.
section_title: FDA Obligations for Regulatory Science Varied Widely across Centers and Offices and Priority Areas
paragraphs: Of the nine centers and offices that obligated funds targeted at regulatory  science from fiscal year 2010 through fiscal year 2014, total obligations  ranged from approximately $450,000 by OMH to about $200 million by  NCTR. The center and offices within the Office of the Chief Scientist— NCTR, OCET, OMH, ORSI, and OWH—accounted for 65 percent of  FDA’s total obligations for projects targeted at regulatory science, with  NCTR accounting for 60 percent of the total Office of the Chief Scientist’s  obligations. Centers with regulatory responsibilities for medical products  (CDER, CBER, and CDRH) accounted for 34 percent of the obligations,  and OIP accounted for the remaining 1 percent. The average obligation  per project by centers and offices ranged from just over $110,000 for  OWH to about $1.1 million for OIP. However, the centers and offices that  had the highest total obligations were not necessarily the ones that had  the highest average obligation at the project level. For instance, OIP had  the highest average obligations per project, more than $1 million, yet it  had the second lowest total obligations, at about $5.4 million. (See fig. 4.)
Similarly, total obligations associated with each FDA regulatory science  priority area varied widely, ranging from about $3 million for projects that  focused on global product safety to approximately $203 million for  projects that focused on the toxicology priority area. (See fig. 5.)  Projects that focused on the clinical evaluations and personalized  medicine and medical countermeasures priority areas were among those  with the greatest obligations. Average obligations per project ranged from  about $250,000 for the projects that focused on the manufacturing and  quality priority area to approximately $1.1 million for the projects that  focused on the global product safety priority area. However, the FDA  priority areas that had the lowest total obligations were not necessarily  the ones that had the lowest average obligation at the project level. For  example, projects that included a focus on global product safety had the  lowest total obligations but they had the highest average obligation per  project. Similarly, projects that included a focus on social and behavioral  science had the second lowest total obligations but they had the second  highest average obligation per project.
FDA’s centers and offices make decisions about funding that determine  which projects to fund, and they generally obligated project funds across  several regulatory science priority areas. Specifically, two-thirds of the  centers and offices provided obligations to five or more priority areas. For  example, CBER provided obligations to projects that collectively focused  on every priority area except global product safety. Nevertheless, there  were four centers and offices that directed at least half of their regulatory  science obligations to a single priority area.
100 percent of OCET obligations were for medical countermeasures.
62 percent of OIP obligations were for global product safety.
56 percent of NCTR obligations were for toxicology.
50 percent of OWH obligations were for clinical evaluations and  personalized medicine.
While funding for each of the FDA priority areas generally came from a  number of different centers and offices, for all but two priority areas, there  was one center or office that accounted for the majority of obligations.
Global product safety: 100 percent of the obligations from OIP.
Toxicology: 82 percent of the obligations from NCTR.
Social and behavioral science: 76 percent of the obligations from  CDER.
Medical countermeasures: 64 percent of the obligations from OCET.
Manufacturing and quality: 56 percent of the obligations from CBER.
Emerging technologies: 55 percent of the obligations from CBER.
Information sciences: 49 percent of the obligations from NCTR.
Clinical evaluations and personalized medicine: 43 percent of the  obligations from NCTR.
section_title: FDA Reported That the Selected Regulatory Science Projects Led to Changes in Agency and External Stakeholder Practices
paragraphs: FDA reported that the 17 regulatory science projects that we selected for  review helped to advance regulatory science. (See appendix I for  additional information about each of these 17 projects.) For each of these  17 projects, FDA identified achievements that we classified as  dissemination of project findings, internal changes at FDA, and changes  by industry and groups outside of FDA. (See fig. 6.)
Dissemination of project findings. For 16 of the 17 selected projects,  FDA reported that it disseminated the project results in scientific  publications, conferences, FDA workshops, or some combination of all  three. Such dissemination provides FDA an opportunity to share new  information and understanding internally and with the larger scientific  community and may contribute to future regulatory science activities.
Internal impacts. FDA described internal impacts resulting from each of  the 17 selected projects, many of which related to advancing FDA’s  scientific understanding in a particular area.
For 12 selected projects, FDA reported that the projects resulted in  new information about the topic that FDA was using or considering  using for future work. For example, one goal of OCET’s project  looking at the feasibility of using electronic health records in public  health emergencies was advancing FDA’s understanding of the  possibilities and limitations of using electronic health data. The  findings from this project indicated that structured data in electronic  health records could help the agency assess the risk of adverse  events, particularly those that are severe. However, the results also  indicated that this near real-time data still had some built-in delays  and that the data search process cannot be fully automated. This  provided FDA with information it can use as it considers using  electronic health records in emergency situations.
For 9 selected projects, FDA reported that the results led the agency  to plan or conduct additional studies or activities that represent the  logical next step in the particular area being studied. For example,  initial results from an ORSI study of the use of social media to provide  early signals of drug safety concerns showed a relationship between  data obtained from social media and from FDA’s adverse event  reporting system. Using that information, FDA then conducted a  retrospective study for 10 safety concerns to see if there was  evidence of those adverse events in social media before FDA became  aware of them. FDA has since reported that the analysis identified  specific limitations of the tool FDA uses in monitoring safety concerns.
Other internal impacts were related to changes in agency practices.
For 8 selected projects, FDA reported that the results led to the  development of standards, methods, tools, or training for FDA internal  use. For example, an OIP project was designed to create a tool to  help secure global supply chains against the infiltration of counterfeit  or substandard products. The project resulted in the production of a  “roadmap” that FDA could use to develop such a system.
For 5 selected projects, FDA reported that the results led to either a  change in guidance or regulation or the decision to not make a  previously proposed change. For example, CDER funded a study of  surrogate endpoints that could speed the development of new  therapies for breast cancer. FDA used results from this study to inform  its development of guidance to industry that described study designs  in which a surrogate endpoint may be accepted by FDA as reasonably  likely to predict the clinical benefit of a drug. In response to a  statutory requirement, CDER also examined whether quantitative  information could be added to drug advertising to maximize consumer  and health care professional understanding of the benefits and risks  of the drug. Based on the results of the study, the Secretary of HHS  concluded that quantitative information cannot be readily applied to  many drugs and therefore it is not appropriate to issue regulations that  would require such information to be added to promotional labeling or  advertising.
For 5 selected projects, FDA reported that the results led the agency  to change aspects of its review process. For example, a CBER study  designed to improve influenza vaccine efficacy provided information to  FDA that has helped in the review of preclinical animal studies that  are included in some drug applications. The study also resulted in  FDA offering training for its lab members in an approach to vaccine  testing that FDA says is a common part of a product review. In  addition, an NCTR project developed a knowledge base about liver  toxicity and has used that to advise CDER reviewers about drug- induced liver injury risk for products they were reviewing.
External impacts. FDA described external impacts for 8 of the 17  selected projects. These projects resulted in the development of tools or  proposed standards for use by industry or the implementation or use of  new tools or standards by industry or outside organizations. According to  FDA, it can take several years for funded research to result in these types  of tools.
For 7 selected projects, FDA reported that the results led to the  development of tools or proposed standards for use by industry. For  example, a CDRH study of radio interference with automated external  defibrillators led FDA to recommend to an international commission  that standards for these types of defibrillators be modified to account  for radio interference. Similarly, FDA officials reported that results  from a study funded by OWH looking at degradation of absorbable  polymers used in some cardiac stents have provided guidance for  industry on the design, manufacturing, and regulation of these  absorbable stents.
For 6 selected projects, FDA reported that industry or outside  organizations have made changes based on the results of these FDA- funded projects. For example, FDA told us NCTR’s studies of  Bisphenol A, a chemical found in certain plastics, helped FDA and the  European Food Safety Authority, resolve public safety concerns.  Similarly, CDRH’s study of total disc replacement devices for the  spine led to the development of a test guide published through the  American Society of Testing and Materials International that is used  by multiple manufacturers. FDA told us that the results from these  tests were then part of the manufacturers’ submissions for approval of  these devices.
section_title: Conclusions
paragraphs: For several years, FDA has been aware of the need to improve its  scientific base and has established multiple regulatory science initiatives,  as well as prioritized areas to address that need. FDA projects targeted at  advancing regulatory science have led to internal and external impacts in  understanding new science associated with medical products. However,  the agency has not identified measurable goals in their strategic plans or  reports on strategic priorities, such as specific targets and time frames,  for regulatory science. Such goals are a best practice for strategic  planning and could enable FDA to assess and report its progress in  addressing its identified priority areas and strategically plan and allocate  resources for its broader regulatory science initiative. The agency faces  another obstacle to its strategic plan without consistent information about  centers and offices’ distribution of targeted regulatory science funding  among those identified priority areas. The individual centers and offices  that decide which projects to fund are either tracking the priority areas in  different ways or not all. Our prior work encourages the proactive  collection of consistent information and standards for internal control  recommend federal agencies have complete and accurate data for  making funding decisions. Systematic tracking by each center or office is  needed for the agency to examine obligations across, or progress within,  specific priority areas and would help the agency to strategically plan for  its regulatory science initiative as a whole.
section_title: Recommendations for Executive Action
paragraphs: In order to improve FDA’s strategic planning for regulatory science efforts,  we recommend the Secretary of Health and Human Services direct the  Commissioner of FDA to take the following two actions:  1.  Develop and document measurable goals, such as targets and time  frames, for its regulatory science efforts so it can consistently assess  and report on the agency’s progress in regulatory science efforts.  2.  Systematically track funding of regulatory science projects across  each of its priority areas.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to HHS. HHS concurred with our  recommendations and provided written comments, which are reprinted in  appendix II. In its written comments, HHS agreed with the importance of  strategic planning for regulatory science. HHS concurred with our  recommendation that FDA should develop and document measurable  goals; HHS suggested that agency documents with a targeted focus,  such as user fee commitment letters and specific planning documents,  are a more appropriate place for such goals than an agency-level  strategic plan. In our recommendation to HHS, we do not specify where  such goals should be documented. We recognize, as HHS noted in its  comments, that advancing regulatory science is an uncertain and non- linear path that can make it challenging to set targets for specific  accomplishments. Nevertheless, FDA should develop measureable goals  that are related to the impacts that are discussed in HHS’s comments,  including the effectiveness and efficiency of FDA’s regulatory review, new  pathways for medical product development, enhancements in the  agency’s ability to provide useful guidance to sponsors, and new  technologies to monitor manufacturing and real world use of approved  medical products. As all but one of the agency-wide priority areas are  being addressed by projects funded by multiple centers and offices, it is  important that FDA develop and document measurable goals that  encompass the efforts of multiple centers and offices. HHS also  concurred with our recommendation to systematically track funding  across FDA’s regulatory science priority areas, and the department  identified recent and planned activities of specific centers to improve such  tracking  We support these efforts and reiterate the importance of FDA  systematically track funds agency-wide for each of the priority areas it  developed. Systematic tracking of both progress on measurable goals  and funding is essential for FDA to strategically plan its regulatory science  initiative across the agency. In addition to these general comments, HHS  provided technical comments, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of Health  and Human Services and other interested parties. In addition, the report  is available at no charge on the GAO website at http://www.gao.gov. If  you or your staff have any questions about this report, please contact me  at (202) 512-7114 or crossem@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix III.
section_title: Appendix I: Description of Projects Included in Our Review of Achievements Resulting from FDA’s Regulatory Science Projects
paragraphs: We examined the achievements related to regulatory science for 17  projects funded by the Food and Drug Administration (FDA). Below is a  brief description of each project, including examples of achievements  resulting from that project, according to information provided by FDA.
Title: Created new approaches to identify and understand critical  product quality attributes of complex products, such as stem cell- derived products (both animal and human) and complex systems of  medical devices.
Center or office funding project: Center for Biologics Evaluation  and Research (CBER)
Priority area(s) covered: Emerging technologies and medical  countermeasures  Funding obligations and time period of funding: $4.513 million  (fiscal years 2010 through 2014)
Description of project: Certain stem cells are being used in clinical  trials for multiple medical conditions. However, multiple factors,  including donor variation and culture conditions may affect the clinical  performance of the stem cells either in terms of safety, efficacy, or  both. This project was designed to identify product attributes that  correlate with specific outcomes and to increase understanding of  factors that might affect stem cell-derived safety and efficacy. This  project also included the development of methods to quantify various  attributes of these cells.
Select achievements reported by FDA: The project’s findings  affected FDA reviewers’ understanding of stem-cell products. Further,  the findings have helped producers of stem cell products with their  studies, specifically noting that some sponsors have adopted  quantitative methods for assessing the stem cells. The findings were  important in the scientific reasoning in two guidance documents.  Additional projects have been planned to further advance  understanding in this area.
Title: Developing new approaches for measuring the quality of next- generation smallpox vaccines.
Center or office funding project: CBER  Priority area(s) covered: Medical countermeasures  Funding obligations and time period of funding: $0.537 million  (fiscal years 2010 through 2014)
Description of project: The goal of the project was to evaluate  factors that affect the safety and efficacy of smallpox vaccines and to  develop new methods to evaluate smallpox vaccine quality— specifically, potency. The studies were aimed at helping the  development of new smallpox vaccines.
Select achievements reported by FDA: Results from the project  informed FDA’s understanding of various characteristics of the  vaccine, as well as factors that influenced immune responses to the  vaccine. FDA also reported that one of the two alternative approaches  for evaluating the potency of smallpox vaccines used in clinical trials  was considerably faster than traditional methods and could be  adapted for future use. In addition, further studies are ongoing in other  tests related to smallpox vaccines.
Title: Correlates of protective immunity against influenza.
Center or office funding project: CBER  Priority area(s) covered: Manufacturing and quality, emerging  technologies, and medical countermeasures  Funding obligations and time period of funding: $1.209 million  (fiscal years 2010 and 2011)
Description of project: This project was designed to support the  development of seasonal and pandemic influenza vaccines by  identifying mechanisms that contribute to immunity and developing  measures of those responses.
Select achievements reported by FDA: This project resulted in the  development of a new method that was useful for testing influenza  vaccines, the discovery of certain characteristics that are important for  producing protective immunity, and the determination of the amount of  a vaccine ingredient that is needed to produce immunity. FDA  reported that the project provided a foundation for training lab  members in development and validation of an approach to vaccine  testing, while also providing the basis for an international study led by  CBER to assess reproducibility in one of the methods developed in  this project.
Title: Evaluating a surrogate endpoint that could speed development  of new therapies for breast cancer.
Center or office funding project: Center for Drug Evaluation and  Research (CDER)
Priority area(s) covered: Clinical evaluations and personalized  medicine  Funding obligations and time period of funding: FDA could not  determine funding as the research was conducted as part of FDA  employees’ regular responsibilities.
Description of project: FDA conducted research to assess and  evaluate the validity and potential applications of a surrogate  endpoint—a measure that can predict, but is not itself a measure of,  benefit—in trials of treatments for women with breast cancer. By  collaborating with an international working group, FDA researchers  were able to use data from more than 12,000 patients.
Select achievements reported by FDA: FDA reported that  researchers found a potential relationship between the surrogate  endpoint and survival. This then provided important information to  drug developers for the design of future drug trials. It also informed  the development of guidance to industry.
Title: Completed three studies and a literature review assessing  whether quantitative information could be successfully added to  television and print advertisements to maximize audience  understanding of benefit information in the piece, including the type of  benefit information, different combinations of statistical format, and  different graphic representations.
Center or office funding project: CDER  Priority area(s) covered: Social and behavioral science  Funding obligations and time period of funding: $0.270 million  (fiscal year 2010)
Description of project: This project was composed of four studies  designed to investigate whether quantitative information in direct-to- consumer advertisements is helpful for consumers. FDA was asked  by Congress to investigate this topic to determine whether such  information about the benefits and risks of prescriptions drugs in a  standardized format would improve health care decision-making by  clinicians, patients, and consumers.
Select achievements reported by FDA: FDA reported that, based  on these studies and other efforts, the inclusion of certain types of  quantitative information can be helpful in some limited circumstances,  but a standardized format cannot be readily applied to many drugs. It  was therefore not appropriate to issue new regulations that would  require such information on promotional labeling or print advertising.  The findings have been used internally by an FDA working group that  explores direct-to-consumer advertising. FDA reported that the  findings have also been used by the external research community.
Title: Experimental study of patient information prototypes.
Center or office funding project: CDER  Priority area(s) covered: Social and behavioral science  Funding obligations and time period of funding: $1.613 million  (fiscal year 2010)
Description of project: This project examined different methods of  presenting prescription drug information to patients who obtained a  prescription. The study was designed to compare the format in which  information is presented, the inclusion of additional context or not, and  the order of information about warnings versus information about the  efficacy of the drug.
Select achievements reported by FDA: The study provided  information about patient preferences for and increased  comprehension of single page prototypes over the currently available  format. These findings have informed FDA’s development of the  Patient Medication Information Initiative, which will consider a new  regulation to require all prescription drugs to have a single document  standardized in content and format that provides prescription  information to patients in an accurate, easily understood, and  balanced form.
Title: Development and validation of a standard test method to  assess for impingement of artificial total disc replacement devices in  order to provide scientific basis for regulatory guidance and better  predict which devices will be clinically successful.
Center or office funding project: Center for Devices and  Radiological Health (CDRH)
Priority area(s) covered: Clinical evaluations and personalized  medicine and emerging technologies  Funding obligations and time period of funding: $0.753 million  (fiscal years 2010 through 2012)
Description of project: The primary goal of the project was to  develop a new scientific tool to characterize impingement— unintended contact between surfaces of the device—of total disc  replacement devices. Impingement can be linked to premature  mechanical device failure and was not accounted for in preclinical test  methods. As a result, bench testing was not accurately mimicking  wear and damage that was being observed clinically.
Select achievements reported by FDA: The project resulted in the  development of an impingement test guide published through a  professional association. As a result, multiple device manufacturers  used the guide to perform impingement testing and FDA incorporated  those results into their decision making on premarket approval  applications and investigational device exemption submissions.
Title: Developed standards to reduce the risk of misconnection  between different types of small-bore connectors used for  intravenous, feeding, neural, blood pressure cuff, and breathing  system tubes to prevent serious adverse events.
Center or office funding project: CDRH  Priority area(s) covered: Manufacturing and quality  Funding obligations and time period of funding: FDA could not  determine funding as the research was conducted as part of FDA  employees’ regular responsibilities.
Description of project: Because devices using small bore  connectors have been accidentally connected with devices that have  different functions and have led to serious adverse events for patients,  including deaths, FDA participated in international efforts to  standardize connector designs for specific medical applications such  that they cannot be interconnected with a device for another medical  application.
Select achievements reported by FDA: FDA has issued guidance  on premarket recommendations for devices that use small-bore  connectors intended for use in the gastrointestinal tract. International  standards are also being finalized based on this work. Device  manufacturers are modifying their devices accordingly. In addition,  FDA has developed a website for highlighting relevant information for  stakeholders.
Title: Created a general testing protocol and test methods for  electromagnetic compatibility of automated external defibrillators.
Center or office funding project: CDRH  Priority area(s) covered: Emerging technologies  Funding obligations and time period of funding: FDA could not  determine funding as the research was conducted as part of FDA  employees’ regular responsibilities.
Description of project: A growing number of adverse events and  voluntary recalls by manufacturers of automated external defibrillators  led FDA to study the effect of electromagnetic interference that had  been related to potentially life-threatening failures of this device.
Select achievements reported by FDA: As a result of this project,  FDA developed test methods to evaluate the susceptibility of  automated external defibrillators to radiofrequency interference. FDA  recommended an international commission’s standards be modified to  account for interference testing at certain frequencies.
Title: Physiologically based pharmacokinetic models for Bisphenol A.
Center or office funding project: National Center for Toxicological  Research (NCTR)
Priority area(s) covered: Toxicology  Funding obligations and time period of funding: $1.049 million  (fiscal years 2010 through 2014)
Description of project: Due to concerns about the safety of  Bisphenol A, which is used in many consumer plastic products, NCTR  developed computational modeling to simulate infant exposure to  Bispehnol A to provide FDA with information necessary to complete a  safety assessment.
Select achievements reported by FDA: The results of the project  allowed FDA to predict how much chemical remained after being  metabolized and would get into the circulatory system of adults and  infants. These models were incorporated into FDA’s updated  assessment on Bisphenol A and allowed FDA’s Center for Food  Safety and Nutrition, as well as other regulatory bodies, to determine  that current uses of Bisphenol A are safe for infants and adults and,  further, led the agency to conclude that the traditional safety  assessment methods used were overly conservative.
Title: Development of liver toxicity knowledge base to empower the  FDA review process.
Center or office funding project: NCTR  Priority area(s) covered: Toxicology and information sciences  Funding obligations and time period of funding: $3.426 million  (fiscal years 2010 through 2013)
Description of project: Drug-induced liver injury is a serious safety  concern that is a frequent cause of denied approvals and “black box”  warnings on drugs. As a result, FDA was interested in developing a  database to improve its understanding and prediction of such liver  injury.
Select achievements reported by FDA: The project has produced a  centralized resource of data and predictive models that are useful for  both research and regulation. NCTR has trained CDER reviewers to  effectively use the software from this database. NCTR also has  received requests from CDER for five consultations to assess the risk  of products that it has reviewed and to incorporate the software into  training for new reviewers. An extension of the project is also currently  under review.
Title: Assessing the feasibility of using electronic health record  systems to conduct near real-time monitoring of health outcomes,  including serious or unexpected adverse events associated with  medical countermeasures used during public health emergencies.
Center or office funding project: Office of Counterterrorism and  Emerging Threats (OCET)
Priority area(s) covered: Medical countermeasures  Funding obligations and time period of funding: $1.419 million  (fiscal year 2013)
Description of project: The project was designed as a proof-of- concept feasibility study to learn whether it is possible to extract  adverse event data from electronic health records and, if so, whether  those data would provide useful information about the safety and  effectiveness of medical countermeasures during a public health  emergency.
Select achievements reported by FDA: FDA learned that electronic  health records data could help inform a risk assessment of medical  countermeasures based on adverse events; however, there are limits  to what can be done. For example, they found it was feasible to detect  severe adverse events, but that less severe adverse events were  likely to be underreported.
Title: Sentinel initiative for surveillance of drugs, vaccines, and blood  products used to prevent and treat pandemic influenza.
Center or office funding project: OCET  Priority area(s) covered: Medical countermeasures  Funding obligations and time period of funding: $9.1 million (fiscal  year 2011)
Description of project: Sentinel is FDA’s system for conducting near  real-time active safety surveillance of FDA-regulated medical products  through routinely collected electronic healthcare data. This project  was designed to expand those capabilities to include preparedness  for safety surveillance in response to the use of medical  countermeasures, including for pandemic influenza.
Select achievements reported by FDA: This project created the  capability for FDA to monitor the safety of medical countermeasures— for example, influenza vaccines used during an emergency, such as a  pandemic. It also increased the efficiency of linkages between  registries of immunization and the Sentinel database, which FDA says  is vital during pandemics.
Title: Collecting spectral information of foods, pharmaceutical  ingredients, and formulated product (as well as its packaging  materials) to establish a comprehensive spectral library accessible  through the internet.
Center or office funding project: Office of International Programs  (OIP)
Priority area(s) covered: Information sciences  Funding obligations and time period of funding: $0.314 million  (fiscal year 2013)
Description of project: The primary goal of this project was to  develop a roadmap for the development of a global library that could  potentially be used to protect consumers against fraudulent and  adulterated products.
Select achievements reported by FDA: A roadmap was produced  from this project and FDA intends to use it in discussions with  domestic and foreign stakeholders, including other regulatory  agencies and manufacturers.
Title: Explored the potential for mining social media and other web  sources to detect adverse event and safety signals.
Center or office funding project: Office of Regulatory Science and  Innovation (ORSI)
Priority area(s) covered: Information sciences  Funding obligations and time period of funding: $0.801 million  (fiscal years 2012 through 2014)
Description of project: This study evaluated the validity and  trustworthiness of using a social media data mining tool to detect drug  safety events. It was also designed to evaluate the potential of social  media data to provide early signals for drug safety events in  postmarketing surveillance and to better understand how these data  can be used.
Select achievements reported by FDA: FDA initially reported that  the data mining tool suggested that user-generated data sources may  identify signals not found in the traditional voluntary reporting systems  and that there was agreement between data obtained from this tool  and that obtained from the FDA Adverse Events Reporting System.  FDA conducted an additional study to evaluate if there was evidence  in social media for 10 recent MedWatch Safety Alerts prior to FDA  becoming aware of them. FDA has since reported that monitoring  social media did not provide early safety concerns for the medical  products that were monitored in the study. The analysis identified  specific limitations of FDA’s tool that was used in monitoring safety  concerns. These limitations related to the natural variability of data  sources and the difficulties in conducting accurate evaluation of the  data. FDA noted that as monitoring social media for safety concerns is  a new approach, the agency still needs to establish best practices in  order to use it effectively.
Title: Consortium for tuberculosis biomarkers.
Center or office funding project: ORSI  Priority area(s) covered: Clinical evaluations and personalized  medicine  Funding obligations and time period of funding: $1.425 million  (fiscal years 2010 through 2012)
Description of project: This project had three main objectives: 1)  create protocols, processes, and standards by which a consortium for  tuberculosis biomarkers—composed of three organizations central to  tuberculosis clinical drug development—would operate; 2) create a  repository for receiving, storing, and shipping samples to designated  investigators; and 3) establish a peer review panel to review  proposals related to the discovery and qualification of tuberculosis  biomarkers, especially some surrogate markers.
Select achievements reported by FDA: The consortium established  protocols for key data elements to be gathered; a consensus set of  operating procedures for sample collection, processing, and storage;  and quality assurance and monitoring for these activities. The  consortium also adopted a peer review process to review applications  for access to samples.
Title: Sex-based differences in the molecular mechanisms of polymer  degradation in drug eluting stents.
Center or office funding project: Office of Women’s Health (OWH)
Priority area(s) covered: Clinical evaluations and personalized  medicine and manufacturing and quality  Funding obligations and time period of funding: $0.2 million (fiscal  years 2011 through2012)
Description of project: This study explored the different breakdown  of materials used in biodegradable stents and examined the potential  effect of sex on the degradation of these materials.
Select achievements reported by FDA: FDA reported that the study  found that stent material breakdown varied in different tissues. The  findings provide information that can guide FDA and the larger  community in the design, manufacture, and evaluation of absorbable  stents. FDA reported that this research will support the development  of guidance for implants containing certain absorbable components.
section_title: Appendix II: Comments from the Department of Health and Human Services
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Marcia Crosse, (202) 512-7114 or crossem@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, William Hadley, Assistant Director;  Carolyn Garvey; Sandra George; Cathleen Hamann; Carolyn Feis  Korman; and Deborah Linares made key contributions to this report.