National Institutes of Health: Better Oversight Needed to Help Ensure Continued Progress Including Women in Health Research
Published: 2015-10-22T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Women make up over half the U.S. population, but historically have been underrepresented in clinical research supported by NIH and others. As a result, differences in the manifestation of certain diseases and reactions to treatment in women compared with men were not identified. For example, there have been instances of women having adverse effects that differed from those of men related to medications and other treatments. NIH's Inclusion Policy established requirements governing women's inclusion in its clinical research.
GAO was asked to provide information on women's participation in NIH research. Among other reporting objectives, GAO examined (1) women's enrollment and NIH's efforts to monitor this enrollment in NIH-funded clinical research; and (2) NIH's efforts to ensure that NIH-funded clinical trials are designed and conducted to analyze potential sex differences, when applicable. To do this, GAO reviewed relevant laws and policies, including the Inclusion Policy, and federal standards for internal control; reviewed and analyzed NIH enrollment data from fiscal years 2005-2014; and interviewed NIH and IC officials and other experts.
section_title: What GAO Found
paragraphs: Data from the National Institutes of Health (NIH) show that more women than men were enrolled in NIH-funded clinical research for fiscal years 2005-2014, but NIH does not make certain enrollment data readily available to interested parties or examine other detailed data to identify potential challenges to enrolling women in specific research and disease or condition areas. In fiscal year 2014, for example, NIH reported that across all of the clinical research studies it funded—including phase III clinical trials, the largest studies involving human subjects— 57 percent of enrollees (16.4 million) were women. NIH collects enrollment data from individual awardees through its Institutes and Centers (IC)—which generally fund studies in different research areas—and publicly reports data on aggregate enrollment as part of its implementation of the Inclusion Policy developed to implement provisions of the NIH Revitalization Act of 1993. However, NIH does not make the IC-level enrollment data from each of the 25 ICs that report data readily available to interested parties, so that interested parties must make an effort to seek out this data. In addition, NIH does not routinely examine more detailed enrollment data, such as enrollment data organized by the disease and condition being studied. As a result, NIH is limited in its ability to identify whether women are sufficiently represented in studies in specific areas—such as cardiovascular disease—or if the agency-wide data inadvertently mask enrollment challenges. By not examining more detailed data on enrollment below the aggregate level, NIH cannot know whether it is adequately including women in all of the research it supports, in a manner consistent with its Inclusion Policy. Further, NIH's reporting and monitoring in this area is inconsistent with federal internal control standards, which call for agencies to have controls to help ensure effective information flow and effective monitoring of agency activities.
NIH requires that phase III clinical trial awardees consider whether analysis of potential differences in outcomes between women and men is needed in their studies—one of the key requirements of its Inclusion Policy; however, the agency does not maintain, analyze, or report summary data to oversee whether analysis of outcomes by sex are planned or conducted. NIH officials told GAO that they rely on peer review and program officer monitoring to ensure awardee compliance with the analysis requirement. However, NIH program officers do not have a required field in a reporting system or other means to record the information they collect to monitor awardees' analysis plans and compliance with the Inclusion Policy requirement. In addition, there is no data element in NIH's data system to indicate whether an awardee's study should or does include plans for an analysis of potential differences in research outcomes by sex. As a result, NIH lacks summary data, such as the percentage of awardees in a given year with trials designed to identify potential differences in clinical outcomes by sex. Without this summary information, NIH cannot report this information in the agency's biennial reports to Congress and other stakeholders. The lack of summary data and reporting compromises NIH's monitoring of its implementation of the Inclusion Policy and conflicts with federal internal control standards, which call for agencies to ensure the flow of information about agency activities, provide for internal and external communication, and conduct periodic monitoring. Further, it limits NIH's assurance that it is supporting research that can be used to shape improved medical practice for both women and men.
section_title: What GAO Recommends
paragraphs: GAO recommends that NIH examine and report more detailed data on women's enrollment in NIH-funded studies, and collect, examine, and report data on the extent to which these studies include analyses of potential differences between women and men. NIH agreed with GAO's recommendations and plans to take action to implement them.

section_title: Letter
paragraphs: The Department of Health and Human Services’ (HHS) National Institutes  of Health (NIH) is the nation’s largest public funder of biomedical  research, with a budget of more than $30 billion for fiscal year 2015.  Although women make up just over half the U.S. population, their health  needs have historically been underrepresented in the research supported  by NIH and others. This underrepresentation in clinical research has led  to failures in recognizing differences between women and men in the  prevalence of certain diseases, in how certain diseases manifest, and in  the reactions to treatments. As a result, there have been notable  instances of women experiencing adverse effects and other poor  outcomes that differed from those experienced by men related to health  care treatments. For example, in 2013, the Food and Drug Administration  lowered the recommended dose of a popular sleep drug for women after  it was determined that women and men metabolize the drug differently.  That difference left women with more of the drug in their bodies the next  morning and therefore at a greater risk of driving while impaired.  Women’s health research advocates have stated that without the routine  consideration of the potential for these and other clinical differences  between women and men, the nation does not receive the full value of its  public investment in biomedical research.
It has been over 2 decades since Congress passed the NIH Revitalization  Act of 1993 (Revitalization Act), which provided statutory requirements for  NIH to follow regarding research on women’s health. Specifically, the  Revitalization Act contained provisions for the enrollment of women in  clinical research studies—including clinical trials—and requiring that  clinical trials be designed and carried out to be able to provide for a valid  analysis of whether the variables being studied affect women and men  differently. The Revitalization Act also required NIH to establish  guidelines for including women in clinical research. In 2000, we examined  NIH’s efforts to conduct research on women’s health over the previous  decade. We reported that NIH had made significant progress in  implementing the Revitalization Act by issuing guidelines to implement  the law, conducting extensive training for scientists and reviewers, and  implementing a centralized data system to monitor enrollment. However,  we also recommended in 2000 that NIH continue to make improvements,  including better implementation of the requirement that some studies be  designed to permit analysis of results by sex, which could reveal whether  interventions affect women and men differently. In response, NIH  developed specific guidance on this aspect of the policy for applicants  and for the reviewers who evaluate applicants for NIH funding.
Partly in response to our recommendation, in October 2001, NIH  amended its Policy and Guidelines on The Inclusion of Women and  Minorities as Subjects in Clinical Research (Inclusion Policy), which is in  place today. The Inclusion Policy establishes two key requirements for  NIH-funded researchers conducting clinical research studies. Specifically,  researchers must (1) design research plans that detail the breakdown of  their studies’ participants by sex and provide a rationale for their planned  enrollment; and (2) for certain clinical trials, include plans for analyzing  outcomes for potential sex differences, when appropriate, as determined  by prior scientific evidence.
Despite these efforts, reports from the Institute of Medicine (IOM)  published in 2010 and 2012 stated that continued efforts—by NIH and  others—are needed to improve both the overall status of women’s health  research and, in particular, the ability of researchers to conduct analysis  of study results by sex to determine whether outcomes differ for women  compared to men. While experts acknowledge that there have been  increases in women’s enrollment in clinical research studies since the  passage of the Revitalization Act, the IOM and others have found that  even when women are included in clinical trials, the results of analyses  are often not reported by sex—even when the overall results are  published in peer-reviewed journals. A recent study found that less than  half of NIH-funded clinical trials overall had been published in a peer- reviewed biomedical journal within 30 months of trial completion, and in  response, NIH officials stated that other means of sharing such results— besides publication in scientific literature—is needed. Experts have  stated that insufficient reporting of clinical trial results by sex limits the  ability of researchers to identify potentially important sex differences that  may ultimately affect patient care.
You asked us to provide information on women’s participation in NIH- funded clinical trials, including NIH’s policies regarding enrollment, and  both the analysis and reporting of clinical trial results. In this report, we  examine NIH’s implementation of the Inclusion Policy, in particular,  1.  the level of women’s enrollment and NIH’s efforts to monitor this  enrollment in NIH-funded clinical research; and  2.  NIH’s efforts to ensure that NIH-funded clinical trials are designed and  conducted to analyze potential sex differences, when applicable.  3.  NIH’s efforts to help ensure that researchers make the results of  clinical trials public, including sex-specific results, when appropriate.
In addition, Appendix I of this report summarizes information on the  factors affecting women’s participation in clinical research, as were most  frequently cited in published literature.
To address these objectives, we reviewed relevant laws and policies and  interviewed NIH officials and other relevant experts. Specifically, we  reviewed laws and policies (including proposed policies)—in particular,  the Revitalization Act and NIH’s Inclusion Policy. We interviewed officials  from NIH’s Office of Research on Women’s Health (ORWH), the Office of  Extramural Research (OER), and the National Library of Medicine. We  also interviewed representatives from selected NIH Institutes and Centers  (ICs) that we identified as among the largest ICs both in terms of the total  amount of award funding provided to researchers and in terms of the total  number of research awards made: the National Cancer Institute (NCI),  the National Institute of Allergy and Infectious Disease (NIAID), and the  National Heart, Lung, and Blood Institute (NHLBI). In addition to  interviewing the leadership of the selected ICs, we also interviewed  program officers from the selected ICs who work directly with NIH  awardees. We also interviewed officials from the Society for Women’s  Health Research, the Women’s Health Research Institute at Northwestern  University, and the editors of one major medical journal, as well as other  individual researchers and experts, to gain perspectives on women’s  inclusion in biomedical research supported by NIH and others and on the  analysis and reporting of research outcomes.
To describe the level of women’s enrollment and NIH’s efforts to monitor  women’s enrollment in NIH-funded clinical research, we obtained and  reviewed enrollment data for all NIH-funded clinical research studies  across NIH and for each of the ICs. For the aggregate NIH data, we  reviewed 10 years of enrollment data (fiscal years 2005-2014), and for IC- level data, we reviewed 4 years of enrollment data (fiscal years 2011- 2014) for the 25 ICs that collect and report such data. We also reviewed  documentation of NIH’s data collection tools, and interviewed NIH and IC  officials—from the offices identified above—regarding their oversight of  enrollment data. Through our review of the data and supporting  documentation and our interviews with NIH and IC officials, we  determined these enrollment data to be sufficiently reliable for our  purposes. As part of our review, we examined the extent to which NIH’s  monitoring efforts were consistent with the standards for internal control in  the federal government—specifically those related to information,  communication, and monitoring.
To describe NIH’s efforts to ensure that NIH-funded clinical trials are  designed and conducted to analyze potential sex differences, when  applicable, we collected and reviewed NIH guidance and tools regarding  how to assess applicants’ and awardees’ compliance with the analysis  requirement of the Inclusion Policy, as it applies to certain clinical trials.  We reviewed NIH reports and documentation related to implementation of  the Inclusion Policy. We also interviewed NIH and IC officials specifically  regarding their efforts to monitor implementation of the analysis  requirement of the Inclusion Policy. As part of our review, we examined  the extent to which NIH’s efforts were consistent with the standards for  internal control in the federal government—specifically those related to  control activities, information, communication, and monitoring.
To describe NIH’s efforts to help ensure that researchers make the  results of clinical trials public, including sex-specific results when  appropriate, we reviewed NIH policies and procedures regarding the  reporting of clinical trial results (particularly through public venues),  including proposed policies, and documentation of NIH activities related  to sharing of clinical trial results. We examined the NIH registry and  results website—ClinicalTrials.gov—and related materials. We also  reviewed the publication policies of selected scientific journals to  determine whether they addressed reporting of results by sex.
We conducted this performance audit from September 2014 to October  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Headed by a presidentially appointed and Senate-confirmed director, NIH  comprises 27 ICs and an Office of the Director. NIH’s ICs both conduct  and support biomedical research specific to their unique missions, which  generally focus on a specific disease, a particular organ, or a stage in life  (e.g., childhood). Each of the ICs has its own director and staff, as well as  its own advisory council or board, which helps to support and oversee the  IC’s work. Within the Office of the Director are offices responsible for  issues, programs, and activities that span NIH components, particularly  research initiatives and issues involving multiple ICs.
subsections: section_title: NIH-Supported Research and Clinical Trials
paragraphs: NIH’s biomedical research that focuses on humans—its clinical research  studies—includes clinical trials of biomedical or behavioral interventions  such as new drugs, medical treatments, and surgical procedures and  devices. Clinical trials are divided into four phases. In phase I clinical  trials, which typically include 20 to 80 people, researchers test a new  biomedical or behavioral intervention on human subjects for the first time  to evaluate safety. In phase II clinical trials, the intervention is given to a  larger group of people, 100 to 300 participants, to further evaluate  efficacy and safety. In phase III clinical trials, the intervention is given to  even larger groups—from several hundred to several thousand  participants—to compare the intervention to commonly used or  experimental interventions. Finally, phase IV studies are conducted after  the intervention has been marketed, in order to gather information on  long-term use.
NIH’s ICs support clinical trials predominantly through “extramural  research”—awarding funds to researchers at universities or other  research entities (awardees) through grants, contracts, and cooperative  agreements. Of NIH’s 27 ICs, almost all fund extramural research  projects. These ICs use a standard peer review process to inform the final  decisions on which extramural research projects to fund. The size and  composition of the ICs’ clinical trial portfolios vary substantially,  depending on such factors as the IC’s budget, mission, and the scientific  goals of any given study. For example, some ICs support few if any  phase III clinical trials.
In fiscal year 2014, NIH’s ICs reported funding of nearly $30 billion for all  biomedical research. Of that amount, NIH estimates that—based on  reporting categories HHS developed for use by all of its agencies—  $23.9 billion (80.3 percent) funded research related to the health of both  women and men, $4 billion (13.2 percent) funded research related to  women’s health, and an estimated $1.9 billion (6.4 percent) funded  research related to men’s health. (See Appendix II for more details on  estimated fiscal year 2014 funding for selected diseases and conditions of  particular relevance for women.) To determine these amounts, NIH  annually assigns its research funding to certain women’s health disease  and condition categories—such as breast cancer or heart disease.  Additionally, NIH classifies this funding as either related to the health of  both sexes or as supporting research on women’s health only or men’s  health only by using HHS’s calculation guidelines. NIH reports these  funding estimates in HHS’s annual congressional budget justification and  in ORWH’s biennial Report of the Advisory Committee on Research on  Women’s Health.
section_title: The NIH Revitalization Act of 1993 and NIH Inclusion Policy
paragraphs: The Revitalization Act required NIH to ensure the appropriate inclusion of  women in NIH-funded clinical research, including clinical trials. The  Revitalization Act contained provisions that required NIH to, among other  things, ensure that women are included in all NIH-funded clinical  research, and report on compliance with the inclusion provisions of the  Act. Additionally, the Revitalization Act requires NIH to ensure that clinical  trials are designed and carried out in a manner sufficient to allow for valid  analysis of the extent to which the outcomes measured in the trial affect  women differently than men. NIH subsequently determined that this  particular requirement only applied to phase III clinical trials, as they are  the largest clinical trials involving human subjects and the closest to  effecting broad changes in public health policy and standards of care.
The Revitalization Act also directed NIH to develop guidelines for  including women in clinical research and report biennially to Congress on  NIH’s compliance with this policy. The resulting Inclusion Policy, the  current version of which has been in place since October 2001, requires  NIH applicants conducting clinical research to  1.  design research plans that detail the breakdown of their studies’  participants by sex and provide a rationale for their planned  enrollment for all clinical research studies; and  2.  include plans for analyzing outcomes for potential sex differences for  NIH-defined phase III clinical trials, when appropriate, as determined  by consideration of prior scientific evidence.
In addition to NIH ICs, several NIH offices play a role in implementing the  Inclusion Policy, particularly ORWH, which was established in 1990—and  codified in the Revitalization Act—to promote women’s health research,  and OER, which administers and manages NIH grants policies,  operations, and data systems. Since November 2011, the implementation  of NIH’s Inclusion Policy has been overseen by the Subcommittee on  Inclusion Governance, which comprises senior NIH officials from ORWH,  OER, and several ICs. This committee, co-chaired by the ORWH  director and staffed by the NIH Inclusion Policy Officer from OER, is  charged with examining and considering current NIH policies related to  the inclusion of women in NIH-funded clinical research. In addition, the  implementation of the Inclusion Policy is monitored by the Advisory  Committee on Research on Women’s Health (Women’s Health Advisory  Committee), whose creation was mandated by the Revitalization Act, as  well as by the individual IC’s directors and advisory councils or boards.
section_title: NIH’s Extramural Awards Management Process
paragraphs: During the peer review process, applicants’ plans for including women, as  appropriate, are reviewed and assessed along with the applicants’ plans  to meet other requirements or considerations that are outlined in the  funding opportunity announcement or research solicitation. The outcome  of these assessments—scores from the peer reviewers—inform the  funding decisions made by the ICs. Prior to awards being made, program  officers or other IC staff may advise awardees on additional information  required before an award can be released, and the resolution of any  concerns raised during the peer review stage—including concerns related  to adherence to the Inclusion Policy. After awards are made, NIH’s  awardees are responsible for managing their day-to-day activities in  accordance with NIH requirements, and the IC making the award is  responsible for the awarded funds and for monitoring progress and  compliance with NIH policies, including the Inclusion Policy. IC program  officers monitor awardees through a variety of methods—including  reviews of reports and correspondence from the awardee, and site  visits—to identify potential problems with scientific progress, compliance,  and areas where technical assistance might be necessary. One such  report that program officers review is the annual progress report—which  includes data on study enrollment, among other things. Awardees submit  information, including enrollment data, through the Electronic Research  Administration (eRA) Commons, which is part of NIH’s electronic data  collection and grants administration system that is used by awardees and  program officers to access and share administrative information related to  research awards.
section_title: More Women Than Men Were Enrolled in NIH Research Overall, but NIH Does Not Examine Detailed Data to Identify Potential Challenges to Enrolling Women
paragraphs: NIH data show that over the last decade more women than men have  been enrolled across all NIH-funded clinical research, including phase III  clinical trials. NIH publicly reports aggregate enrollment numbers on a  biennial basis; however, it does not routinely make detailed IC enrollment  data readily available or examine more detailed enrollment data by  disease and condition, in order to identify potential challenges to enrolling  women in certain research and disease or condition areas.
subsections: section_title: NIH Data Show More Women than Men Enrolled in NIH-Funded Clinical Research
paragraphs: NIH requires each awardee to report enrollment, including enrollment by  sex, for each of their NIH-funded research awards, including phase III  clinical trials. As of fiscal year 2015, these data are reported to NIH  through its Inclusion Monitoring System—one part of NIH’s awardee data  system. Program officers review these data at least annually to determine  whether actual enrollment is consistent with the enrollment planned in the  research design. The program officers use a designated checklist,  among other tools, to document their monitoring as part of the annual  progress report review process. According to NIH officials, awardee  enrollment data are aggregated by each IC and presented in and  discussed during meetings held by each IC’s advisory board or council,  which are open to the public. The IC-level enrollment data are certified as  being compliant with the Inclusion Policy by the IC’s advisory board or  council and by the IC Director and included in an IC-level enrollment  report. The certified IC enrollment reports are submitted to ORWH and  OER, where, according to NIH officials, the data are checked for  consistency and errors as part of a quality control process. NIH  aggregates the enrollment data across the agency and reports this  aggregate data to the Women’s Health Advisory Committee, Congress,  and the public in NIH’s biennial inclusion report.
According to the data collected by NIH, in each fiscal year from 2005  through 2014, more women than men were enrolled in all NIH-funded  clinical research studies, including phase III trials. (See fig. 1.)  Specifically, in fiscal year 2014, among all NIH-funded clinical   research studies, 57 percent of enrollees (16.4 million) were women,   and 39 percent (11 million) were men. For all NIH-funded phase III  clinical trials, in fiscal year 2014, 60 percent of enrollees were women  (about 480,000) and 39 percent were men (about 314,000).
At the individual IC level, data show that for each IC, women’s enrollment  for all clinical research studies, including phase III clinical trials, was  generally higher than men’s enrollment in most years from fiscal year  2011 through fiscal year 2014. (See app. III for enrollment data at the NIH  and IC level over this period.) Of the 25 ICs reporting enrollment in fiscal  years 2011 through 2014, 13 ICs enrolled more women than men in the  ICs’ clinical research studies during all four years. An additional 4 ICs  enrolled more women than men in 3 of the fiscal years between 2011 and  2014. According to NIH, 10 of the 25 ICs regularly support phase III  clinical trials, and of these, 3 (about one third) enrolled more women than  men in each year in fiscal years 2011 through 2014, and another 2  enrolled more women than men in 3 of the 4 fiscal years.
section_title: NIH Does Not Make IC- Level Enrollment Data Readily Available or Examine Detailed Enrollment Data to Identify Potential Challenges to Enrolling Women in Specific Research and Disease Areas
paragraphs: NIH collects and reviews aggregated enrollment data from the ICs;  however, NIH officials do not make these IC-level enrollment data readily  available to the public or other interested parties. Specifically, the IC-level  enrollment data are not published as part of the overall NIH biennial  report on enrollment, are not shared with the Women’s Health Advisory  Committee, and are not available for download from the ORWH website.  Individual official IC enrollment reports were also not available through  the websites of the three ICs in our review. Beginning in the fiscal year  2013-2014 reporting period, NIH required all ICs to submit their IC  enrollment report numbers in a standard format, which may allow for  easier public sharing of these data going forward.
Additionally, NIH officials do not routinely examine detailed enrollment  data by sex beyond the IC level—such as by a specific research area or  disease or condition being studied—to identify potential challenges to  enrolling women in these areas, because enrollment data are currently  not available in this format. The guidance NIH provided to the ICs for  submitting their fiscal year 2013-2014 enrollment data to ORWH and  OER explains that ICs may choose to further break out their reported  enrollment data by disease area, portfolio area, or in some other manner.  According to NIH officials, certain ICs analyze enrollment data by various  disease categories when necessary.  However, the IC officials we spoke  with told us that their data systems are not capable of systematically  aggregating enrollment data in this manner for routine reporting. When  asked if it would be possible to aggregate enrollment data by disease and  condition, IC officials said that they would be able to do so, but it would be  a time intensive, manual process given current data system limitations.  Further, NIH officials told us they expect NIH’s new enrollment data  system deployed in October 2014 to increase functionality in examining  enrollment data in different ways, but as of July 2015 the officials did not  have specific plans or details available. In addition, NIH officials stated  that because an individual IC’s research generally focuses on a specific  disease, a particular organ, or a stage in life, the enrollment data that are  aggregated at the IC level would roughly correspond with major disease  and condition categories and could be used, to some extent, as a proxy  for disease and condition enrollment data. However, this proxy method  does not take into account the fact that many ICs are responsible for  research that includes multiple diseases, organs, or stages in life; for  example, NHLBI’s research portfolio includes studies of heart, lung, and  blood related diseases and conditions. In addition, research on certain  diseases and conditions—such as obesity—falls under the purview of  multiple ICs.
NIH’s practices of not sharing the IC-level enrollment data and not  examining detailed data on enrollment by sex—by specific research area  or the disease or condition being studied—are inconsistent with several  federal standards for internal control. Specifically, the internal control  standards for information and communications state that for an entity to  run and control its operations, it must have relevant, reliable, and timely  communications relating to internal as well as external events.  Information is needed throughout the agency to achieve all of its  objectives, and effective communication should occur in a broad sense,  with information flowing down, across, and up the organization. In  addition to internal communications, management should ensure there  are adequate means of communicating with, and obtaining information  from, external stakeholders who may have a significant impact on the  agency achieving its goals (e.g., the Women’s Health Advisory  Committee). Federal internal control standards for monitoring call for  management to assess the quality of agency performance over time and  ensure that the findings of audits and other reviews are promptly  resolved.
Because NIH does not readily share IC-level enrollment data with the  public and other interested parties, such as the Women’s Health Advisory  Committee—through means such as the ORWH website or the biennial  NIH enrollment report—those interested in reviewing IC-level enrollment  information would have to attend—or watch online if webcast—each  individual IC’s advisory board or council meeting or specifically seek out  or request any public record resulting from these meetings to have  access to these data. In addition, by not routinely examining more  detailed enrollment data that is aggregated by sex—such as data at the  disease and condition level—NIH is limited in its ability to identify whether  women are sufficiently represented in studies in specific areas that cross  ICs—such as obesity. Further, NIH does not have information of sufficient  detail to monitor and determine if the aggregate enrollment data from  across NIH inadvertently mask low enrollment for particular research  areas or diseases or conditions. At an April 2015 Women’s Health  Advisory Committee meeting, some committee members raised such  concerns, noting that published studies on clinical trials of specific  diseases and conditions, such as cardiovascular disease, appeared to  show that women’s enrollment was lower than the enrollment that NIH  had reported in the aggregate. The committee members acknowledged  that there could be many reasons for such discrepancies, but noted that  they would like to see more detailed enrollment data to improve their  understanding of the data and ensure that women are being appropriately  included in NIH-funded clinical trials.
section_title: NIH Does Not Record Whether Clinical Trials Will Analyze Sex Differences to Allow Summary Analysis and Reporting for Oversight of Its Inclusion Policy
paragraphs: NIH’s Inclusion Policy requires that individual awardees conducting phase  III clinical trials consider whether analysis of potential differences in study  outcomes between women and men is needed in their studies, consistent  with the Revitalization Act’s provisions regarding the design of certain  clinical trials. However, the agency does not maintain, analyze, or report  summary data to oversee whether analysis of outcomes by sex are  planned or conducted, when applicable, across all NIH-funded clinical  trials.
Under the Inclusion Policy, applicants seeking funds for phase III clinical  trials must consider prior scientific evidence and assess whether an  analysis of potential sex differences is merited, and if so, develop a plan  to analyze study results accordingly. Both this consideration and the  plan for analysis, if appropriate, are to be included in the awardee’s  application for funding. To ensure awardees’ compliance with this  requirement, NIH officials told us they rely on the agency’s peer review  process for reviewing applications, and after awards are made, on IC  program officers’ monitoring of individual awardees. NIH has guidelines  for peer reviewers to use when assessing applications and rating  applicants’ inclusion plans during peer review. This review typically  includes an evaluation of the proposed study design and assessment of  whether any plans for conducting an analysis of potential sex differences  are “acceptable” or “unacceptable.” The assessment and rating are based  on consideration of prior scientific evidence that either supports or  negates the existence of differences in outcomes by sex. Peer reviewers  document this assessment in a summary statement provided to the ICs  for final award determinations. According to NIH officials, if reviewers  determine that an applicant’s plans are not acceptable, the applicant is  barred from funding until the plans are addressed and deemed  acceptable by IC officials. After awards are made, program officers from  the ICs that fund the studies are responsible for monitoring awardees’  overall progress on an ongoing basis. Specifically, program officers are to  review awardees’ annual progress reports, which the Inclusion Policy  states should address analysis of potential sex differences, as  appropriate. Officials from one IC told us that, through their regular  interactions with awardees, program officers are very familiar with the  design of their awardees’ trials and whether any such analysis is planned  or underway.
NIH program officers monitor individual awardees’ compliance with the  analysis requirement of the Inclusion Policy; however, the agency lacks  summary data on awardees’ analysis plans, including the percentage of  awardees in a given year with trials designed to identify potential sex  differences, when applicable. Currently, program officers review  awardees’ progress reports—including any information reported regarding  the analysis of potential sex differences—but do not have means, such as  a written checklist or a required field in an electronic reporting system, for  recording the information obtained through this monitoring, as they do for  monitoring enrollment. NIH’s awardee data system includes information  on whether individual awards include phase III trials. However, the data  system does not have a data element that denotes whether an awardee’s  study should include or has plans for an analysis of potential sex  differences. Although such information is included in the narrative  included in awardees’ funding applications, this information cannot be  easily aggregated for summary reporting, NIH officials explained, and  therefore they do not estimate the proportion of trials being conducted at  any one time that are designed and explicitly intended to identify  differences in study outcomes by sex. NIH officials also told us that they  plan to add a question for this type of monitoring to the existing electronic  checklist used by program officers in the fall of this year, to be  implemented for awards funded in fiscal year 2016.
Because NIH does not have summary data regarding the analysis  requirement of the Inclusion Policy, it has not reported summary  information on this aspect of the Inclusion Policy to key stakeholders— including the Women’s Health Advisory Committee and the Congress.  Notably, NIH’s agency-wide biennial reports on the status of the Inclusion  Policy do not include information on the extent to which NIH-funded  phase III trials included plans to conduct analyses of potential sex  differences, or on the overall status of this aspect of the Inclusion Policy.  Instead, the report focuses primarily on NIH-wide aggregate enrollment.  NIH officials told us they rely on program officers’ monitoring of individual  awardees to ensure that the analysis requirement of the Inclusion Policy  is being implemented appropriately after awards are made, because part  of the program officer’s role is to ensure satisfactory scientific progress as  well as compliance with NIH policies. The officials added that they were  not sure of the utility of summary reporting in this case.
NIH’s lack of summary data and reporting regarding the analysis  requirement of the Inclusion Policy conflicts with federal internal control  standards. First, federal internal control standards require that federal  agencies have control activities in place to ensure that management’s  directives are carried out and that these controls are monitored. These  standards also state that information should be recorded and  communicated to management and other responsible officials in a form  and within a time frame that enables them to carry out their internal  control and other responsibilities.
Without summary data on the funded phase III clinical trials that are  intended to provide information on potential sex differences—including  the number of such trials funded in a given year—senior NIH officials are  limited in their ability to effectively oversee the implementation of the  Inclusion Policy to assess whether changes are needed to their  procedures. Further, NIH cannot provide this information to stakeholders  such as the Women’s Health Advisory Committee and Congress in its  regular reporting on other aspects of the Inclusion Policy. As a result,  these stakeholders lack assurance that the agency is implementing the  Inclusion Policy as intended and in a manner consistent with the  Revitalization Act’s provisions regarding the design of certain clinical  trials. In its fiscal year 2011-2012 Report of the Advisory Committee on  Research on Women’s Health, NIH previously acknowledged that  inclusion is not just a matter of having women and men included in clinical  studies; rather, the scientific value of research studies is greatly  enhanced by providing knowledge about differences and/or similarities  between different populations affected by the diseases under study.  However, without assurance that its clinical trials are being designed and  conducted as directed under the law and its implementing policy, NIH’s  insight regarding the interpretation, validation, and generalizability of  findings resulting from the research it supports—as these findings apply  to both women and men—is diminished, potentially limiting the value of  NIH-funded research.
section_title: Proposed NIH Policy Intended to Increase Public Reporting of Clinical Trial Results by Awardees and Could Increase Reporting of Sex- Specific Results, According to NIH Officials
paragraphs: NIH has developed and proposed a policy intended to increase public  reporting of clinical trial results by requiring all NIH-funded clinical trials to  be registered and have results submitted to its registry and results  database, ClinicalTrials.gov. ClinicalTrials.gov contains accessible and  searchable information on publicly and privately supported clinical trials  and observational studies that is provided and updated by NIH awardees  and other researchers. Currently, all NIH awardees are encouraged to  register their trials with ClinicalTrials.gov at the beginning of the study, but  only certain trials—those of certain drugs and devices regulated by the  Food and Drug Administration—must register and provide summary  results as required by law. According to NIH officials, as of July 2015,  there were approximately 195,000 studies registered in ClinicalTrials.gov,  and almost 18,000 of these have summary results information posted.  Under the proposed NIH policy, all NIH-funded clinical trials, regardless of  trial phase or type of intervention being studied, would have to be  registered and submit summary results—including participant flow,  baseline demographics such as the sex and age of participants, primary  and secondary outcomes, and adverse events—to ClinicalTrials.gov.  According to NIH, if the proposed policy goes into effect, compliance may  be enforced through possible suspension or termination of funding and  noncompliance could impact future funding decisions. NIH sought public  comments on the proposed policy from November 2014 through March  2015 and, as of August 2015, was analyzing the comments it received.  NIH anticipates that the final policy will be issued in the first quarter of  fiscal year 2016.
NIH officials told us that the proposed policy for clinical trial registration  and results submission was not intended to increase reporting of sex- specific results to ClinicalTrials.gov; however, the officials also said that  there is the potential for more reporting of sex-specific results, given the  overall increase expected in the number of reported studies. Since the  proposed policy would require awardees to report results for their  prespecified primary and secondary outcome measures as part of the  summary results submission, the reporting of sex-specific results would  depend on the design of the trial, according to NIH officials. Specifically, if  sex differences were among the prespecified primary and secondary  outcomes studied in a specific trial, officials said, then that would be  reflected in the results submitted to ClinicalTrials.gov.
In issuing the proposed policy, NIH stated that its awardees are expected  to make their trial results available to the research community and to the  public at large in order to contribute to scientific knowledge and,  ultimately, public health. NIH proposed the policy partly in response to a  recent study, which found that within 30 months of trial completion, the  results of less than half of NIH-funded clinical trials had been published in  a peer-reviewed biomedical journal, the traditional method for sharing  results. NIH stated that because journal publication of clinical trials  results is not always possible, it is important to provide other ways for  clinical trial results to be disseminated and publicly available to  researchers, health care providers, and others.
NIH has made efforts to encourage sex-specific reporting of clinical trial  results. In NIH’s fiscal years 2011-2012 Report on the Advisory  Committee for Research on Women’s Health, ORWH stated that it is only  through sex-specific reporting that full information becomes available to  the public and to scientists who can then use such data to inform future  studies, thereby building the knowledge base in a manner that takes into  consideration the influences of sex on health and disease. Specifically,  NIH has worked with journal editors and others to encourage reporting of  results by sex. Specifically, in 2011, NIH asked IOM to convene a  workshop of researchers, journal editors, and others on the topic to  discuss the importance of reporting results by sex and the implications of  this issue for journals’ reporting policies. According to Stanford  University’s Gendered Innovations project, 32 peer-reviewed journals  worldwide have editorial policies requiring that clinical trial researchers  include information on results by sex when they submit articles for  publication. However, editors from one medical journal that we spoke  with stated that when evaluating whether results should be reported by  sex, it is important to consider whether examining sex differences is a  primary outcome of the study, and whether the trial was big enough for a  valid subgroup analysis—i.e., analysis of the effect of the intervention on  two or more different groups of participants, such as women and men.  They emphasized that if a study was not designed for a subgroup  analysis by sex and one was performed, the results could be erroneous.
NIH has also made efforts to facilitate the sharing of clinical trials results,  including sex-specific information, through venues other than journals and  ClinicalTrials.gov. NIH has a number of policies that promote the  dissemination of research results—and the underlying data—and guide  awardees in disseminating their results, including the NIH Data Sharing  Policy, among others. Additionally, related to the sharing of sex-specific  information, NIH hosts a Women’s Health Resources portal and a  Women’s Health topic page on the Medline Plus webpage, which includes  links to other information about women’s health from journal articles and  ClinicalTrials.gov. The agency also reports summaries of research related  to women’s health in the biennial Report of the Advisory Committee on  Research on Women’s Health.
section_title: Conclusions
paragraphs: Although NIH has made progress in the 2 decades since the 1993  Revitalization Act regarding the inclusion of women in NIH-funded clinical  research, opportunities remain for NIH to further extend the value of its  investment in medical research. NIH is responsible for ensuring that the  nation receives the greatest benefit of the large federal investment in  clinical research by fully implementing its Inclusion Policy, such that  women are adequately included in NIH-funded clinical trials when  appropriate, and that potential sex differences may be identified. By not  readily sharing IC-level enrollment data, NIH limits the public’s and other  interested parties’ ability to gain insight into enrollment issues at each of  the ICs, putting the onus of obtaining these data on the interested parties  themselves to attend or view online up to 25 individual IC board or council  meetings or request any public record of these meetings. By not  examining more detailed enrollment data—such as data aggregated by  research area or specific to various diseases and conditions—NIH cannot  know whether it is adequately including women across all of the research  it supports. Without this greater insight into enrollment for specific to  diseases and conditions, NIH is limited in its ability to assess whether its  programs that support cross-cutting research spanning multiple ICs are  successfully including women in clinical research or facing challenges  that the agency should address. Further, the lack of summary data and  reporting about the extent to which awardees plan to conduct or perform  analyses of potential sex differences in phase III clinical trials  compromises NIH’s oversight and jeopardizes the agency’s ability to  provide assurances over the Act’s provisions regarding the design of  certain clinical trials and meet the purposes of its Inclusion Policy. Without  summary data, such as the proportion of trials being conducted that  intend to analyze differences in outcomes for men and women, and  reporting on that data, NIH and Congress cannot know whether or to what  extent current efforts are helping to ensure that differences in clinical  outcomes by sex are identified and that NIH is supporting research that  can be used to shape improved medical practices for both women and  men.
The overall increase in the enrollment of women in NIH-funded clinical  research studies, such that women have been a significant proportion of  research subjects for nearly 2 decades, is a noteworthy achievement for  NIH. To continue to build on this achievement—and consistent with  federal internal control standards—NIH should turn its focus to assessing  whether the agency is meeting the purposes of the Inclusion Policy, and if  it is not, take the needed corrective actions.
section_title: Recommendations for Executive Action
paragraphs: To ensure effective implementation of the Inclusion Policy in a manner  consistent with the Revitalization Act’s provisions regarding the design of  certain clinical trials, the NIH Director should take the following five  actions: make IC-level enrollment data readily available through public means,  such as NIH’s regular biennial report to Congress on the inclusion of  women in research, or through NIH’s website; examine approaches for aggregating more detailed enrollment data at  the disease and condition level, and report on the status of this  examination to key stakeholders and through its regular biennial  report to Congress on the inclusion of women in research; ensure that program officers have a means for recording information  obtained from monitoring awardees’ plans for and progress in  conducting analyses of potential differences in outcomes by sex; on a regular basis, systematically collect and analyze summary data  regarding awardees’ plans to conduct analyses of potential sex  differences, such as the proportion of trials being conducted that  intend to analyze differences in outcomes for men and women; and  report on this summary data and the results of this analysis in NIH’s  regular biennial report to Congress on the inclusion of women in  research.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this product to HHS for comment, and HHS  responded with comments provided by NIH. In its written comments,  reproduced in appendix IV, NIH generally concurred with our findings and  recommendations. NIH also provided technical comments that were  incorporated into the final report, as appropriate.
In commenting on our first recommendation to make IC-level enrollment  data readily available to the public, NIH agreed and indicated that there  are opportunities for the agency to increase the accessibility of IC-level  enrollment data. NIH also stated that the agency has begun to  standardize IC enrollment reporting and will continue this effort by  standardizing data tables and graphics for ICs to provide for the NIH-wide  biennial reports. NIH did not provide a timeline for making this information  readily available to the public.
NIH agreed with our second recommendation to examine approaches for  aggregating more detailed enrollment information at the disease and  condition level, and to report on the status of this examination to key  stakeholders. In its comments, NIH also reiterated what we describe in  our report: some ICs conduct analysis of enrollment by disease or  condition on an as-needed basis. NIH noted that the agency is working on  ways to analyze enrollment at the disease and condition level across the  ICs. NIH did not provide information on when the agency would be able to  analyze these enrollment data, but it did state that when the agency is  able to perform the analysis, NIH would make the results readily available  through NIH’s biennial inclusion reports or other means.
NIH agreed with our third recommendation to ensure that program  officers have a means for recording their monitoring of awardees’ plans  for and progress in conducting analysis of potential sex differences, and  confirmed that the agency plans to add questions that would facilitate this  type of monitoring into the existing checklist program officers use to  document other types of monitoring beginning in fiscal year 2016.
In commenting on our fourth and fifth recommendations regarding  collecting and reporting summary data on awardees’ plans for sex- differences analysis, NIH agreed that it is critical to obtain more  information on which clinical trials involve analyses of sex differences,  and described some alternative data collection approaches for improving  oversight of this issue. We maintain that thoughtful, useful analysis and  summary reporting would improve NIH’s oversight of this aspect of the  Inclusion Policy. Our recommendation was not intended to prescribe or  limit the type of analysis performed or the data collected by NIH; instead  we provided an example that NIH could adapt as needed, and we  encourage the agency to explore the best alternatives for their analyses.
In other general comments, NIH also noted other opportunities that  support oversight, such as the importance of peer reviewers in examining  applicants’ plans for including women prior to funding decisions, and  expanded reporting in ClinicalTrials.gov. In addition, the agency noted  that ClinicalTrials.gov could engender greater transparency of clinical trial  results and help assure that the analyses required under the Inclusion  Policy are being completed.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  Secretary of the Department of Health and Human Services, the Director  of the National Institutes of Health, and other interested parties. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix V.
section_title: Appendix I: Literature Review of Factors Affecting Women’s Participation in Clinical Research
paragraphs: We reviewed 34 published journal articles that specifically identified  barriers to or reasons for women’s participation in clinical research. We  conducted an initial literature search that identified 168 studies published  from January 2004 through October 2014 on the topic of women in  clinical research. After reviewing abstracts and the full text of some  articles, we narrowed this group down to the 34 articles included in this  report that focus on the factors specifically affecting women’s participation  in clinical research. We also spoke with or received written responses  from officials and program officers from 3 institutes and centers (ICs) of  the National Institutes of Health (NIH) about the factors affecting women’s  participation in clinical research reported by NIH awardees.
In our review of the literature, the three most commonly cited barriers  women face when considering whether to participate in clinical research  were fear of experimentation/trust issues, health related concerns, and  transportation/convenience issues. (See table 1.) Fear of  experimentation/trust issues included distrust of physicians and the  medical community and fear that the experimental treatment may be  inferior to the conventional treatment(s). Health related concerns cited in  the articles included concerns about the potential side effects or other  adverse events associated with the experimental treatment. Some of the  transportation/convenience concerns cited in the articles included  difficulty traveling to a clinic (no transportation or a long distance to the  facility) and general inconvenience associated with participating in the  research.
The three most common reasons for women’s participation in clinical  research identified in our literature review were personal/health benefits,  altruism, and a general category of “other” reasons. (See table 2.)  Personal/health benefits included access to new treatment and drugs,  while “other” reasons included the fact that the research was being done  at a clinic where the participant had already received care or was being  conducted by clinical staff with whom the participants were familiar.  Participation due to altruism included the desire to help science.
The NIH IC officials and program officers we spoke with generally agreed  that the barriers and reasons for participation that we identified through  our literature review were consistent with those they have encountered  when working with their awardees. In addition, officials from one IC we  spoke with identified other reasons women participate in clinical research,  as cited by awardees. These reasons included the opportunity to learn  about the disease being studied and how to manage it, and a sense of  pride associated with participation. In contrast, these officials stated that  for some diseases, the effect or perceived effect on future fertility could  be a barrier to women’s participation.
section_title: Appendix II: National Institutes of Health (NIH) Funding for Women’s Health Research
paragraphs: To determine the amount of NIH funding for research on women’s health,  we collected and reviewed NIH women’s health budget information for  fiscal years 2009-2014. We also interviewed officials from NIH and the  Department of Health and Human Services (HHS) regarding the budget  categories that NIH uses in its women’s health budget, which were  originally developed for use in reporting by all HHS agencies.
NIH, like all HHS agencies, annually compiles a summary table that  estimates funding for women’s health research. Per HHS’s guidance, NIH  classifies its funding into 16 categories and about 120 sub-categories  representing specific diseases and conditions; and three categories by  sex—funding for women’s health research, funding for men’s health  research, and funding for research related to both women’s and men’s  health. Since the sex categorization of funding is based on enrollment, a  portion of funding for research studies for certain diseases that primarily  affect women—such as cervical cancer—but are reported as including  male participants may be categorized as related to men’s health or both  sexes. NIH reports these funding estimates in HHS’s annual  congressional budget justification and in the biennial Report of the  Advisory Committee on Research on Women’s Health.
To more closely examine the amount NIH funded for research on  selected diseases and conditions with a particular relevance to women,  we developed a list of diseases and conditions using the top 10 diseases  and conditions in each of the Centers for Disease Control and  Prevention’s 2012 lists of leading causes of death for women and leading  chronic diseases for women. We supplemented this information with the  diseases and conditions with a particular relevance to women that were  identified in a 2010 Institute of Medicine report, Women’s Health  Research: Progress, Pitfalls, and Promise.
We worked with NIH and several of its institutes and centers (IC) to  assign each of the diseases and conditions in our list to the NIH budget  categories that were included in the women’s health budget. These  matches allowed us to provide an estimate of funding for these selected  diseases and conditions, which are shown in Table 3.
NIH officials told us that determining the amount NIH funded for women’s  health research overall—as well as for specific diseases and conditions— is difficult, and the resulting information are estimates, rather than actual  amounts, due to challenges compiling the funding data. These challenges  include: (1) methodological issues in assigning research funding to a sex  category, especially for basic research, which does not include human  subjects, (2) difficulties assigning NIH research funding to broad HHS- determined disease categories, (3) research projects that overlap disease  categories, but must be assigned to a single disease category, and   (4) variation in data collection processes at the IC level.
section_title: Appendix III: National Institutes of Health (NIH) and Institute and Center (IC) Research Study Enrollment
paragraphs: Tables 4 and 5 below present NIH and IC-level enrollment data, by sex,  for fiscal years 2011 through 2014—for all clinical research studies and  for phase III clinical trials, respectively.
section_title: Appendix IV: Comments from the Department of Health and Human Services
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Karen Doran, Assistant   Director; Amanda Cherrin; Emily Loriso; and Julie T. Stewart made   key contributions to this report. Jennie F. Apter, Leia Dickerson,   Krister Friday, and Jacquelyn Hamilton also contributed to the  development of this report.