National Institutes of Health: Better Oversight Needed to Help Ensure Continued Progress Including Women in Health Research
Released: 2015-10-23T00:00:00

Letter

Background

Headed by a presidentially appointed and Senate-confirmed director, NIH comprises 27 ICs and an Office of the Director. NIH’s ICs both conduct and support biomedical research specific to their unique missions, which generally focus on a specific disease, a particular organ, or a stage in life (e.g., childhood). Each of the ICs has its own director and staff, as well as its own advisory council or board, which helps to support and oversee the IC’s work. Within the Office of the Director are offices responsible for issues, programs, and activities that span NIH components, particularly research initiatives and issues involving multiple ICs.

NIH-Supported Research and Clinical Trials

NIH’s biomedical research that focuses on humans—its clinical research studies—includes clinical trials of biomedical or behavioral interventions such as new drugs, medical treatments, and surgical procedures and devices. Clinical trials are divided into four phases. In phase I clinical trials, which typically include 20 to 80 people, researchers test a new biomedical or behavioral intervention on human subjects for the first time to evaluate safety. In phase II clinical trials, the intervention is given to a larger group of people, 100 to 300 participants, to further evaluate efficacy and safety. In phase III clinical trials, the intervention is given to even larger groups—from several hundred to several thousand participants—to compare the intervention to commonly used or experimental interventions. Finally, phase IV studies are conducted after the intervention has been marketed, in order to gather information on long-term use.

NIH’s ICs support clinical trials predominantly through “extramural research”—awarding funds to researchers at universities or other research entities (awardees) through grants, contracts, and cooperative agreements. Of NIH’s 27 ICs, almost all fund extramural research projects. These ICs use a standard peer review process to inform the final decisions on which extramural research projects to fund. The size and composition of the ICs’ clinical trial portfolios vary substantially, depending on such factors as the IC’s budget, mission, and the scientific goals of any given study. For example, some ICs support few if any phase III clinical trials.

In fiscal year 2014, NIH’s ICs reported funding of nearly $30 billion for all biomedical research. Of that amount, NIH estimates that—based on reporting categories HHS developed for use by all of its agencies— $23.9 billion (80.3 percent) funded research related to the health of both women and men, $4 billion (13.2 percent) funded research related to women’s health, and an estimated $1.9 billion (6.4 percent) funded research related to men’s health. (See Appendix II for more details on estimated fiscal year 2014 funding for selected diseases and conditions of particular relevance for women.) To determine these amounts, NIH annually assigns its research funding to certain women’s health disease and condition categories—such as breast cancer or heart disease. Additionally, NIH classifies this funding as either related to the health of both sexes or as supporting research on women’s health only or men’s health only by using HHS’s calculation guidelines. NIH reports these funding estimates in HHS’s annual congressional budget justification and in ORWH’s biennial Report of the Advisory Committee on Research on Women’s Health.

The NIH Revitalization Act of 1993 and NIH Inclusion Policy

The Revitalization Act required NIH to ensure the appropriate inclusion of women in NIH-funded clinical research, including clinical trials. The Revitalization Act contained provisions that required NIH to, among other things, ensure that women are included in all NIH-funded clinical research, and report on compliance with the inclusion provisions of the Act. Additionally, the Revitalization Act requires NIH to ensure that clinical trials are designed and carried out in a manner sufficient to allow for valid analysis of the extent to which the outcomes measured in the trial affect women differently than men. NIH subsequently determined that this particular requirement only applied to phase III clinical trials, as they are the largest clinical trials involving human subjects and the closest to effecting broad changes in public health policy and standards of care.

The Revitalization Act also directed NIH to develop guidelines for including women in clinical research and report biennially to Congress on NIH’s compliance with this policy. The resulting Inclusion Policy, the current version of which has been in place since October 2001, requires NIH applicants conducting clinical research to 1. design research plans that detail the breakdown of their studies’ participants by sex and provide a rationale for their planned enrollment for all clinical research studies; and 2. include plans for analyzing outcomes for potential sex differences for NIH-defined phase III clinical trials, when appropriate, as determined by consideration of prior scientific evidence.

In addition to NIH ICs, several NIH offices play a role in implementing the Inclusion Policy, particularly ORWH, which was established in 1990—and codified in the Revitalization Act—to promote women’s health research, and OER, which administers and manages NIH grants policies, operations, and data systems. Since November 2011, the implementation of NIH’s Inclusion Policy has been overseen by the Subcommittee on Inclusion Governance, which comprises senior NIH officials from ORWH, OER, and several ICs. This committee, co-chaired by the ORWH director and staffed by the NIH Inclusion Policy Officer from OER, is charged with examining and considering current NIH policies related to the inclusion of women in NIH-funded clinical research. In addition, the implementation of the Inclusion Policy is monitored by the Advisory Committee on Research on Women’s Health (Women’s Health Advisory Committee), whose creation was mandated by the Revitalization Act, as well as by the individual IC’s directors and advisory councils or boards.

NIH’s Extramural Awards Management Process

During the peer review process, applicants’ plans for including women, as appropriate, are reviewed and assessed along with the applicants’ plans to meet other requirements or considerations that are outlined in the funding opportunity announcement or research solicitation. The outcome of these assessments—scores from the peer reviewers—inform the funding decisions made by the ICs. Prior to awards being made, program officers or other IC staff may advise awardees on additional information required before an award can be released, and the resolution of any concerns raised during the peer review stage—including concerns related to adherence to the Inclusion Policy. After awards are made, NIH’s awardees are responsible for managing their day-to-day activities in accordance with NIH requirements, and the IC making the award is responsible for the awarded funds and for monitoring progress and compliance with NIH policies, including the Inclusion Policy. IC program officers monitor awardees through a variety of methods—including reviews of reports and correspondence from the awardee, and site visits—to identify potential problems with scientific progress, compliance, and areas where technical assistance might be necessary. One such report that program officers review is the annual progress report—which includes data on study enrollment, among other things. Awardees submit information, including enrollment data, through the Electronic Research Administration (eRA) Commons, which is part of NIH’s electronic data collection and grants administration system that is used by awardees and program officers to access and share administrative information related to research awards.

More Women Than Men Were Enrolled in NIH Research Overall, but NIH Does Not Examine Detailed Data to Identify Potential Challenges to Enrolling Women

NIH data show that over the last decade more women than men have been enrolled across all NIH-funded clinical research, including phase III clinical trials. NIH publicly reports aggregate enrollment numbers on a biennial basis; however, it does not routinely make detailed IC enrollment data readily available or examine more detailed enrollment data by disease and condition, in order to identify potential challenges to enrolling women in certain research and disease or condition areas.

NIH Data Show More Women than Men Enrolled in NIH-Funded Clinical Research

NIH requires each awardee to report enrollment, including enrollment by sex, for each of their NIH-funded research awards, including phase III clinical trials. As of fiscal year 2015, these data are reported to NIH through its Inclusion Monitoring System—one part of NIH’s awardee data system. Program officers review these data at least annually to determine whether actual enrollment is consistent with the enrollment planned in the research design. The program officers use a designated checklist, among other tools, to document their monitoring as part of the annual progress report review process. According to NIH officials, awardee enrollment data are aggregated by each IC and presented in and discussed during meetings held by each IC’s advisory board or council, which are open to the public. The IC-level enrollment data are certified as being compliant with the Inclusion Policy by the IC’s advisory board or council and by the IC Director and included in an IC-level enrollment report. The certified IC enrollment reports are submitted to ORWH and OER, where, according to NIH officials, the data are checked for consistency and errors as part of a quality control process. NIH aggregates the enrollment data across the agency and reports this aggregate data to the Women’s Health Advisory Committee, Congress, and the public in NIH’s biennial inclusion report.

According to the data collected by NIH, in each fiscal year from 2005 through 2014, more women than men were enrolled in all NIH-funded clinical research studies, including phase III trials. (See fig. 1.) Specifically, in fiscal year 2014, among all NIH-funded clinical research studies, 57 percent of enrollees (16.4 million) were women, and 39 percent (11 million) were men. For all NIH-funded phase III clinical trials, in fiscal year 2014, 60 percent of enrollees were women (about 480,000) and 39 percent were men (about 314,000).

At the individual IC level, data show that for each IC, women’s enrollment for all clinical research studies, including phase III clinical trials, was generally higher than men’s enrollment in most years from fiscal year 2011 through fiscal year 2014. (See app. III for enrollment data at the NIH and IC level over this period.) Of the 25 ICs reporting enrollment in fiscal years 2011 through 2014, 13 ICs enrolled more women than men in the ICs’ clinical research studies during all four years. An additional 4 ICs enrolled more women than men in 3 of the fiscal years between 2011 and 2014. According to NIH, 10 of the 25 ICs regularly support phase III clinical trials, and of these, 3 (about one third) enrolled more women than men in each year in fiscal years 2011 through 2014, and another 2 enrolled more women than men in 3 of the 4 fiscal years.

NIH Does Not Make IC- Level Enrollment Data Readily Available or Examine Detailed Enrollment Data to Identify Potential Challenges to Enrolling Women in Specific Research and Disease Areas

NIH collects and reviews aggregated enrollment data from the ICs; however, NIH officials do not make these IC-level enrollment data readily available to the public or other interested parties. Specifically, the IC-level enrollment data are not published as part of the overall NIH biennial report on enrollment, are not shared with the Women’s Health Advisory Committee, and are not available for download from the ORWH website. Individual official IC enrollment reports were also not available through the websites of the three ICs in our review. Beginning in the fiscal year 2013-2014 reporting period, NIH required all ICs to submit their IC enrollment report numbers in a standard format, which may allow for easier public sharing of these data going forward.

Additionally, NIH officials do not routinely examine detailed enrollment data by sex beyond the IC level—such as by a specific research area or disease or condition being studied—to identify potential challenges to enrolling women in these areas, because enrollment data are currently not available in this format. The guidance NIH provided to the ICs for submitting their fiscal year 2013-2014 enrollment data to ORWH and OER explains that ICs may choose to further break out their reported enrollment data by disease area, portfolio area, or in some other manner. According to NIH officials, certain ICs analyze enrollment data by various disease categories when necessary. However, the IC officials we spoke with told us that their data systems are not capable of systematically aggregating enrollment data in this manner for routine reporting. When asked if it would be possible to aggregate enrollment data by disease and condition, IC officials said that they would be able to do so, but it would be a time intensive, manual process given current data system limitations. Further, NIH officials told us they expect NIH’s new enrollment data system deployed in October 2014 to increase functionality in examining enrollment data in different ways, but as of July 2015 the officials did not have specific plans or details available. In addition, NIH officials stated that because an individual IC’s research generally focuses on a specific disease, a particular organ, or a stage in life, the enrollment data that are aggregated at the IC level would roughly correspond with major disease and condition categories and could be used, to some extent, as a proxy for disease and condition enrollment data. However, this proxy method does not take into account the fact that many ICs are responsible for research that includes multiple diseases, organs, or stages in life; for example, NHLBI’s research portfolio includes studies of heart, lung, and blood related diseases and conditions. In addition, research on certain diseases and conditions—such as obesity—falls under the purview of multiple ICs.

NIH’s practices of not sharing the IC-level enrollment data and not examining detailed data on enrollment by sex—by specific research area or the disease or condition being studied—are inconsistent with several federal standards for internal control. Specifically, the internal control standards for information and communications state that for an entity to run and control its operations, it must have relevant, reliable, and timely communications relating to internal as well as external events. Information is needed throughout the agency to achieve all of its objectives, and effective communication should occur in a broad sense, with information flowing down, across, and up the organization. In addition to internal communications, management should ensure there are adequate means of communicating with, and obtaining information from, external stakeholders who may have a significant impact on the agency achieving its goals (e.g., the Women’s Health Advisory Committee). Federal internal control standards for monitoring call for management to assess the quality of agency performance over time and ensure that the findings of audits and other reviews are promptly resolved.

Because NIH does not readily share IC-level enrollment data with the public and other interested parties, such as the Women’s Health Advisory Committee—through means such as the ORWH website or the biennial NIH enrollment report—those interested in reviewing IC-level enrollment information would have to attend—or watch online if webcast—each individual IC’s advisory board or council meeting or specifically seek out or request any public record resulting from these meetings to have access to these data. In addition, by not routinely examining more detailed enrollment data that is aggregated by sex—such as data at the disease and condition level—NIH is limited in its ability to identify whether women are sufficiently represented in studies in specific areas that cross ICs—such as obesity. Further, NIH does not have information of sufficient detail to monitor and determine if the aggregate enrollment data from across NIH inadvertently mask low enrollment for particular research areas or diseases or conditions. At an April 2015 Women’s Health Advisory Committee meeting, some committee members raised such concerns, noting that published studies on clinical trials of specific diseases and conditions, such as cardiovascular disease, appeared to show that women’s enrollment was lower than the enrollment that NIH had reported in the aggregate. The committee members acknowledged that there could be many reasons for such discrepancies, but noted that they would like to see more detailed enrollment data to improve their understanding of the data and ensure that women are being appropriately included in NIH-funded clinical trials.

NIH Does Not Record Whether Clinical Trials Will Analyze Sex Differences to Allow Summary Analysis and Reporting for Oversight of Its Inclusion Policy

NIH’s Inclusion Policy requires that individual awardees conducting phase III clinical trials consider whether analysis of potential differences in study outcomes between women and men is needed in their studies, consistent with the Revitalization Act’s provisions regarding the design of certain clinical trials. However, the agency does not maintain, analyze, or report summary data to oversee whether analysis of outcomes by sex are planned or conducted, when applicable, across all NIH-funded clinical trials.

Under the Inclusion Policy, applicants seeking funds for phase III clinical trials must consider prior scientific evidence and assess whether an analysis of potential sex differences is merited, and if so, develop a plan to analyze study results accordingly. Both this consideration and the plan for analysis, if appropriate, are to be included in the awardee’s application for funding. To ensure awardees’ compliance with this requirement, NIH officials told us they rely on the agency’s peer review process for reviewing applications, and after awards are made, on IC program officers’ monitoring of individual awardees. NIH has guidelines for peer reviewers to use when assessing applications and rating applicants’ inclusion plans during peer review. This review typically includes an evaluation of the proposed study design and assessment of whether any plans for conducting an analysis of potential sex differences are “acceptable” or “unacceptable.” The assessment and rating are based on consideration of prior scientific evidence that either supports or negates the existence of differences in outcomes by sex. Peer reviewers document this assessment in a summary statement provided to the ICs for final award determinations. According to NIH officials, if reviewers determine that an applicant’s plans are not acceptable, the applicant is barred from funding until the plans are addressed and deemed acceptable by IC officials. After awards are made, program officers from the ICs that fund the studies are responsible for monitoring awardees’ overall progress on an ongoing basis. Specifically, program officers are to review awardees’ annual progress reports, which the Inclusion Policy states should address analysis of potential sex differences, as appropriate. Officials from one IC told us that, through their regular interactions with awardees, program officers are very familiar with the design of their awardees’ trials and whether any such analysis is planned or underway.

NIH program officers monitor individual awardees’ compliance with the analysis requirement of the Inclusion Policy; however, the agency lacks summary data on awardees’ analysis plans, including the percentage of awardees in a given year with trials designed to identify potential sex differences, when applicable. Currently, program officers review awardees’ progress reports—including any information reported regarding the analysis of potential sex differences—but do not have means, such as a written checklist or a required field in an electronic reporting system, for recording the information obtained through this monitoring, as they do for monitoring enrollment. NIH’s awardee data system includes information on whether individual awards include phase III trials. However, the data system does not have a data element that denotes whether an awardee’s study should include or has plans for an analysis of potential sex differences. Although such information is included in the narrative included in awardees’ funding applications, this information cannot be easily aggregated for summary reporting, NIH officials explained, and therefore they do not estimate the proportion of trials being conducted at any one time that are designed and explicitly intended to identify differences in study outcomes by sex. NIH officials also told us that they plan to add a question for this type of monitoring to the existing electronic checklist used by program officers in the fall of this year, to be implemented for awards funded in fiscal year 2016.

Because NIH does not have summary data regarding the analysis requirement of the Inclusion Policy, it has not reported summary information on this aspect of the Inclusion Policy to key stakeholders— including the Women’s Health Advisory Committee and the Congress. Notably, NIH’s agency-wide biennial reports on the status of the Inclusion Policy do not include information on the extent to which NIH-funded phase III trials included plans to conduct analyses of potential sex differences, or on the overall status of this aspect of the Inclusion Policy. Instead, the report focuses primarily on NIH-wide aggregate enrollment. NIH officials told us they rely on program officers’ monitoring of individual awardees to ensure that the analysis requirement of the Inclusion Policy is being implemented appropriately after awards are made, because part of the program officer’s role is to ensure satisfactory scientific progress as well as compliance with NIH policies. The officials added that they were not sure of the utility of summary reporting in this case.

NIH’s lack of summary data and reporting regarding the analysis requirement of the Inclusion Policy conflicts with federal internal control standards. First, federal internal control standards require that federal agencies have control activities in place to ensure that management’s directives are carried out and that these controls are monitored. These standards also state that information should be recorded and communicated to management and other responsible officials in a form and within a time frame that enables them to carry out their internal control and other responsibilities.

Without summary data on the funded phase III clinical trials that are intended to provide information on potential sex differences—including the number of such trials funded in a given year—senior NIH officials are limited in their ability to effectively oversee the implementation of the Inclusion Policy to assess whether changes are needed to their procedures. Further, NIH cannot provide this information to stakeholders such as the Women’s Health Advisory Committee and Congress in its regular reporting on other aspects of the Inclusion Policy. As a result, these stakeholders lack assurance that the agency is implementing the Inclusion Policy as intended and in a manner consistent with the Revitalization Act’s provisions regarding the design of certain clinical trials. In its fiscal year 2011-2012 Report of the Advisory Committee on Research on Women’s Health, NIH previously acknowledged that inclusion is not just a matter of having women and men included in clinical studies; rather, the scientific value of research studies is greatly enhanced by providing knowledge about differences and/or similarities between different populations affected by the diseases under study. However, without assurance that its clinical trials are being designed and conducted as directed under the law and its implementing policy, NIH’s insight regarding the interpretation, validation, and generalizability of findings resulting from the research it supports—as these findings apply to both women and men—is diminished, potentially limiting the value of NIH-funded research.

Proposed NIH Policy Intended to Increase Public Reporting of Clinical Trial Results by Awardees and Could Increase Reporting of Sex- Specific Results, According to NIH Officials

NIH has developed and proposed a policy intended to increase public reporting of clinical trial results by requiring all NIH-funded clinical trials to be registered and have results submitted to its registry and results database, ClinicalTrials.gov. ClinicalTrials.gov contains accessible and searchable information on publicly and privately supported clinical trials and observational studies that is provided and updated by NIH awardees and other researchers. Currently, all NIH awardees are encouraged to register their trials with ClinicalTrials.gov at the beginning of the study, but only certain trials—those of certain drugs and devices regulated by the Food and Drug Administration—must register and provide summary results as required by law. According to NIH officials, as of July 2015, there were approximately 195,000 studies registered in ClinicalTrials.gov, and almost 18,000 of these have summary results information posted. Under the proposed NIH policy, all NIH-funded clinical trials, regardless of trial phase or type of intervention being studied, would have to be registered and submit summary results—including participant flow, baseline demographics such as the sex and age of participants, primary and secondary outcomes, and adverse events—to ClinicalTrials.gov. According to NIH, if the proposed policy goes into effect, compliance may be enforced through possible suspension or termination of funding and noncompliance could impact future funding decisions. NIH sought public comments on the proposed policy from November 2014 through March 2015 and, as of August 2015, was analyzing the comments it received. NIH anticipates that the final policy will be issued in the first quarter of fiscal year 2016.

NIH officials told us that the proposed policy for clinical trial registration and results submission was not intended to increase reporting of sex- specific results to ClinicalTrials.gov; however, the officials also said that there is the potential for more reporting of sex-specific results, given the overall increase expected in the number of reported studies. Since the proposed policy would require awardees to report results for their prespecified primary and secondary outcome measures as part of the summary results submission, the reporting of sex-specific results would depend on the design of the trial, according to NIH officials. Specifically, if sex differences were among the prespecified primary and secondary outcomes studied in a specific trial, officials said, then that would be reflected in the results submitted to ClinicalTrials.gov.

In issuing the proposed policy, NIH stated that its awardees are expected to make their trial results available to the research community and to the public at large in order to contribute to scientific knowledge and, ultimately, public health. NIH proposed the policy partly in response to a recent study, which found that within 30 months of trial completion, the results of less than half of NIH-funded clinical trials had been published in a peer-reviewed biomedical journal, the traditional method for sharing results. NIH stated that because journal publication of clinical trials results is not always possible, it is important to provide other ways for clinical trial results to be disseminated and publicly available to researchers, health care providers, and others.

NIH has made efforts to encourage sex-specific reporting of clinical trial results. In NIH’s fiscal years 2011-2012 Report on the Advisory Committee for Research on Women’s Health, ORWH stated that it is only through sex-specific reporting that full information becomes available to the public and to scientists who can then use such data to inform future studies, thereby building the knowledge base in a manner that takes into consideration the influences of sex on health and disease. Specifically, NIH has worked with journal editors and others to encourage reporting of results by sex. Specifically, in 2011, NIH asked IOM to convene a workshop of researchers, journal editors, and others on the topic to discuss the importance of reporting results by sex and the implications of this issue for journals’ reporting policies. According to Stanford University’s Gendered Innovations project, 32 peer-reviewed journals worldwide have editorial policies requiring that clinical trial researchers include information on results by sex when they submit articles for publication. However, editors from one medical journal that we spoke with stated that when evaluating whether results should be reported by sex, it is important to consider whether examining sex differences is a primary outcome of the study, and whether the trial was big enough for a valid subgroup analysis—i.e., analysis of the effect of the intervention on two or more different groups of participants, such as women and men. They emphasized that if a study was not designed for a subgroup analysis by sex and one was performed, the results could be erroneous.

NIH has also made efforts to facilitate the sharing of clinical trials results, including sex-specific information, through venues other than journals and ClinicalTrials.gov. NIH has a number of policies that promote the dissemination of research results—and the underlying data—and guide awardees in disseminating their results, including the NIH Data Sharing Policy, among others. Additionally, related to the sharing of sex-specific information, NIH hosts a Women’s Health Resources portal and a Women’s Health topic page on the Medline Plus webpage, which includes links to other information about women’s health from journal articles and ClinicalTrials.gov. The agency also reports summaries of research related to women’s health in the biennial Report of the Advisory Committee on Research on Women’s Health.

Conclusions

Although NIH has made progress in the 2 decades since the 1993 Revitalization Act regarding the inclusion of women in NIH-funded clinical research, opportunities remain for NIH to further extend the value of its investment in medical research. NIH is responsible for ensuring that the nation receives the greatest benefit of the large federal investment in clinical research by fully implementing its Inclusion Policy, such that women are adequately included in NIH-funded clinical trials when appropriate, and that potential sex differences may be identified. By not readily sharing IC-level enrollment data, NIH limits the public’s and other interested parties’ ability to gain insight into enrollment issues at each of the ICs, putting the onus of obtaining these data on the interested parties themselves to attend or view online up to 25 individual IC board or council meetings or request any public record of these meetings. By not examining more detailed enrollment data—such as data aggregated by research area or specific to various diseases and conditions—NIH cannot know whether it is adequately including women across all of the research it supports. Without this greater insight into enrollment for specific to diseases and conditions, NIH is limited in its ability to assess whether its programs that support cross-cutting research spanning multiple ICs are successfully including women in clinical research or facing challenges that the agency should address. Further, the lack of summary data and reporting about the extent to which awardees plan to conduct or perform analyses of potential sex differences in phase III clinical trials compromises NIH’s oversight and jeopardizes the agency’s ability to provide assurances over the Act’s provisions regarding the design of certain clinical trials and meet the purposes of its Inclusion Policy. Without summary data, such as the proportion of trials being conducted that intend to analyze differences in outcomes for men and women, and reporting on that data, NIH and Congress cannot know whether or to what extent current efforts are helping to ensure that differences in clinical outcomes by sex are identified and that NIH is supporting research that can be used to shape improved medical practices for both women and men.

The overall increase in the enrollment of women in NIH-funded clinical research studies, such that women have been a significant proportion of research subjects for nearly 2 decades, is a noteworthy achievement for NIH. To continue to build on this achievement—and consistent with federal internal control standards—NIH should turn its focus to assessing whether the agency is meeting the purposes of the Inclusion Policy, and if it is not, take the needed corrective actions.

Recommendations for Executive Action

To ensure effective implementation of the Inclusion Policy in a manner consistent with the Revitalization Act’s provisions regarding the design of certain clinical trials, the NIH Director should take the following five actions: make IC-level enrollment data readily available through public means, such as NIH’s regular biennial report to Congress on the inclusion of women in research, or through NIH’s website; examine approaches for aggregating more detailed enrollment data at the disease and condition level, and report on the status of this examination to key stakeholders and through its regular biennial report to Congress on the inclusion of women in research; ensure that program officers have a means for recording information obtained from monitoring awardees’ plans for and progress in conducting analyses of potential differences in outcomes by sex; on a regular basis, systematically collect and analyze summary data regarding awardees’ plans to conduct analyses of potential sex differences, such as the proportion of trials being conducted that intend to analyze differences in outcomes for men and women; and report on this summary data and the results of this analysis in NIH’s regular biennial report to Congress on the inclusion of women in research.

Agency Comments and Our Evaluation

We provided a draft of this product to HHS for comment, and HHS responded with comments provided by NIH. In its written comments, reproduced in appendix IV, NIH generally concurred with our findings and recommendations. NIH also provided technical comments that were incorporated into the final report, as appropriate.

In commenting on our first recommendation to make IC-level enrollment data readily available to the public, NIH agreed and indicated that there are opportunities for the agency to increase the accessibility of IC-level enrollment data. NIH also stated that the agency has begun to standardize IC enrollment reporting and will continue this effort by standardizing data tables and graphics for ICs to provide for the NIH-wide biennial reports. NIH did not provide a timeline for making this information readily available to the public.

NIH agreed with our second recommendation to examine approaches for aggregating more detailed enrollment information at the disease and condition level, and to report on the status of this examination to key stakeholders. In its comments, NIH also reiterated what we describe in our report: some ICs conduct analysis of enrollment by disease or condition on an as-needed basis. NIH noted that the agency is working on ways to analyze enrollment at the disease and condition level across the ICs. NIH did not provide information on when the agency would be able to analyze these enrollment data, but it did state that when the agency is able to perform the analysis, NIH would make the results readily available through NIH’s biennial inclusion reports or other means.

NIH agreed with our third recommendation to ensure that program officers have a means for recording their monitoring of awardees’ plans for and progress in conducting analysis of potential sex differences, and confirmed that the agency plans to add questions that would facilitate this type of monitoring into the existing checklist program officers use to document other types of monitoring beginning in fiscal year 2016.

In commenting on our fourth and fifth recommendations regarding collecting and reporting summary data on awardees’ plans for sex- differences analysis, NIH agreed that it is critical to obtain more information on which clinical trials involve analyses of sex differences, and described some alternative data collection approaches for improving oversight of this issue. We maintain that thoughtful, useful analysis and summary reporting would improve NIH’s oversight of this aspect of the Inclusion Policy. Our recommendation was not intended to prescribe or limit the type of analysis performed or the data collected by NIH; instead we provided an example that NIH could adapt as needed, and we encourage the agency to explore the best alternatives for their analyses.

In other general comments, NIH also noted other opportunities that support oversight, such as the importance of peer reviewers in examining applicants’ plans for including women prior to funding decisions, and expanded reporting in ClinicalTrials.gov. In addition, the agency noted that ClinicalTrials.gov could engender greater transparency of clinical trial results and help assure that the analyses required under the Inclusion Policy are being completed.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of the Department of Health and Human Services, the Director of the National Institutes of Health, and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Literature Review of Factors Affecting Women’s Participation in Clinical Research

We reviewed 34 published journal articles that specifically identified barriers to or reasons for women’s participation in clinical research. We conducted an initial literature search that identified 168 studies published from January 2004 through October 2014 on the topic of women in clinical research. After reviewing abstracts and the full text of some articles, we narrowed this group down to the 34 articles included in this report that focus on the factors specifically affecting women’s participation in clinical research. We also spoke with or received written responses from officials and program officers from 3 institutes and centers (ICs) of the National Institutes of Health (NIH) about the factors affecting women’s participation in clinical research reported by NIH awardees.

In our review of the literature, the three most commonly cited barriers women face when considering whether to participate in clinical research were fear of experimentation/trust issues, health related concerns, and transportation/convenience issues. (See table 1.) Fear of experimentation/trust issues included distrust of physicians and the medical community and fear that the experimental treatment may be inferior to the conventional treatment(s). Health related concerns cited in the articles included concerns about the potential side effects or other adverse events associated with the experimental treatment. Some of the transportation/convenience concerns cited in the articles included difficulty traveling to a clinic (no transportation or a long distance to the facility) and general inconvenience associated with participating in the research.

The three most common reasons for women’s participation in clinical research identified in our literature review were personal/health benefits, altruism, and a general category of “other” reasons. (See table 2.) Personal/health benefits included access to new treatment and drugs, while “other” reasons included the fact that the research was being done at a clinic where the participant had already received care or was being conducted by clinical staff with whom the participants were familiar. Participation due to altruism included the desire to help science.

The NIH IC officials and program officers we spoke with generally agreed that the barriers and reasons for participation that we identified through our literature review were consistent with those they have encountered when working with their awardees. In addition, officials from one IC we spoke with identified other reasons women participate in clinical research, as cited by awardees. These reasons included the opportunity to learn about the disease being studied and how to manage it, and a sense of pride associated with participation. In contrast, these officials stated that for some diseases, the effect or perceived effect on future fertility could be a barrier to women’s participation.

Appendix II: National Institutes of Health (NIH) Funding for Women’s Health Research

To determine the amount of NIH funding for research on women’s health, we collected and reviewed NIH women’s health budget information for fiscal years 2009-2014. We also interviewed officials from NIH and the Department of Health and Human Services (HHS) regarding the budget categories that NIH uses in its women’s health budget, which were originally developed for use in reporting by all HHS agencies.

NIH, like all HHS agencies, annually compiles a summary table that estimates funding for women’s health research. Per HHS’s guidance, NIH classifies its funding into 16 categories and about 120 sub-categories representing specific diseases and conditions; and three categories by sex—funding for women’s health research, funding for men’s health research, and funding for research related to both women’s and men’s health. Since the sex categorization of funding is based on enrollment, a portion of funding for research studies for certain diseases that primarily affect women—such as cervical cancer—but are reported as including male participants may be categorized as related to men’s health or both sexes. NIH reports these funding estimates in HHS’s annual congressional budget justification and in the biennial Report of the Advisory Committee on Research on Women’s Health.

To more closely examine the amount NIH funded for research on selected diseases and conditions with a particular relevance to women, we developed a list of diseases and conditions using the top 10 diseases and conditions in each of the Centers for Disease Control and Prevention’s 2012 lists of leading causes of death for women and leading chronic diseases for women. We supplemented this information with the diseases and conditions with a particular relevance to women that were identified in a 2010 Institute of Medicine report, Women’s Health Research: Progress, Pitfalls, and Promise.

We worked with NIH and several of its institutes and centers (IC) to assign each of the diseases and conditions in our list to the NIH budget categories that were included in the women’s health budget. These matches allowed us to provide an estimate of funding for these selected diseases and conditions, which are shown in Table 3.

NIH officials told us that determining the amount NIH funded for women’s health research overall—as well as for specific diseases and conditions— is difficult, and the resulting information are estimates, rather than actual amounts, due to challenges compiling the funding data. These challenges include: (1) methodological issues in assigning research funding to a sex category, especially for basic research, which does not include human subjects, (2) difficulties assigning NIH research funding to broad HHS- determined disease categories, (3) research projects that overlap disease categories, but must be assigned to a single disease category, and (4) variation in data collection processes at the IC level.

Appendix III: National Institutes of Health (NIH) and Institute and Center (IC) Research Study Enrollment

Tables 4 and 5 below present NIH and IC-level enrollment data, by sex, for fiscal years 2011 through 2014—for all clinical research studies and for phase III clinical trials, respectively.

Appendix IV: Comments from the Department of Health and Human Services

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Karen Doran, Assistant Director; Amanda Cherrin; Emily Loriso; and Julie T. Stewart made key contributions to this report. Jennie F. Apter, Leia Dickerson, Krister Friday, and Jacquelyn Hamilton also contributed to the development of this report.