VA Health Care: VA Spends Millions on Post-Traumatic Stress Disorder Research and Incorporates Research Outcomes into Guidelines and Policy for Post-Traumatic Stress Disorder Services
Released: 2011-01-24T00:00:00

Letter

Background

VA manages its intramural research program through ORD. According to ORD’s 2009 to 2014 strategic plan, ORD has 10 research priority areas, which are topics of research that are considered important to VA. The research priority areas are the health care needs of veterans who have served in Operation Enduring Freedom and Operation Iraqi Freedom, aging-related conditions, mental health care and well-being, chronic diseases, long-term care and caregiving, deployment-related exposure to hazardous environmental agents, equity in care, access in rural areas, women’s health, and personalized medicine. According to VA officials, all of these research priority areas could include PTSD research.

VA funds intramural research through the following: VA’s Merit Review Program: This program supports research studies typically conducted by one VA investigator at one VA facility and is administered by ORD’s four research and development services, each of which has a different research focus. (See table 1.) Each research and development service is responsible for soliciting, reviewing, selecting, and funding research proposals submitted to the service.

VA’s CSP: This program, which is administered by Clinical Science, funds larger-scale, multisite clinical trials and epidemiological research studies on key diseases that impact veterans.

The Merit Review Program has research award funding limits, which are set by VA. In some cases, intramural research awards may only be funded for a certain number of years. See table 2 for more information.

In addition to individual studies conducted at VA facilities, VA has several research centers and programs that conduct or support PTSD research. For example, the National Center for PTSD focuses on PTSD research. VA also has Research Enhancement Award Programs, which help support PTSD research by providing staff and other resources to investigators. (For more information on VA research centers and research programs that conduct or support PTSD research, see app. I.) According to a VA official from the National Center for PTSD, VA does not fund most of the PTSD research that is being conducted today.

Origin and Criteria for Intramural Research Proposals

Intramural research proposals may be service-directed—solicited by ORD on specific topics—or investigator-initiated— submitted by investigators to ORD on their own initiative. Investigators submit proposals either in response to a request for proposals on a specific topic (for service- directed proposals) or to an open request for proposals (for investigator- initiated proposals). For both the Merit Review Program and CSP, proposals are typically evaluated in two review cycles per year.

To be considered for intramural research funding: The proposal must be veteran-centric.

The proposal must have received approval from the director of the medical center and the research and development office of the medical center where the lead investigator, known as a principal investigator, is based.

The principal investigator and any coprincipal investigators must demonstrate a primary professional commitment to VA, as demonstrated by at least a 5/8 time VA appointment at the time the funding is awarded and previous VA experience, including experience in research and patient care.

Research must be conducted primarily on VA premises. The principal investigator and any coprincipal investigators must have designated research space within a VA medical center.

PTSD Research Funding Increased from Fiscal Year 2005 through Fiscal Year 2009

Overall intramural PTSD research funding from VA’s medical and prosthetic research appropriation increased from $9.9 million in fiscal year 2005 to $24.5 million in fiscal year 2009. The number of intramural PTSD research studies funded through the Merit Review Program and CSP increased from 47 in fiscal year 2005 to 96 in fiscal year 2009.

VA Intramural PTSD Research Funding Increased from $9.9 Million in Fiscal Year 2005 to $24.5 Million in Fiscal Year 2009

Based on the VA data we obtained and summarized, we found that overall intramural PTSD research funding from VA’s medical and prosthetic research appropriation increased from about $9.9 million in fiscal year 2005 to about $24.5 million in fiscal year 2009, or by about 150 percent (see fig. 1). Overall intramural PTSD research funding included funding for specific PTSD studies as well as for other PTSD research-related funding, such as career development awards provided to junior VA investigators to conduct PTSD studies, salaries for VA investigators who are not VA clinicians, funding for PTSD research conducted within ORD research centers, and PTSD research meetings.

Of the $80.2 million provided for PTSD studies from fiscal year 2005 through fiscal year 2009, $51.3 million, or about 64 percent, was for studies funded through the Merit Review Program. The remaining approximately $28.9 million, or about 36 percent, was for CSP studies. (See fig. 2.)

From fiscal year 2005 through fiscal year 2009, intramural PTSD research funding ranged from 2.5 percent to 4.8 percent of VA’s medical and prosthetic research appropriation. (See table 3 for VA intramural PTSD research funding and VA’s medical and prosthetic research appropriations from fiscal year 2005 through fiscal year 2009.) For comparison, according to a 2009 report prepared by ORD staff for VA’s National Research Advisory Council, for fiscal year 2009, funding for intramural traumatic brain injury research was about $14.6 million, 2.9 percent of the medical and prosthetic research appropriation. Funding for spinal cord injury research was $27.2 million, 5.3 percent of the medical and prosthetic research appropriation. Funding for intramural cardiovascular disease and stroke research was $53.1 million, 10.4 percent of the medical and prosthetic research appropriation.

The Number of PTSD Studies Funded through VA’s Intramural Research Program Increased from 47 in Fiscal Year 2005 to 96 in Fiscal Year 2009

Similarly, we found that the number of PTSD studies funded from VA’s medical and prosthetic research appropriations through VA’s intramural research program increased from fiscal year 2005 through fiscal year 2009. (See fig. 3.) Specifically, in fiscal year 2005, 47 intramural PTSD research studies were funded while in fiscal year 2009, 96 intramural PTSD research studies were funded. This represented an increase of more than 100 percent. Of all the studies funded each fiscal year, only a small number were CSP studies.

VA Intramural PTSD Research Studies Are Funded Primarily According to Scientific Merit

According to VA officials, intramural research proposals, including those on PTSD, are reviewed and funded in VA’s Merit Review Program and VA’s CSP primarily according to scientific merit.

VA’s Merit Review Program Intramural research proposals submitted to VA’s Merit Review Program are reviewed through a series of steps prior to funding. See figure 4 for an overview of the submission, review, and funding process for proposals submitted to the Merit Review Program. (For more detailed information on this process, see app. II.)

First, investigators submit proposals electronically. Investigators typically submit Merit Review Program proposals to grants.gov, the government’s central grant identification and proposal portal, in response to a request for proposals. Submitted proposals are then transferred to eRA Commons, an electronic system for grant administration functions, for VA processing and review.

Second, each proposal is assigned to a merit review panel for evaluation. Each merit review panel reviews proposals in a specific research topic area, and is composed of panelists, typically associate-level professors, who are selected based on their expertise in this area. According to VA documents, as of 2010, there were a total of 35 merit review panels across VA’s research and development services. The merit review panels evaluate each proposal based on its scientific merit. Panelists consider several criteria in evaluating the overall scientific merit of a proposal. (See table 4 for criteria used to determine scientific merit.)

Third, the merit review panelists score the proposals to determine their rank. Each panelist provides a score to each of the proposals reviewed by the panel. The scores are averaged to create a “priority score” for the proposal. (See app. II for specific scoring guidelines given to panelists in all research and development services.) All proposals scored by the merit review panel are then ranked by priority score among all of the proposal scores recently assigned by the merit review panel. The rank of the proposal is used to determine the “percentile” of the proposal.

Finally, research and development service directors determine how many proposals to fund. All of the proposals scored by all merit review panels in a research and development service in the review cycle are ranked together by their percentiles to be considered for funding. According to VA officials, research and development service directors typically fund up to the 25th percentile of proposals in a review cycle, beginning with those with the most scientific merit, although the number of proposals funded may vary depending on the budget. According to VA, research and development service directors may also choose to fund a small number of additional proposals at the margin that respond to research priority areas. For example, if the fundable range determined by a research and development service director was up to the 25th percentile, proposals at the 26th percentile related to research priority areas could also be considered for funding.

VA’s Cooperative Studies Program VA intramural research proposals submitted to CSP are reviewed and scored in a process similar to that of the Merit Review Program prior to consideration for funding. To help develop the CSP proposal, investigators are assisted by members of a CSP center, a VA entity that provides guidance and support for research across multiple sites. (See fig. 5 for an overview of the process for submitting, reviewing, and funding a CSP proposal. For more information on ORD’s CSP review process, see app. III.)

Before submitting a research proposal, investigators submit a letter of intent, or a preliminary outline of a proposal, to the Director of Clinical Science to be approved for planning a CSP proposal. Based on the merit of the letter of intent, as determined by three or more reviewers, the Clinical Science Director decides whether to fund planning efforts to develop a CSP proposal.

When the principal investigator receives approval to begin planning efforts, the Clinical Science Director assigns a CSP center to provide statistical and methodological guidance to the investigator. The director of the CSP center designates a project manager and methodologist, such as a person with expertise in biostatistics, to provide guidance to the principal investigator. The Clinical Science Director, with recommendations from the principal investigator, then forms a planning committee of additional experts to assist in developing a CSP proposal. The planning committee develops a CSP proposal over the course of two planning meetings.

Once a proposal is developed, the CSP center, on behalf of the principal investigator, submits a hard copy proposal to the CSP central office for evaluation by the Cooperative Studies Scientific Merit Review Board. This board consists of reviewers who have extensive experience in clinical research and the conduct of clinical trials or epidemiology studies. Reviewers evaluate CSP proposals based on scientific merit. According to VA, the scientific merit of a CSP proposal is defined by the importance of the proposal, its feasibility, the clarity and achievability of its objectives, the adequacy of the plan of investigation, the correctness of the technical details, and the adequacy of the safeguards for the welfare of the patients. Based on these criteria, reviewers discuss the general scientific merit of the proposal. Reviewers vote on whether to unconditionally approve, conditionally approve, reject or defer with recommendation for resubmittal, or reject each proposal. (See table 5 for an overview of funding recommendations provided by the board.)

After the reviewers vote, they each provide scores for a proposal recommended for funding based on scientific merit. The scores are then averaged to provide a priority score for a proposal. Finally, the Clinical Science Director considers the priority scores of all the proposals in that review cycle and selects the proposals with the strongest priority scores for funding. According to VA officials, the number of proposals funded may vary depending on the budget.

VA Has Established a Standardized Process to Review All Relevant PTSD Research Outcomes and Incorporate Them into a PTSD CPG

The VA/DOD Evidence-Based Practice Work Group, which is responsible for developing and updating all of VA’s CPGs, has a standardized and reproducible process to review all relevant research outcomes when developing or updating all CPGs, including the PTSD CPG. To develop or update a CPG, the VA/DOD Evidence-Based Practice Work Group identifies and assigns a group of VA and DOD clinical leaders and experts who are knowledgeable in the subject area to work on the CPG. Generally, the process to develop or update a CPG consists of the following steps.

First, the assigned group of VA and DOD clinical leaders and experts identifies “clinical questions” that will be answered in the CPG. According to VA officials, clinical questions can be either broad or specific. For example, the 2004 PTSD CPG contained clinical questions regarding whether early intervention is more effective than later intervention, and whether certain interventions, such as different psychotherapies, are more effective than others.

Second, in order to minimize bias, an external contractor conducts a systematic review of relevant research and selects and summarizes the most methodologically rigorous research studies that are applicable to each of the clinical questions.

Third, after receiving summaries of the studies with the highest level of evidence, the VA and DOD group of clinical leaders and experts rates the research using an established grading scheme that considers the level of evidence of each research study—the scope and methodological rigor of an individual study; the overall quality of evidence—the overall quality of all of the research that addresses a particular clinical question, considering the level of evidence of all the studies considered; and the net effect of an intervention—according to the collective results of the studies considered, the intervention’s benefits minus the intervention’s harms.

Finally, the assigned group of VA and DOD clinical leaders and experts assigns a grade to each evidence-based recommendation based on an assessment of the overall quality of evidence and the net effect of the intervention. (See app. IV for a detailed description of the process used to develop evidence-based VA/DOD CPGs.)

The process for conducting a systematic review of research outcomes to develop or update a CPG is repeated as often as is deemed necessary by the VA/DOD Evidence-Based Practice Work Group according to its written procedures and designated time frames. According to VA/DOD Evidence- Based Practice Work Group documents, routine updates to the CPGs should ideally occur approximately every 2 years. However, updates to CPGs often do not occur every 2 years, and VA officials told us that some CPGs are updated more frequently than others based on availability of resources and priority areas. Additionally, VA officials reported that a CPG will be immediately updated if any evidence-based recommendation contained in it is identified as harmful to patients.

According to VA, the VA/DOD Evidence-Based Practice Work Group approved an update to the 2004 PTSD CPG on October 25, 2010, and published the update on VA’s Web site on November 17, 2010. According to VA officials, the systematic process outlined above was used to review all relevant research outcomes and make evidence-based recommendations for PTSD services to both develop and update the PTSD CPG.

VA Reviewed Research Outcomes and Used Existing Resources in Determining Which PTSD Services to Require Its Facilities to Provide or Make Available to Veterans

According to VA officials, the decision to require that cognitive processing therapy and prolonged exposure therapy be made available to veterans diagnosed with PTSD at VA facilities—as indicated in the Handbook, which established certain requirements for mental health services within VA—was based on a review of research outcomes and the availability of existing resources.

Review of research outcomes. According to VA, agency officials and qualified subject matter experts reviewed relevant research outcomes and the quality of the research to determine the most efficacious PTSD treatments available when determining which PTSD services to include in the Handbook and make available to veterans. Specifically, VA officials told us that their decision to include cognitive processing therapy and prolonged exposure therapy in the Handbook was influenced by the fact that both of these had been graded as level “A” treatments in the 2004 PTSD CPG (indicating that the intervention is always indicated and acceptable). Furthermore, VA officials said that these two therapies had greater evidence supporting their effectiveness than other PTSD services also graded as level “A” in the 2004 PTSD CPG. In addition, VA officials added that their decision was validated by the results of a VA- commissioned Institute of Medicine study published in 2008 that reviewed the evidence for existing PTSD treatments. According to VA, the study found that cognitive processing therapy and prolonged exposure therapy were considered efficacious treatments for PTSD. While the Institute of Medicine report was released after VA had already decided to include cognitive processing therapy and prolonged exposure therapy in the Handbook, VA officials explained that the Institute of Medicine report was the basis for the decision not to include other PTSD services in the Handbook.

Availability of existing resources. VA officials told us that prior to issuing the Handbook in 2008, VA had already begun investing considerable resources to implement national training programs for cognitive processing therapy and prolonged exposure therapy in 2006 and 2007, respectively. VA officials said that they decided to implement the national training programs because VA realized the need to create sufficient capacity so that evidence-based PTSD treatments could be available to veterans throughout the VA system. VA explained that the national training programs were rolled out in advance of the Handbook’s issuance as part of the implementation of VA’s Comprehensive Veterans Health Administration Strategic Plan for Mental Health Services, which called for rapid implementation of evidence-based treatments. VA did this to ensure that it had the capacity to provide cognitive processing therapy and prolonged exposure therapy to all veterans with PTSD for whom these treatments were clinically appropriate. VA officials said that they were able to begin implementing national training programs for cognitive processing therapy in 2006 and prolonged exposure therapy in 2007 because VA had qualified instructors to administer the programs and money available to fund them.

Unlike the written and standardized process that the VA/DOD Evidence- Based Practice Work Group established to develop CPGs, VA does not have a formal written process or framework to explain its decision for including cognitive processing therapy and prolonged exposure therapy in the Handbook. VA officials explained that they followed a process when choosing cognitive processing therapy and prolonged exposure therapy, but added that clinical decision-making processes are not typically expected to be documented in a formal manner.

VA officials told us that they plan to assess the implementation of the Handbook and will update PTSD requirements in it as needed or as new information or unexpected obstacles arise in the future. VA officials stated that they are currently clarifying the language regarding some of the requirements, but do not plan to revise any of the requirements relating to PTSD services at this time.

Agency Comments

We provided a draft of this report to VA and received technical comments, which we incorporated into our report as appropriate.

We are sending a copy of this report to the Secretary of Veterans Affairs. The report also is available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix V.

Appendix I: VA Research Centers and Programs That Conduct or Support PTSD Research

In addition to post-traumatic stress disorder (PTSD) research studies that are conducted by individual Department of Veterans Affairs (VA) investigators, or researchers, VA also funds a number of research centers or programs that conduct or support PTSD research. See table 6 for a description of these VA research centers and programs.

Appendix II: The Review Process for VA’s Merit Review Program

Research proposals submitted to the Department of Veterans Affairs’ (VA) Merit Review Program are evaluated in merit review panels that each review proposals in a specific research topic area. Each merit review panel is comprised of panelists, typically associate-level professors, who are selected based on their expertise in the area. Panelists are responsible for scoring proposals based on scientific merit to provide funding recommendations. See figure 6 for a detailed description of the Merit Review Program’s process for reviewing research proposals and table 7 for the Merit Review Program’s scoring guidelines.

Appendix III: The Review Process for VA’s Cooperative Studies Program

Research proposals submitted to the Department of Veterans Affairs’ (VA) Cooperative Studies Program (CSP) are reviewed and scored by the Cooperative Studies Scientific Merit Review Board. Reviewers on the board are chosen based on their expertise in clinical or epidemiological research. They typically serve 4-year terms, and ad hoc members can be added depending on specific expertise that may be needed to review a proposal. According to VA, as of October 2010, there were six reviewers on the board.

During the research proposal review process, the study team—which includes the lead researcher (referred to as the principal investigator) and a methodologist, such as a person with expertise in biostatistics—has an interactive discussion with the board regarding the proposal. Reviewers evaluate CSP proposals based on scientific merit and provide scores to reflect their funding recommendations. See figure 7 for a detailed description of the review process for CSP research proposals.

Appendix IV: The Process for Developing VA/DOD Evidence-Based Clinical Practice Guidelines

In 1999, the Department of Veterans Affairs (VA) and the Department of Defense (DOD) formed the VA/DOD Evidence-Based Practice Work Group to issue joint VA/DOD clinical practice guidelines (CPG)—tools that provide guidance and evidence-based recommendations to clinicians regarding the most effective interventions and services for a variety of health care topics. To develop or update a CPG, the VA/DOD Evidence- Based Practice Work Group has a standardized process to ensure that systematic reviews of relevant research outcomes are conducted in order to formulate evidence-based recommendations for prevention, assessment, and treatment services.

To develop or update a CPG, the VA/DOD Evidence-Based Practice Work Group identifies two clinical leaders—one from VA and one from DOD— who then help identify not more than 15 to 20 other experts in the subject area to form a “guideline working group.” A member of the VA/DOD Evidence-Based Practice Work Group is also selected to be an evidence chaperone for each CPG to ensure that conformity to prevailing standards for conducting high-quality systematic reviews is upheld.

To determine the scope of the CPG, the guideline working group, the evidence chaperone, and a facilitator are responsible for identifying clinical questions that are to be answered by a systematic review of relevant research outcomes. According to VA officials, clinical questions can be both broad and specific. For example, the 2004 post-traumatic stress disorder CPG contained clinical questions regarding whether early intervention is more effective than later intervention and whether certain interventions, such as different psychotherapies, are more effective than others.

According to VA, in order to answer these clinical questions, an external evidence center—an entity that conducts systematic reviews of research on a variety of topics—is contracted to collect and review all relevant research (including, but not limited to, VA- and DOD-sponsored research) to assess its applicability to each clinical question under consideration using explicit and reproducible methods. The evidence center then focuses its review on the best available research, that is, high-quality, methodologically rigorous studies that address health issues that impact VA and DOD populations and consider the effectiveness as well as the harms and benefits of the intervention at issue. According to VA officials, the evidence center provides summaries of only the best available research to the guideline working group for review.

After receiving the summaries, the guideline working group reviews the research in sequential steps using an established rating scheme developed by the U.S. Preventive Services Task Force to formulate evidence-based recommendations. See figure 8 for an overview of the steps that the guideline working group uses to formulate evidence-based recommendations.

Level of evidence. First, the guideline working group reviews the summaries to identify the level of evidence, or the level of methodological rigor. For example, research studies that have the highest quality are categorized as “I” (indicating at least one properly done randomized controlled trial), while research studies of the lowest quality are categorized as “III” (indicating that the research reflects the opinion of respected authorities, descriptive studies, case reports, and expert committees). (See table 8.)

Overall quality of research. After determining the level of evidence of individual research studies, the guideline working group makes a determination regarding the overall quality of all of the research that addresses a particular clinical question. The overall quality takes into account the number, quality, and size of all of the individual research studies together as well as the consistency of the results between research outcomes to determine the collective overall strength of the research. Based on this review, the guideline working group determines the overall quality of the evidence to be good, fair, or poor. (See table 9.)

Net effect of the intervention. For interventions that were supported by studies of “fair” or “good” overall quality, the guideline working group evaluates the benefits and the potential harms to determine the net effect of the intervention. The net effect of an intervention takes into account the benefits of the intervention minus the harms to determine the overall potential clinical benefit that the intervention may provide to patients. The net effect of the intervention ranges from “substantial” (meaning the benefit substantially outweighs the harm) to “zero or negative” (meaning it has no impact or a negative impact on patients). (See table 10.)

Grade of evidence-based recommendation. In the final step, the guideline working group uses its assessment of the overall quality of the evidence and the net effect of the intervention to grade evidence-based recommendations. (See table 11.)

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Mary Ann Curran, Assistant Director; Susannah Bloch; Stella Chiang; Martha R. W. Kelly; Melanie Krause; Lisa Motley; Michelle Paluga; Rebecca Rust; and Suzanne Worth made key contributions to this report.

Related GAO Products

VA Health Care: Progress and Challenges in Conducting the National Vietnam Veterans Longitudinal Study. GAO-10-658T. Washington, D.C.: May 5, 2010.

VA Health Care: Status of VA’s Approach in Conducting the National Vietnam Veterans Longitudinal Study. GAO-10-578R. Washington, D.C.: May 5, 2010.

VA Health Care: Preliminary Findings on VA’s Provision of Health Care Services to Women Veterans. GAO-09-899T. Washington, D.C.: July 16, 2009.

DOD and VA Health Care: Challenges Encountered by Injured Servicemembers during Their Recovery Process. GAO-07-589T. Washington, D.C.: March 5, 2007.

VA Health Care: Spending for Mental Health Strategic Plan Initiatives Was Substantially Less Than Planned. GAO-07-66. Washington, D.C.: November 21, 2006.

VA Health Care: Preliminary Information on Resources Allocated for Mental Health Strategic Plan Initiatives. GAO-06-1119T. Washington, D.C.: September 28, 2006.

VA Health Care: VA Should Expedite the Implementation of Recommendations Needed to Improve Post-Traumatic Stress Disorder Services. GAO-05-287. Washington, D.C.: February 14, 2005.

VA and Defense Health Care: More Information Needed to Determine If VA Can Meet an Increase in Demand for Post-Traumatic Stress Disorder Services. GAO-04-1069. Washington, D.C.: September 20, 2004.